Implement Metrics in EDC and IdentityHub

[ma3u]

WHAT

The EDC owner need to define the main KPIs to measure the Data Exchange:

1. Number of offered Assets in the catalog
2. Number of contracts per month
3. Unique connections between two BPN
4. Amount of File transfers (incoming, outgoing)

WHY

Data Providers want to create Dashboards to track and audit the data negotiations and data transfers.

Positive and negative activities of the Dataspace Protocol describe catalog, contract negotation, transfer process.

HOW

In the EDC documentation the recommendation is to working with Asynchronous Messaging and Events via cloud event data specification.

Following Event implementation exists:

1. AssetEvent - create an Asset
2. ContractDefinitionEvent - create a contract definition
3. SecretEvent - create a secret
4. PolicyDefintionEvent - create a policy definition
5. ContractNegotiationEvent - create contract negotiation
6. TransferProcessEvent - start the transfer process

We want to store the relevant events as time series data (history data) in the log storage like Prometheus as persist activity log events and visualize this data in dashboard with Grafana.

To implement this feature we need to write a monitor service that subscribe the required events from the EventRouter. The example PolicyMonitorExtension shows how this can implemented. Following scopes we can subscribe the events:

• TransferProcessPolicyContext.TRANSFER_SCOPE
• ContractNegotiationPolicyContext.NEGOTIATION_SCOPE
• CatalogPolicyContext.CATALOG_SCOPE

The PrometheusMonitorServiceExtension need to inject the EventRouter:

    @Inject
    private EventRouter eventRouter;

And provide a subscriber that implements EventSubscriber like StartMonitoring.java

Next activities

• Make a architectural decision Metric framework decision #1745
• Create a branch OpenTelemetry for TRACTUS-X EDC. We work with pull requests.
• Create a Umbrella Helmchart for OpenTelemetry monitoring.

Other resources

There are two architectural decision records for the eventing framework with more technical details:

• 2022-06-03-event-framework
• 2023-03-09-event-framework-refactoring

The Open Telemetry example on Micrometer/Jaeger scape the requests and response of the EDC endpoints only. This example provides traceability and observability information.

Context

Part of the Catena-X NEXT TAP7.2 discussion