Making ContractOffer and its contained types serializable to JSON

[jimmarino]

@denisneuling @DominikPinsel I think we will need to make ContractOffer and its contained types, e.g. OfferedAsset, Policy, Asset, etc. serializable to JSON so they can be persisted for reliability and auditing. What do you guys think?

[DominikPinsel]

@jimmarino
Maybe it's just me, but I would try to avoid adding JSON attributes to classes, that are not serialized. It makes the code harder to read/maintain. Also another committer would probably (wrongly) assume that the code is serialized, if we add these attributes.
And, as we are always able to add these attributes in the future when we need them, I would not do that now.

[jimmarino]

I think those classes will need to be serialized. How would auditing be supported?

[denisneuling]

My opinion event-sourcing would be applicable here.
Means: On each domain state change the system must emit an object - so to say an event-object - expressing the changed state. Those emitted events are listened to by an "auditing-service". The auditing service then uses the properties from the event object and maps the data to a representation fitting its retention mechanism. This has the advantage of having domain objects and audit/event-log decoupled. This also means we can evolve both our domain objects and the auditing schema without breaking each others.

_{Denis Neuling [email protected], Daimler TSS GmbH, legal info/Impressum}

[jimmarino]

That would be one way of providing access to the audit state. However, the state still needs to be serialized, and this gets tricky with extensible type systems, specifically the one in the EDC.

To take a concrete example, TransferProcess and its contained types are serializable so that they can be persisted. This is required for auditing (however that is achieved) and reliability. Contained within TransferProcess are extensible subtypes, e.g. those of ProvisionedResource. Those subtypes need to be serialized as part of the TransferProcess, so we used Jackson's polymorphic serialization capabilities to achieve this (TypeManager handles this).

If we don't have that mechanism, there is no way to persist TransferProcess unless each extension handles specific subtypes, which will become too complex. Similarly, mapping to a distinct representation will run into issues when encountering extensible types such as ProvisionedResource. An example of this would be, "determine which infrastructure and geography data were stored in, e.g. S3 region." If there were two distinct domain models, each extension would be required to implement a mapping between its ProvisionedResource type and another type used for auditing (in this case, the S3 data).

These same issues will need to be solved for the contract negotiation subsystem to achieve reliability and auditing.

Does that make sense or confuse the issue?

[MoritzKeppler]

Let us describe (and document) that as a pattern: Where do we expect Jackson annotations and for which reasons? Also the consideration/pros&cons of binding domain objects to a technical library. And some consequences to be taken into account ("no secret must get serialized").
If it's for persistence I assume we also have to clarify how version upgrades or loading data (e.g. a TransferProcess) that does not fit the set of loaded extensions will be handled, e.g. a ProvisionedResource needs to be able to perform a sanity check on load etc.
_{Moritz Keppler [email protected], Daimler TSS GmbH, legal info/Impressum}

[DominikPinsel]

Maybe this is a good starting point. From my point of view I would expect Jackson annotations at data, that is persistent in some kind of system or serialized to be send to another system. As the EDC is open source and highly extendable, there could arise problems if you burden our entities to fulfill requirements, that come from serialization features, we cannot know yet.

As far as I see there are currently two business cases. Persistence for reliability and persistence for auditing. If we would implement both cases separately, we could probably come up with a better solution, then making everything serializable.

Here are some problems, we probably should solve before the EDC is used in production:

Persistence for reliability

• It may become necessary to (de-)serialize different persistent versions. For the reliability this problem becomes even larger when you look at the entities like the Resource Definitions, that are implemented by extensions. This means, also all the provision extensions must be able to support all their versions. Something which is probably difficult to realize dependable over all extensions.
• The EDC probably wants some kind of defined behavior, in case problems occur within the de-serialization. This does not only happen when an old version is no longer supported, but also when an extension is missing after a restart or the transfer process was stored by another connector. A problem, where the solution might look very different, depending on the data model.
• Persistence works primarily document based. As the EDC is trying to be very extensible, I think that’s an unnecessary constraint.

Persistence for audible

• If this audible feature is implemented by serializing our complete domain objects and everything another EDC extensions might have put in there, it’s very difficult to control what kind of data is stored. This could become a problem if there is a very large amount of data stored or the data is not GDPR compliant. As there is no separately defined data model for auditing, fixing an issue after years, might become a huge problem.
• Just serializing our EDC entities and storing them for a long period of time, is something that would probably not be de-serializable after a few iterations. It is also something that is very hard (or even impossible) to test.
• Changing extensions could destroy the de-serialization of auditable entities.
• Persistence works primarily document based. As the EDC is trying to be very extensible, I think that’s an unnecessary constraint. Maybe even more so for auditing, where the amount of data could be very large. This may become an issue regarding performance and costs in the cloud (e.g. Azure Cosmos DB).
• In general, the amount of data might become a problem if always the whole data set is persistent and not only the off-set.

That’s just a few thoughts and probably there are many more problems if we implement persistence for reliability and audibility. But I hope you get my point. There might be problems in the future. And these problems are a lot easier to tackle, if we follow separation of concerns.

For now we can go with serializable entities. But in the long run this is not a real solution and we should address the requirements for reliability and audible separatally.
How we solve those problems could also have a big impact on our architecture (e.g. if we want to use event sourcing for audibility), so we should have these discussions in the near future.

_{Dominik Pinsel [email protected], Daimler TSS GmbH, legal info/Impressum}