Contract code entry provides catalog of app protocols and events supported

[sreuland]

Background

• Contracts reside and execute on-chain on the Stellar network and can implement application protocols which stipulate expectations for a contract's behavior, interface methods and also optionally any events the contract should emit. Examples of application protocols a contract can implement being custom tokens, defi dapps, dex's, liquidity pools, etc.
• Contracts can emit any events during function Invocations, the events are not bound or correlated specific to the invoked function which emitted it.
• An event has a schema-less model composed of a list of topics and a payload all being ScVal, there are no requirements on format of values set in topics.
• Events emitted by contracts are opaque to Soroban Runtime, the runtime does not inspect or enforce any aspect of the event data.
• Soroban runtime serializes all events emitted by contracts during a Transaction InvokeHostFn operation into TransactionResultMeta xdr per transaction.
• Downstream applications are any remote processes which run off-chain from the Stellar network and parse the TransactionResultMeta xdr emitted from Stellar network upon each closed ledger to retrieve all events.
• Downstream applications want to consume the data model related to application protocols, they iterate all events, inspecting the topics of each event to determine the type of event and possibly inferring the overall application protocol the event/contract are related.
• the notion of an event catalog initially came up in discussion of custom, non-SAC token contract mint event discussion, and seeing more discussions emerge related to other events which contracts adhering to a token protocol should emit.

Problem
Downstream applications cannot determine what app protocols a soroban contract participates or events it may emit from current state of ledger. Instead downstream apps employ various custom approaches in order to identify what contracts adhere to an application protocol and therefore their system can process ensuing events, often requires an ingestion pipeline to parse historical and ongoing event flows to identify contracts which are emitting events with names that align to the application protocol of interest. Or the downstream application maintains a static list of contract addresses for which it knows are related to the protocol of interest.

App protocols which contracts can adhere to such as for tokenization, defi, etc may require the contract implementing programmatic interfaces but the protocols may also require a series or sequence of events to be emitted by the contract during the invocations and lifetime cycles. Each protocol can stipulate these events are required through documentation only, possibly even with overlapping event names of other protocols.

There is no mechanism for the contract programmatic interfaces to enforce the event behaviors, therefore, a contract could implement the protocol's interface but not emit all of the events during invocation, leading to inconsistency for downstream applications to correctly process the contract data relevant to a protocol.

Proposal
Enable key use cases:
(1) Downstream apps can identify latest app protocols and events a contract instance supports while on the network based on its executable reference to the contract's code ledger entry via (ContractCodeEntry) which could provide a standardized catalog of protocol and event names supported by the contract's code.

(2) Downstream apps can focus ingestion pipeline on this catalog from the contract's code ledger entry as a standard way to determine a contract instance's protocol support, and therefore any related event parsing. Allowing app data pipelines to more dynamically parse incoming events accurately to a protocol spec, and remove the need for custom, out-of-band type approaches to determine event parsing rules.

(3) Downstream apps could leverage the catalog ledger entry as a standard way to dynamically discover contracts on the network which are compliant with an app protocol and therefore can invoke functions from the protocol interface as such.

(4) The existing convention of the first topic of an event emitted by contract is the logical name of the event must continue to be followed, as it's the only way to identify discrete event types.

(5) Only the contract code can assert the protocols and events it's able to participate in by specifying the optional catalog at wasm upload time, enhancing the CAP-00460-02 host function to accept the catalog at same time as the wasm upload_wasm(wasm: Bytes, catalog: AppProtocolCatalog) and same update to allow catalog in related InvokeHostFn HOST_FUNCTION_TYPE_UPLOAD_CONTRACT_WASM.

Example of the contract application protocol catalog as an extension of the ContractCodeEntry:

type AppProtocolCatalogEntry struct {
         // if ProtocolName is empty string, then the Events are custom to the contract, and are non-protocol oriented
	ProtocolName        ScSymbol   
        // list of event names as ScSymbol's
        // this depends upon the standard convention of first topic of any emitted event is always this same event name.
        Events                     ScVec<ScSymbol> 
}

type AppProtocolCatalog struct {
        // list of catalog entries
        AppProtocolCatalog        ScVec<AppProtocolCatalogEntry>
}

type ContractCodeEntryV2 struct {
	Ext        ExtensionPoint
	AppProtocolCatalog AppProtocolCatalog
}

[leighmcculloch]

I think contracts already have the components in place to solve problems like this. Contract WASMs have two custom sections that can contain entries that describe both the interface and meta information about the contract within. Any system that indexes contracts can extract that data as needed.

Both the spec and meta live in the custom sections and are unprocessed by the network because they have no impact on the network. In much the same way the app catalogue as proposed above has no use to the network itself, it's just more meta to be interpreted by off-chain systems.

The two custom sections that already exist today are:

• contractspecv0 which contains a stream of SCSpecEntry XDR values. These values today contain functions and types. There's an open issue on the rs-soroban-sdk repo to explore how to include the schema for events as well: Explore including events in contract interfaces / spec rs-soroban-sdk#1097

• contractmetav0 which contains a stream of SCMetaEntry XDR values. These values are key-value pairs. Developers can specify any keys they wish and so standards / patterns for capturing what standards a contract adheres to can be developed by anyone in the ecosystem if it is needed. The values can be specified by calling within the contract code contractmeta!(key = ..., value = ...), or by specifying meta at build time using stellar contract build --meta key=value.

I believe a solution to the problem can be built using the existing meta, and maybe the existing spec if the spec is expanded to capture event schemas (stellar/rs-soroban-sdk#1097). No one has proposed this yet, but it could be as simple as specifying, for example:

• meta key=seps val=40,41 where this indicates a contract implements SEP-40 an SEP-41.

@sreuland If this is something you'd like to experiment with, extracting this information, lmk and we can do a walk through of the logic to get at it and parse it, which is relatively small amount of logic and would be easy to replicate in another application or indexer you're building.

[sreuland]

Not suggesting duplicate, assuming SCSpecEventV0 is created based on rs-soroban-sdk/issues/1097. The proposal here is narrowed to just SCSpecProtocolCatalog and SCSpecProtocolEntry which would refer to existing stream of SCSpecFunctionand SCSpecEvent entries via the Name attribute as a foreign key.

The gain from using SCSpecProtocolCatalog and SCSpecProtocolEntry as the model for contracts to express their protocol assertions rather than SCMeta key-values, has some benefits:

• strongly typed model promotes catalog as real domain entity, with a standard interface to read/write protocol meta information which is self-documenting and all participants, contract authors, upstream/downstream apps converge on through the Spec. Otherwise, in SCMeta key/value schema, the read/write rules for each protocol needs some out-of-band docs at some layer which state what key and value are in SCMeta entries and all participants need to sync on.
• association of events and fn's to a protocol, by enabling the contract author to associate Spec'd events and fn's for the contract to a protocol.
  • acts as an assertion of contract's implementation of the protocol, if author inadvertently missed implementing an event or fn, the absence would be captured in catalog, or the author notices it at catalog creation time and can resolve.
  • contract could emit other custom events than the just the protocol it implements and/or could potentially implement more than one protocol, the catalog would be able help clarify what fn/events belong to each protocol or custom.
  • upstream/downstream apps can verify their implementation of protocol for fn/event expectations matches up to the contract's assertion of same protocol in the catalog.
• scalability, if need to express more protocol information later, the Spec schema accommodates migration, whereas key/value will entail more intricate/complex rules to parse.

[leighmcculloch]

Otherwise, in SCMeta key/value schema, the read/write rules for each protocol needs some out-of-band docs at some layer which state what key and value are in SCMeta entries and all participants need to sync on.

This would be defined in a SEP, which is the same layer at which Specs and Meta live at themselves. It's the same layer that things like contract verification are building on too. Meta allows for greater freedom to innovate which is largely why that's where we're moving to solve most problems like this.

potentially implement more than one protocol, the catalog would be able help clarify what fn/events belong to each protocol or custom.

Over time SEPs will likely layer, for example a contract might impl SEP-41 plus a future SEP that provides additional functions or slightly modifies the interpretation. Very likely and hopefully SEPs will avoid overlapping event names, so it's unclear what the purpose of reiterating that information would be.

Something to consider and weigh up is that contracts today work on a duck typing model. If they quack like a duck, they're probably a duck. And both approaches, duck typing vs the structural model defined in this discussion, have tradeoffs. We'd benefit from concrete examples / use cases that would benefit from adding more structure and therefore trading off on flexibility.

[sreuland]

How do we expect non-SEP protocols to be coded as Spec's by contracts, for example, if a contract implements the ERC-721 standard for NFT, it stipulates fn's and events, would that protocol need a corresponding Stellar protocol SEP to define the Spec Meta key/value adaptation for it? The catalog structure seems like it would address scalability of app protocol encoding to Specs, doesn't require a dependency on SEP if that were the case.

[dmkozh]

FWIW it's not possible to implement any ERC in Soroban directly (because of the difference between the platforms), so I don't think that's the right motivation. I don't think we should promote using ERC terminology in whatever metadata format proposed. SEP is the mechanism for standardization on Stellar and anything else is arbitrary due to ambiguity. I think the next, more granular level of abstraction is the interface, as Leigh has pointed above. I'm not sure it's worth pursuing something 'in-between' these two.

[sreuland]

I don't think we should promote using ERC terminology in whatever metadata format proposed

yes, agreed, ERC encoding is not an intention of the discussion, the example was intended more for NFT as some app protocol, was looking to understand the expectations for non-SEP protocol use cases. App protocols boil down to functions and events and some docs that state when events should be emitted, and it sounds like some protocols could take a formal path as SEP's and contracts encode expected ScSpec/ScMeta entries from that, and potentially there could be other app protocols developed/shared by contracts which publish similar documentation for using the protocol as SEP does but somewhere else like on the author's GitHub repo and these contracts would ideally encode similar ScSpec/ScMeta entries.

We'd benefit from concrete examples / use cases that would benefit from adding more structure and therefore trading off on flexibility.

One type of use case is data warehouse or indexing data pipelines, they will ingest events generically from the network and would like to filter them and index by each unique app protocol on the network, including SEP and non-SEP protocols, so they can categorize network usage metrics by protocol, number of unique contracts implementing the protocol, tx counts, most active source accounts, etc.

Another use case is an indexing data pipeline that depends on parsing events from contracts that correctly implement an app protocol's entire expected event lifecycle. It's been observed anecdotally on pubnet that some custom token contracts are not emitting all the expected events during token lifecycle.

In both of these cases, ideally the backend pipelines can develop a single routine which can automate dynamic determination of the protocol the event belongs to. Which is where the proposal is either to:

• parseScMeta from the corresponding contract which emitted the event for specific key/values denoting the protocol identity with specific key/values which are documented somewhere.
• parse a typed struct of ScSpecProtocolCatalog from the corresponding contract which emitted the event looking for a protocol name that is documented somewhere.

A bit further on the second use case of ingesting events when a contract partially implements the protocol's event lifecycle, the contract catalog asserts the events per protocol that a contract emits, backend applications can automate further against the catalog, such as dropping events from contracts that don't assert the same list of events for a protocol that the backend requires, enabling backend to quarantine the event, redirect it to separate review space, rather than parse it and lead to inconsistent aggregated protocol data, maybe the contract author just missed that event implementation detail from docs.

[janewang]

How do we expect non-SEP protocols to be coded as Spec's by contracts, for example, if a contract implements the ERC-721 standard for NFT, it stipulates fn's and events, would that protocol need a corresponding Stellar protocol SEP to define the Spec Meta key/value adaptation for it? The catalog structure seems like it would address scalability of app protocol encoding to Specs, doesn't require a dependency on SEP if that were the case.

I believe we need a SEP to define the standard for NFT contracts. This SEP should also canonicalize the events emitted by an NFT contract, ensuring consistency across implementations.

For contracts that do not adhere to an existing SEP, they should define their event schemas in the custom WASM section, as Leigh suggested, so consumers know what to expect regarding event structure. Event schema should be required part of a contract interface description. Also Errors too (we will leave that for another day).

[chowbao]

Just curious; What situations would there be NFT contracts that don't adhere to the standard if there is a standard defined?

Like if everyone wants to deviate from the defined standard then the standard is probably too limiting or not well defined.

[sreuland]

For contracts that do not adhere to an existing SEP, they should define their event schemas in the custom WASM section

yes, i think that example is a subset of the wider case of any contract that emits events and how can the contract's WASM section encode protocol information about the events to enable downstream consumers to properly implement event stream handling at the protocol level and apply it accurately to all contract event streams on the network which match up to the protocol. The proposal in this discussion has converged down to slightly different models that could be encoded in the contract's WASM:

• ScMeta key/values provide protocol names that a contract implements.

  • protocol names are not directly associated toSCSpecEvent events defined in the WASM.
  • The keys and values needed to encode protocol name are defined out-of-band from the WASM in a SEP or for non-SEP custom protocols wherever the contract author chooses such as in a GH repo, etc.
  • The downstream consumers either assume the contract implements all events for a stated protocol or downstream additionally inspects the list of ScSpecEvent's implying the assumption they all belong to the protocol to confirm protocol support.
  • This approach doesn't require adding any additional wasm spec.

• ScSpecProtocolCatalog provides protocol names that a contract implements.

  • protocol names are linked to a list of associated ScSpecEvent which the contract asserts it will emit for each protocol name.
  • The protocol name is defined out-of-band from the WASM in a SEP or for non-SEP custom protocols wherever the contract author chooses such as in a GH repo, etc.
  • The downstream consumers either assume the contract implements all events for a stated protocol or downstream additionally inspects the associated list of ScSpecEvent to confirm protocol support.
  • This approach proposes adding new ScSpecProtocolCatalog wasm spec entry.

it's really a matter of which would be most intuitive for contract authors and downstream consumers to use, possibly upstream consumers like wallets also, as the wallet would be coded to interact with contracts at the protocol's fn level and could discover all contracts which assert support to the protocol.