SPIKE: Parsing Ledger Close Meta in Stellar Blockchain to Generate Token Transfer Events

[karthikiyer56]

WIP-------

This issue deals with capturing the task of creating a generic token transfer processor library/package that parses the LedgerCloseMeta to generate a list of TokenTransfer events that detail how balances have been updated for Ledger Entries of the type LedgerEntryTypeAccount or LedgerEntryTypeTrustline

The endgoal for this issue is to define and agree upon a data model for the type TokenTransferEvent struct

Scope:

• Identify classic operations that can cause balance updates. Include relevant information about what was the operation and result that caused the balance to be updated - either debited or credited.
• For token transfers that occur in smart contracts - so long as they conform to SEP-41, track events of the nature: Transfer, Mint, Clawback, and Burn, by parsing the LCM

Not in Scope:

• Custom smart tokens created in smart contracts that don't conform to SEP-41 are not in scope for this work (because those can't be tangibly tracked to begin with)

Basic TokenTransferEvent DataModel

It is intuitive to think of a token transfer as an event that impacts an entity's balance. The entity could be a trustline, an account, or a smart contract.
Transfers can be classified as either a Debit or a Credit, with the reason for the transfer varying widely—stemming from classic operations in Stellar, a transaction fee, or one of the [Transfer, Mint, Burn, Clawback] events generated by a smart contract.

The following represents the basic fields that are common to all TokenTransferEvent, no matter the cause

type TokenTransferEvent struct {
	EntityID string // Represents an account, trustline, or smart contract

	Asset xdr.Asset // or some other representation of the asset, perhaps string??  // The asset involved in the transfer

	Amount float64 // Always positive, represents the quantity transferred

	LedgerSequence uint32 // The Ledger associated that brought about this token transfer

	TxHash Hash // The Hash of the transactin that brought about this token transfer

	TransferType Enum // Type can be Credit or Debit

	Reason Enum // Reason can be Fee, SmartContract, or ClassicOperation.

	/*
	   Interface implemented by specific actions causing this event.
	   Each of classic operations will have different information that they would show
	   For e.g -
	   a ManageOffer Buy/Sell operation might show the claim atoms that were filled as a part of this tokenTransfer
	   a Simple Payment might show the "from" and "to" fields
	   a burn smart contract event will have the details of the the address holding the balance of tokens that was burned
	*/
	Details TokenTransferDetails
}

What causes an account's balance to change

This section highlights all the reasons that can cause an account/trustline/smart contract's balance to be updated, and what can be included in the Details TokenTransferDetails field for each reason.

Broadly, reasons for account balance updates can be categorized as:

• Transaction Fees
• Changes caused by classic operations.
• Events generated by smart contracts

Transaction Fees

Paid for by the source of the transaction when submitted to Horizon or Soroban
Sample Details to be included:

type TransactionFeeDetails struct {
	sourceAccount string
	txhash        string
}

Classic Operations

1. Create Account
   This will cause the source's balance to ⬇ and newly created account's balance to ⬆

type CreateAccountDetails struct {
	sourceAccount      string
	destinationAccount string
}

2. Simple Payment

type SimplePaymentDetails struct {
	sourceAccount      string
	destinationAccount string
}

3. Account Merge
   This will cause the sourceAccount to stop existing (and thus cause it's balance to go ⬇ ), and the destination account's balance to go ⬆

type AccountMergeDetails struct {
	sourceAccount      string
	destinationAccount string
}

4. Clawback
   Note: There is nothing here in details

type ClawbackDetails struct {
}

5. Create Claimable Balance
   This will cause the source account's balance to ⬇

type CreateClaimableBalanceDetails struct {
	destinationAccounts []string
}

6. Claim Claimable Balance
   This will cause the account's (that is claiming this CB) balance to go ⬆

type ClaimClaimableBalanceDetails struct {
	balanceId string
}

7. Clawback Claimable Balance
   NOTE - A ClawbackClaimableBalance operation does not cause any account or trustline's balance to go up or down. Meaning, there never will be a TokenTransferEvent generated in response to a ClawbackClaimableBalanceOperation. I am simply adding this section for articulating the above point.

8. LiquidityPool Deposit

9. LiquidityPool Withdraw

10. Manage Offer Buy
    The kind of events to expect when a ManageBuyOffer operation is successful is best understood with an example.
    Existing State of the Ledger:
    Suppose 3 sell offers of BTC-USD exist on the orderbook.
    1. offerId = 123, sellerID = account X
    2. offerId = 234, sellerID = account Y
    3. offerID = 456, sellerId = account Z

    Given a ManageOfferBuy operation for BTC-USD with a sourceAccount = Account P, that fills against the 3 existing offers, we will generate 6 events for account P, and 2 each for account X, Y, Z, so a total of 12 events.
    In general, number of transfer events = len(claimAtom list) * 2 * 2

    1. eventType = debit, account = P, asset = USD, amount = (amountBought from the claimAtom corresponding to offerId=123), fromAccount = P, toAccount = X
    2. eventType = debit, account = P, asset = USD, amount = (amountBought from the claimAtom corresponding to offerId=234), fromAccount = P, toAccount = Y
    3. eventType = debit, account = P, asset = USD, amount = (amountBought from the claimAtom corresponding to offerId=456), fromAccount = P, toAccount = Z
    4. eventType = credit, account = P, asset = BTC, amount = (amountSold from the claimAtom corresponding to offerId=123), fromAccount = X, toAccount = P
    5. eventType = credit, account = P, asset = BTC, amount = (amountSold from the claimAtom corresponding to offerId=234), fromAccount = Y, toAccount = P
    6. eventType = credit, account = P, asset = BTC, amount = (amountSold from the claimAtom corresponding to offerId=456), fromAccount = Z, toAccount = P
    7. eventType = credit, account = X, asset = USD, amount = (amountBought from the claimAtom corresponding to offerId=123), fromAccount = P, toAccount = X (inverse of 1)
    8. eventType = credit, account = Y, asset = USD, amount = (amountBought from the claimAtom corresponding to offerId=234), fromAccount = P, toAccount = Y (inverse of 2)
    9. eventType = credit, account = Z, asset = USD, amount = (amountBought from the claimAtom corresponding to offerId=456), fromAccount = P, toAccount = Z (inverse of 3)
    10. eventType = debit, account = X, asset = BTC, amount = (amountSold from claimAtom corresponding to offerId=123), fromAccount = X, toAccount = P (inverse of 4)
    11. eventType = debit, account = Y, asset = BTC, amount = (amountSold from claimAtom corresponding to offerId=234), fromAccount = Y, toAccount = P (inverse of 5)
    12. eventType = debit, account = Z, asset = BTC, amount = (amountSold from claimAtom corresponding to offerId=456), fromAccount = Z, toAccount = P (inverse of 6)

This is what will be included in the Details section

type ManageOfferBuyDetails struct {
	fromAccount string    // AccountId that is debited
	toAccount   string       // AccountId that is credited
	offerId int64                // The resting offerId that was filled as a part of this transfer
}

11. Manage Sell Offer

12. Path Payment Strict Send

13. Path Payment Strict Receive

Smart Contract Events

Smart Contract events are emiited as a part of the transaction meta when the an InvokeHostFn operation is executed.
There are four types of events emitted that deal with token movement, as described in SEP-41 and CAP-46-6

**Note: The names of the fields in these events are named so as to conform to the topic and field names as specified in CAP-46-6 **

Refer to these snippets in CAP-46-6

    /// Emits an event with:
    /// - topics - `["transfer", from: Address, to: Address]`
    /// - data - `amount: i128`
....
    /// Emits an event with:
    /// - topics - `["burn", from: Address]`
    /// - data - `amount: i128`
....
/// Emits an event with topics `["mint", admin: Address, to: Address, sep0011_asset: String], data
/// = amount: i128`
....
/// Emits an event with topics `["clawback", admin: Address, from: Address, sep0011_asset: String],
/// data = amount: i128`

1. Transfer

type SmartContractTransferEventDetails struct {
	from string  // represents either a contractID or an accountId
	to string
}

2. Mint

type SmartContractMintEventDetails struct {
	admin string // represents either a contractID or an accountId
	to    string
}

3. Burn

type SmartContractBurnEventDetails struct {
	from string // represents either a contractID or an accountId
}

4. ClawBack

type SmartContractClawbackEventDetails struct {
	admin string // represents either a contractID or an accountId
	from    string
}

[Shaptic]

The Amount should be a string to avoid imprecision, or a whole integer without a decimal involved. You can leverage our amount package to make this easier.

For SAC events, note that the support/contractevents package should make this easier, as well.

[sreuland]

Reason could be relocated internally as attribute within Details specific to each transfer type.
Type is a bit ambiguous, it's really indicating the direction of transfer, maybe Basis

thinking further on potential for event portability over-the-wire and model evolutions over time, might be able to address those concerns with serialization pattern and defer the real transfer event schema to an IDL rather than manual code binding:

TokenTransferEvent {
     SchemaName string 
     Payload []byte
}

It would use a deser lib and its associated tooling, to define an IDL per transfer event schema and leverage code-gen from tooling for client-side code binding to the IDL in various languages. rather than manual code binding(struct per detail transfer type per language). FlatBuffers as a suggestion. Could store the transfer event IDL's in GH repo as the 'schema registry', clients in various langauges would refer to that for code-gen.

[karthikiyer56]

Reason could be relocated internally as attribute within Details specific to each transfer type.

That is a good idea. I will make the change

Type is a bit ambiguous, it's really indicating the direction of transfer, maybe Basis

I agree that Type is very ambigous. But. I'd argue so is Basis. Perhaps a more descriptive name would be TransferDirection ?

Re portablility and Use of IDL + schema registry:
If you ask me, the best way to use would have to use ProtoBuf. There is no better IDL than protbuf - Define your message schema and run it through a code generator and you will get bindings for each language.
No schema registry of any sorts required. Your code/proto definitions act as the schema themselves and you ensure backwards compatiblity by following proper field naming conventions.

Protobufs are better than creating the Event as a payload of bytes and embedding the schema name in it, as you are suggesting.

That being said, that is a bit too expansive for this ask. By ask, I mean the "new processors library" in general
Besides, that still doestn take away from the fact that you still need to write SER/DESER code, and maintain it to be able to parse the message/event from the wire (assuming you want it to be portable across languages). You'd have to write bindings for each language that you choose to support regardless.

TL;dr - I think that might be overkill

[Shaptic]

If each field has the appropriate src/dst (or equivalent, if possible) in its details, then there's no need for a TransferDirection since it's implied by amount always being positive.

[karthikiyer56]

If each field has the appropriate src/dst (or equivalent, if possible) in its details, then there's no need for a TransferDirection since it's implied by amount always being positive.

Three reasons to have the TransferDirection field in the result:

• Imagine a usecase where you are a wallet like Freighter and using a list of TokenTransferEvents to build your own datastores. Now on the Freighter UI, you want to only show credits (or debits) for an account, and the only thing you will need to filter on at is TransferDirection.

• It is not always true that a transferEvent will have a src/dst.

  • For e.g, in case of clawbacks, claimable clawbacks and smart contract burn type events.

  • Consider a create claimable balance operation. In this case the source is known. i.e the amount will be debited from the source's account. We dont know the destination yet, especially if you specify more than one destinations. Following that train of thought, when you have a claim claimable balance operation, all that you have (currently) is the balanceId. So, the account that is specified in the operation is credited with the amount, but there is no way to identify what the source might be in this case. We only have the balanceId field to go by, and that doesnt give an indication of who the creator (debited account) of the CB was.

[Shaptic]

True, agreed that having the field makes filtering and rendering far simpler. Otherwise, like in the situations you outlined, while you could still infer direction (from: me, to: null, is leaving my account but it's unclear where), it would require introspection of the details to a degree that defeats the very purpose of this simplified event structure.

[sreuland]

Was going to suggest AssetAccountingBasis for what is currently TransferDirection, b/c at least in general accounting, asset accounts have debit balances, a debit increases the asset's balance, and vice versa for credits.

There is no better IDL than protbuf - Define your message schema and run it through a code generator and you will get bindings for each language.
No schema registry of any sorts required.

The Protobuffers .protobuf files are the schema registry, those are external and need to exist somewhere, same as .fbs files for IDL source in FlatBuffers. Both tools are similar, they propose an IDL file and provide codegen tooling for deser purposes, either would be usable in this kind of proposal, Avro probably would be another option too.

you still need to write SER/DESER code, and maintain it to be able to parse the message/event from the wire (assuming you want it to be portable across languages). You'd have to write bindings for each language that you choose to support regardless.

not sure what is meant since that is what the codegen'd stubs from an IDL deser tooling provide, maybe you are inferring to how the app selects which generated code to use for deser of payloads. The token transfer processor will be implemented in go, what is the intended interface, i.e. is it a pure function with an input of LCM and an output of []TokenTransferEvent, assuming something similar to that and using CDP, can sketch out depiction of IDL usage, it could be applicable for just TokenTransferDetails or the entire TokenTransferEvent:

[sreuland]

another variation that could happen in parallel and leverages the usage of IDL is to package the go token transformer implementation into an executable cli tool which can run as o/s process which just emits serialized TokenTransferEvents to some type of external store, such as MQ. In this design, the downstream application can be written in any programming language which IDL tooling has a compiler and application focuses on smaller coding effort to just consume TokenTransferEvent from the MQ, it doesn't have to implement additional pipeline concerns to run a ledger backend consumer(cdp or captive core) and the token transfer transformer.

[gouthamp-stellar]

Nice work Karthik! I'm taking a closer look at this and will add some more thoughts later, but on first jump, I noticed that the TokenTransferEvent does not have the operation/transaction that caused the token transfer, which is something we need for the state change indexer wallet backend will be building. Maybe add that as a field to TokenTransferDetails?

[karthikiyer56]

but on first jump, I noticed that the TokenTransferEvent does not have the operation/transaction that caused the token transfer

The Reason enum is going to give you information about operation, if the change is precipitated by a classic operation.
It will have about 10-15 values.
Something like this

type TokenTranferEventReason uint32

const (
	TokenTransferEventReasonUnknown TokenTranferEventReason = iota
	TokenTransferEventReasonTxFees
	TokenTransactionEventReasonSmartContractEvent
	TokenTransferEventReasonCreateAccount
	TokenTransferEventReasonSimplePayment
	TokenTransferEventReasonPathPaymentStrictSend
	TokenTransferEventReasonPathPaymentStrictReceive
	TokenTransferEventReasonMergeAccount
	....
	....
	
)

Other than that, you dont need the operation or the operation result, since you will use the Details field to grab the cause and effect of that operation.
And those details have enough information about the operation result.
for e.g if reason == TokenTransferEventReasonSimplePayment, then Details will be of the Type SimplePaymentDetails, which has a sourceAccount and destinationAccount

You really shouldnt be having to work with the operation itself when going through a processor

---

transaction that caused the token transfer

Thanks for bringing that up. The txHash and LedgerSequence will be added to the TokenTransferEvent. I forgot to add that.
Have updated the base TokenTransferEvent struct in the description with these 2 fields now

[gouthamp-stellar]

I understand that the Details field captures the cause and effect of an operation, but wallet backend would like the option of presenting to a client the operation as well as the resulting state change. Clients may want to inspect this operation for their own purposes/reasons. They may want more details than what is captured by the Details field. You could argue that the txhash may be sufficient for this, but we don't want to dig into the transaction and try to figure out the exact operation that caused the change

It may also reduce the burden of having to model complex operations like path strict send. The actual change can only capture some key details of the token transfer and the rest of the details clients can get by deserializing the operation xdr string

[gouthamp-stellar]

You mention a source account in your CreateClaimableBalanceDetails struct and yet I don't see it in the struct definition:

type CreateClaimableBalanceDetails struct {
destinationAccounts []string
}

Is there a reason for this?

Also if it is a source account's balance that goes down, why list all the destinationAccounts here (whose balance only goes up when they claim this claimable balance), although I get that this information is available in the operation ?

[gouthamp-stellar]

For ClaimClaimableBalanceDetails, how do we know the account that actually claimed the balance?

type ClaimClaimableBalanceDetails struct {
balanceId string
}

[gouthamp-stellar]

To answer your question about ClawbackClaimableBalanceDetails

we'd like to model (at least for now) getting assigned as a claimant on the balance as a state change, so yes, we'd like this to be included as well

[JakeUrban]

Maybe we can use "pending_credit" or "pending_debit" for claimable balances, since they aren't crediting or debiting your balance until they're claimed.

[karthikiyer56]

You mention a source account in your CreateClaimableBalanceDetails struct and yet I don't see it in the struct definition:

This particular TokenTransferEvent that has this CreateClaimableBalanceDetails section is generated in response to a CreateClaimableBalance operation.
The entityId in the top-level TokenTransferEvent is the source account, whose balance was immediately debited by the said amount after the operation was completed. So, there is no need for sourceAccount to show up in the details

---

why list all the destinationAccounts here

The TokenTransferEvent was caused in response to the operation CreateClaimableBalanceOperation, and will be claimed by one of the destinationAccounts presented in the Details in due time.
To rephrase it, think of this usecase in Freighter UI - "I can see that my balance was deducted because of a claimableBalance operation I created. And the recipients can are one amongst this list"

---

For ClaimClaimableBalanceDetails, how do we know the account that actually claimed the balance?

Same reasoning as above. This particular TokenTransferEvent that has this ClaimClaimableBalanceDetails section is generated in response to a ClaimClaimableBalance operation. The entityId is the destination account that claimed this claimable balance

--

Maybe we can use "pending_credit" or "pending_debit" for claimable balances, since they aren't crediting or debiting your balance until they're claimed.

This is not true.
The moment a CB is created, the source account's balance is reduced by that much. So, there is no notion of pending per se. And when you claim it, the claimant account's balance is increased by that much. When working on these events, I am working in isolation. Meaning, I cannot look at a BalanceId in the input operation and identify who created the claimable balance entry.

---

To answer your question about ClawbackClaimableBalanceDetails
we'd like to model (at least for now) getting assigned as a claimant on the balance as a state change, so yes, we'd like this to be included as well

That comment was more of a footnote for me. A ClawbackClaimableBlaance operation does not generate a token transfer of any sort. Meaning, no account or trustline's balance is debited or credited. And so, as such, it doesnt contribute or generate a tokenTransferEvent. It was incorrect of me to include that operation as something that can cause somethig's balance to go up or down. I am going to remove that operation from the list of operations.

EDIT: I didnt remove it, but rather left a note detailing as to why this operation wont generate a TokenTransferEvent

[gouthamp-stellar]

Also, instead of a txHash, can you add the actual tx xdr string instead? I feel like that's a more useful field since it can be deserialized and the transaction examined if need be by anyone using this library.

[gouthamp-stellar]

Question - would it be possible to track WHEN a TokenTransferEvent took place? In the indexer we are building for the wallet backend, we'd like to have this piece of information so users can get state changes for their accounts for a particular time slice

[gouthamp-stellar]

Your design for CreateClaimableBalanceDetails and ClaimClaimableBalanceDetails makes sense, but now I'm curious - why have a source account for

type CreateAccountDetails struct {
sourceAccount string
destinationAccount string
}

type SimplePaymentDetails struct {
sourceAccount string
destinationAccount string
}

and

type AccountMergeDetails struct {
sourceAccount string
destinationAccount string
}

Couldn't the above events be modeled the same way? For example, for SimplePaymentDetails, can't the entityId be the sourceAccount?