RPC Branch for Persistent Entry Eviction

docs/integration.md

@@ -19,7 +19,7 @@ stellar-core generates several types of data that can be used by applications, d
 
 Full [Ledger](ledger.md) snapshots are available in both:
   * [history archives](history.md) (checkpoints, every 64 ledgers, updated every 5 minutes)
-  * in the case of captive-core (enabled via the `--in-memory` command line option) the ledger is maintained within the stellar-core process and ledger-state need to be tracked as it changes via "meta" updates.
+* in the case of captive-core the ledger is maintained within the stellar-core process and ledger-state need to be tracked as it changes via "meta" updates.
 
 ## Ledger State transition information (transactions, etc)
 

docs/quick-reference.md

@@ -147,27 +147,16 @@ transactions or ledger states) must be downloaded and verified sequentially. It
 worthwhile to save and reuse such a trusted reference file multiple times before regenerating it.
 
 ##### Experimental fast "meta data generation"
-`catchup` has a command line flag `--in-memory` that when combined with the
-`METADATA_OUTPUT_STREAM` allows a stellar-core instance to stream meta data instead
-of using a database as intermediate store.
+`catchup` when combined with the
+`METADATA_OUTPUT_STREAM` allows a stellar-core instance to stream meta data.
 
 This has been tested as being orders of magnitude faster for replaying large sections
 of history.
 
 If you don't specify any value for stream the command will just replay transactions
 in memory and throw away all meta. This can be useful for performance testing the transaction processing subsystem.
 
-The `--in-memory` flag is also supported by the `run` command, which can be used to
-run a lightweight, stateless validator or watcher node, and this can be combined with
-`METADATA_OUTPUT_STREAM` to stream network activity to another process.
-
-By default, such a stateless node in `run` mode will catch up to the network starting from the
-network's most recent checkpoint, but this behaviour can be further modified using two flags
-(that must be used together) called `--start-at-ledger <N>` and `--start-at-hash <HEXHASH>`. These
-cause the node to start with a fast in-memory catchup to ledger `N` with hash `HEXHASH`, and then
-replay ledgers forward to the current state of the network.
-
-A stateless and meta-streaming node can additionally be configured with
+A meta-streaming node can additionally be configured with
 `EXPERIMENTAL_PRECAUTION_DELAY_META=true` (if unspecified, the default is
 `false`).  If `EXPERIMENTAL_PRECAUTION_DELAY_META` is `true`, then the node will
 delay emitting meta for a ledger `<N>` until the _next_ ledger, `<N+1>`, closes.

docs/software/commands.md

@@ -159,13 +159,7 @@ apply.
   checkpoint from a history archive.
 * **run**: Runs stellar-core service.<br>
   Option **--wait-for-consensus** lets validators wait to hear from the network
-  before participating in consensus.<br>
-  (deprecated) Option **--in-memory** stores the current ledger in memory rather than a
-  database.<br>
-  (deprecated) Option **--start-at-ledger <N>** starts **--in-memory** mode with a catchup to
-  ledger **N** then replays to the current state of the network.<br>
-  (deprecated) Option **--start-at-hash <HASH>** provides a (mandatory) hash for the ledger
-  **N** specified by the **--start-at-ledger** option.
+  before participating in consensus.
 * **sec-to-pub**:  Reads a secret key on standard input and outputs the
   corresponding public key.  Both keys are in Stellar's standard
   base-32 ASCII format.
@@ -608,6 +602,8 @@ enabled by specifying a port via the HTTP_QUERY_PORT config setting.
 ### Query Commands
 
 * **`getledgerentryraw`**<br>
+  Used to query arbitrary key-value pairs in the live BucketList.<br>
+
   A POST request with the following body:<br>
 
   ```
@@ -640,3 +636,126 @@ enabled by specifying a port via the HTTP_QUERY_PORT config setting.
   value of the `TTL` entry to the current ledger sequence number.
 
   `ledgerSeq` gives the ledger number on which the query was performed.
+
+* **`getledgerentry`**<br>
+  Used to query both live and archived LedgerEntries. While `getledgerentryraw` does simple key-value lookup
+  on the live ledger, `getledgerentry` will query a given key in both the live BucketList, as well as archives.
+  It will also check to see what proofs are required for the given key, if any, report back whether an entry is
+  archived or live, and return the entry's current TTL value.
+
+  A POST request with the following body:<br>
+
+  ```
+  ledgerSeq=NUM&key=Base64&key=Base64...
+  ```
+
+  * `ledgerSeq`: An optional parameter, specifying the ledger snapshot to base the query on.
+  If the specified ledger in not available, a 404 error will be returned. If this parameter
+  is not set, the current ledger is used.
+  * `key`: A series of Base64 encoded XDR strings specifying the `LedgerKey` to query. TTL keys
+  must not be queried and will return 404 if done so.
+
+  A JSON payload is returned as follows:
+
+  ```
+  {
+    "entries": [
+      {"e": "Base64-LedgerEntry", "state": "live", /*optional*/ "ttl": uint32 },
+      {"e": "Base64-LedgerKey", "state": "new_entry_no_proof"},
+      {"e": "Base64-LedgerKey", "state": "new_entry_proof"},
+      {"e": "Base64-LedgerEntry", "state": "archived_no_proof"},
+      {"e": "Base64-LedgerEntry", "state": "archived_proof"},
+  ],
+  "ledger": ledgerSeq
+  }
+  ```
+
+  * `entries`: A list of entries for each queried LedgerKey. Every `key` queried is guarenteed to
+  have a corresponding entry returned.
+  * `e`: Either the `LedgerEntry` or `LedgerKey` for a given key encoded as a Base64 string. If a key
+  is live or archived, `e` contains the corresponding `LedgerEntry`. If a key does not exist
+  (including expired temporary entries) `e` contains the corresponding `LedgerKey`.
+  * `state`: One of the following values:
+    * `live`: Entry is live.
+    * `new_entry_no_proof`: Entry does not exist and does not require creation proof.
+    * `new_entry_proof`: Entry does not exist and requires creation proof.
+    * `archived_no_proof`: Entry is archived and can be restored without a proof.
+    * `archived_proof`: Entry is archived and requires restoration proof.
+  * `ttl`: An optional value, only returned for live Soroban entries. Contains
+  a uint32 value for the entry's `liveUntilLedgerSeq`.
+  * `ledgerSeq`: The ledger number on which the query was performed.
+
+  Classic entries will always return a state of `live` or `new_entry_no_proof`.
+  If a classic entry does not exist, it will have a state of `new_entry_no_proof`.
+
+  Similarly, temporary Soroban entries will always return a state of `live` or
+  `new_entry_no_proof`. If a temporary entry does not exist or has expired, it
+  will have a state of `new_entry_no_proof`.
+
+  This endpoint will always give correct information for archived entries. Even
+  if an entry has been archived and evicted from validators, this endpoint will
+  still the archived entry's full `LedgerEntry` as well as the proper state.
+
+* **`getinvokeproof`**<br>
+
+  Used to generate creation proofs, required when writing new keys that have an archival filter false positive. For testing, use `ARTIFICIALLY_SIMULATE_ARCHIVE_FILTER_MISS` to simulate this.
+
+  A POST request with the following body:<br>
+
+  ```
+  ledgerSeq=NUM&key=Base64&key=Base64...
+  ```
+
+  * `ledgerSeq`: An optional parameter, specifying the ledger snapshot to base the query on.
+  If the specified ledger in not available, a 404 error will be returned. If this parameter
+  is not set, the current ledger is used.
+  * `key`: A series of Base64 encoded XDR strings specifying the `LedgerKey` prove. TTL keys
+  must not be queried and will return 404 if done so.
+
+  A JSON payload is returned as follows:
+
+  ```
+  {
+  "ledger": uint32,  // Required
+  "proof": Base64XDR // optional
+  }
+  ```
+
+  * `proof`: A Base64 encoded XDR string of the proof payload required for the associated
+  `InvokeHostFunctionOp`. If no proof is required, this field is ommited.
+  * `ledgerSeq`: The ledger number on which the query was performed.
+
+  Note that only a single proof object is required for all keys in a given TX.
+
+* **`getrestoreproof`**<br>
+
+  Used to generate restoration proofs, required when writing new keys that have an archival filter false positive. For testing, use `REQUIRE_PROOFS_FOR_ALL_EVICTED_ENTRIES` for easier simulation.
+
+  A POST request with the following body:<br>
+
+  ```
+  ledgerSeq=NUM&key=Base64&key=Base64...
+  ```
+
+  * `ledgerSeq`: An optional parameter, specifying the ledger snapshot to base the query on.
+  If the specified ledger in not available, a 404 error will be returned. If this parameter
+  is not set, the current ledger is used.
+  * `key`: A series of Base64 encoded XDR strings specifying the `LedgerKey` prove. TTL keys
+  must not be queried and will return 404 if done so.
+
+  A JSON payload is returned as follows:
+
+  ```
+  {
+  "ledger": uint32,  // Required
+  "proof": Base64XDR // optional
+  }
+  ```
+
+  * `proof`: A Base64 encoded XDR string of the proof payload required for the associated
+  `RestoreFootprintOp`. If no proof is required, this field is ommited.
+  * `ledgerSeq`: The ledger number on which the query was performed.
+
+  Note that only a single proof object is required for all keys in a given TX.
+  This endpoint will always give correct information for all archived entries, even if
+  an entry has been evicted.

docs/stellar-core_example.cfg

@@ -229,14 +229,6 @@ FLOOD_DEMAND_BACKOFF_DELAY_MS = 500
 #   against each other.
 MAX_DEX_TX_OPERATIONS_IN_TX_SET = 0
 
-# DEPRECATED_SQL_LEDGER_STATE (bool) default false
-# When set to true, SQL is used to store all ledger state instead of
-# BucketListDB. This is not recommended and may cause performance degregradation.
-# This is deprecated and will be removed in the future. Note that offers table
-# is still maintained in SQL when this is set to false, but all other ledger
-# state tables are dropped.
-DEPRECATED_SQL_LEDGER_STATE = false
-
 # BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT (Integer) default 14
 # Determines page size used by BucketListDB for range indexes, where
 # pageSize == 2^BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT. If set to
@@ -258,11 +250,6 @@ BUCKETLIST_DB_INDEX_CUTOFF = 20
 # this value is ingnored and indexes are never persisted.
 BUCKETLIST_DB_PERSIST_INDEX = true
 
-# BACKGROUND_EVICTION_SCAN (bool) default true
-# Determines whether eviction scans occur in the background thread. Requires
-# that DEPRECATED_SQL_LEDGER_STATE is set to false.
-BACKGROUND_EVICTION_SCAN = true
-
 # EXPERIMENTAL_BACKGROUND_OVERLAY_PROCESSING (bool) default false
 # Determines whether some of overlay processing occurs in the background
 # thread.
@@ -601,17 +588,12 @@ MAX_SLOTS_TO_REMEMBER=12
 # only a passive "watcher" node.
 METADATA_OUTPUT_STREAM=""
 
-# Setting EXPERIMENTAL_PRECAUTION_DELAY_META to true causes a stateless node
+# Setting EXPERIMENTAL_PRECAUTION_DELAY_META to true causes a node
 # which is streaming meta to delay streaming the meta for a given ledger until
 # it closes the next ledger. This ensures that if a local bug had corrupted the
 # given ledger, then the meta for the corrupted ledger will never be emitted, as
 # the node will not be able to reach consensus with the network on the next
 # ledger.
-#
-# Setting EXPERIMENTAL_PRECAUTION_DELAY_META to true in combination with a
-# non-empty METADATA_OUTPUT_STREAM (which can be configured on the command line
-# as well as in the config file) requires an in-memory database (specified by
-# using --in-memory on the command line).
 EXPERIMENTAL_PRECAUTION_DELAY_META=false
 
 # Number of ledgers worth of transaction metadata to preserve on disk for

docs/stellar-core_example_validators.cfg

@@ -4,7 +4,6 @@ PUBLIC_HTTP_PORT=false
 NETWORK_PASSPHRASE="Example configuration"
 
 DATABASE="sqlite3://example.db"
-DEPRECATED_SQL_LEDGER_STATE = false
 
 NODE_SEED="SA7FGJMMUIHNE3ZPI2UO5I632A7O5FBAZTXFAIEVFA4DSSGLHXACLAIT a3"
 NODE_HOME_DOMAIN="domainA"

docs/stellar-core_standalone.cfg

@@ -12,7 +12,6 @@ NODE_IS_VALIDATOR=true
 
 #DATABASE="postgresql://dbname=stellar user=postgres password=password host=localhost"
 DATABASE="sqlite3://stellar.db"
-DEPRECATED_SQL_LEDGER_STATE = false
 
 COMMANDS=["ll?level=debug"]
 

docs/stellar-core_testnet.cfg

@@ -4,7 +4,6 @@ PUBLIC_HTTP_PORT=false
 NETWORK_PASSPHRASE="Test SDF Network ; September 2015"
 
 DATABASE="sqlite3://stellar.db"
-DEPRECATED_SQL_LEDGER_STATE = false
 
 # Stellar Testnet validators
 [[HOME_DOMAINS]]

docs/stellar-core_testnet_legacy.cfg

@@ -9,7 +9,6 @@ KNOWN_PEERS=[
 "core-testnet3.stellar.org"]
 
 DATABASE="sqlite3://stellar.db"
-DEPRECATED_SQL_LEDGER_STATE = false
 UNSAFE_QUORUM=true
 FAILURE_SAFETY=1
 

docs/stellar-core_testnet_validator.cfg

@@ -4,7 +4,6 @@ PUBLIC_HTTP_PORT=false
 NETWORK_PASSPHRASE="Test SDF Network ; September 2015"
 
 DATABASE="sqlite3://stellar.db"
-DEPRECATED_SQL_LEDGER_STATE = false
 
 # Configuring the node as a validator
 # note that this is an unsafe configuration in this particular setup: