Merge remote-tracking branch 'origin/feat/oracle-v5' into electra-churn

README.md

@@ -3,7 +3,8 @@
 [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 [![Tests](https://github.com/lidofinance/lido-oracle/workflows/Tests/badge.svg?branch=daemon_v2)](https://github.com/lidofinance/lido-oracle/actions)
 
-Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update reports to the Lido smart contracts.
+Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update
+reports to the Lido smart contracts.
 
 ## How it works
 
@@ -15,7 +16,8 @@ There are 3 modules in the oracle:
 
 ### Accounting module
 
-Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck validators and processes user withdrawal requests.
+Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck
+validators and processes user withdrawal requests.
 Also Accounting module makes decision to turn on/off the bunker.
 
 **Flow**
@@ -37,7 +39,8 @@ The frame includes these stages:
 
 ### Ejector module
 
-Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user withdrawals.
+Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user
+withdrawals.
 
 **Flow**
 
@@ -58,14 +61,16 @@ Only Oracle:
 - Memory - 8 GB
 
 Oracle + KAPI:
+
 - vCPU - 4
 - Memory - 16 GB
 
 ## Dependencies
 
 ### Execution Client Node
 
-To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node.
+To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on
+historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node.
 Oracle needs two weeks of archived data.
 
 | Client                                          | Tested | Notes                                                                                                                                                                           |
@@ -77,19 +82,21 @@ Oracle needs two weeks of archived data.
 
 ### Consensus Client Node
 
-Also, to calculate some metrics for bunker mode Oracle needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node.
+Also, to calculate some metrics for bunker mode Oracle
+needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node.
 
-| Client                                            | Tested | Notes                                                                                                                                           |
-|---------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| [Lighthouse](https://lighthouse.sigmaprime.io/)   |   🟢   | Use `--reconstruct-historic-states` param                                                                                                       |
-| [Lodestar](https://lodestar.chainsafe.io)         |   🔴   | Not tested yet                                                                                                                                  |
-| [Nimbus](https://nimbus.team)                     |   🔴   | Not tested yet                                                                                                                                  |
-| [Prysm](https://github.com/prysmaticlabs/prysm)   |   🟢   | Use <br> `--grpc-max-msg-size=104857600` <br> `--enable-historical-state-representation=true` <br> `--slots-per-archive-point=1024` <br> params |
-| [Teku](https://docs.teku.consensys.net)           |   🟢   | Use <br> `--data-storage-mode=archive` <br>`--data-storage-archive-frequency=1024`<br> `--reconstruct-historic-states=true`<br> params          |
+| Client                                          | Tested | Notes                                                                                                                                           |
+|-------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Lighthouse](https://lighthouse.sigmaprime.io/) |   🟢   | Use `--reconstruct-historic-states` param                                                                                                       |
+| [Lodestar](https://lodestar.chainsafe.io)       |   🔴   | Not tested yet                                                                                                                                  |
+| [Nimbus](https://nimbus.team)                   |   🔴   | Not tested yet                                                                                                                                  |
+| [Prysm](https://github.com/prysmaticlabs/prysm) |   🟢   | Use <br> `--grpc-max-msg-size=104857600` <br> `--enable-historical-state-representation=true` <br> `--slots-per-archive-point=1024` <br> params |
+| [Teku](https://docs.teku.consensys.net)         |   🟢   | Use <br> `--data-storage-mode=archive` <br>`--data-storage-archive-frequency=1024`<br> `--reconstruct-historic-states=true`<br> params          |
 
 ### Keys API Service
 
-This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in database.
+This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in
+database.
 
 [Lido Keys API repository.](https://github.com/lidofinance/lido-keys-api)
 
@@ -102,9 +109,11 @@ Pull the image using the following command:
 docker pull lidofinance/oracle:{tag}
 ```
 
-Where `{tag}` is a version of the image. You can find the latest version in the [releases](https://github.com/lidofinance/lido-oracle/releases)
+Where `{tag}` is a version of the image. You can find the latest version in
+the [releases](https://github.com/lidofinance/lido-oracle/releases)
 **OR**\
-You can build it locally using the following command (make sure build it from latest [release](https://github.com/lidofinance/lido-oracle/releases)):
+You can build it locally using the following command (make sure build it from
+latest [release](https://github.com/lidofinance/lido-oracle/releases)):
 
 ```bash
 docker build -t lidofinance/oracle .
@@ -124,16 +133,17 @@ Full variables list could be found [here](https://github.com/lidofinance/lido-or
    and your environment is ready to run the oracle.
 
 ## Run the oracle
+
 1. By default, the oracle runs in *dry mode*. It means that it will not send any transactions to the Ethereum network.
-    To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable:
+   To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable:
     ```
     MEMBER_PRIV_KEY={value}
     ```
-    Where `{value}` is a private key of the Oracle member account or:
+   Where `{value}` is a private key of the Oracle member account or:
     ```
     MEMBER_PRIV_KEY_FILE={path}
     ```
-    Where `{path}` is a path to the private key of the Oracle member account.
+   Where `{path}` is a path to the private key of the Oracle member account.
 2. Run the container using the following command:
 
    ```bash
@@ -203,6 +213,7 @@ In manual mode all sleeps are disabled and `ALLOW_REPORTING_IN_BUNKER_MODE` is T
 | `CACHE_PATH`                                           | Directory to store cache for CSM module                                                                                                                                  | False    | `.`                                     |
 
 ### Mainnet variables
+
 > LIDO_LOCATOR_ADDRESS=0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb
 > ALLOW_REPORTING_IN_BUNKER_MODE=False
 
@@ -281,9 +292,12 @@ Special metrics for ejector oracle:
 
 Special metrics for CSM oracle:
 
-| Metric name                       | Description                                 | Labels |
-|-----------------------------------|---------------------------------------------|--------|
-| TBD                               | TBD                                         |        |
+| Metric name                     | Description                            | Labels |
+|---------------------------------|----------------------------------------|--------|
+| csm_current_frame_range_l_epoch | Left epoch of the current frame range  |        |
+| csm_current_frame_range_r_epoch | Right epoch of the current frame range |        |
+| csm_unprocessed_epochs_count    | Unprocessed epochs count               |        |
+| csm_min_unprocessed_epoch       | Minimum unprocessed epoch              |        |
 
 # Development
 

fixtures/tests/modules/accounting/test_withdrawal_integration.py/test_happy_path.json

@@ -6,7 +6,7 @@
         "to": "0x8D49f1b4AF30598679D4D37Be4B094da1b459b82",
         "data": "0xa3a3fd5d"
       },
-      "0x8ee61584b9d3e010c55f1fa77a803051f5f783385ec75b4e3fc71e199a86184d"
+      "latest"
     ],
     "response": {
       "jsonrpc": "2.0",
@@ -195,7 +195,7 @@
         "to": "0x4c1F6cA213abdbc19b27f2562d7b1A645A019bD9",
         "data": "0x29fd065d"
       },
-      "0x8ee61584b9d3e010c55f1fa77a803051f5f783385ec75b4e3fc71e199a86184d"
+      "latest"
     ],
     "response": {
       "jsonrpc": "2.0",
@@ -218,4 +218,4 @@
       "result": "0x0000000000000000000000000000000000000000000000037d3047cdfd698705000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000030af00000000000000000000000000000000000000000000000000000000000030c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002"
     }
   }
-]
+]

src/constants.py

@@ -12,7 +12,7 @@
 PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX = 3
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#gwei-values
 EFFECTIVE_BALANCE_INCREMENT = 2**0 * 10**9
-MAX_EFFECTIVE_BALANCE = 32 * 10**9
+MAX_EFFECTIVE_BALANCE = Gwei(32 * 10**9)
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/capella/beacon-chain.md#execution
 MAX_WITHDRAWALS_PER_PAYLOAD = 2**4
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes
@@ -32,6 +32,8 @@
 MIN_ACTIVATION_BALANCE = Gwei(2**5 * 10**9)
 MAX_EFFECTIVE_BALANCE_ELECTRA = Gwei(2**11 * 10**9)
 
+LIDO_DEPOSIT_AMOUNT = MIN_ACTIVATION_BALANCE
+
 # Local constants
 GWEI_TO_WEI = 10**9
 SHARE_RATE_PRECISION_E27 = 10**27

src/main.py

@@ -31,7 +31,6 @@
 
 from src.web3py.contract_tweak import tweak_w3_contracts
 
-
 logger = logging.getLogger(__name__)
 
 
@@ -42,22 +41,10 @@ def main(module_name: OracleModule):
         'variables': {
             **build_info,
             'module': module_name,
-            'ACCOUNT': variables.ACCOUNT.address if variables.ACCOUNT else 'Dry',
-            'LIDO_LOCATOR_ADDRESS': variables.LIDO_LOCATOR_ADDRESS,
-            'CSM_MODULE_ADDRESS': variables.CSM_MODULE_ADDRESS,
-            'FINALIZATION_BATCH_MAX_REQUEST_COUNT': variables.FINALIZATION_BATCH_MAX_REQUEST_COUNT,
-            'EL_REQUESTS_BATCH_SIZE': variables.EL_REQUESTS_BATCH_SIZE,
-            'MAX_CYCLE_LIFETIME_IN_SECONDS': variables.MAX_CYCLE_LIFETIME_IN_SECONDS,
+            **variables.PUBLIC_ENV_VARS,
         },
     })
-    ENV_VARIABLES_INFO.info({
-        "ACCOUNT": str(variables.ACCOUNT.address) if variables.ACCOUNT else 'Dry',
-        "LIDO_LOCATOR_ADDRESS": str(variables.LIDO_LOCATOR_ADDRESS),
-        "CSM_MODULE_ADDRESS": str(variables.CSM_MODULE_ADDRESS),
-        "FINALIZATION_BATCH_MAX_REQUEST_COUNT": str(variables.FINALIZATION_BATCH_MAX_REQUEST_COUNT),
-        "EL_REQUESTS_BATCH_SIZE": str(variables.EL_REQUESTS_BATCH_SIZE),
-        "MAX_CYCLE_LIFETIME_IN_SECONDS": str(variables.MAX_CYCLE_LIFETIME_IN_SECONDS),
-    })
+    ENV_VARIABLES_INFO.info(variables.PUBLIC_ENV_VARS)
     BUILD_INFO.info(build_info)
 
     logger.info({'msg': f'Start healthcheck server for Docker container on port {variables.HEALTHCHECK_SERVER_PORT}'})

src/modules/accounting/accounting.py

@@ -216,11 +216,13 @@ def get_updated_modules_stats(
     def _get_consensus_lido_state(self, blockstamp: ReferenceBlockStamp) -> tuple[ValidatorsCount, ValidatorsBalance]:
         lido_validators = self.w3.lido_validators.get_lido_validators(blockstamp)
 
-        count = len(lido_validators)
-        total_balance = Gwei(sum(int(validator.balance) for validator in lido_validators))
+        validators_count = len(lido_validators)
+        active_balance = sum(int(validator.balance) for validator in lido_validators)
+        pending_deposits = self.w3.lido_validators.calculate_pending_deposits_sum(lido_validators)
+        total_balance = Gwei(active_balance + pending_deposits)
 
-        logger.info({'msg': 'Calculate lido state on CL. (Validators count, Total balance in gwei)', 'value': (count, total_balance)})
-        return ValidatorsCount(count), ValidatorsBalance(total_balance)
+        logger.info({'msg': f'Calculate Lido state on CL. {validators_count=}, {active_balance=}, {pending_deposits=}, {total_balance=} (Gwei)'})
+        return ValidatorsCount(validators_count), ValidatorsBalance(total_balance)
 
     def _get_finalization_data(self, blockstamp: ReferenceBlockStamp) -> tuple[FinalizationShareRate, FinalizationBatches]:
         simulation = self.simulate_full_rebase(blockstamp)
@@ -326,7 +328,7 @@ def get_extra_data(self, blockstamp: ReferenceBlockStamp) -> ExtraData:
         logger.info({'msg': 'Calculate stuck validators.', 'value': stuck_validators})
         exited_validators = self.lido_validator_state_service.get_lido_newly_exited_validators(blockstamp)
         logger.info({'msg': 'Calculate exited validators.', 'value': exited_validators})
-        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits(blockstamp.block_hash)
+        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits()
 
         if consensus_version == 1:
             return ExtraDataService.collect(
@@ -350,7 +352,7 @@ def _get_generic_extra_data(self, blockstamp: ReferenceBlockStamp) -> GenericExt
         logger.info({'msg': 'Calculate stuck validators.', 'value': stuck_validators})
         exited_validators = self.lido_validator_state_service.get_lido_newly_exited_validators(blockstamp)
         logger.info({'msg': 'Calculate exited validators.', 'value': exited_validators})
-        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits(blockstamp.block_hash)
+        orl = self.w3.lido_contracts.oracle_report_sanity_checker.get_oracle_report_limits()
         return stuck_validators, exited_validators, orl
 
     def _calculate_report_v1(self, blockstamp: ReferenceBlockStamp) -> ReportData: