From 580c2e1d96c87b9442ad79b14037b024a1bb0429 Mon Sep 17 00:00:00 2001 From: Shawn Reuland Date: Tue, 28 Nov 2023 18:43:49 -0800 Subject: [PATCH] #243: include examples of parallel worker configs --- docs/run-platform-server/ingestion.mdx | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/run-platform-server/ingestion.mdx b/docs/run-platform-server/ingestion.mdx index f380d00bd..5f549b7c7 100644 --- a/docs/run-platform-server/ingestion.mdx +++ b/docs/run-platform-server/ingestion.mdx @@ -51,15 +51,24 @@ You can run historical ingestion in parallel in background while your main Horiz #### Parallel ingestion workers -You can break up historical date range into slices and run each in parallel as a separate process: +You can parallelize the ingestion of target historical ledger range by dividing it into sequential slices of smaller ranges and run the db reingest range command for each sub range in parallel as a separate process on the same or a different machine. The shorthand rule for best performance is to identify the number of CPU cores available per target machine, if multi-core, then add `--parallel-workers ` to the command, this will enable the command to further parallelize internally within a single process using multiple threads and sub-divided smaller ranges. ``` -horizon1> stellar-horizon db reingest range 1 10000 -horizon2> stellar-horizon db reingest range 10001 20000 -horizon3> stellar-horizon db reingest range 20001 30000 -# ... etc. +# target range 1 30000, on single machine with 1 CPU core +horizon1> stellar-horizon db reingest range 1 30000 + +# target range 1 30000, on single machine with 4 CPU cores +horizon1> stellar-horizon db reingest range 1 30000 --parallel-workers 4 + +# target range 1 30000, on single machine with 4 CPU cores +horizon1> stellar-horizon db reingest range 1 15000 --parallel-workers 2 +horizon1> stellar-horizon db reingest range 15001 30000 --parallel-workers 2 + +# target range 1 30000, on two machines, each has 2 CPU cores +horizon1> stellar-horizon db reingest range 1 15000 --parallel-workers 2 +horizon2> stellar-horizon db reingest range 15001 30000 --parallel-workers 2 ```