Skip to content

Commit

Permalink
#243: include examples of parallel worker configs
Browse files Browse the repository at this point in the history
  • Loading branch information
sreuland committed Nov 29, 2023
1 parent 440fba8 commit 580c2e1
Showing 1 changed file with 14 additions and 5 deletions.
19 changes: 14 additions & 5 deletions docs/run-platform-server/ingestion.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,24 @@ You can run historical ingestion in parallel in background while your main Horiz

#### Parallel ingestion workers

You can break up historical date range into slices and run each in parallel as a separate process:
You can parallelize the ingestion of target historical ledger range by dividing it into sequential slices of smaller ranges and run the db reingest range command for each sub range in parallel as a separate process on the same or a different machine. The shorthand rule for best performance is to identify the number of CPU cores available per target machine, if multi-core, then add `--parallel-workers <num of cores>` to the command, this will enable the command to further parallelize internally within a single process using multiple threads and sub-divided smaller ranges.

<CodeExample>

```
horizon1> stellar-horizon db reingest range 1 10000
horizon2> stellar-horizon db reingest range 10001 20000
horizon3> stellar-horizon db reingest range 20001 30000
# ... etc.
# target range 1 30000, on single machine with 1 CPU core
horizon1> stellar-horizon db reingest range 1 30000
# target range 1 30000, on single machine with 4 CPU cores
horizon1> stellar-horizon db reingest range 1 30000 --parallel-workers 4
# target range 1 30000, on single machine with 4 CPU cores
horizon1> stellar-horizon db reingest range 1 15000 --parallel-workers 2
horizon1> stellar-horizon db reingest range 15001 30000 --parallel-workers 2
# target range 1 30000, on two machines, each has 2 CPU cores
horizon1> stellar-horizon db reingest range 1 15000 --parallel-workers 2
horizon2> stellar-horizon db reingest range 15001 30000 --parallel-workers 2
```

</CodeExample>
Expand Down

0 comments on commit 580c2e1

Please sign in to comment.