Query analytics for FoundationDB. fdbprof tails the client transaction profiles a
cluster records once profile client set is on, writes them to ClickHouse, and a
Grafana dashboard slices them: which key prefixes cost the most time, which keys and
ranges are hot, which ranges transactions conflict on, and latency percentiles per
operation. It is like Percona's Query Analytics, but for FoundationDB 7.3 and later.
fdbprof only reads from the cluster and caps how much it reads per poll. When it
falls behind, it gives up profile history (and the dashboard shows the loss) rather
than add load to FoundationDB.
# Tail a cluster into ClickHouse until stopped; a restart resumes from a cursor kept in ClickHouse
fdbprof -config /etc/fdbprof/config.yaml
# Drain what is in the profile keyspace right now, then exit
fdbprof -config /etc/fdbprof/config.yaml -once
# The same daemon from the published image, with the cluster file passed in as contents
docker run -d --name fdbprof \
-e FDB_CLUSTER_FILE_CONTENTS="$(cat /etc/foundationdb/fdb.cluster)" \
-e FDB_CLUSTER_NAME=prod-east \
-e CLICKHOUSE_ADDRESS=ch-1.internal:9000 \
-e CLICKHOUSE_PASSWORD="$CH_PASSWORD" \
-e INGESTER_ID=fdbprof-prod-east \
-v /etc/fdbprof/config.yaml:/etc/fdbprof/config.yaml:ro \
quay.io/tigrisdata/fdb-profile-analyzer:latest -config /etc/fdbprof/config.yaml
# No ClickHouse: decode the profile keyspace and print the hottest keys and 12-byte key regions
go run ./cmd/profdump -cluster-file /etc/foundationdb/fdb.cluster -top-keys 20 -range-depth 12You can also query ClickHouse directly. This lists the most expensive prefixes over the last hour:
SELECT key_prefix, op_type,
count() AS ops,
sum(latency) / 3600 AS load, -- average operations in flight
quantile(0.99)(latency) AS p99_seconds
FROM fdb_profiles.operations
WHERE cluster = 'prod-east' AND event_time > now() - INTERVAL 1 HOUR
GROUP BY key_prefix, op_type
ORDER BY load DESC
LIMIT 10Setup has four steps: turn on profiling in FoundationDB, create the schema in
ClickHouse, point fdbprof at the cluster file, and import the dashboard into Grafana.
fdbcli --exec 'profile client set 0.01 100MB' # sample 1% of transactions, cap the buffer at 100MB
fdbcli --exec 'profile client get'fdbprof never changes these settings. Every count on the dashboard comes from
transactions sampled at this rate. The size sets the cap on the ring buffer, which
FoundationDB trims oldest-first. If the buffer is too small for your traffic, or the
ingester stops, records are trimmed before fdbprof reads them, and the dashboard
marks the loss as a history gap.
The tables are _local ReplacingMergeTrees behind Distributed tables, which route
over a ClickHouse cluster named fdb. Your remote_servers config must define
that cluster, even on a single node:
<!-- config.d/fdb-cluster.xml: one <shard> per node -->
<clickhouse>
<remote_servers>
<fdb>
<shard>
<replica><host>ch-1.internal</host><port>9000</port><user>fdbprof</user><password>secret</password></replica>
</shard>
</fdb>
</remote_servers>
</clickhouse>Then, on every node (add --host, --user and --password as your setup needs):
clickhouse-client --query 'CREATE DATABASE IF NOT EXISTS fdb_profiles'
for f in schema/*.sql; do clickhouse-client --database fdb_profiles --multiquery < "$f"; done- Keep the database name
fdb_profiles. The dashboard's queries name it explicitly. - Re-running is safe, but nothing migrates. The files only
CREATE ... IF NOT EXISTSand never alter an existing table, so after a schema change, drop and recreate the tables. - Retention is set by TTLs: operations are kept for 30 days, mutations and conflict ranges for 7, and profiling config for 90.
- Every column is documented in docs/SCHEMA.md.
There are three ways to get fdbprof:
- the
fdbprof-linux-amd64binary from releases, which needs FoundationDB's client package forlibfdb_c, like any FoundationDB client; - the image
quay.io/tigrisdata/fdb-profile-analyzer:<version>(linux/amd64,libfdb_cincluded); make buildfrom source.
The config file only needs the values you change from the defaults. config.yaml documents every key.
fdb:
cluster_file: /etc/foundationdb/fdb.cluster
cluster_name: prod-east # the label the dashboard's cluster dropdown selects
clickhouse:
address: ch-1.internal:9000 # native protocol
username: fdbprof
password: secret
prefixes: # your keyspace map; the dashboard groups by it
- /users
- /orders
- /events- Cluster file.
fdbproffinds the cluster file in this order, first match wins:FDB_CLUSTER_FILE_CONTENTS(written to$TMPDIR/fdb.cluster, the easiest option in a container), thenFDB_CLUSTER_FILE, thenfdb.cluster_file, then FoundationDB's default location.fdbprofmust be able to reach the coordinators listed in it. - Run one daemon per FoundationDB cluster, each with its own
fdb.cluster_name(the default isdefault). Any number of daemons can write to the same ClickHouse. - Give each daemon a stable ingester id. The read cursor is stored in ClickHouse
per cluster and ingester id, and the id defaults to the hostname. In a container, set
INGESTER_IDor-ingester-id. Otherwise every recreated container starts over from the oldest record still in the buffer. - Prefixes are the dashboard's main grouping. When prefixes overlap, the longest
match wins. A key that matches no prefix is labelled with its first 16 bytes plus
(unknown), which points at the next prefix worth adding. - Environment variables override the file:
FDB_CLUSTER_NAME,CLICKHOUSE_ADDRESS,CLICKHOUSE_PASSWORD,LOG_FORMAT(jsonortext) andINGESTER_ID. - Two settings worth knowing:
fdb.max_poll_bytescaps how muchfdbprofreads from the cluster per poll (32MB by default).ingest.capture_mutation_values: falsekeeps written values out of ClickHouse. Those values are application payload and may include personal data. Hotspot analysis only needs the keys.
- Flags:
-config,-once,-ingester-id,-verbose,-version.
- Install the ClickHouse plugin with
grafana cli plugins install grafana-clickhouse-datasource, or withGF_INSTALL_PLUGINS=grafana-clickhouse-datasourcein Docker. - Add a ClickHouse data source that uses the native protocol (port 9000). A read-only user is enough.
- Import fdb-profiles.json under Dashboards → New → Import, or put it in a file-provisioned dashboards folder.
The dashboard is at /d/fdb-profile-analyzer. Its first dropdown picks the ClickHouse
data source and the second picks the FoundationDB cluster. It is tested on Grafana 13.0.
The filters at the top are data source, cluster, key prefix, operation and data centre, plus a latency-percentile picker (p99 by default) and the range depth used by Hot keyspace regions. Every panel follows the filters, except the profiling-configuration row, which applies to the whole cluster. These features appear throughout the dashboard:
- Est. columns. Counts cover sampled transactions only. Each Est. column scales its count up by the sample rate to estimate the cluster-wide total.
- Activity sparklines. Each row in a table has one, drawn across the selected time window, which shows whether the row is steadily hot or had a burst.
- Hex keys. Each key is shown both as text and as hex, because FoundationDB keys are arbitrary bytes.
- History gaps. A red annotation across every panel marks where FoundationDB trimmed
records before
fdbprofread them. Charts under-report around a gap.
Profiling configuration
- Client sample rate: the fraction of transactions FoundationDB is sampling, over time. Drag-select a window to zoom the whole dashboard to it.
- Profile buffer: how much of the profile buffer's size cap is in use. When usage nears the cap, FoundationDB starts trimming the oldest records.
- History loss: how many history gaps occurred in the window, and how many records were dropped because their chunks were trimmed before reassembly. Zero is healthy.
Overview
- Operations, Operations/sec, Error rate, p99 latency: headline numbers for the filtered window. The error rate is the share of events with an error code. This p99 is always p99 and ignores the percentile picker.
Query analyzer
- Operations by prefix: one row per prefix and operation, ranked by Load. Load is total time divided by the window length, which is the average number of these operations in flight, so it ranks by cost rather than by frequency. Other columns are count, ops/sec, total, average, p95, p99 and max latency, and errors. Clicking a prefix opens its recent operations.
Throughput and latency
- Throughput by operation:
get,get_range,get_version,commitand the threeerror_*types over time. A flat zero on the errors is healthy. - Latency by operation: the picked percentile per operation type, with errors excluded.
Hotspots
- Hot read keys: the top 50 keys by
getcount, with each key's share, a running cumulative share, average latency and bytes read. "The top 10 keys are 60% of reads" is the kind of result to look for. A flat distribution is a result too. - Hot write keys: the same ranking for
SET_VALUEandADD_VALUEmutations, with bytes written. - Hot scanned ranges: the top 25
get_rangecalls, grouped by exact begin and end keys, with average bytes and latency. One wide scan can cost more than thousands of point reads. - Hot keyspace regions: reads and writes bucketed by their first
range_depthbytes, which shows where traffic lands regardless of call shape. A low Distinct keys count means one hot key; a high count means a hot region. Raise the depth to zoom in.
Conflicts
- Commits and conflicts: successful commits against commits that lost a conflict
(
error_commit) over time. - Conflicting ranges: the conflict ranges carried by losing transactions, most
frequent first.
readrows are what the loser had read before another transaction changed it, andwriterows are what it was trying to write. The range at the top is where contention is worst.
Recent operations
- Recent operations: the latest 200 raw events behind the aggregates, with key, latency, value and range sizes, mutation count, commit bytes, error code and transaction id.
| Path | What |
|---|---|
| cmd/fdbprof | the ingest daemon |
| cmd/profdump | reads, decodes and ranks profile records without ClickHouse |
| cmd/loadgen | profiled load generator for the test bed |
| schema | ClickHouse DDL, column by column in docs/SCHEMA.md |
| deploy/testenv/grafana/dashboards | the dashboard JSON |
| test/integration | end-to-end tests against the test bed |
| docs/contributing.md | commit convention, pull-request checks, how releases are cut |
Every commit subject starts with major:, minor: or patch: (make hooks enforces
this locally). Each merge to main is released as the version those prefixes add up to:
a tag, the fdbprof-linux-amd64 binary with checksums, and the Quay image.
make testenv_up # FoundationDB, ClickHouse and Grafana in Docker, with profiling on and the schema applied
make load_start # continuous profiled load, plus a containerised fdbprof ingesting it
make load_logs # follow bothThen open http://localhost:3000/d/fdb-profile-analyzer, where you are logged in as
admin automatically; the test bed's cluster is named test. Host-side tools use
.testenv/fdb.cluster. make load_stop stops the load, and make testenv_down
removes everything, including data. make integration_test runs the full suite on a
fresh bed and records results in test-results/. make lists every target.