Skip to content

monitoring.md

Latest commit

 

History

History
89 lines (65 loc) · 3.79 KB

monitoring.md

File metadata and controls

89 lines (65 loc) · 3.79 KB
Raw

Monitoring & Metrics

Each node plugin exposes Prometheus metrics on metrics.port (default 9100). Enable scraping either through your own scrape config or via the chart's metrics.serviceMonitor.enabled=true (Prometheus Operator), with the scrape interval set by metrics.serviceMonitor.interval (default 1m). Metrics can be disabled entirely with metrics.enabled=false.

Gauges are organized in three aggregation levels: node, pool and volume.

Node-level

Metric Labels Description
rawfile_remaining_capacity_bytes node Free capacity for new volumes on this node (excluding reserved storage).

Per-pool

Metric Labels Description
rawfile_pool_capacity_bytes node, pool Capacity allocated to the pool for provisioning (backing FS size minus reserved capacity).
rawfile_pool_remaining_capacity_bytes node, pool Free capacity for new volumes on the pool (excluding reserved storage).
rawfile_pool_reserved_capacity_bytes node, pool Bytes reserved per the pool's reservedCapacity configuration.
rawfile_pool_backing_fs_capacity_bytes node, pool Total size of the filesystem backing the pool (statvfs).
rawfile_pool_backing_fs_available_bytes node, pool Available space on the backing FS — includes the reserved slice and non-rawfile space.
rawfile_pool_backing_fs_usage_bytes node, pool Used space on the backing FS — includes non-rawfile tenants (kubelet ephemeral storage, logs, …).
rawfile_pool_volumes_physical_bytes node, pool Sum of physical (on-disk) sizes of all volumes in the pool.
rawfile_pool_volumes_logical_bytes node, pool Sum of logical (provisioned) sizes of all volumes in the pool.
rawfile_pool_volume_count node, pool Number of volumes provisioned on the pool.
rawfile_pool_info node, pool, mode, default_pool Static pool info; always 1, use labels for joins.

Per-volume

Metric Labels Description
rawfile_volume_used_bytes node, volume Actual disk space used inside the volume.
rawfile_volume_total_bytes node, volume Space allocated (provisioned) to the volume.
rawfile_volume_physical_bytes node, volume Physical (on-disk) size of the volume's backing file.
rawfile_volume_info node, volume, pool, sparse, thin_provision Static volume info; always 1, use labels for joins.

Useful queries

Volume fullness (alert when a volume nears its limit):

rawfile_volume_used_bytes / rawfile_volume_total_bytes > 0.9

Pool overcommit ratio (thin provisioning):

sum by (node, pool) (rawfile_pool_volumes_logical_bytes)
  / rawfile_pool_capacity_bytes

Real free space left in a pool (guard against thin-provisioning exhaustion):

rawfile_pool_remaining_capacity_bytes < 10 * 1024^3

Join volume usage with its pool via info metrics:

rawfile_volume_used_bytes
  * on (node, volume) group_left (pool) rawfile_volume_info

Renamed metrics (v0.14.1)

rawfile_pool_available_bytes and rawfile_pool_usage_bytes (introduced in v0.14.0) were renamed — update dashboards/alerts:

rawfile_pool_available_bytes  →  rawfile_pool_backing_fs_available_bytes
rawfile_pool_usage_bytes      →  rawfile_pool_backing_fs_usage_bytes

Logs

Structured logs (logFormat: json by default, pretty for humans) with levels from TRACE to CRITICAL (logLevel chart value). Every gRPC request/response is logged with context at debug levels.

REST API

When capabilities.apiServer.enabled=true, a REST API provides cluster-wide introspection of nodes, pools, volumes and background tasks — see Architecture § REST API Server.