coroot is not seeing all docker containers running on the same host as itself

[ovizii]

Here are some it is not picking up:

https://github.com/bulwarkmail/webmail
https://github.com/maxmind/geoipupdate
https://github.com/ColinIanKing/stress-ng
https://github.com/traefik/whoami

Any ideas what/where to check?

Anything from the clickhouse logs possibly related to this?

❯ docker compose logs -f coroot-clickhouse
coroot-clickhouse  | /entrypoint.sh: explicitly skip changing user 'default'
coroot-clickhouse  | ClickHouse Database directory appears to contain a database; Skipping initialization
coroot-clickhouse  | Processing configuration file '/etc/clickhouse-server/config.xml'.
coroot-clickhouse  | Merging configuration file '/etc/clickhouse-server/config.d/clickhouse-conf.xml'.
coroot-clickhouse  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
coroot-clickhouse  | Merging configuration file '/etc/clickhouse-server/config.d/logging_rules.xml'.
coroot-clickhouse  | Merging configuration file '/etc/clickhouse-server/config.d/user_logging.xml'.
coroot-clickhouse  | Logging warning to /var/log/clickhouse-server/clickhouse-server.log
coroot-clickhouse  | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
coroot-clickhouse  | 2026.04.03 16:36:37.531710 [ 1 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
coroot-clickhouse  | 2026.04.03 16:36:37.531940 [ 1 ] {} <Warning> Application: Integrity check of the executable skipped because the reference checksum could not be read.
coroot-clickhouse  | 2026.04.03 16:36:37.532940 [ 1 ] {} <Error> Application: Disabling cgroup memory observer because of an error during initialization: Code: 107. DB::Exception: Cannot find cgroups v1 or v2 current memory file. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
coroot-clickhouse  | 
coroot-clickhouse  | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd45ee8
coroot-clickhouse  | 1. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000007728154
coroot-clickhouse  | 2. DB::CgroupsMemoryUsageObserver::CgroupsMemoryUsageObserver(std::chrono::duration<long long, std::ratio<1l, 1l>>) @ 0x000000000be3fea0
coroot-clickhouse  | 3. DB::Server::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000be9dd80
coroot-clickhouse  | 4. Poco::Util::Application::run() @ 0x0000000012d44dbc
coroot-clickhouse  | 5. DB::Server::run() @ 0x000000000be92d70
coroot-clickhouse  | 6. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000012d4e064
coroot-clickhouse  | 7. mainEntryClickHouseServer(int, char**) @ 0x000000000be8f3d0
coroot-clickhouse  | 8. main @ 0x00000000077197f8
coroot-clickhouse  | 9. ? @ 0x0000000000020e10
coroot-clickhouse  |  (version 24.3.18.7 (official build))
coroot-clickhouse  | 2026.04.03 16:36:37.600409 [ 1 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.18.7 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

Here are the relevant node-agent logs:

coroot-node-agent  | I0405 19:29:06.907348  348856 container.go:528] TCP listen open pid=148626 id=/system.slice/docker.service addr=127.0.0.11:38207
coroot-node-agent  | I0405 19:29:06.925611  348856 registry.go:368] calculated container id 4109906 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.925753  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109906
coroot-node-agent  | I0405 19:29:06.925835  348856 registry.go:368] calculated container id 4109908 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.925850  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109908
coroot-node-agent  | I0405 19:29:06.925884  348856 registry.go:368] calculated container id 4109909 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.925906  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109909
coroot-node-agent  | I0405 19:29:06.925942  348856 registry.go:368] calculated container id 4109910 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.925951  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109910
coroot-node-agent  | I0405 19:29:06.925993  348856 registry.go:368] calculated container id 4109912 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.926004  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109912
coroot-node-agent  | I0405 19:29:06.926036  348856 registry.go:368] calculated container id 4109913 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.926047  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109913
coroot-node-agent  | I0405 19:29:06.926094  348856 registry.go:368] calculated container id 4109914 -> /system.slice/systemd-udevd.service/udev -> /system.slice/systemd-udevd.service
coroot-node-agent  | I0405 19:29:06.926106  348856 registry.go:387] "skipping system service" id="/system.slice/systemd-udevd.service" unit="systemd-udevd.service" type="notify-reload" triggered_by="systemd-udevd-kernel.socket" pid=4109914
coroot-node-agent  | I0405 19:29:06.986619  348856 container.go:562] TCP listen close pid=148626 id=/system.slice/docker.service addr=127.0.0.11:38207
coroot-node-agent  | I0405 19:29:07.024607  348856 registry.go:368] calculated container id 4109951 ->  -> 
coroot-node-agent  | I0405 19:29:07.024751  348856 registry.go:373] "ignoring" cg="" pid=4109951
coroot-node-agent  | I0405 19:29:07.024806  348856 registry.go:368] calculated container id 4109952 ->  -> 
coroot-node-agent  | I0405 19:29:07.024815  348856 registry.go:373] "ignoring" cg="" pid=4109952
coroot-node-agent  | I0405 19:29:16.078109  348856 remote_writer.go:159] sent metrics in 14ms
coroot-node-agent  | E0405 19:29:20.232333  348856 tls.go:68] pid=4110239 golang_app=/usr/local/bin/cscli golang_version=go1.25.8: failed to get symbol crypto/tls.(*Conn).Write: no symbols found
coroot-node-agent  | E0405 19:29:20.232333  348856 tls.go:68] pid=4110239 golang_app=/usr/local/bin/cscli golang_version=go1.25.8: failed to get symbol crypto/tls.(*Conn).Write: no symbols found
coroot-node-agent  | E0405 19:29:20.232333  348856 tls.go:68] pid=4110239 golang_app=/usr/local/bin/cscli golang_version=go1.25.8: failed to get symbol crypto/tls.(*Conn).Write: no symbols found
coroot-node-agent  | E0405 19:29:20.232333  348856 tls.go:68] pid=4110239 golang_app=/usr/local/bin/cscli golang_version=go1.25.8: failed to get symbol crypto/tls.(*Conn).Write: no symbols found
coroot-node-agent  | I0405 19:29:22.940549  348856 registry.go:368] calculated container id 4110326 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:22.940591  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110326
coroot-node-agent  | I0405 19:29:22.940668  348856 registry.go:368] calculated container id 4110322 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:22.940698  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110322
coroot-node-agent  | I0405 19:29:22.940731  348856 registry.go:368] calculated container id 4110309 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:22.940747  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110309
coroot-node-agent  | I0405 19:29:22.940796  348856 registry.go:368] calculated container id 4110310 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:22.940815  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110310
coroot-node-agent  | I0405 19:29:22.940851  348856 registry.go:368] calculated container id 4110311 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:22.940861  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110311
coroot-node-agent  | I0405 19:29:22.940888  348856 registry.go:368] calculated container id 4110312 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:22.940895  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110312
coroot-node-agent  | I0405 19:29:23.039767  348856 registry.go:368] calculated container id 4110340 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:23.039802  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110340
coroot-node-agent  | I0405 19:29:24.824771  348856 registry.go:368] calculated container id 4110407 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:24.824934  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110407
coroot-node-agent  | I0405 19:29:24.825183  348856 registry.go:368] calculated container id 4110423 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:24.825202  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=4110423
coroot-node-agent  | I0405 19:29:24.825382  348856 registry.go:368] calculated container id 78752 -> /user.slice/user-0.slice/session-8.scope -> 
coroot-node-agent  | I0405 19:29:24.825398  348856 registry.go:373] "ignoring" cg="/user.slice/user-0.slice/session-8.scope" pid=78752

[sueun-dev]

hey, so coroot-node-agent discovers containers reactively through eBPF tracing. it only "sees" a container when one of its processes generates a traceable event like a TCP connection, a listen call, a process fork, or a file open. if a container is sitting idle with no network activity or new process starts, it wont show up until something happens.

a few things to check:

1. docker socket mount - the node-agent needs access to /run/docker.sock on the host to call docker inspect and get container metadata (name, labels, etc). check your docker-compose.yml for the node-agent and make sure you have something like:

volumes:
  - /run/docker.sock:/host/run/docker.sock:ro

if this mount is missing or the path is wrong, the agent will log a warning during startup like failed to interact with dockerd. scroll to the very top of your node-agent logs to check for that.

2. trigger some activity - try curling your whoami container from the host (curl http://localhost:<port>) and see if it suddenly appears in coroot. same with bulwarkmail/webmail - hit it with a request. if they show up after that, it confirms the eBPF event-driven discovery is the reason.

3. geoipupdate - this one is expected to be invisible most of the time. it runs periodically, updates the db, then exits. the node-agent will only catch it during those brief windows when its actually running.

4. stress-ng - if you're running CPU-only stress tests (no network), theres a good chance it wont generate any TCP events for the agent to hook into. it should still get picked up on process-start events though, so check if maybe its being filtered. do you have CONTAINER_NAMES_FILTER or any similar env var set on the node-agent?

5. check the full startup logs - run docker compose logs coroot-node-agent | head -50 and look for any warnings about dockerd/containerd init failing, or any "failed to get container metadata" messages. that would point to a socket access issue.

the clickhouse errors in your logs (cgroup memory observer, IPv6 listen) are unrelated - those are just ClickHouse complaining about container environment limitations and wont affect container discovery.

lmk what you find!

[def]

This comment is completely wrong and misleading, likely AI-generated slop.

[ovizii]

Thanks. In that case, I'd be happy about any other pointers.

[sueun-dev]

No, i think im correct. I checked twice. If not just tell me know. I wrote my knowlege

[sueun-dev]

im sorry, looked into it for the past 4 hours and you're right. node-agent discovers containers through cgroup scanning, not eBPF event-driven discovery. i had the mechanism wrong. my bad.

[sueun-dev]

Sorry, my original answer was wrong, and my follow-up was not accurate either. In node-agent v1.31.0 the agent walks /proc at startup and picks up new processes through an eBPF task_newtask hook, so idle containers are tracked. It reaches the Docker socket via /proc/1/root (the official compose runs it privileged with pid: host), so no docker.sock mount is needed. What you saw comes from the UI: in Coroot v1.19.0 the Applications list skips apps outside Kubernetes/Nomad that have no connections to or from other apps in the selected time range. They should still be findable through search.

coroot/api/views/overview/applications.go

Lines 48 to 57 in 44a4f6e

	func renderApplications(w *model.World) []*ApplicationStatus {
	var res []*ApplicationStatus
	for _, app := range w.Applications {
	switch {
	case app.IsK8s():
	case app.Id.Kind == model.ApplicationKindNomadJobGroup:
	case !app.IsStandalone():
	default:
	continue
	}

[ovizii]

Since nobody is going to add any comments, here is my feedback before closing the discussion.
stalwart and bulwark have started showing up so the above explanation about them only being picked up when actual network traffic occurs seem to be true.