Errors while trying to view logs

[sati-max]

Bug description

Randomly log view crashes and displays the "Something went wrong" information, sometimes it crashes the browser and the browser displays: "RESULT_CODE_KILLED_BAD_MESSAGE" error.

Digging inside query-service logs, there are 3 types of errors:

1/ error while reading time series result
{"level":"ERROR","timestamp":"2025-01-08T10:40:01.304Z","caller":"clickhouseReader/reader.go:4776","msg":"error while reading time series result","error":"context canceled","stacktrace":"go.signoz.io/signoz/pkg/query-service/app/clickhouseReader.(*ClickHouseReader).GetTimeSeriesResultV3\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:4776\ngo.signoz.io/signoz/pkg/query-service/app/querier.(*querier).execClickHouseQuery\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/querier/querier.go:97\ngo.signoz.io/signoz/pkg/query-service/app/querier.(*querier).runBuilderQuery\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/querier/helper.go:231"}

2/ error writing response
{"level":"ERROR","timestamp":"2025-01-08T10:46:09.218Z","caller":"app/http_handler.go:307","msg":"error writing response","bytesWritten":0,"error":"write tcp 10.x.x.x:8080->10.x.x.x:35518: write: broken pipe","stacktrace":"go.signoz.io/signoz/pkg/query-service/app.writeHttpResponse\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/http_handler.go:307\ngo.signoz.io/signoz/pkg/query-service/app.(*APIHandler).Respond\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/http_handler.go:344\ngo.signoz.io/signoz/pkg/query-service/app.(*APIHandler).queryRangeV3\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/http_handler.go:3767\ngo.signoz.io/signoz/pkg/query-service/app.(*APIHandler).QueryRangeV3\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/http_handler.go:3869\ngo.signoz.io/signoz/pkg/query-service/app.(*APIHandler).RegisterQueryRangeV3Routes.(*AuthMiddleware).ViewAccess.func7\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/auth.go:50\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\ngo.signoz.io/signoz/ee/query-service/app.loggingMiddleware.func1\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:387\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\ngo.signoz.io/signoz/ee/query-service/app.(*Server).analyticsMiddleware-fm.(*Server).analyticsMiddleware.func1\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:557\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\ngo.signoz.io/signoz/ee/query-service/app.setTimeoutMiddleware.func1\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:589\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\ngo.signoz.io/signoz/pkg/query-service/app.LogCommentEnricher.func1\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/server.go:392\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212\ngo.signoz.io/signoz/ee/query-service/app.(*Server).createPublicServer.(*Cors).Handler.func2\n\t/home/runner/go/pkg/mod/github.com/rs/[email protected]/cors.go:289\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\ngo.signoz.io/signoz/ee/query-service/app.(*Server).createPublicServer.CompressHandler.CompressHandlerLevel.func3\n\t/home/runner/go/pkg/mod/github.com/gorilla/[email protected]/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2141\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2943\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/net/http/server.go:2014"}

3/ Couldn't report query progress
{"level":"ERROR","timestamp":"2025-01-08T10:39:20.528Z","caller":"clickhouseReader/reader.go:4762","msg":"Couldn't report query progress","queryId":"26039d7a-aa31-461f-a352-21d7d2daa964","error":"query 26039d7a-aa31-461f-a352-21d7d2daa964 doesn't exist","stacktrace":"go.signoz.io/signoz/pkg/query-service/app/clickhouseReader.(*ClickHouseReader).GetTimeSeriesResultV3.func1.1\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:4762"}

Checking the last error query_id inside clickhouse shows 0 rows, so the query wasn't send to clickhouse, yet query-service expected results?
SELECT * FROM system.query_log WHERE initial_query_id = '26039d7a-aa31-461f-a352-21d7d2daa964'; SELECT * FROM system.query_log WHERE query_id = '26039d7a-aa31-461f-a352-21d7d2daa964';

It started to happen randomly today so I can't provide steps to reproduce.

Version information

• Signoz version: 0.53.1 (installed from 0.51.1 chart)
• Browser version: Edge latest version
• Your OS and version: Windows 11
• Your CPU Architecture(ARM/Intel): Intel

Additional context

Thank you for your bug report – we love squashing them!