Question regarding collector configuration and performance

[teddy-puffenbarger]

I have some general questions around configuration setup for the collectors and debugging. I'm running a simple java application auto-instrumented with the java agent with a local collector on the same instance. The local collector is configured to send data to a central gateway collector. The gateway collector then exports the spans to NewRelic.

I'm running Jmeter to hit the java application and I've noticed under high load (~3k transactions per second, each call should create 6 spans), the local collector starts to fail to export spans to the gateway collector and ends up dropping spans. The gateway collector has no errors in it's logs, but NewRelic no longer receives spans after a short time (less than a minute) and the local collector continues to fail to send spans to the gateway collector going forward.

When I configure the local collector to export spans directly to NewRelic instead of exporting to the gateway collector, there are no issues. I read through the Performance Doc and it seems that this should be within the limits specified there (?).

I'm having difficulty debugging what's the limiting factor on the gateway collector side that causes the local collector to start failing to export spans and drop the data. I've looked at Retry Documentation for the exporter, but it doesn't seem to answer my questions around what's happening on the gateway collector to where it stops sending new spans to NewRelic after a short amount of time and seems to become unresponsive?

Any insight is appreciated, thanks!

The collector is v0.43

My configuration for the local and gateway collectors are:

local:

receivers:
  hostmetrics:
    collection_interval: 10m
    scrapers:
      cpu:
      ..... (couple others)
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: <gateway collector loadbalancer>

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]
      processors: [batch]
    metrics:
      receivers: [otlp]
      exporters: [logging]
      processors: [batch]

gateway:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:8443

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: https://otlp.nr-data.net:443
    headers:
      api-key: XXXXXXXX
    compression: gzip

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]
      processors: [batch]
    metrics:
      receivers: [otlp]
      exporters: [logging, otlp]
      processors: [batch]
    logs:
      receivers: [otlp]
      exporters: [logging]
      processors: [batch]

[pmm-sumo]

The performance issues you're observing might be due to specifics of Jaeger wire format. It's worth trying adding groupbyattrs processor to your pipeline in compaction mode