Unable to Filter Metrics Using Resource Attributes - resource_attributes

[mskhan-devops]

Hi Everyone,
As per the example on page opentelemetry-collector-contrib/processor/filterprocessor, I am trying to configure ADOT Collector Pipeline for EKS cluster to publish metrics to CloudWatch Container Insights:

processors:
  filter/1:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - pod_cpu_.*

This works fine and I see filtered metrics against my apps in the Container Insights.

The problem is as soon as I add resource_attributes to the above configuration to filter out metrics against the specific app running in the EKS cluster, like below:

processors:
  filter/1:
    metrics:
      include:
        match_type: regexp
        metric_names:
          - pod_cpu_.*
        resource_attributes:
          - Key: k8s.pod.name
            Value: myapp-pod-with-hash.*

AWS ADOT Collector stops publishing any metrics to Container Insights. Can anyone please help with this?

I am using this guide and sample config file:
Cost savings by customizing metrics sent by Container Insights in Amazon EKS

Sample config file:
otel-container-insights-infra.yaml

Thanks

[trasc]

Hi, are you sure that k8s.pod.name is a resource attribute and not a data point attribute?

Can you add a logger in your pipeline (without resource_attributes filtering) and provide a sample ResourceMetrics?

[mskhan-devops]

Hi @trasc - Thank you for your response.

I am not 100% sure ifk8s.pod.name is a resource attribute but looking at the opentelemetry-specifications document, I think it is a resource_attributes:
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/k8s.md

I have also added a logger to the pipeline but it did not throw anything informative. Could you please help with respect to the implementation of AWS OTEL Collector for Container Insights with resource_attributes

I am following this page:
https://aws.amazon.com/blogs/containers/cost-savings-by-customizing-metrics-sent-by-container-insights-in-amazon-eks/

Please note that the is a full example of AWS OTEL implementation but they did not mention anything about the resource_attributesusage on the above page.

Thank you very much!

[pmm-sumo]

I think this question is duplicated here: open-telemetry/opentelemetry-collector-contrib#8106

Perhaps k8s.pod.name is assigned in a consecutive processor? If that's the case, order in processors section needs to be modified.

Hi, are you sure that k8s.pod.name is a resource attribute and not a data point attribute?

In case of the that, you can use groupbyattrs processor to move it to resource, e.g.:

processors:
  groupbyattrs:
    keys:
      - k8s.pod.name

I have also added a logger to the pipeline but it did not throw anything informative.

You need to set the debug level, e.g.

exporters:
  logging/debug:
    loglevel: debug