Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rdkafka v0.15.0+ incompatibility #1325

Open
arielvalentin opened this issue Jan 7, 2025 · 2 comments
Open

rdkafka v0.15.0+ incompatibility #1325

arielvalentin opened this issue Jan 7, 2025 · 2 comments

Comments

@arielvalentin
Copy link
Collaborator

arielvalentin commented Jan 7, 2025

The instrumentation has not been compatible since 0.15.0. The monkey patches do not honor the current method signature of the Producer class and ends up reverting it back to earlier versions.

It also looks like we can no longer instrument each_batch starting in v0.20.0:

https://github.com/open-telemetry/opentelemetry-ruby-contrib/actions/runs/12657654638/job/35272889337?pr=1323

    1) Error:
  OpenTelemetry::Instrumentation::Rdkafka::Patches::Consumer::#each_batch#test_0001_traces each_batch call:
  NotImplementedError: `each_batch` has been removed due to data consistency concerns.
  
  This method was removed because it did not properly handle partition reassignments,
  which could lead to processing messages from partitions that were no longer owned
  by this consumer, resulting in duplicate message processing and data inconsistencies.
  
  Recommended alternatives:
  
  1. Implement your own batching logic using rebalance callbacks to properly handle
     partition revocations and ensure message processing correctness.
  
  2. Use a high-level batching library that supports proper partition reassignment
     handling out of the box (such as the Karafka framework).
  
      /opt/hostedtoolcache/Ruby/3.3.6/x64/lib/ruby/gems/3.3.0/gems/rdkafka-0.20.0/lib/rdkafka/consumer.rb:614:in `each_batch'
      lib/opentelemetry/instrumentation/rdkafka/patches/consumer.rb:46:in `each_batch'
      test/opentelemetry/instrumentation/rdkafka/patches/consumer_test.rb:196:in `block (3 levels) in <top (required)>'
  
  7 runs, 30 assertions, 0 failures, 1 errors, 0 skips
  rake aborted!
  Command failed with status (1)
  /opt/hostedtoolcache/Ruby/3.3.6/x64/bin/bundle:25:in `load'
  /opt/hostedtoolcache/Ruby/3.3.6/x64/bin/bundle:25:in `<main>'
  Tasks: TOP => test
  (See full trace by running task with --trace)

We must ensure that instrumentations do not negatively impact gems or introduce regressions in functionality.

@arielvalentin
Copy link
Collaborator Author

cc: @andyfleming I could use your help with getting updates to this instrumentation to make it compatible with v0.15+

@arielvalentin
Copy link
Collaborator Author

The build was stabilized but we need help with supporting rdkafka v0.15+

@arielvalentin arielvalentin changed the title rdkafka v0.20.0 incompatibility build failures rdkafka v0.15.0+ incompatibility Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant