Logging rework

[tmcqueen-materials]

The process of debugging OpenMSIStream zombie states, where everything is "running" but message consumption has ceased (probably due to an underlying bug in the rdkafka library, see confluentinc/librdkafka#4916 ) was complicated by the suppression of most log information from the Kafka producers/consumers.

This PR removes the log suppression and propagates log level sets appropriately. Part of doing this is removing a dependency on (undocumented, internal) parts of KafkaCrypto, which made me realize message size accounting associated with heartbeats is incorrect.

Creating as a draft PR to see what the runners think. Requires openmsi/openmsitoolbox#8 .

[davidelbert]

I merged the openmsitoolbox PR so the OpenMSILogger class is now updated. Would you please run the run-tests-no-kafka test again to clear the checks before I merge? (I hope that is all it was.). Thanks.

[tmcqueen-materials]

@davidelbert : thanks. That did not fix it because although a new v1.2.4 was tagged for openmsitoolbox, it was not pushed to PyPI (and the code version string was not updated beyond v1.2.3). I've submitted a second PR over there for that: openmsi/openmsitoolbox#10 , but not sure the process to issue a new PyPI release.

[davidelbert]

@tmcqueen-materials, openmsitoolbox has new version on PyPi now. It's tagged "openmsitoolbox 1.2.4.post1" as you suggested. Are you now able to rerun the CircleCI pipeline? Thanks.

[Xarthisius]

@tmcqueen-materials you'll also have to bump version of openmsistream itself to make tests pass, e.g.:

diff --git a/openmsistream/version.py b/openmsistream/version.py
index 320141d..ef4e23e 100644
--- a/openmsistream/version.py
+++ b/openmsistream/version.py
@@ -1 +1 @@
-__version__ = "1.8.2"
+__version__ = "1.8.3.dev1"

[tmcqueen-materials]

@Xarthisius : Thanks. It looks like this has surfaced a different long-standing nit in the ssl ca location handling code that I'm going to go fix now...

[tmcqueen-materials]

@Xarthisius , @davidelbert : At this point, this should be ready for review. The failing tests are associated with what looks like a CI issue and trying to test S3 bucket functionality. Let me know of any changes that are needed.

[Xarthisius]

LGTM! I would only add explicit version dependency to:

https://github.com/openmsi/openmsistream/blob/9df39e5cf5ee798e026edf4f3330495c1c824b66/setup.py#L62C9-L62C33

so that it picks up openmsitoolbox with proper changes.

[tmcqueen-materials]

@Xarthisius : great idea. Just pushed that change.

[davidelbert]

Merging without the python matrix CircleCI tests passing. They are failing because of lack of S3 bucket writing test endpoint that I can fix later once JetStream2 is back from service or the local allocations are up.