v2.9.1

This release brings mostly dependency upgrades, internal refactoring and documentation improvements.

Thanks to @AdrielC for making their first contributions!

🚀 Changes

• Add config for transactional consuming @erikvanoosten (#1433)
• Convert commitAsync callback handling to ZIO sooner @svroonland (#1404)
• Extract Committer and RebalanceCoordinator classes from Runloop + unit tests @svroonland (#1375)
• [Internal] Micro-optimisation: Reuse previously calculated java.langSystem.nanoTime @guizmaii (#1407)
• Add even more zio-kafka resources @erikvanoosten (#1415)
• Add more zio-kafka resources @erikvanoosten (#1414)
• Document when zio-kafka is faster than raw java kafka @erikvanoosten (#1403)
• Remove 'experimental' label from FetchStrategy @erikvanoosten (#1393)

🌱 Dependency Updates

• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.30 @zio-scala-steward (#1430)
• Update README.md @zio-assistant (#1417)
• Update README.md @zio-assistant (#1413)
• Update sbt-native-packager to 1.11.0 @zio-scala-steward (#1429)
• Update logback-classic to 1.5.16 @zio-scala-steward (#1428)
• Update README.md @zio-assistant (#1427)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.29 @zio-scala-steward (#1426)
• Update sbt, scripted-plugin to 1.10.7 @zio-scala-steward (#1424)
• Update zio, zio-streams, zio-test, ... to 2.1.14 @zio-scala-steward (#1422)
• Update logback-classic to 1.5.15 @zio-scala-steward (#1423)
• Update logback-classic to 1.5.14 @zio-scala-steward (#1421)
• Update logback-classic to 1.5.13 @zio-scala-steward (#1420)
• Revert "Update zio, zio-streams, zio-test, ... to 2.1.13" @erikvanoosten (#1400)
• Update zio, zio-streams, zio-test, ... to 2.1.13 @zio-scala-steward (#1399)
• Update embedded-kafka to 3.9.0 @zio-scala-steward (#1398)
• Update zio-logging-slf4j, ... to 2.4.0 @zio-scala-steward (#1387)
• Update sbt, scripted-plugin to 1.10.6 @zio-scala-steward (#1410)

🔧 Maintenance

• Fix scala-steward flow @erikvanoosten (#1418)
• Documents the different benchmarks @erikvanoosten (#1402)
• ZioKafkaConsumerbenchmark.throughput: increase count per batch @svroonland (#1409)
• Use same amount of messages for all consumer benchmarks @erikvanoosten (#1382)
• Improve Test Reliability by Replacing ZIO.sleep with Synchronization Primitives in ConsumerSpec @AdrielC (#1401)
• Fix timing issue in ConsumerSpec test by adding delay in consumer stream @AdrielC (#1388)
• Remove unnecessary workflow input @erikvanoosten (#1394)
• Run profiler only on release and on demand @erikvanoosten (#1392)
• Add sbt to github action workflows, use ubuntu-latest @erikvanoosten (#1416)

v2.9.0

This release brings important fixes in the rebalancing mechanism, as well as other minor improvements.

A couple of methods are now deprecated:

• Serde#inmapM, renamed to inmapZIO
• Deserializer#contramapM, renamed to contramapZIO
• Serializer#mapM, renamed to mapZIO
• Consumer#fromJavaConsumer(javaConsumer), replaced by fromJavaConsumerWithPermit

If you are using scala-steward, the renamed methods are automatically changed.

🚀 Changes

• Add example dashboard to docs by @erikvanoosten in #1357
• Fix slow rebalanceSafeCommits behavior by @svroonland in #1358
• Remove attemptBlocking from AdminClient by @svroonland in #1359
• Fix concurrent modification exception in Consumer release by @svroonland in #1365
• Fix stream interrupt when no data, Add tests for PartitionStreamControl by @svroonland in #1371
• More logging around rebalancing when rebalanceSafeCommits is true by @svroonland in #1360
• Require a semaphore in Consumer.fromJavaConsumer by @svroonland in #1368
• Rename Serde methods for consistency with ZIO 2 by @svroonland in #1378
• Fix deprecation notice version on Consumer.fromJavaConsumer by @svroonland in #1379
• Decouple stream halt detection timeout from max poll interval by @svroonland in #1376

🌱 Dependency Updates

• Update kafka-clients and embedded-kafka to 3.8.1 by @zio-scala-steward in #1352
• Update zio-kafka, zio-kafka-testkit to 2.8.3 by @zio-scala-steward in #1353
• Update zio, zio-streams, zio-test, ... to 2.1.12 by @zio-scala-steward in #1364

🔧 Maintenance

• Update README.md by @zio-assistant in #1355
• Update sbt to 1.10.5 by @zio-scala-steward in #1362
• Automate pruning of benchmark history by @erikvanoosten in #1370
• Fix automatic pruning of benchmark history by @erikvanoosten in #1372
• Fix automatic pruning of benchmark history (attempt 2) by @erikvanoosten in #1373
• Bump scala-steward-org/scala-steward-action from 2.70.0 to 2.71.0 by @dependabot in #1377
• Use max.poll.records i.s.o. config name by @erikvanoosten in #1381
• Formatting by @svroonland in #1383
• Disable mima check for new minor releases by @erikvanoosten in #1384

Full Changelog: v2.8.3...v2.9.0

v2.8.3

This release fixes a bug where the Consumer could get stuck when partitions were lost after connection issues with the broker. In addition, the Producer can now give more detailed errors and got some performance optimizations.

From now on new releases are automatically checked for binary compatibility against the latest minor release (eg 2.8.0 in case of this release).

Thanks to @ytalashko and @ashwinbhaskar for their contributions!

🚀 Changes

• Enable mima check during release @erikvanoosten (#1351)
• Remove lost partitions from assigned streams @svroonland (#1350)
• Document Serdes in more depth @erikvanoosten (#1337)
• Document scala 3 plus zio-kafka-testkit @erikvanoosten (#1329)
• Optimize Producer sendFromQueue implementation @ytalashko (#1326)
• More precise batch publish errors @ytalashko (#1321)
• Fixes #1318 Adds documentation about auto commit behaviour @ashwinbhaskar (#1319)
• Produce sequentially and in parallel in benchmark @erikvanoosten (#1314)
• Use AtomicInteger instead of AtomicLong inside Producer.sendFromQueue implementation @ytalashko (#1315)
• Simple producer benchmark @svroonland (#1310)

🌱 Dependency Updates

• Update scala3-library to 3.3.4 @zio-scala-steward (#1336)
• Update sbt-scalafix to 0.13.0 @zio-scala-steward (#1335)
• Update scala-library to 2.13.15 @zio-scala-steward (#1334)
• Update README.md @zio-assistant (#1330)
• Update sbt to 1.10.2 @zio-scala-steward (#1327)
• Update logback-classic to 1.5.8 @zio-scala-steward (#1322)
• Update zio-streams, zio-test-sbt to 2.1.9 @zio-scala-steward (#1320)
• Update zio, zio-streams, zio-test, ... to 2.1.9 @zio-scala-steward (#1317)
• Update zio-streams, zio-test-sbt to 2.1.8 @zio-scala-steward (#1316)
• Update sbt-mima-plugin to 1.1.4 @zio-scala-steward (#1296)
• Update zio, zio-streams, zio-test, ... to 2.1.8 @zio-scala-steward (#1312)
• Update to Kafka 3.8.0 @erikvanoosten (#1313)
• Update zio-kafka, zio-kafka-testkit to 2.8.2 @zio-scala-steward (#1306)
• Update README.md @zio-assistant (#1309)
• Update README.md @zio-assistant (#1308)
• Update sbt-native-packager to 1.10.4 @zio-scala-steward (#1293)
• Update zio-logging-slf4j, ... to 2.3.1 @zio-scala-steward (#1307)
• Update sbt to 1.10.4 @zio-scala-steward (#1349)
• Update logback-classic to 1.5.12 @zio-scala-steward (#1347)
• Update sbt to 1.10.3 @zio-scala-steward (#1346)
• Update zio-streams, zio-test-sbt to 2.1.11 @zio-scala-steward (#1344)
• Update logback-classic to 1.5.11 @zio-scala-steward (#1341)
• Update zio-logging-slf4j, ... to 2.3.2 @zio-scala-steward (#1343)
• Update zio, zio-streams, zio-test, ... to 2.1.11 @zio-scala-steward (#1342)
• Update logback-classic to 1.5.9 @zio-scala-steward (#1340)
• Bump benchmark-action/github-action-benchmark from 1.20.3 to 1.20.4 @dependabot (#1348)
• Bump scala-steward-org/scala-steward-action from 2.69.0 to 2.70.0 @dependabot (#1338)
• Bump scala-steward-org/scala-steward-action from 2.65.0 to 2.69.0 @dependabot (#1328)

v2.8.2

This release reverts the producer changes of v2.8.1 because they has a huge impact on performance. We will reconsider the changes and make sure the performance tests accurately cover producing events before applying this change again.

What's Changed

• Revert back to 2.8.0 sendFromQueue implementation in producer by @ytalashko in #1304

Full Changelog: v2.8.1...v2.8.2

v2.8.1

Version 2.8.1 has an important bugfix that prevents partitions from being consumed twice in some circumstances (#1294). Many thanks to @ytalashko for finding the root cause, providing a fix, and tirelessly discussing the problem until it was fully understood and the fix applied.

🚀 Noteworthy changes

• Fix assign followed by fast revoke during rebalance @ytalashko (#1294)
• Add alternative fetch strategy for many partitions @erikvanoosten (#1281)
• Document metrics and consumer tuning based on metrics @erikvanoosten (#1280)

🚅 More changes

• Prevent users from enabling auto commit @erikvanoosten (#1290)
• Alternative producer implementation @erikvanoosten (#1285)
• Small improvements to the Producer @erikvanoosten (#1272)
• Improve example app @erikvanoosten (#1279)

🌱 Maintenance

• Update sbt-tpolecat to 0.5.2 @zio-scala-steward (#1301)
• Update embedded-kafka to 3.7.1.1 @zio-scala-steward (#1300)
• Update logback-classic to 1.5.7 @zio-scala-steward (#1299)
• Update zio-streams, zio-test-sbt to 2.1.7 @zio-scala-steward (#1297)
• Update zio, zio-streams, zio-test, ... to 2.1.7 @zio-scala-steward (#1295)
• Update scalafmt-core to 3.8.3 @zio-scala-steward (#1291)
• Update README.md @zio-assistant (#1278)
• Update README.md @zio-assistant (#1277)
• Update zio-kafka, zio-kafka-testkit to 2.8.0 @zio-scala-steward (#1274)
• Update zio-streams, zio-test-sbt to 2.1.6 @zio-scala-steward (#1275)

v2.8.0

Version 2.7.5 was not fully compatible with 2.7.4 which we missed during the release. Therefore, with some delay, we bump the minor version.

Moving back to pre-2.7.0 behavior: in this release, upon a lost partition the stream ends without error. To detect lost partitions, you can configure a rebalance listener.

🚀 Changes

• Allow for some auth exceptions during poll @erikvanoosten (#1270)
• A lost partition is no longer fatal @erikvanoosten (#1252)
• Update consumer docs @petern-sc (#1269)

🌱 Maintenance

• Update zio, zio-streams, zio-test, ... to 2.1.6 @zio-scala-steward (#1273)
• Update sbt to 1.10.1 @zio-scala-steward (#1271)
• Update kafka-clients to 3.7.1 @zio-scala-steward (#1268)
• Update zio-streams, zio-test-sbt to 2.1.5 @zio-scala-steward (#1267)
• Update zio, zio-streams, zio-test, ... to 2.1.5 @zio-scala-steward (#1266)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.28 @zio-scala-steward (#1263)
• Update zio-streams, zio-test-sbt to 2.1.4 @zio-scala-steward (#1261)
• Update zio, zio-streams, zio-test, ... to 2.1.4 @zio-scala-steward (#1260)
• Update scalafmt-core to 3.8.2 @zio-scala-steward (#1259)
• Update zio-streams, zio-test-sbt to 2.1.3 @zio-scala-steward (#1257)
• Update zio, zio-streams, zio-test, ... to 2.1.3 @zio-scala-steward (#1256)
• Aborts builds when a newer build is ready @erikvanoosten (#1255)
• Update README.md @zio-assistant (#1254)
• Update zio-kafka, zio-kafka-testkit to 2.7.5 @zio-scala-steward (#1253)

v2.7.5

Minor breaking changes

• breaking: Do not run user rebalance listener on same thread runtime @svroonland (#1205)
  In case you register a rebalance listener, it now runs on the default zio runtime and no longer on the special same-thread-runtime. Access to the underlying java client is removed. If this affects you, please let us know your use case.
• Remove no longer needed collection-compat @erikvanoosten (#1221)
  The library collection-compat is no longer a dependency. If you are using it, please add it to your own dependencies.

🚀 Changes

• Allow stream to be interrupted when there is no traffic @erikvanoosten (#1251)
  Lost partitions during no traffic situations are now correctly detected. That is, the stream for the partition that is lost is interrupted, the other streams are closed gracefully, the consumer aborts with an error. Note that in the next release we will change this behavior, then the stream for the partition that is lost is still ended, but the other streams will be unaffected.
• Initialize Java Producer in blocking threadpool @kyri-petrou (#1230)
• Call removeMembersFromConsumerGroup without members @sderosiaux (#1227)
• Document snapshots and Update resources @erikvanoosten (#1223)
• Drop experimental label for rebalanceSafeCommits @erikvanoosten (#1222)
• Correct prune benchmark history instructions @erikvanoosten (#1217)
• Add MIMA checks to ensure binary compatibility @erikvanoosten (#1214)

🌱 Maintenance

• Update zio-streams, zio-test-sbt to 2.1.2 @zio-scala-steward (#1249)
• Update zio, zio-streams, zio-test, ... to 2.1.2 @zio-scala-steward (#1248)
• Update zio-logging-slf4j, ... to 2.3.0 @zio-scala-steward (#1246)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.27 @zio-scala-steward (#1247)
• Update zio-logging-slf4j, ... to 2.2.4 @zio-scala-steward (#1244)
• Update zio-streams, zio-test-sbt to 2.1.1 @zio-scala-steward (#1243)
• Update zio, zio-streams, zio-test, ... to 2.1.1 @zio-scala-steward (#1242)
• Update zio-streams, zio-test-sbt to 2.1.0 @zio-scala-steward (#1241)
• Update zio, zio-streams, zio-test, ... to 2.1.0 @zio-scala-steward (#1240)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.26 @zio-scala-steward (#1237)
• Update sbt-scalafix to 0.12.1 @zio-scala-steward (#1234)
• Update sbt to 1.10.0 @zio-scala-steward (#1236)
• Update scala-library to 2.13.14 @zio-scala-steward (#1235)
• Update zio-logging-slf4j, ... to 2.2.3 @zio-scala-steward (#1226)
• Regenerate readme @erikvanoosten (#1225)
• Update logback-classic to 1.5.6 @zio-scala-steward (#1224)
• Update scala-collection-compat to 2.12.0 @zio-scala-steward (#1220)
• Revert "Bump coursier/cache-action from 6.4.5 to 6.4.6" @erikvanoosten (#1216)
• Update zio-streams, zio-test-sbt to 2.0.22 @zio-scala-steward (#1215)
• Update logback-classic to 1.5.5 @zio-scala-steward (#1211)
• Update sbt-native-packager to 1.10.0 @zio-scala-steward (#1212)
• Update zio, zio-streams, zio-test, ... to 2.0.22 @zio-scala-steward (#1213)
• Update logback-classic to 1.5.4 @zio-scala-steward (#1210)
• Update sbt-tpolecat to 0.5.1 @zio-scala-steward (#1206)
• Update scalafmt-core to 3.8.1 @zio-scala-steward (#1204)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.25 @zio-scala-steward (#1200)
• Update zio-kafka, zio-kafka-testkit to 2.7.4 @zio-scala-steward (#1197)
• Update README.md @zio-assistant (#1195)
• Bump benchmark-action/github-action-benchmark from 1.20.1 to 1.20.3 @dependabot (#1245)
• Bump coursier/cache-action from 6.4.5 to 6.4.6 @dependabot (#1209)
• Bump benchmark-action/github-action-benchmark from 1.18.0 to 1.20.1 @dependabot (#1208)

v2.7.4

Noteworthy Changes

• Update kafka-clients to 3.7.0 @zio-scala-steward (#1191)

Changes

• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.24 @zio-scala-steward (#1194)
• Update kafka-clients to 3.7.0 @zio-scala-steward (#1191)
• Update scala3-library to 3.3.3 @zio-scala-steward (#1181)
• Update sbt-scalafix to 0.12.0 @zio-scala-steward (#1184)
• Update logback-classic to 1.5.3 @zio-scala-steward (#1190)
• Update logback-classic to 1.5.2 @zio-scala-steward (#1185)
• Update logback-classic to 1.5.1 @zio-scala-steward (#1178)
• Fix a merge error and regenerate actions @erikvanoosten (#1177)
• Update README.md @zio-assistant (#1176)
• Update README.md @zio-assistant (#1175)
• Update README.md @zio-assistant (#1174)
• Update zio-streams, zio-test-sbt to 2.0.21 @zio-scala-steward (#1167)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.23 @zio-scala-steward (#1166)
• Update README.md @zio-assistant (#1173)
• Update scalafmt-core to 3.8.0 @zio-scala-steward (#1171)
• Update README.md @zio-assistant (#1172)
• Update sbt to 1.9.9 @zio-scala-steward (#1170)
• Update scala-library to 2.13.13 @zio-scala-steward (#1169)
• Update zio-logging-slf4j, ... to 2.2.2 @zio-scala-steward (#1165)
• Update logback-classic to 1.5.0 @zio-scala-steward (#1163)
• Update zio-kafka, zio-kafka-testkit to 2.7.3 @zio-scala-steward (#1164)

🌱 Dependency Updates

• Bump scala-steward-org/scala-steward-action from 2.64.0 to 2.65.0 @dependabot (#1192)
• Bump scala-steward-org/scala-steward-action from 2.63.0 to 2.64.0 @dependabot (#1186)
• Bump release-drafter/release-drafter from 5 to 6 @dependabot (#1189)
• Bump scala-steward-org/scala-steward-action from 2.61.0 to 2.63.0 @dependabot (#1133)

v2.7.3

Noteworthy Changes

• Collect consumer metrics @erikvanoosten (#1143)

Zio-kafka now passes a lot of metrics to zio. Use any zio metrics backend to expose them! See the issue #1143 for more information on which metrics are collected.

Changes

• Update zio, zio-streams, zio-test, ... to 2.0.21 @zio-scala-steward (#1144)
• Add scala steward badge (2nd attempt) @erikvanoosten (#1157)
• Update README.md @zio-assistant (#1155)
• Ensure that fibers forked in Runloop.make are interruptible @svroonland (#1154)
• Add scala steward badge @erikvanoosten (#1145)
• Collect consumer metrics @erikvanoosten (#1143)
• Update zio-logging-slf4j, ... to 2.2.0 @zio-scala-steward (#1152)
• Update zio-logging-slf4j, ... to 2.1.17 @zio-scala-steward (#1151)
• Update README.md @zio-assistant (#1150)
• Update zio-kafka, zio-kafka-testkit to 2.7.2 @zio-scala-steward (#1149)

v2.7.2

Note worthy changes

• Add partitionsFor method to producer @kyri-petrou (#1147)
• retry Consumer.CommitTimeout exception @myazinn (#1140)

Other changes

• Use Semaphore @adamgfraser (#1142)
• Make tests faster @erikvanoosten (#1139)
• Delete RunloopTimeout and small improvements @erikvanoosten (#1141)
• Update kafka-clients to 3.6.1 @zio-scala-steward (#1131)
• Improve support and documentation for manual offsets @erikvanoosten (#1135)
• Update sbt to 1.9.8 @zio-scala-steward (#1138)
• Update zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1137)
• Update zio, zio-concurrent, zio-streams, ... to 2.0.20 @zio-scala-steward (#1136)
• Update logback-classic to 1.4.14 @zio-scala-steward (#1129)
• Update logback-classic to 1.3.14 @zio-scala-steward (#1128)
• Update zio-kafka, zio-kafka-testkit to 2.7.1 @zio-scala-steward (#1127)
• Remove unused config for Mergify @erikvanoosten (#1124)
• Adds test for stopping consumption while doing async commits @erikvanoosten (#1123)
• Update README.md @zio-assistant (#1125)
• Update logback-classic to 1.4.13 @zio-scala-steward (#1122)