Skip to content

fix: commit all messages in a batch and rely on throttled strategy to commit offsets

Jasper Vaneessen requested to merge fix/kafka-commit-strategy into next

Use of acks and commits didn't work as expected. This MR changes sink flow in the following way:

  • After successful batch insertion, all messages in the batch are acked individually (ack is in-memory, no network calls)
  • throttled commit strategy results in commit of offsets after x amount of acks or y amount of time passed

Default settings result in ack size if 50K and ack max age of 60 seconds.

mp.messaging.incoming.metric-events-in.auto.commit.interval.ms=5000
mp.messaging.incoming.metric-events-in.throttled.unprocessed-record-max-age.ms=60000

See https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.4/kafka/kafka.html#_acknowledgement

Edited by Jasper Vaneessen

Merge request reports

Loading