Skip to content

feat: Kafka message data components - #298

Draft
chrisk314 wants to merge 1 commit into
feat/message-data-classesfrom
feat/kafka-message-data
Draft

chrisk314 wants to merge 1 commit into
feat/message-data-classesfrom
feat/kafka-message-data

Conversation

@chrisk314

Copy link
Copy Markdown
Contributor

Draft while the testing story is unresolved. There is no Kafka broker available to integration-test against, and standing one up is an open decision (managed MSK/Confluent versus a container such as Redpanda in compose.yaml). It is also not yet clear that Kafka support is wanted enough to warrant the time needed to test it properly. This stays a draft until a testing setup is determined and there is appetite to invest in it.

What it adds

KafkaDataReader and KafkaDataWriter on the MessageDataReader/MessageDataWriter bases from #284 - the Apache Kafka equivalent of the PubSub and SQS/SNS components.

  • Consumer: consumer group with enable_auto_commit=False; acknowledging a batch commits TopicPartition -> OffsetAndMetadata(offset + 1) for the records actually processed, taking the highest offset per partition, so records still waiting in the buffer remain eligible for redelivery. An empty getmany poll returns no messages rather than ending the stream, matching the base class's "idle is not exhaustion" rule.
  • Producer: a batch is submitted with send() for every record and flushed once, so aiokafka's internal batching is used instead of one request/ack round trip per record.
  • Configuration: bootstrap_servers accepts a string or list and falls back to KAFKA_BOOTSTRAP_SERVERS; KAFKA_BOOTSTRAP_SERVERS also lands in Settings.kafka.
  • Errors: authorization and record-too-large failures map to MessageBrokerPermanentError (no retries burnt on them); connection, coordinator and rebalance errors map to MessageBrokerTransientError and are retried with backoff.
  • Packaging: the kafka extra (aiokafka>=0.11,<1) plus aiokafka in the test dependency group so the unit tests import the real error and offset types.
  • Docs: Kafka rows, install extra, settings and example in the Message Data usage page.

Split out of #284 as requested so that PR stays scoped to what can be tested: the base classes, PubSub and SQS/SNS.

Testing

24 unit tests (128 across the message data suite with this branch), all mock-based against unittest.mock doubles patched into plugboard.library.kafka_io - no broker required. They cover offset commit contents (including out-of-order records spanning partitions), empty polls, multi-partition flattening, batched sends with exactly one flush, error mapping, settings resolution and JSON vs raw encoding.

ruff, ty and tests/unit pass. No integration test against a real Kafka broker exists, which is the gap the draft status above is about; the behaviour most likely to bite is exactly the part mocks cannot show - group rebalances, session timeouts, and commit semantics under real partition assignment.

Notes for whoever picks this up

  • This PR targets feat/message-data-classes, not main. Retarget it to main once feat: Message data classes #284 merges (GitHub retargets automatically if the base branch is deleted after a squash-merge, but double-check).
  • uv.lock conflicts with feat: Message data classes #284 are likely: this branch re-adds aiokafka (and its async-timeout dependency) to the lockfile that feat: Message data classes #284 removes them from. Resolving by re-running uv lock on top of the merged base is the safest route.
  • If only a container-based setup is on the table, compose.yaml already provides the pattern used for RabbitMQ/Redis, and the CI job that starts it exports the broker URLs - Kafka would follow the same shape (KAFKA_BOOTSTRAP_SERVERS plus a test_kafka_io.py integration test that round-trips a record and asserts the commit).

Adds `KafkaDataReader` and `KafkaDataWriter` on the `MessageDataReader` /
`MessageDataWriter` bases:

- Consumer uses a consumer group with auto-commit disabled; acknowledging a batch
  commits the offsets following the processed records, highest per partition, so
  records still buffered remain eligible for redelivery.
- Producer submits a batch's records together and flushes once, keeping the
  client's internal batching instead of one round trip per record.
- `bootstrap_servers` falls back to `KAFKA_BOOTSTRAP_SERVERS` and accepts a string
  or a list; aiokafka failures are mapped onto the broker exception hierarchy
  (authorization / record-too-large are permanent, connection and rebalance errors
  are retried).
- The `kafka` extra declares aiokafka, and aiokafka is in the test dependency group
  so the unit tests can import the real error and offset types.

372 lines of tests cover offset commits (including out-of-order records across
partitions), empty polls returning no messages, multi-partition flattening, batched
sends with a single flush, error mapping, settings resolution and JSON vs raw
encoding.

Split out of #284, where it landed with the PubSub and SQS/SNS implementations.
@github-actions

Copy link
Copy Markdown

Benchmark comparison for 1be92fad (base) vs 861fa1d6 (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.14-64bit/0001_pr)         291.9942 (1.0)      297.7815 (1.00)     294.7383 (1.00)     2.2927 (1.09)     294.1210 (1.0)      3.5155 (1.0)           2;0  3.3928 (1.00)          5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.14-64bit/0001_base)     292.0238 (1.00)     296.5186 (1.0)      294.3717 (1.0)      2.1120 (1.0)      295.3671 (1.00)     3.8098 (1.08)          3;0  3.3971 (1.0)           5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant