Kafka KRaft Now In Production
Written by Kay Ewbank   
Thursday, 06 October 2022

Apache Kafka has been updated with a new way to run Apache Kafka with self-managed metadata. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

kafka

The main improvement to Kafka is the marking of KRaft mode as production ready, though only for new clusters. This gives a way to run Apache Kafka with self-managed metadata, so eliminating the need to run an Apache Zookeeper cluster alongside every Kafka cluster. The new KRaft mode improves partition scalability and resiliency while simplifying deployments of Apache Kafka.

There are some features that are currently supported by Apache ZooKeeper mode that are not yet supported by KRaft mode. These include configuring SCRAM users via the administrative API, supporting JBOD configurations with multiple storage directories, and modifying certain dynamic configurations on the standalone KRaft controller. The plan is that these features will be added over the next few months.

Other additions include transactional EOS for source connectors. Exactly Once Semantics (EOS) is a framework that allows stream processing applications such as Kafka Streams to process data through Kafka without loss or duplication. The second improvement (which is also part of the work supporting EOS for source connectors) is support for atomic commit of source connector records and offsets, meaning that their offsets are committed if and only if all records up to that point have been ack'd by the producer.

The other new feature is the ability for the admin API to use batched offset fetch requests.

Kafka 3.3 is available now. 

kafka

More Information

Kafka Website

Related Articles

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


PostgreSQL 15 Released - What's New
27/10/2022

The long awaited release of the most popular advanced open source database is finally here carrying many bells and whistles. It includes performance improvements on managing workloads in both loc [ ... ]



Deno 1.28 Stabilizes NPM Compatibility
22/11/2022

Deno 1.28 has been released with stabilizied compatibility for npm. Deno is the JavaScript and TypeScript runtime from the creator of Node.js.


More News

picobook

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Thursday, 13 October 2022 )