Kafka KRaft Now In Production
Written by Kay Ewbank   
Thursday, 06 October 2022

Apache Kafka has been updated with a new way to run Apache Kafka with self-managed metadata. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

kafka

The main improvement to Kafka is the marking of KRaft mode as production ready, though only for new clusters. This gives a way to run Apache Kafka with self-managed metadata, so eliminating the need to run an Apache Zookeeper cluster alongside every Kafka cluster. The new KRaft mode improves partition scalability and resiliency while simplifying deployments of Apache Kafka.

There are some features that are currently supported by Apache ZooKeeper mode that are not yet supported by KRaft mode. These include configuring SCRAM users via the administrative API, supporting JBOD configurations with multiple storage directories, and modifying certain dynamic configurations on the standalone KRaft controller. The plan is that these features will be added over the next few months.

Other additions include transactional EOS for source connectors. Exactly Once Semantics (EOS) is a framework that allows stream processing applications such as Kafka Streams to process data through Kafka without loss or duplication. The second improvement (which is also part of the work supporting EOS for source connectors) is support for atomic commit of source connector records and offsets, meaning that their offsets are committed if and only if all records up to that point have been ack'd by the producer.

The other new feature is the ability for the admin API to use batched offset fetch requests.

Kafka 3.3 is available now. 

kafka

More Information

Kafka Website

Related Articles

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


The Appeal of Google Summer of Code
21/03/2024

With the list of participating organizations now published, it is time for would-be contributors to select among them and apply for Google Summer of Code (GSoC). Rust has joined in the program fo [ ... ]



GR00T Could Be The Robot You Have Always Wanted
27/03/2024

We may not have flying cars, but we could well soon have robots that match up to predictions for the 21st century. Nvidia has announced GR00T, a cleverly named project to build robots using foundation [ ... ]


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Thursday, 13 October 2022 )