Apache Kafka 2.7 Updates Broker
Written by Kay Ewbank   
Tuesday, 29 December 2020

Apache Kafka, the big data and datastreams platform, has been updated in a new version that the developers say moves closer to replacing ZooKeeper in Kafka with a self-managed metadata quorum.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

kafka

It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

Currently, Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller.  The developers are working to remove this dependency to enable them to manage metadata in a more scalable and robust way, enabling support for more partitions.  It will also simplify the deployment and configuration of Kafka.

The latest work on this large task involves a new AlterIsr API to replace the notification znode so the controller has the exclusive ability to update Leader and ISR state. Leaders will use this API to request an ISR change from the controller rather than directly mutating the underlying state.

The work on replacing ZooKeeper is far from over. There are currently seven tasks in active development to provide support for more partitions per cluster, simpler operation, and tighter security.

 The new release, Kafka 2.7, also provides the addition of the Core Raft Implementation. This is part of a larger set of work to improve the architecture of Kafka to be based on a replicated metadata log which is maintained by a self-managed quorum. Kafka relies heavily on log replication, and the developers want to use the ideas behind  is at the heart of Kafka, and it is also at the heart of consensus protocols such as Raft. The latest work adds a separate "raft" module containing the core consensus protocol. 

Other improvements adds metrics to Kafka Streams to report properties of RocksDB, and support for sliding windows aggregations in the DSL.

 

kafka

More Information

Kafka Website

Related Articles

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

Kafka Webview Released

Comparing Kafka To RabbitMQ

Apache Kafka Adds New Streams API

 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Rust Twice As Productive As C++
03/04/2024

Google director of engineering, Lars Bergstrom, gave a talk at the recent Rust Nation UK conference and claimed that Rust was twice as productive as C++. Given how good Google is at C++, this is quite [ ... ]



Android 15 Developer Preview Updated
25/03/2024

Google has released Android 15 Developer Preview 2 with changes including better handling of automatic language switching and updates for OpenJDK 17.


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Tuesday, 29 December 2020 )