Apache Kafka 2.7 Updates Broker
Written by Kay Ewbank   
Tuesday, 29 December 2020

Apache Kafka, the big data and datastreams platform, has been updated in a new version that the developers say moves closer to replacing ZooKeeper in Kafka with a self-managed metadata quorum.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

kafka

It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

Currently, Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller.  The developers are working to remove this dependency to enable them to manage metadata in a more scalable and robust way, enabling support for more partitions.  It will also simplify the deployment and configuration of Kafka.

The latest work on this large task involves a new AlterIsr API to replace the notification znode so the controller has the exclusive ability to update Leader and ISR state. Leaders will use this API to request an ISR change from the controller rather than directly mutating the underlying state.

The work on replacing ZooKeeper is far from over. There are currently seven tasks in active development to provide support for more partitions per cluster, simpler operation, and tighter security.

 The new release, Kafka 2.7, also provides the addition of the Core Raft Implementation. This is part of a larger set of work to improve the architecture of Kafka to be based on a replicated metadata log which is maintained by a self-managed quorum. Kafka relies heavily on log replication, and the developers want to use the ideas behind  is at the heart of Kafka, and it is also at the heart of consensus protocols such as Raft. The latest work adds a separate "raft" module containing the core consensus protocol. 

Other improvements adds metrics to Kafka Streams to report properties of RocksDB, and support for sliding windows aggregations in the DSL.

 

kafka

More Information

Kafka Website

Related Articles

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

Kafka Webview Released

Comparing Kafka To RabbitMQ

Apache Kafka Adds New Streams API

 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, Facebook or Linkedin.

Banner


NetBeans Improves Java Support
05/01/2021

Apache NetBeans has been updated to offer improved support for Java 14 and 15 with support for code coloring and formatting alongside handling of new Java language features by autocomplete. The releas [ ... ]



Colorization Of Early Films Good or Bad?
17/01/2021

Here's another example of the use of AI for colorizing and breathing new life into photos, or rather in this case early film footage. This one goes an extra step - adding sound effects. While the resu [ ... ]


More News

square

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Tuesday, 29 December 2020 )