Kafka Graphs Framework Extends Kafka Streams
Written by Kay Ewbank   
Friday, 03 August 2018

A new graph processing framework for Apache Kafka extends Kafka Streams to provide distributed graph analytics based only on what is already provided by the Kafka layers. Kafka Graphs is a client layer for distributed processing of graphs.

Kafka Graphs provides a library for graph transformations alongside a distributed platform for executing graph algorithms. The developers say it was inspired by other platforms for graph analytics such as Apache Flink Gelly, Apache Spark GraphX, and Apache Giraph, but unlike these other frameworks it does not require anything other than what is already provided by the Kafka abstraction funnel.

The Kafka abstraction tunnel is the name given by the developer of Kafka Graphs, Robert Yokota, to four layers of the Kafka ecosystem  - Confluent's KSQL, the Kafka Streams DSL, the Kafka Streams Processor API, and the Producer-Consumer API. These form a hierarchy of abstractions that handle different parts of the stream being processed, with most cases being handled by KSQL, and subsequent layers handling smaller volumes to form the 'funnel'.

Graphs in Kafka Graphs are represented by two tables from Kafka Streams, one for vertices and one for edges. The vertex table is comprised of an ID and a vertex value, while the edge table is comprised of a source ID, target ID, and edge value. Once a graph is created, graph transformations can be performed on it.

Kafka Graphs provides a number of graph algorithms based on the approach taken by Pregel, a system for large-scale graph processing. Algorithms in Graphs include breadth-first search, local clustering coefficient, single- and multiple-source shortest path,weakly-connected components, and pagerank.  Custom Pregel-based graph algorithms can also be added by implementing the ComputeFunction interface.

Because Kafka Graphs is built on top of Kafka Streams, it is able to use the underlying partitioning scheme of Kafka Streams to support distributed graph processing. When multiple instantiations of the REST application are started on different hosts they will automatically work together to partition the set of vertices when executing a graph algorithm.

The developers say Kafka Graphs is still in its early stages, but it is available to try on GitHub.

kakfalogo

More Information

Kafka Graphs on GitHub

Pregel

Related Articles

Apache Kafka Adds New Streams API

Open Source GraphQL Engine Launched

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, Facebook or Linkedin.

Banner


Linux Doubles Market Share
08/05/2020

Linux usage more than doubled during April, increasing from a share of 1.36 percent of all desktop operating systems in March to 2.87 percent in April. Much of the increase was seen in Ubunt [ ... ]



Python Developer Survey
07/05/2020

Insights from 24,000 Python developers from more than 150 different countries reveal that JavaScript is the most common language used in combination with Python, data analysis is the main motivat [ ... ]


More News

graphics

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Friday, 03 August 2018 )