GoKa Stream Processing For Kafka
GoKa Stream Processing For Kafka
Written by Kay Ewbank   
Friday, 26 May 2017

There's a new library for writing distributed stream processing applications in Go backing them with data in Kafka. GoKa aims to reduce the complexity of building highly scalable and highly available microservices.

Kafka is an Apache project that can be used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, fast, and runs in production in thousands of companies. Goka  uses Kafka for message passing, fault-tolerant state storage and workload partitioning

Goka works by binding a state table to Kafka consumer groups, and persisting them in Kafka. The Goka group table represents the state of the group. A microservice modifies and serves the content of a table employing two complementary object types: processors and views.

A processor is a set of callback functions that modify the group table when messages arrive. Processors can also emit messages into other topics. Goka distributes the partitions of the input topics across all processor instances in a processor group to enable scaling and fault-tolerance. If a processor instance fails, its partitions and state are reassigned to the remaining healthy members of the processor group.

A view is a persistent cache of a group table that subscribes for the updates of all partitions of a group table and keeps local disk storage in sync with the group topic. Views provide read-only access to the group tables and can be used to provide external services.

Emitters deliver key-value messages into Kafka. The example used in the documentation is of a database handler emitting the state changes into Kafka for other interested applications to consume.

Goka handles all the message input and output. You provide one or more callback functions that handle messages from any of the Kafka topics you are interested in, and then can work with only the deserialized messages.

Goka is on GitHub complete with samples and good documentation.

kakfalogo

More Information

Goka On GitHub

Related Articles

Apache Kafka Adds New Streams API

Apache Beam Moves To Top Level

HBase Adds MultiWAL Support

Spark BI Gets Fine Grain Security

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

 

Banner


Oracle Database Programming Interface for C
15/08/2017

Oracle's Database Programming Interface for C (ODPI-C) is now available on GitHub. ODPI-C is an open source library of C code that simplifies access to Oracle Database for applications written in C or [ ... ]



Top 10 From Around The Web: eCommerce Development
28/07/2017

eCommerce is the focus of this trawl around the web to bring you items of interest that you otherwise might not have found. We start with a look at integrating a payment gateway and continue with [ ... ]


More News

 

 
 

 

blog comments powered by Disqus

Last Updated ( Friday, 26 May 2017 )
 
 

   
Banner
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.