|KSQL Adds Avro Support|
|Written by Kay Ewbank|
|Thursday, 28 December 2017|
The developers of KSQL, the streaming SQL engine for Apache Kafka, have released version 0.3 with improvements for robustness and resource utilization.
KSQL is designed to make it easier to read, write, and process streaming data in real-time, at scale, using SQL-like semantics. It supports stream processing operations including aggregations, joins, windowing, and session management.
KSQL is now on a monthly release schedule. The latest version, KSQL 0.3, has a mix of some new features based on user requests, alongside improvements to the way the language is written to make it more robust and to use resources better.
The first feature of interest in the updated version is Avro support and integration with the Confluent Schema Registry. Until now, KSQL has been able to use data in JSON and delimited formats. The developers say that understandably, they've been getting requests for support other data formats, and Avro has been by far the most requested.
Apache Avro is a data serialization system that was developed as part of the Apache Hadoop project, and that is also used by Kafka. Acro uses JSON for defining its data types and protocols. It serializes data in a compact binary format.
The new support for Avro in this release of KSQL comes via integration with the Confluent Schema Registry, which is part of open source Confluent Platform. This means you can now run KSQL queries that read and write Avro data.
The use of the Confluent Schema Registry means the support for Avro is more complete than you might expect. If you want to create a STREAM or TABLE, KSQL infers the necessary information from the associated Avro schema in the Confluent Schema Registry. This means you don't have to manually define Avro schemas then map them to KSQL’s columns and types in your DDL statements.
You can also convert between Acro, JSON and delimited data formats in real-time using just a single line of KSQL. You can also create joins between streams and tables in KSQL regardless of the underlying data formats. The developers say that there’s no special syntax needed; joining different data sources “just works” because KSQL’s internal data model translates automatically between the various data formats for you.
The other main improvements to this version of KSQL are to add the beginnings of metrics and the ability to observe what's happening in KSQL. For streams and tables, the language has a new
statement that shows statistics, such as number of messages processed per second, total messages, the time when the last message was received, as well as corresponding failure metrics.
The developers have also improved the
or email your comment to: firstname.lastname@example.org
|Last Updated ( Thursday, 28 December 2017 )|