A connector that integrates Apache NiFi and Apache Pulsar has been made open source by Cloudera and StreamNative.

The connector will be available as part of the Cloudera platform. When used together, NiFi and Pulsar can be used to create a cloud-native streaming data platform that can ingest, transform, and analyze massive amounts of data. 

The Cloudera team includes some of the original developers of Apache NiFi, while StreamNative was founded by the original creators of Apache Pulsar. NiFi has been developed from the “Niagara Files” technology used by the NSA and made available to the Apache Software Foundation through the NSA Technology Transfer Program. NiFi is a visual tool for flow-based programming that can be used to create data flows that move data from one technological platform, such as databases, cloud-storage, and messaging systems, to another.

NiFi provides event level data provenance and traceability. The NiFi platform includes a collection of over 100 pre-built processors.

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project. Pulsar uses a replicated distributed ledger to provide durable stream storage.

The new tool provides a way to consume and produce messages from Pulsar topics at scale with simple configuration settings within Apache NiFi. Once the data has been stored inside Pulsar, it can be made available to stream processing engines such as Flink or Spark.

The tool will be available on Cloudera starting with version 7.2.14 of CDF on the Public Cloud. Developers wanting to use the processors in other Apache NiFi clusters can download the files from a maven central repository, or can build them directly from the source code on GitHub.

More Information

Apache NiFi

Cloudera CDF

Maven Central Repository

Pulsar NiFi Tool On GitHub

Related Articles

Apache Daffodil Now Top Level Project  

Apache Flink ML 2.0 Released

Spark 3 Improves Python and SQL Support

Apache Flink 1.9 Adds New Query Engine

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info