| Cloudera And StreamNative Open Source NiFi Pulsar Connector |
| Thursday, 10 March 2022 |
|
A connector that integrates Apache NiFi and Apache Pulsar has been made open source by Cloudera and StreamNative. The connector will be available as part of the Cloudera platform. When used together, NiFi and Pulsar can be used to create a cloud-native streaming data platform that can ingest, transform, and analyze massive amounts of data.
The Cloudera team includes some of the original developers of Apache NiFi, while StreamNative was founded by the original creators of Apache Pulsar. NiFi has been developed from the “Niagara Files” technology used by the NSA and made available to the Apache Software Foundation through the NSA Technology Transfer Program. NiFi is a visual tool for flow-based programming that can be used to create data flows that move data from one technological platform, such as databases, cloud-storage, and messaging systems, to another. NiFi provides event level data provenance and traceability. The NiFi platform includes a collection of over 100 pre-built processors. Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project. Pulsar uses a replicated distributed ledger to provide durable stream storage.
The new tool provides a way to consume and produce messages from Pulsar topics at scale with simple configuration settings within Apache NiFi. Once the data has been stored inside Pulsar, it can be made available to stream processing engines such as Flink or Spark. The tool will be available on Cloudera starting with version 7.2.14 of CDF on the Public Cloud. Developers wanting to use the processors in other Apache NiFi clusters can download the files from a maven central repository, or can build them directly from the source code on GitHub. More InformationRelated ArticlesApache Daffodil Now Top Level Project Spark 3 Improves Python and SQL Support Apache Flink 1.9 Adds New Query Engine Apache Flink 1.5.0 Adds Support For Broadcast State Flink Gets Event-time Streaming To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin. 
Comments
or email your comment to: comments@i-programmer.info |
