Cloudera And StreamNative Open Source NiFi Pulsar Connector
Thursday, 10 March 2022

A connector that integrates Apache NiFi and Apache Pulsar has been made open source by Cloudera and StreamNative.

The connector will be available as part of the Cloudera platform. When used together, NiFi and Pulsar can be used to create a cloud-native streaming data platform that can ingest, transform, and analyze massive amounts of data. 

cloudera

The Cloudera team includes some of the original developers of Apache NiFi, while StreamNative was founded by the original creators of Apache Pulsar. NiFi has been developed from the “Niagara Files” technology used by the NSA and made available to the Apache Software Foundation through the NSA Technology Transfer Program. NiFi is a visual tool for flow-based programming that can be used to create data flows that move data from one technological platform, such as databases, cloud-storage, and messaging systems, to another.

NiFi provides event level data provenance and traceability. The NiFi platform includes a collection of over 100 pre-built processors.

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project. Pulsar uses a replicated distributed ledger to provide durable stream storage.

nifitool

The new tool provides a way to consume and produce messages from Pulsar topics at scale with simple configuration settings within Apache NiFi. Once the data has been stored inside Pulsar, it can be made available to stream processing engines such as Flink or Spark.

The tool will be available on Cloudera starting with version 7.2.14 of CDF on the Public Cloud. Developers wanting to use the processors in other Apache NiFi clusters can download the files from a maven central repository, or can build them directly from the source code on GitHub.

 cloudera

More Information

Apache NiFi

Cloudera CDF

Maven Central Repository

Pulsar NiFi Tool On GitHub

Related Articles

Apache Daffodil Now Top Level Project  

Apache Flink ML 2.0 Released

Spark 3 Improves Python and SQL Support

Apache Flink 1.9 Adds New Query Engine

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


GitLab Releases Duo Chat
22/04/2024

GitLab has announced that Duo Chat is now generally available in GitLab 16.11, offering a range of AI features in a single natural language chat experience.



OpenAI Introduces GPT-4o, Loses Sutskever
15/05/2024

It's an eventful week for OpenAI, the research company dedicated to making advances towards Artificial General Intelligence that are both safe and beneficial to all. A day after it showcased its lates [ ... ]


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info