Cloudera And StreamNative Open Source NiFi Pulsar Connector
Thursday, 10 March 2022

A connector that integrates Apache NiFi and Apache Pulsar has been made open source by Cloudera and StreamNative.

The connector will be available as part of the Cloudera platform. When used together, NiFi and Pulsar can be used to create a cloud-native streaming data platform that can ingest, transform, and analyze massive amounts of data. 

cloudera

The Cloudera team includes some of the original developers of Apache NiFi, while StreamNative was founded by the original creators of Apache Pulsar. NiFi has been developed from the “Niagara Files” technology used by the NSA and made available to the Apache Software Foundation through the NSA Technology Transfer Program. NiFi is a visual tool for flow-based programming that can be used to create data flows that move data from one technological platform, such as databases, cloud-storage, and messaging systems, to another.

NiFi provides event level data provenance and traceability. The NiFi platform includes a collection of over 100 pre-built processors.

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project. Pulsar uses a replicated distributed ledger to provide durable stream storage.

nifitool

The new tool provides a way to consume and produce messages from Pulsar topics at scale with simple configuration settings within Apache NiFi. Once the data has been stored inside Pulsar, it can be made available to stream processing engines such as Flink or Spark.

The tool will be available on Cloudera starting with version 7.2.14 of CDF on the Public Cloud. Developers wanting to use the processors in other Apache NiFi clusters can download the files from a maven central repository, or can build them directly from the source code on GitHub.

 cloudera

More Information

Apache NiFi

Cloudera CDF

Maven Central Repository

Pulsar NiFi Tool On GitHub

Related Articles

Apache Daffodil Now Top Level Project  

Apache Flink ML 2.0 Released

Spark 3 Improves Python and SQL Support

Apache Flink 1.9 Adds New Query Engine

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Rust's Rapid Rise on TIOBE Index
10/07/2024

Rust is making spectacular progress up the TIOBE index and JavaScript is also on the up and experiencing a personal best. Kotlin is maintaining its inclusion in the top 20 and the gap at the very top  [ ... ]



NSA Refuses To Release Grace Hopper Tapes
14/07/2024

A lecture by Grace Hopper with the title “Future Possibilities: Data, Hardware, Software, and People” was recorded on videotape. More than 40 years later NSA is refusing to release it.


More News

kotlin book

 

Comments




or email your comment to: comments@i-programmer.info