Ibis 8 Adds Streaming
Written by Kay Ewbank   
Tuesday, 05 March 2024

Ibis 8.0 has been released with stream processing backends. The new release includes Apache Flink as a streaming backend, and RisingWave, a streaming database backend. There's also a new batch backend with Exasol, bringing the total number of backends Ibis supports to 20. 

The Ibis team says that most geospatial operations are now supported in the DuckDB backend, making Ibis a local option for geospatial analytics.

Ibis is a portable Python dataframe library. It provides support for scaling data transformation code in Python and can be used to work with data in multiple data platforms. Ibis has been downloaded more than 10 million times.

ibis

Ibis is an open source project and is independently governed. While four out of five members of the steering committee are employed by Voltron Data, it is not owned by Voltron Data. The fifth member of the steering committee is at Alphabet working on Google BigQuery.

Ibis has the ability to work with multiple dataframes, and provides a common API for data manipulation in Python, and compiles that API into the backend's native language. The Ibis team acknowledges that portability with Ibis isn't perfect, but say that commonalities across backends and SQL dialects combined with years of engineering effort produce a full-featured and robust framework for data manipulation in Python.

The new support for stream processing opens up the option of using Ibis for working with high-throughput, low-latency data processing with time semantics for applications such as fraud detection, real-time analytics, and IoT. The inclusion of Apache Flink means Ibis has the most popular open-source stream processing framework, while RisingWave is an open-source Postgres-compatible streaming database with a cloud offering.

With the release of version 8.0, Ibis now supports 20 different query engines, accommodating a wide range of data processing needs from small-scale queries with DuckDB to large, distributed preprocessing/ETL jobs with engines like BigQuery, Spark, Theseus, and more.

The updated list of supported backends includes Google BigQuery, ClickHouse, Dask, DataFusion, Druid, DuckDB, Exasol, Flink, Impala, Microsoft SQL Server, MySQL, Oracle, pandas, Polars, PostgreSQL, PySpark, RisingWave, Snowflake, SQLite and Trino.

Ibis 8 is available now.

ibis

More Information

Ibis Website

Related Articles

Google Builds Data Lake On BigQuery

Apache Flink ML 2.0 Released

Apache Flink 1.9 Adds New Query Engine

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

Google BigQuery Updated

Pandas Reaches 1.0

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


C23 ISO Standard Is Here But You Probably Won't Read It
06/11/2024

At last ISO C23 has been published, but at $250 you probably aren't going to read it. Can we really tolerate this sort of profiteering on the work of others? This is worse than academic publishing!



DuckDB And Hydra Partner To Get DuckDB Into PostgreSQL
11/11/2024

The offspring of that partnership is pg_duckdb, an extension that embeds the DuckDB engine into the PostgreSQL database, allowing it to handle analytical workloads.


More News

espbook

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Tuesday, 05 March 2024 )