Apache Iceberg Rust Released
Written by Alex Denham   
Monday, 09 September 2024

Apache has released version 0.3 of Iceberg-rust, an official Rust implementation of the Iceberg high-performance format for huge analytic tables.

Iceberg was originally developed by Netflix, was made an opensource Apache incubator project in 2018, and graduated to be a top level project in 2020.

Iceberg is widely used and is supported by nearly all query engines. Users can write SQL to query data from an Iceberg table stored at object storage services without ingesting it into a database first.

iceberg

Iceberg Rust 0.3 is built and tested with stable rust, and will keep a rolling MSRV(minimum supported rust version).  This release is being used by Databend, an open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake; and iceberg-catalog, a Rust implementation of the Iceberg REST Catalog specification.

This release is described by the developers as including most of the functionality users need to read and write an Iceberg table, though currently only the REST catalog has been implemented. Other catalogs are under active development.

The main changes to this version start with the addition of Sync and Send to the Catalog trait. Send and Sync are fundamental to Rust's concurrency handling, and are used to work out whether a type can be sent to or shared with other threads. 

This release has also implemented OAuth for the catalog's REST client. OAuth is an open-standard authorization protocol that applications can use to provide secure designated access.

There's also new Parquet writer and reader capabilities with support for data projection, and memory catalog and memory file IO support. A SQL Catalog has been established and there's now a subproject for this.

Support has been added for Google Cloud Storage (GCS) and AWS session tokens to ensure Iceberg-rust can be used with GCS and AWS sessions.

Iceberg-rust's concurrency support has been enhanced with the implementation of concurrent table scans and data file fetching, and there are enhanced predicate builders and expression evaluators. The developers have also added support for timestamp columns in row filters.

The contributors say that plans for the next release include the integration of native support for DataFusion, and the ability to connect other query engines with Iceberg, such as Databend.

Iceberg-rust 0.30 is available now.

iceberg

More Information

Apache Iceberg

Related Articles

Snowflake Support For Apache Iceberg Goes GA

Apache Iceberg Improves Spark Support

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


A Turing Machine In LEGO
06/10/2024

A Turing Machine is one of those abstract ideas that is much easier to undertand if you can see it working. Now there's a proposal on the LEGO Ideas site for a Working Turing Machine. Lend it your sup [ ... ]



Paul Allen's Living Computers Sold For Record Prices
04/10/2024

Auction house Christie's of New York broke existing records for sales of rare and iconic computers when it sold the collection that the late Paul Allen, co-founder of Microsoft, had assembled to  [ ... ]


More News

kotlin book

 

Comments




or email your comment to: comments@i-programmer.info