Google has announced improvements to BigLake and BigQuery, including with the general availability of BigLake Metastore; new high-performance Iceberg-native Cloud Storage; and native support with Dataplex Universal Catalog, providing unified and fine-grained access controls across all supported engines. 

Google BigQuery metastore was announced in beta earlier in the year, and has now been renamed as BigLake Metastore. It's a highly scalable runtime metadata service that works with multiple engines including BigQuery, Apache Spark, Apache Hive and Apache Flink, and supports the open Apache Iceberg table format.

Google BigQuery is a distributed, serverless SQL engine that provides a way to query petabytes of data. It has built-in machine learning, is serverless, and supported by Google Cloud.

BigLake is a storage engine that provides a foundation for building open data lakehouses on Google Cloud that bring the best of Google’s infrastructure to Apache Iceberg. The data platform is built on Google BigQuery and Dataplex and can be used for data analysis for both structured and unstructured data.

The newly available BigLake Metastore, formerly known as BigQuery metastore, is fully managed, serverless, and scalable. It is works with BigQuery as well as other Iceberg compatible engines, and Google says removes the need to manage custom metastore deployments. The BigLake metastore can be used to access all Cloud Storage and BigQuery storage data across multiple runtimes including BigQuery, AlloyDB (preview), and open-source, Iceberg-compatible engines such as Spark and Flink. Google is also introducing support for the Iceberg REST Catalog API in preview. 

This release also adds new, high-performance Iceberg-native Cloud Storage, including automatic table maintenance (including compaction and garbage collection) and integration with Google Cloud Storage management tools, including auto-class tiering and encryption.

BigLake tables for Apache Iceberg in BigQuery will enable high-throughput streaming, auto-reclustering, and support for multi-table transactions is being added soon. BigQuery also adds native integration with Vertex AI, and Google says the BigLake updates are natively supported with Dataplex Universal Catalog, so benefit from unified and fine-grained access controls across all supported engines. 

Google BigLake Metastore is available now.

More Information

Google BigQuery

Google BigLake

Related Articles

Google BigQuery Updated

BigQuery Updated and Repriced

Google BigQuery Updated

BigQuery Now Open to All

Google BigQuery Service

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info