TileDB Improves Sparse Array Support
Written by Kay Ewbank   
Monday, 25 May 2020

Designed to give data scientists a more powerful way to store, update, analyze, and share large sets of data, TileDB lets you model data as either a dense or a sparse multi-dimensional array. New features in version 2 include support for heterogeneous dimensions and string dimensions in sparse arrays.

TileDB consists of a multi-dimensional array data format, a fast embeddable, open-source C++ storage engine with data science tooling integrations, and its own propriatary cloud service, TileDB Cloud, which was launched in November 2019. So while the Developer Edition of the database engine is free, you pay as you go for compute time. There is also TileDB Enterprise which is the same as TileDB Cloud delivered as licensable software with support. TileDB Enterprise can be deployed on-premises or on a private cloud with LDAP and SAML authentication.

 

 

TileDB's embeddable C++ library includes APIs for C, C++, Python, R, Java and Go. The library is integrated with Spark, Dask, PrestoDB, MariaDB, Arrow and geospatial libraries like PDAL, GDAL and Rasterio. The new release has a completely revamped TileDB R API that was worked on by Dirk Eddelbuettel, a developer known for his work on several popular R packages and a board member of the R Foundation. The TileDB team says:

"We want to make TileDB an integral part of the R ecosystem and are just getting going on integrations with other key R packages, such as the tidyverse and Bioconductor."

TileDB 2.0 adds support for Google Cloud Storage and Azure Blob Storage to the existing AWS S3 support together with a completely revamped R API. The other main improvement to the new release is support for heterogenous and string dimensions. The previous release of TileDB only supported homogeneous dimensions, dimensions with the same data type. This worked well for some data, but the developers realized it was limiting for dataframes where the columns were made up of different data types such as Date (Datetime) and Price (Double).. In addition, many dataframes also have String columns that users need to slice such as Name. TileDB 2.0 adds heterogeneous and string dimensions, so now fully supports use with dataframes.

The new release is available on GitHub or from the TileDB website.

 

tiledb

 

More Information

TileDB Homepage

TileDB On GitHub

Related Articles

Databricks Delta Adds Faster Parquet Import

Databricks Delta Lake Now Open Source

RAPIDS GPU Data Analysis Platform Launched

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, Facebook or Linkedin.

Banner


Neo4j Announces Graph Machine Learning Framework
27/10/2020

Neo4j has announced a graph machine learning framework that the developers describe as making advanced graph-based machine learning techniques more accessible.



Angular 11 Released With Byelog Goal Complete
17/11/2020

Angular 11 has been released with updates across the platform including the framework, the CLI and components. More specifically, the new version enforces stricter types and has automatic inlining of  [ ... ]


More News

square

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 25 May 2020 )