|Apache Arrow 4 Adds New C++ Compute Functions|
|Written by Kay Ewbank|
|Tuesday, 18 May 2021|
Apache Arrow has been updated to version 4.0. It has extra C++ compute functions for numeric and string data, and improves the performance of Arrow Datasets.
The C++ support has been improved in this release, with support for automatic implicit casting in compute kernels, and new compute functions fir numeric data including quantile and power. Several new functions for string processing have been added, providing ways to trim characters, extract substrings captured by a regex pattern, and matching strings against redex patterns. There are also new functions for computing UTF8 string lengths, and replacing non-overlapping substrings that match a literal pattern or regular expression.
Improvements to the Python support include the ability to create a dataset from a Python iterator of record batches. The Dataset interface has also been improved for Python, and can now use custom projections using expressions when scanning.
Rust support has seen the most changes in this release, with new features and performance improvements. The developers say that they have concentrated largely on the necessary details to make it possible to release the Rust versions to cargo at a more regular rate. In addition, the Ballista distributed compute project has been officially included.
Rust support for Arrow includes JSON reader improvements and a new JSON writer, as well as improved schema inference for nested list and struct types. Rust support for Arrow DataFusion has better SQL support including the ability to use Union, Having, Extract, Show Tables and Interval. You can use Group By with more data types, and user defined functions can now provide specialized implementations for scalar values. There are also several new SQL metrics.
Performance improvements include Constant folding, a partitioned hash join, and improved parallelism using repartitioning pass. Hash aggregate performance is also better with large numbers of grouping values, and there's predicate pushdown support for table scans.
or email your comment to: email@example.com
|Last Updated ( Tuesday, 18 May 2021 )|