|PipelineDB Released As PostgreSQL Extension|
|Written by Kay Ewbank|
|Thursday, 01 November 2018|
PipelineDB 1.0 has been released as a PostgreSQL extension. It is intended to be used for high-performance time-series aggregation based on continuous SQL queries.
The original version of PipelineDB was released as a fork of PostgreSQL, but user demand was for an extension rather than a standalone fork, and having made money from the original version, the developers have invested into reworking PipelineDB as a standard PostgreSQL extension.
PipelineDB is open-source, and is designed for use in large scale data sets where storing lots of raw time-series data and aggregating it over and over again becomes inefficient. The developers say the amount of value that PipelineDB adds is directly proportional to the amount of continuous aggregation that an analytics use case can benefit from. They are also clear that PipelineDB should be used for analytics use cases that only require summary data, like realtime reporting dashboards.
PipelineDB's strength lies in being used for running continuous aggregations over streaming time-series data, and only storing the compact output of these continuous queries as incrementally updated table rows that can be evaluated with minimal query latency. The ideal situation is where you know in advance what queries you're going to want to run, so that the analytics are simpler, faster and cheaper because you've already done the pre-querying. This means summary data is always available for low-latency lookups.
The developers say that:
"Even if billions and billions of rows are written to
This way of working enables extremely large levels of raw event throughput on modest hardware footprints, and extremely low read query latencies.
Alongside time-series analysis, PipelineDB can be used for what the developers call continuous transforms:
"Unlike continuous views -- which store aggregate state in incrementally updated tables -- continuous transforms are stateless and simply apply a transformation to a stream, writing out the result to another stream."
PipelineDB ships with a number of built-in aggregates including:
or email your comment to: firstname.lastname@example.org
|Last Updated ( Thursday, 01 November 2018 )|