|PeerDB Brings Real Time Streaming To PostgreSQL
|Written by Nikos Vaggalis
|Thursday, 23 November 2023
PeerDB is an ETL/ELT tool built for PostgreSQL. It makes all tasks that require streaming data from PostgreSQL to third party counterparts as effortless as it gets.
But the basics first;. Why the need to stream data from PostgreSQL, or any database for the matter?
For a start, it's what streaming data's most popular technique CDC is used for. CDC is a way to capture changes made in the database and forwarding them in real-time to external applications (such as Kafka) through connectors such as the ones offered by Debezium, the open source distributed platform that turns your existing databases into event streams. There are many ways to implemented CDC like row versioning, pubsub, triggers and log monitoring, with the log-based one being the most popular and automated. The use cases of CDC include real-time analytic, replication to Data Warehouses, Queues and Storages or any other customized solutions.
The most popular tool for enabling CDC is of course open source Debezium. Compared to Debezium, PeerDB is significantly simpler to set up and manage.
For instance, to mirror data from a Postgres instance to a Snowflake one you just have to :
CREATE PEER postgres_peer
CREATE PEER snowflake_peer
CREATE MIRROR real_time_cdc
Transactions and users table are now replicated in realtime from Postgres to Snowflake, so that when you Insert/Update or Delete from the Postgres tables, the same operation is mirrored on the Snowflake ones too.
Besides sporting a developer friendly API as seen above, PeerDB is also performant in comparison to similar tools:
Since PeerDB talks "Postgres" it also supports native Postgre features such as :
Nativity also means that you can use the tools you are familiar with on PeerDB as well:
PeerDB support a number of different modes of streaming like log based (CDC), cursor based (timestamp or integer) and XMIN, while at the time of writing it supports the following connectors :
Of course it is free and open source and available as a docker image. There's also a Cloud and Enterprise offering which is fully managed and hosted on AWS, Azure and GCP, and requires a paid subscription.
To conclude, PostgreSQL never ceases to amaze. With PeerDB included, its ecosystem goes from strength to strength.
or email your comment to: email@example.com