From: Rory Campbell-Lange <rory@campbell-lange.net>
To: Robert Blayzor <rblayzor.bulk@inoc.net>
Cc: pgsql-performance@postgresql.org
Subject: Re: PgSQL 14 - Logical Rep - Single table multiple publications?
Date: Tue, 2 Aug 2022 15:57:39 +0100
Message-ID: <Yuk7YwXqd2ETishS@campbell-lange.net> (raw)
In-Reply-To: <3d5996ac-a181-21a5-62a0-1cd34e6b3f1a@inoc.net>
References: <3d5996ac-a181-21a5-62a0-1cd34e6b3f1a@inoc.net>
On 02/08/22, Robert Blayzor (rblayzor.bulk@inoc.net) wrote:
> Is it possible to have a single subscriber table contact multiple publishers
> and just insert all of the data into a single table on the subscriber? ie:
> merge type replication. There are no primary/FK constraints, etc. The
> records are just time based audit log type data...
Your use case meets, I think, the third "typical use case" listed at
https://www.postgresql.org/docs/current/logical-replication.html, namely
"Consolidating multiple databases into a single one (for example for
analytical purposes)."
I've just been testing aggregating all the data in one schema across 300
publisher databases into 5 subscriber schemas on two Postgresql 14 clusters on
the same machine. Each of 60 publisher tables are aggregating into a
single table on the subscriber.
Special care must be taken with the "replica identity" of published
tables, as set out at
https://www.postgresql.org/docs/current/logical-replication-publication.html.
For example, you may need a unique identifying column for each source
table in addition to the normal row identifier to differentiate *this*
table's id 1 row from the *other* table's id 1 row, otherwise the
subscriber won't be able to identify the row to delete if this table's
id 1 row is deleted (for example).
Although this seems to work fine with native replication, the pglogical
extension has more knobs. For instance, the
pglogical.wait_for_subscription_sync_complete function is useful to ensure that
sync finishes when part of a migration.
Rory
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: rory@campbell-lange.net, rblayzor.bulk@inoc.net
Subject: Re: PgSQL 14 - Logical Rep - Single table multiple publications?
In-Reply-To: <Yuk7YwXqd2ETishS@campbell-lange.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox