Date: Tue, 2 Aug 2022 15:57:39 +0100
From: Rory Campbell-Lange <rory@campbell-lange.net>
To: Robert Blayzor <rblayzor.bulk@inoc.net>
Cc: pgsql-performance@postgresql.org
Subject: Re: PgSQL 14 - Logical Rep - Single table multiple publications?
Message-ID: <Yuk7YwXqd2ETishS@campbell-lange.net>
References: <3d5996ac-a181-21a5-62a0-1cd34e6b3f1a@inoc.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3d5996ac-a181-21a5-62a0-1cd34e6b3f1a@inoc.net>
Archived-At: 
 <https://www.postgresql.org/message-id/Yuk7YwXqd2ETishS%40campbell-lange.net>
Precedence: bulk

On 02/08/22, Robert Blayzor (rblayzor.bulk@inoc.net) wrote:
> Is it possible to have a single subscriber table contact multiple publishers
> and just insert all of the data into a single table on the subscriber? ie:
> merge type replication. There are no primary/FK constraints, etc.  The
> records are just time based audit log type data...

Your use case meets, I think, the third "typical use case" listed at
https://www.postgresql.org/docs/current/logical-replication.html, namely
"Consolidating multiple databases into a single one (for example for
analytical purposes)."

I've just been testing aggregating all the data in one schema across 300
publisher databases into 5 subscriber schemas on two Postgresql 14 clusters on
the same machine. Each of 60 publisher tables are aggregating into a
single table on the subscriber.

Special care must be taken with the "replica identity" of published
tables, as set out at
https://www.postgresql.org/docs/current/logical-replication-publication.html.
For example, you may need a unique identifying column for each source
table in addition to the normal row identifier to differentiate *this*
table's id 1 row from the *other* table's id 1 row, otherwise the
subscriber won't be able to identify the row to delete if this table's
id 1 row is deleted (for example).

Although this seems to work fine with native replication, the pglogical
extension has more knobs. For instance, the
pglogical.wait_for_subscription_sync_complete function is useful to ensure that
sync finishes when part of a migration.

Rory