From: Andres Freund <andres@2ndquadrant.com>
To: pgsql-hackers@postgresql.org
Subject: Changeset Extraction v7.0 (was logical changeset generation)
Date: Wed, 15 Jan 2014 01:22:23 +0100
Message-ID: <20140115002223.GA17204@awork2.anarazel.de> (raw)
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Hello Everyone,
I am pleased to announce the next version of the the changeset
extraction feature.
There's more changes than I can remember, but that's what comes to my
tired mind:
* Initial userlevel docs (including an example session!).
* generalization of the "replication slot" system to also work for
streaming rep, although the user interface for that is mostly missing.
* don't remove WAL still required by a replication slot, be it a slot for
changeset extraction or streaming rep.
* New output plugin interface with one _PG_output_plugin_init dlsym()ed
function filling out the callbacks.
* Renaming of the init and _cleanup output plugins to startup and shutdown.
* Simplification of the prepare_write/write interface for output plugins
(no need to specify LSNs anymore).
* Renaming of the changeset extraction operations to
create_replication_slot/drop_replication_slot/start_replication
... logical.
* moving the SQL changeset functions from a contrib module into core
* Addition of peeking functions for changeset extraction.
* revised error messages
* revised comments
* ...
I've followed Robert's wishes with generalizing the replication slot
interface to not only work for changeset generation, but also streaming
rep - not sure whether that was the right choice, it's been more work
than I expected blocking things a bit but we're there now....
There's no clientside support included except as in the pg_receivexlog
hack attached as the last patch, but I also have tested it via streaming
rep and it mostly works (minus a hot_standby_feedback bug, will report
tomorrow).
What I think is missing:
* The user docs need more work, even though we're in a much better state
than before.
* Replication slots are stored in binary files. I think it might make
sense to store them as text files instead, for easier
extensibility. Especially since we want to use them for streaming rep,
I am pretty sure new attributes will soon come. I don't think it's
critical enough performancwise to store them in binary.
* Contrary to what Robert and I'd discussed I've named the SQL functions
outputting changes decoding_slot_(get|peek)_[binary_]changes instead of
decoding_stream_* - I can change that, but the SQL functions don't
actually support streaming, so I thought that might be
confusing. Opinions?
* Robert complained earlier about the way historical catalog snapshots
are forced in tqual.c - I don't really know yet what the better way
would be here.
* Some functionality probably needs to move between the patches - it's a
bit hard to see where the best boundaries are.
The sources are in my git tree at:
http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=summary
branch xlog-decoding-rebasing-remapping.
The last two patches are *not* indendet to be actually applied, but are
useful for testing.
If you want to test, you'll need a clean initdb, set wal_level=logical
and max_replication_slots>0. There's a example SQL session showing how
things can be used....
Testing, Review, Questions welcome!
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackersAttachments:
0001-wal_decoding-Log-xl_running_xact-s-at-a-higher-frequ.patch.gz (application/x-patch-gzip, 3.1K) download
0002-wal_decoding-Introduce-the-replication-slot-interfac.patch.gz (application/x-patch-gzip, 11.2K) download
0003-wal_decoding-Introduce-changeset-extraction.patch.gz (application/x-patch-gzip, 73.7K) download
0004-wal_decoding-Only-peg-the-xmin-horizon-for-catalog-t.patch.gz (application/x-patch-gzip, 5.3K) download
0005-wal_decoding-Allow-walsenders-to-connect-to-a-specif.patch.gz (application/x-patch-gzip, 4.0K) download
0006-wal_decoding-logical-changeset-extraction-walsender-.patch.gz (application/x-patch-gzip, 11.9K) download
0007-wal_decoding-pg_recvlogical-Introduce-pg_receivexlog.patch.gz (application/x-patch-gzip, 9.1K) download
0008-wal_decoding-test_decoding-Add-a-simple-decoding-mod.patch.gz (application/x-patch-gzip, 24.6K) download
0009-wal_decoding-design-document-v2.4-and-snapshot-build.patch.gz (application/x-patch-gzip, 12.9K) download
0010-wal_decoding-Initial-cut-at-sgml-docs-section.patch.gz (application/x-patch-gzip, 8.7K) download
0011-wal_decoding-Temporarily-add-logical-decoding-regres.patch.gz (application/x-patch-gzip, 1.4K) download
0012-slot-hack-up-pg_receivexlog-support.patch.gz (application/x-patch-gzip, 1.6K) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: andres@2ndquadrant.com
Subject: Re: Changeset Extraction v7.0 (was logical changeset generation)
In-Reply-To: <20140115002223.GA17204@awork2.anarazel.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox