public inbox for [email protected]
help / color / mirror / Atom feedFrom: Craig Ringer <[email protected]>
To: Robert Haas <[email protected]>
Cc: Simon Riggs <[email protected]>
Cc: Konstantin Knizhnik <[email protected]>
Cc: pgsql-hackers <[email protected]>
Subject: Re: Logical replication and multimaster
Date: Fri, 11 Dec 2015 18:12:55 +0800
Message-ID: <CAMsr+YFf4hA=hLTe2hvPZ95-Fxf3bT80-_8CWwrDpDzva3LTOw@mail.gmail.com> (raw)
In-Reply-To: <CA+TgmoaqJJsPNb7JDAHTtGZbH7CSenuf=0trQ_j1+_scLE-c0w@mail.gmail.com>
References: <[email protected]>
<CANP8+jLmPXKxDsqXwf6RNSp9Q-6uc7UcZo=KNfCzH0PeehhE=w@mail.gmail.com>
<CAMsr+YEfTEp35bQ5ia__nDwc2MwzYUQcuQ__vT_b=UNOHyooMA@mail.gmail.com>
<CA+TgmoaqJJsPNb7JDAHTtGZbH7CSenuf=0trQ_j1+_scLE-c0w@mail.gmail.com>
List-Unsubscribe: <mailto:[email protected]?body=unsub%20pgsql-hackers>
On 10 December 2015 at 03:19, Robert Haas <[email protected]> wrote:
> On Sun, Dec 6, 2015 at 10:24 PM, Craig Ringer <[email protected]>
> wrote:
>
> > * A way to securely make a libpq connection from a bgworker without
> messing
> > with passwords etc. Generate one-time cookies, sometihng like that.
>
> Why would you have the bgworker connect to the database via TCP
> instead of just doing whatever it wants to do directly?
pg_dump and pg_restore, mainly, for copying the initial database state.
PostgreSQL doesn't have SQL-level function equivalents, nor
pg_get_tabledef() etc, and there's been strong opposition to adding
anything of the sort when it's been raised before. We could read a dump in
via pg_restore's text conversion and run the appropriate queries over the
SPI, doing the query splitting, COPY parsing and loading, etc ourselves in
a bgworker. It'd be ugly and duplicate a lot, but it'd work. However, it
wouldn't be possible to do restores in parallel that way, and that's
necessary to get good restore performance on big DBs. For that we'd also
basically rewrite pg_restore's parallel functionality using a bgworker
pool.
The alternative is a massive rewrite of pg_dump and pg_restore to allow
them to be used as libraries, and let them use either libpq or the SPI for
queries, presumably via some level of abstraction layer. As well as further
abtraction for pipelining parallel work. Not very practical, and IIRC
whenever library-ifing pg_dump and pg_restore has been discussed before
it's been pretty firmly rejected.
Also, parallelism at apply time. There are two ways to do apply work in
parallel - a pool of bgworkers that each use the SPI, or using regular
backends managing async libpq connections. At this point I think
Konstantin's approach, with a bgworker pool that processes a work queue, is
probably better for this, and want to explore making that a re-usable
extension for 9.5 and possibly a core part of 9.6 or 9.7.
So it's mainly for pg_restore.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
view thread (38+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: Logical replication and multimaster
In-Reply-To: <CAMsr+YFf4hA=hLTe2hvPZ95-Fxf3bT80-_8CWwrDpDzva3LTOw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox