From: Adrian Klaver <adrian.klaver@aklaver.com>
To: Emiel Mols <emiel@crisp.nl>
To: daniel@yesql.se
Cc: Pavel Stehule <pavel.stehule@gmail.com>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Fastest way to clone schema ~1000x
Date: Mon, 26 Feb 2024 08:10:06 -0800
Message-ID: <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com> (raw)
In-Reply-To: <CAF5w505X-OVN_EW9nsOYjBPLQ1auAdcDLKzreCeY5TO_YJEAtA@mail.gmail.com>
References: <CAF5w505Cc_Ft6AWSNnqAfF5=2erkf4ubRtYewKhq2-XHUBYjvQ@mail.gmail.com>
<CAFj8pRC8rGpVOZYkcsv7cx6i4MmqiD5eGF=GdCKRODKXjVSb4Q@mail.gmail.com>
<CAF5w5042yWdX=CAVkbmkbWzN_WWtCZRLyUdRCwwvWgPh6udwgA@mail.gmail.com>
<CAFj8pRAHTkG2F7qs1HD-rSn4JnJ4rC2x_rx3L8xfok58NoPq9w@mail.gmail.com>
<CAF5w506QcmmE70Nctp+=JS4VP_DJT4es9k7NpCqy3kRbu0KKqA@mail.gmail.com>
<D07A9D78-E3BB-408C-AB87-E41314D52029@yesql.se>
<CAF5w505X-OVN_EW9nsOYjBPLQ1auAdcDLKzreCeY5TO_YJEAtA@mail.gmail.com>
On 2/26/24 01:06, Emiel Mols wrote:
> On Mon, Feb 26, 2024 at 3:50 PM Daniel Gustafsson <daniel@yesql.se
> <mailto:daniel@yesql.se>> wrote:
>
> There is a measurable overhead in connections, regardless of if they
> are used
> or not. If you are looking to squeeze out performance then doing
> more over
> already established connections, and reducing max_connections, is a
> good place
> to start.
>
>
> Clear, but with database-per-test (and our backend setup), it would have
> been *great* if we could have switched database on the same connection
> (similar to "USE xxx" in mysql). That would limit the connections to the
> amount of workers, not multiplied by tests.
That is because:
https://dev.mysql.com/doc/refman/8.3/en/glossary.html#glos_schema
"In MySQL, physically, a schema is synonymous with a database. You can
substitute the keyword SCHEMA instead of DATABASE in MySQL SQL syntax,
for example using CREATE SCHEMA instead of CREATE DATABASE. "
>
> Even with a pooler, we're still going to be maintaining 1000s of
> connections from the backend workers to the pooler. I would expect this
> to be rather efficient, but still unnecessary. Also, both
> pgbouncer/pgpool don't seem to support switching database in-connection
> (they could have implemented the aforementioned "USE" statement I
> think!). [Additionally we're using PHP that doesn't seem to have a good
> shared memory pool implementation -- pg_pconnect is pretty buggy].
>
> I'll continue with some more testing. Thanks for now!
--
Adrian Klaver
adrian.klaver@aklaver.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: adrian.klaver@aklaver.com, emiel@crisp.nl, daniel@yesql.se, pavel.stehule@gmail.com
Subject: Re: Fastest way to clone schema ~1000x
In-Reply-To: <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com>
References: <CAF5w505Cc_Ft6AWSNnqAfF5=2erkf4ubRtYewKhq2-XHUBYjvQ@mail.gmail.com> <CAFj8pRC8rGpVOZYkcsv7cx6i4MmqiD5eGF=GdCKRODKXjVSb4Q@mail.gmail.com> <CAF5w5042yWdX=CAVkbmkbWzN_WWtCZRLyUdRCwwvWgPh6udwgA@mail.gmail.com> <CAFj8pRAHTkG2F7qs1HD-rSn4JnJ4rC2x_rx3L8xfok58NoPq9w@mail.gmail.com> <CAF5w506QcmmE70Nctp+=JS4VP_DJT4es9k7NpCqy3kRbu0KKqA@mail.gmail.com> <D07A9D78-E3BB-408C-AB87-E41314D52029@yesql.se> <CAF5w505X-OVN_EW9nsOYjBPLQ1auAdcDLKzreCeY5TO_YJEAtA@mail.gmail.com> <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox