pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Adrian Klaver <adrian.klaver@aklaver.com>
To: Emiel Mols <emiel@crisp.nl>
To: daniel@yesql.se
Cc: Pavel Stehule <pavel.stehule@gmail.com>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Fastest way to clone schema ~1000x
Date: Mon, 26 Feb 2024 08:10:06 -0800
Message-ID: <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com> (raw)
In-Reply-To: <CAF5w505X-OVN_EW9nsOYjBPLQ1auAdcDLKzreCeY5TO_YJEAtA@mail.gmail.com>
References: <CAF5w505Cc_Ft6AWSNnqAfF5=2erkf4ubRtYewKhq2-XHUBYjvQ@mail.gmail.com>
	<CAFj8pRC8rGpVOZYkcsv7cx6i4MmqiD5eGF=GdCKRODKXjVSb4Q@mail.gmail.com>
	<CAF5w5042yWdX=CAVkbmkbWzN_WWtCZRLyUdRCwwvWgPh6udwgA@mail.gmail.com>
	<CAFj8pRAHTkG2F7qs1HD-rSn4JnJ4rC2x_rx3L8xfok58NoPq9w@mail.gmail.com>
	<CAF5w506QcmmE70Nctp+=JS4VP_DJT4es9k7NpCqy3kRbu0KKqA@mail.gmail.com>
	<D07A9D78-E3BB-408C-AB87-E41314D52029@yesql.se>
	<CAF5w505X-OVN_EW9nsOYjBPLQ1auAdcDLKzreCeY5TO_YJEAtA@mail.gmail.com>

On 2/26/24 01:06, Emiel Mols wrote:
> On Mon, Feb 26, 2024 at 3:50 PM Daniel Gustafsson <daniel@yesql.se 
> <mailto:daniel@yesql.se>> wrote:
> 
>     There is a measurable overhead in connections, regardless of if they
>     are used
>     or not.  If you are looking to squeeze out performance then doing
>     more over
>     already established connections, and reducing max_connections, is a
>     good place
>     to start.
> 
> 
> Clear, but with database-per-test (and our backend setup), it would have 
> been *great* if we could have switched database on the same connection 
> (similar to "USE xxx" in mysql). That would limit the connections to the 
> amount of workers, not multiplied by tests.

That is because:

https://dev.mysql.com/doc/refman/8.3/en/glossary.html#glos_schema

"In MySQL, physically, a schema is synonymous with a database. You can 
substitute the keyword SCHEMA instead of DATABASE in MySQL SQL syntax, 
for example using CREATE SCHEMA instead of CREATE DATABASE. "


> 
> Even with a pooler, we're still going to be maintaining 1000s of 
> connections from the backend workers to the pooler. I would expect this 
> to be rather efficient, but still unnecessary. Also, both 
> pgbouncer/pgpool don't seem to support switching database in-connection 
> (they could have implemented the aforementioned "USE" statement I 
> think!). [Additionally we're using PHP that doesn't seem to have a good 
> shared memory pool implementation -- pg_pconnect is pretty buggy].
> 
> I'll continue with some more testing. Thanks for now!

-- 
Adrian Klaver
adrian.klaver@aklaver.com






view thread (9+ messages)  latest in thread

Message-ID: <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com>
Permalink:  ../3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com/
Also on:    postgresql.org/message-id/3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: adrian.klaver@aklaver.com, emiel@crisp.nl, daniel@yesql.se, pavel.stehule@gmail.com
  Subject: Re: Fastest way to clone schema ~1000x
  In-Reply-To: <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com>
  References: <CAF5w505Cc_Ft6AWSNnqAfF5=2erkf4ubRtYewKhq2-XHUBYjvQ@mail.gmail.com> <CAFj8pRC8rGpVOZYkcsv7cx6i4MmqiD5eGF=GdCKRODKXjVSb4Q@mail.gmail.com> <CAF5w5042yWdX=CAVkbmkbWzN_WWtCZRLyUdRCwwvWgPh6udwgA@mail.gmail.com> <CAFj8pRAHTkG2F7qs1HD-rSn4JnJ4rC2x_rx3L8xfok58NoPq9w@mail.gmail.com> <CAF5w506QcmmE70Nctp+=JS4VP_DJT4es9k7NpCqy3kRbu0KKqA@mail.gmail.com> <D07A9D78-E3BB-408C-AB87-E41314D52029@yesql.se> <CAF5w505X-OVN_EW9nsOYjBPLQ1auAdcDLKzreCeY5TO_YJEAtA@mail.gmail.com> <3064222d-bcf2-4eaf-ad9a-357a61c4348a@aklaver.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox