pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Mark Dilger <mark.dilger@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Geoghegan <pg@bowt.ie>
Cc: Andrey M. Borodin <x4mmm@yandex-team.ru>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Amul Sul <sulamul@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: new heapcheck contrib module
Date: Mon, 1 Mar 2021 13:57:16 -0800
Message-ID: <AFC09B65-A354-4A43-B1D4-2CC313BF3CCD@enterprisedb.com> (raw )
In-Reply-To: <CA+TgmobMWqHV092t5Jezdse_dEaQ4x4oJ3h5KzixV0nUYRGQnw@mail.gmail.com >
References: <12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com >
<20200420194210.sevnpi4lyghoanxi@alap3.anarazel.de >
<23E60C3C-C20A-4508-84AC-E0CEA59E784A@enterprisedb.com >
<CA+TgmobAm8vB1MNQ1taO1et+HxTN8ULnQvEG+grNDqNQTBjbtA@mail.gmail.com >
<F845AC50-A479-43F3-ADFB-861A4D2AD287@enterprisedb.com >
<5F743835-3399-419C-8324-2D424237E999@enterprisedb.com >
<CAFiTN-syyHc3jZoou51v0SR8z0POoNfktqEO6MaGig4YS8mosA@mail.gmail.com >
<E6F4258A-B661-4F2C-8A35-278C9436930A@enterprisedb.com >
<CAFiTN-s-VX3DLfqv7OOo0zGCTOZ3xVU9UZ6GgzBZdJTPNnYFNg@mail.gmail.com >
<885D2EC9-7D06-4903-A17D-72C184A1E1C5@enterprisedb.com >
<CAFiTN-uYg_PiZ854=5R-D9skratapAHKhaDAG7fhJv-Vn5UhaQ@mail.gmail.com >
<5CC85F28-F210-47AC-8DA8-EC4E82AF3FD5@enterprisedb.com >
<CAFiTN-sP5iFJGERTq4a4OLTCzjZKhLC5P6_k2bxQKXnuvdmzGA@mail.gmail.com >
<CAFiTN-tTg0R5AHvXC1ro9XphroX1wUtc5t91hU_+7DeGTpE5VQ@mail.gmail.com >
<0C0349EB-DD85-4DA8-82C7-DF70AB714592@enterprisedb.com >
<BA6E870F-241C-4DDB-B92D-DB39E679C7D4@enterprisedb.com >
<C4A3E5AE-CBB2-4E3F-8BAE-29EE216762C2@enterprisedb.com >
<CAAJ_b96_Xr2a+D0j8xKbOnCzURcVQm0V2DS+T_pfc+wt3BpL8g@mail.gmail.com >
<9DF93B83-FE46-4EC4-A99D-77A60729FD2D@enterprisedb.com >
<CA+TgmoZT3kCOaKq6=m5wHGKh_9XQDbCC1LZXL6w_+NMB-n1+mw@mail.gmail.com >
<CAAJ_b94hsrF4x7UKQj03416fAEXJorUpZ9WhsZZjqgv-4+Wg9A@mail.gmail.com >
<50F6468F-5A5E-412A-86EC-DF24B549D733@enterprisedb.com >
<CAAJ_b94+A1Gk5r5sBYbYM2K5hH_gyp_o=+gwJuJhn5ZEm5_rqw@mail.gmail.com >
<CAAJ_b94Wh_7kM9JuJk7wnNsYSNPdSzz35cOEaE_ZRnPcqvupCg@mail.gmail.com >
<7D603B53-8078-4AF0-851B-803FF4416501@enterprisedb.com >
<CA+TgmoZbPtTgkQBGoco39uqKUWuv=0+TOkiNcCuqWWvUkJVuxQ@mail.gmail.com >
<A4C9E0AE-46BA-457F-A445-4527E5AA5E3C@enterprisedb.com >
<98265AA9-7BDC-4708-9348-CC1B173BBE1D@enterprisedb.com >
<CA+Tgmoam+pnGKhWS+RAYCk1eqjhbAPX7Vm69g1VF56xwyxXw+g@mail.gmail.com >
<209783.1603313248@sss.pgh.pa.us >
<77A60E99-6054-40FD-A5D6-769C287D4A0F@enterprisedb.com >
<CA+TgmobGxdm8Tsxawt7LW-qigHRHxD+b17PC+ft9EE+6pMhC9w@mail.gmail.com >
<721BFA83-EA6F-463D-A9DF-CE78A318B73A@enterprisedb.com >
<2A7DA1A8-C4AA-43DF-A985-3CA52F4DC775@enterprisedb.com >
<146814.1603463293@sss.pgh.pa.us >
<221029.1603476265@sss.pgh.pa.us >
<CA+TgmoZsuhZroXdwCNUb5+9QEWSFQMH9=aA1hx-SDeArppH76Q@mail.gmail.com >
<C001DE91-CF4A-44A7-85B6-8BBAB1881196@enterprisedb.com >
<CA+TgmoZJ53=joSyJuR7WmXEOfhbBK+0hQBrOO4mWftwekaHS6w@mail.gmail.com >
<153EEBAD-2673-4367-B36D-4A6AE8EE7A9F@enterprisedb.com >
<85BE72E5-A926-4F23-9FD5-64239C0C6BA5@enterprisedb.com >
<CA+Tgmoa7sNcw15=soG1pLBgByMnn-w70YdfKZmhmYOb43KzzRw@mail.gmail.com >
<B5CF5BFF-0A92-4792-868E-3DB4E8F1C389@enterprisedb.com >
<10677BFE-3B45-4EF3-82E5-E45CA1445CA6@enterprisedb.com >
<CA+hUKG+mX5sb6=ZG9=zHaYofPeLOxO910XHkahcDgP-XzEMLxQ@mail.gmail.com >
<A4C2DFB8-436F-4802-BE8D-7A30290D52FA@enterprisedb.com >
<CA+TgmoY=+bKef-MUb1nKnyL8nV_26cn7NJ8q50AkGg0fV_v_XA@mail.gmail.com >
<411C72E5-23A6-485F-A101-FBF7CFA6D2D5@enterprisedb.com >
<CA+Tgmoa_SfoePsmXfj9FkDFqOCdJyb8XpXUzDo7GahuyPrPnpA@mail.gmail.com >
<70A7BABE-EDC1-4217-8FEB-F446F2938CE1@enterprisedb.com >
<70655DF3-33CE-4527-9A4D-DDEB582B6BA0@enterprisedb.com >
<CA+TgmoZTWOFREQQhE4r4_AJnNorpNq5hZZvXWbNShtAaA3RE9w@mail.gmail.com >
<2E0C3552-DAFC-451E-AB0C-FFE184D11DC5@enterprisedb.com >
<CA+Tgmobp_aurqbXAUM_Da1Srwm6bc0KoYGhqc=0VrCVw9zfZPQ@mail.gmail.com >
<FDA6D0A6-F652-4E76-82B3-EEB7E5ED8625@enterprisedb.com >
<CA+Tgmob2c0eM8+5kzkXaqdc9XbBCkHmtihSOSk-jCzzT4BFhFQ@mail.gmail.com >
<C67CBC40-512A-4CB7-8AF1-8810C6615238@enterprisedb.com >
<CA+TgmoZ4rr+aHgAUrC8wXXkgnoFtc6h0p2v5-d-JyhR++gxjGA@mail.gmail.com >
<B36B01A3-1958-495B-BA05-167046B4A773@enterprisedb.com >
<CA+TgmobMWqHV092t5Jezdse_dEaQ4x4oJ3h5KzixV0nUYRGQnw@mail.gmail.com >
> On Mar 1, 2021, at 1:14 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Wed, Feb 24, 2021 at 1:55 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> [ new patches ]
>
> Regarding 0001:
>
> There seem to be whitespace-only changes to the comment for select_loop().
>
> I wonder if the ParallelSlotsSetupOneDB()/ParallelSlotsSetupMinimal()
> changes could be simpler. First idea: Suppose you had
> ParallelSlotsSetup(numslots) that just creates the slot array with 0
> connections, and then ParallelSlotsAdopt(slots, conn, cparams) if you
> want to make it own an existing connection. That seems like it might
> be cleaner. Second idea: Why not get rid of ParallelSlotsSetupOneDB()
> altogether, and just let ParallelSlotsGetIdle() connect the other
> slots as required? Preconnecting all slots before we do anything is
> good because ... of what?
Mostly because, if --jobs is set too high, you get an error before launching any work. I don't know that it's really a big deal if vacuumdb or reindexdb have a bunch of tasks kicked off prior to exit(1) due to not being able to open connections for all the slots, but it is a behavioral change.
> I also wonder if things might be simplified by introducing a wrapper
> object, e.g. ParallelSlotArray. Suppose a ParallelSlotArray stores the
> number of slots (num_slots), the array of actual PGconn objects, and
> the ConnParams to be used for new connections, and the initcmd to be
> used for new connections. Maybe also the progname. This seems like it
> would avoid a bunch of repeated parameter passing: you could just
> create the ParallelSlotArray with the right contents, and then pass it
> around everywhere, instead of having to keep passing the same stuff
> in. If you want to switch to connecting to a different DB, you tweak
> the ConnParams - maybe using an accessor function - and the system
> figures the rest out.
The existing version of parallel slots (before any of my changes) could already have been written that way, but the author chose not to. I thought about making the sort of change you suggest, and decided against, mostly on the principle of stare decisis. But the idea is very appealing, and since you're on board, I think I'll go make that change.
> I wonder if it's really useful to generalize this to a point of caring
> about all the ConnParams fields, too. Like, if you only provide
> ParallelSlotUpdateDB(slotarray, dbname), then that's the only field
> that can change so you don't need to care about the others. And maybe
> you also don't really need to keep the ConnParams fields in every
> slot, either. Like, couldn't you just say something like: if
> (strcmp(PQdb(conn) , slotarray->cparams->dbname) != 0) { wrong DB,
> can't reuse without a reconnect }? I know sometimes a dbname is really
> a whole connection string, but perhaps we could try to fix that by
> using PQconninfoParse() in the right place, so that what ends up in
> the cparams is just the db name, not a whole connection string.
I went a little out of my way to avoid that, as I didn't want the next application that uses parallel slots to have to refactor it again, if for example they want to process in parallel databases listening on different ports, or to process commands issued under different roles.
> This is just based on a relatively short amount of time spent studying
> the patch, so I might well be off-base here. What do you think?
I like the ParallelSlotArray idea, and will go do that now. I'm happy to defer to your judgement on the other stuff, too, but will wait to hear back from you.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
view thread (218+ messages) latest in thread
Message-ID: <AFC09B65-A354-4A43-B1D4-2CC313BF3CCD@enterprisedb.com>
Permalink: ../AFC09B65-A354-4A43-B1D4-2CC313BF3CCD@enterprisedb.com/
Also on: postgresql.org/message-id/AFC09B65-A354-4A43-B1D4-2CC313BF3CCD@enterprisedb.com
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: mark.dilger@enterprisedb.com, robertmhaas@gmail.com, thomas.munro@gmail.com, tgl@sss.pgh.pa.us, pg@bowt.ie, x4mmm@yandex-team.ru, sfrost@snowman.net, michael@paquier.xyz, sulamul@gmail.com, dilipbalaut@gmail.com, andres@anarazel.de
Subject: Re: new heapcheck contrib module
In-Reply-To: <AFC09B65-A354-4A43-B1D4-2CC313BF3CCD@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox