Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1itMLr-000155-8C for pgsql-bugs@arkaria.postgresql.org; Mon, 20 Jan 2020 01:59:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1itMLo-0005aH-70 for pgsql-bugs@arkaria.postgresql.org; Mon, 20 Jan 2020 01:59:36 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1itMLm-0005aA-BT for pgsql-bugs@lists.postgresql.org; Mon, 20 Jan 2020 01:59:35 +0000 Received: from [66.111.4.25] (helo=out1-smtp.messagingengine.com) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1itMLh-0003WE-V0 for pgsql-bugs@lists.postgresql.org; Mon, 20 Jan 2020 01:59:32 +0000 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 7EBFF21C7A; Sun, 19 Jan 2020 20:59:18 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sun, 19 Jan 2020 20:59:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm3; bh=HQgBrX+RBOtqX7kdM9nXpNhLZWa AQ1IXqmng5spV25U=; b=Tw+lhkx1G1GtubswCwtycTPAvkHB7glyPCIlVu/RnDI TUD6zn45xujANK8ps4sAGBK7bFaKZjs82i6NaMSmC0EdFitC3rQ+/2yXH/boHARl FXhy4Kv2eCvMuKi26u9tU80Woy8+7IUq2+tcs52BDxv7xNEW5KEDV63G0ZOx4gTL lcX8W9qE58wsfFetECtDHVp2mCJaCMV2CVERtEtwl3nzERJCunWM4WS9mmyNmqiM XvPVsQeqxVklAqQ3VMihNcEX6HKSIf/wc3P9SuPkKajTTsEXbKIlXG5kdxNfK6Ce /WmSxDxb5k2ulY7jvw3cFSzqK/1hPd5pUeuG7Pisq9A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=HQgBrX +RBOtqX7kdM9nXpNhLZWaAQ1IXqmng5spV25U=; b=rfg7lP4GDifpUzXns3au4j d75FTqp+SVeqbwEZiZdhY+zvPkpFU06qryVDCsuMqYXcHGnk7k0Ql4dzsIZw7Y0H GmIHXXybLcBksVRxzg23Q+3+74E2bocvaz86sjx4zr559f0dD8XwS7wi4YtAk2ub HOnJ1lwMTahHXqVl07ql6edZ0gSGhepY+QRUZj1+Le31bltr1uN5oRGAIA34A3+R oM3SNHF+qUel0FDBhL8oReEfmdr2WwiYLPe9EcSFR4x3/nPxQVSjuaAO9NV/9QUV w9Q/1ujOsSkg+41Ax5jEXxpRP+cjmaLzcUVR9IjHGfd3j1au2vfsMGM6Rx0qxzWg == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedrudeggdeflecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenfg hrlhcuvffnffculdejtddmnecujfgurhepfffhvffukfhfgggtuggjsehgtderredttddv necuhfhrohhmpefoihgthhgrvghlucfrrghquhhivghruceomhhitghhrggvlhesphgrqh huihgvrhdrgiihiieqnecukfhppeduuddurddutddvrddukedtrddukeehnecurfgrrhgr mhepmhgrihhlfhhrohhmpehmihgthhgrvghlsehprghquhhivghrrdighiiinecuvehluh hsthgvrhfuihiivgeptd X-ME-Proxy: Received: from paquier.xyz (ee0822lan1.rev.em-net.ne.jp [111.102.180.185]) by mail.messagingengine.com (Postfix) with ESMTPA id 5F0B13060B16; Sun, 19 Jan 2020 20:59:16 -0500 (EST) Date: Mon, 20 Jan 2020 10:59:13 +0900 From: Michael Paquier To: Heikki Linnakangas Cc: Andres Freund , Tom Lane , Manuel Rigger , PostgreSQL mailing lists Subject: Re: REINDEX CONCURRENTLY unexpectedly fails Message-ID: <20200120015913.GA1471@paquier.xyz> References: <20191120035408.GA4243@paquier.xyz> <20191212213709.neopqccvdo724eha@alap3.anarazel.de> <20191213034536.GD1942@paquier.xyz> <20191223015956.GB34339@paquier.xyz> <8f0bde7b-2d96-fb5c-cd0e-553694c1d376@iki.fi> <20200108081930.GF3413@paquier.xyz> <20200109030619.GC2251@paquier.xyz> <20200115013824.GA2243@paquier.xyz> <39c1e415-96e6-9b91-79a8-aa1cbd25e801@iki.fi> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="AqsLC8rIMeq19msA" Content-Disposition: inline In-Reply-To: <39c1e415-96e6-9b91-79a8-aa1cbd25e801@iki.fi> X-Host-Lookup-Failed: Reverse DNS lookup failed for 66.111.4.25 (deferred) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --AqsLC8rIMeq19msA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jan 17, 2020 at 02:53:11PM +0200, Heikki Linnakangas wrote: > On 15/01/2020 03:39, Michael Paquier wrote: >> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c >> index 3e59e647e5..4139232b51 100644 >> --- a/src/backend/catalog/index.c >> +++ b/src/backend/catalog/index.c >> @@ -2016,6 +2016,13 @@ index_drop(Oid indexId, bool concurrent, bool con= current_lock_mode) >> LOCKTAG heaplocktag; >> LOCKMODE lockmode; >> + /* >> + * A relation not supporting concurrent indexing should never do >> + * a concurrent index drop or try to use a concurrent lock mode. >> + */ >> + Assert(RelationSupportsConcurrentIndexing(indexId) || >> + (!concurrent && !concurrent_lock_mode)); >> + >> /* >> * To drop an index safely, we must grab exclusive lock on its parent >> * table. Exclusive lock on the index alone is insufficient because >=20 > Or maybe decide to do non-current drop within index_drop() itself, instead > of requiring the caller to set 'concurrent' differently for temporary > tables? A portion which stresses me with your approach (and that I actually used in the first versions of my patch upthread), is that we have some extra work related to PERFORM_DELETION_CONCURRENTLY being done in dependency.c, which becomes basically useless if you enforce the flag only in index_drop() without making sure that the root flag is set in RemoveRelations() (see for example the top of deleteOneObject()), and this causes the index drop to actually mix more concurrent and non-concurrent concepts than just the lock upgrade issue. >> +bool >> +RelationSupportsConcurrentIndexing(Oid relid) >> +{ >=20 > Sorry to beat a dead hore, but I still don't like this function: >=20 > * Does it take a table OID or index OID? (Answer: both) Yes, the persistency is inherited. The function mentioned a relation, so that applied to any relkind actually. Perhaps the function should have made that clearer with a assertion using a relkind check or such. But the original sounded pretty clear to me. > * There's a hidden assumption that if RelationSupportsConcurrentIndexing() > returns false, then it's OK to upgrade the lock. That's true today, but if > we added other conditions when RelationSupportsConcurrentIndexing() retur= ned > false, there would be trouble. It seems like a bad abstraction. > > This would be better if the function was renamed to something like "Is it= OK > to upgrade a CONCURRENTLY build to non-CONCURRENTLY?", but meh. I underst= and > that it's nice to have a place for this comment, so that it doesn't need = to > be repeated in so many places. But I feel that a little bit of repetition= is > better in this case. The reasoning isn't exactly the same for CREATE INDE= X, > DROP INDEX, and REINDEX anyway. Okay, I see your points. So I am fine with your line of arguments. >> + /* >> + * Enforce non-concurrent build if the relation does not support this >> + * option. Do this before any use of the concurrent option is done. >> + */ >> + if (!RelationSupportsConcurrentIndexing(relationId)) >> + stmt->concurrent =3D false; >> + >=20 > Is it OK to scribble on the original 'stmt' here? Doesn't seem kosher, > although it probably works fine in practice. The idea was to avoid any future issues if any code refactored in this area passes down IndexStmt to a subroutine and uses the concurrent flag. I guess that would be hard to miss if using a local variable as you do anyway.. > Do we care whether the *table* supports concurrent indexing, rather than > individual indexes? I guess that's academic, since you can't have tempora= ry > indexes on a permanent table, or vice versa. I cared about that enough, but that's a very defensive position :) But I agree that having a check only for individual indexes would be just but fine. >> + /* >> + * Also check for active uses of the relation in the current >> + * transaction, including open scans and pending AFTER trigger >> + * events. >> + */ >> + CheckTableNotInUse(indexRel, "REINDEX"); >> + >> pgstat_progress_start_command(PROGRESS_COMMAND_CREATE_INDEX, >=20 > I don't understand why this is required for this patch. It seems like a g= ood > thing to check, I think otherwise you get an error from the > CheckTableNotInUse() call in index_drop(), in phase 6 where the old index= es > are dropped. But it seems unrelated to the rest of the patch. Maybe commit > it as a separate patch? I added that per a suggestion from Andres while touching this area of the code. But you are right that it would make sense to remove it =66rom this patch, and get that committed separately. I don't mind starting a new thread for this matter instead of having this discussing buried within this thread. Does it make sense? > @@ -943,8 +962,11 @@ DefineIndex(Oid relationId, > /* > * A valid stmt->oldNode implies that we already have a built form of t= he > * index. The caller should also decline any index build. > + * > + * FIXME: should this check 'concurrent' or 'stmt->concurrent'? I don't > + * quite understand the conditions here. > */ > - Assert(!OidIsValid(stmt->oldNode) || (skip_build && !stmt->concurrent)); > + Assert(!OidIsValid(stmt->oldNode) || (skip_build && !concurrent)); [ .. thinks .. ] It seems to me that this should be using the local variable. SKIP_BUILD is used in some cases for ALTER TABLE, CIC and REINDEX CONCURRENTLY (for the creation of the duplicate index entry). oldNode gets used in ALTER TABLE when attempting to reuse an existing index when changing a column type for example. > - if (concurrent) > + /* > + * Obtain the current persistence of the existing table. We already ho= ld > + * lock on it. > + */ > + heapRel =3D table_open(heapOid, NoLock); > + persistence =3D heapRel->rd_rel->relpersistence; > + table_close(heapRel, NoLock); Why not just using get_rel_persistence() here, as done in ReindexMultipleTables()? > + /* This function shouldn't be called for temporary relations. */ > + if (indexRel->rd_rel->relpersistence =3D=3D RELPERSISTENCE_TEMP) > + elog(ERROR, "cannot reindex a temporary table concurrently"); You are right that an elog() is better than an assertion here as this uses a catalog data. > I came up with the attached version. It seems a bit more clear to me. I'm > not 100% wedded to this, though, so if you want to proceed based on your > version instead, feel free. The docs and the tests are unchanged. Except for the part with index_drop() where I would rather do the decision-making for the concurrent behavior in RemoveRelations()=20 rather than index_drop() (as I did in v4), what you have here looks fine to me. Would you prefer wrapping up this stuff yourself or should I? This needs a backpatch down to 9.4 for the CIC/DIC part. -- Michael --AqsLC8rIMeq19msA Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAl4lCXEACgkQnvQgOdby QH2aqQ/+Ir4EfVZQHAwGKReOWyYY//mAPHA4eHWk978ynp3HlYefiLgOW/Fajfh2 HAQecUcc5EWUn/y3ca3QFtaR3fwx/3CLCZondR8RT57GTyRVhtmdmXwjIceTaUGq 032jqYojNVBNOBCRgnFF3I+J2eu8MSHxs91PTBZuxa5BYYmYvkoag/ZTHSZ6RvUz tkQU73aGTiprV6M8y6sMqulysZL9BHeP8JKYiw25xYbThBEPrVrRiyzuLrlQg3Hr Z2Fgj+Yr/Kowv7+QGynuAaOW365B8qSMWDmDQbXxXDx+bo7Yi8uYEfbUS6EFF6Yz NIOP3kOGmbkBt1KKdH3oZ79oIYxEQ8QL0tcuhWRCRlxl2JaTw/lm6FIStwpwf1k2 uvNdA4MWRKcu+4QlcLJgRHALnMcjGafvxpqUqDWSG9xUCx6EzMQaDOXDhqQ90+Kd 08OoW3bYHECegEaC4MfP9mg1SfHfomh5abxoum9omi2ZvZxbgC9XcTUUvYdnFyi7 7mtY7My7TIoU0L5xy004fvRgoT9BuinfEBw3HaLarXmIbmjzVngF8WJytPRK413d I+rpPL/MNIDX7InUGsJU0kcPy77Mjg0s8T14ZHybc9w7y+Xv00ZgjvjfD2mfZAX9 4GLvBGOnigprmp6IhGMdXWqt84Wv7lzUnlFHeQWruHpW0LfrB7A= =tjgL -----END PGP SIGNATURE----- --AqsLC8rIMeq19msA--