public inbox for [email protected]help / color / mirror / Atom feed
preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) 78+ messages / 13 participants [nested] [flat]
* preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 15:56 Robert Haas <[email protected]> 0 siblings, 3 replies; 78+ messages in thread From: Robert Haas @ 2021-08-17 15:56 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Wed, Aug 11, 2021 at 3:41 AM Shruthi Gowda <[email protected]> wrote: > I have fixed all the issues and now the patch is working as expected. Hi, I'm changing the subject line since the patch does something which was discussed on that thread but isn't really related to the old email subject. In general, I think this patch is uncontroversial and in reasonably good shape. However, there's one part that I'm not too sure about. If Tom Lane happens to be paying attention to this thread, I think his feedback would be particularly useful, since he knows a lot about the inner workings of pg_dump. Opinions from anybody else would be great, too. Anyway, here's the hunk that worries me: + + /* + * Need a separate entry, otherwise the command will be run in the + * same transaction as the CREATE DATABASE command, which is not + * allowed. + */ + ArchiveEntry(fout, + dbCatId, /* catalog ID */ + dbDumpId, /* dump ID */ + ARCHIVE_OPTS(.tag = datname, + .owner = dba, + .description = "SET_DB_OID", + .section = SECTION_PRE_DATA, + .createStmt = setDBIdQry->data, + .dropStmt = NULL)); + To me, adding a separate TOC entry for a thing that is not really a separate object seems like a scary hack that might come back to bite us. Unfortunately, I don't know enough about pg_dump to say exactly how it might come back to bite us, which leaves wide open the possibility that I am completely wrong.... I just think it's the intention that archive entries correspond to actual objects in the database, not commands that we want executed in some particular order. If that criticism is indeed correct, then my proposal would be to instead add a WITH OID = nnn option to CREATE DATABASE and allow it to be used only in binary upgrade mode. That has the disadvantage of being inconsistent with the way that we preserve OIDs everywhere else, but the only other alternatives are (1) do something like the above, (2) remove the requirement that CREATE DATABASE run in its own transaction, and (3) give up. (2) sounds hard and (3) is unappealing. The rest of this email will be detailed review comments on the patch as presented, and thus probably only interesting to someone actually working on the patch. Feel free to skip if that's not you. - I suggest splitting the patch into one portion that deals with database OID and another portion that deals with tablespace OID and relfilenode OID, or maybe splitting it all the way into three separate patches, one for each. This could allow the uncontroversial parts to get committed first while we're wondering what to do about the problem described above. - There are two places in the patch, one in dumpDatabase() and one in generate_old_dump() where blank lines are removed with no other changes. Please avoid whitespace-only hunks. - If possible, please try to pgindent the new code. It's pretty good what you did, but e.g. the declaration of binary_upgrade_next_pg_tablespace_oid probably has less whitespace than pgindent is going to want. - The comments in dumpDatabase() claim that "postgres" and "template1" are handled specially in some way, but there seems to be no code that matches those comments. - heap_create()'s logic around setting create_storage looks slightly redundant. I'm not positive what would be better, but ... suppose you just took the part that's currently gated by if (!IsBinaryUpgrade) and did it unconditionally. Then put if (IsBinaryUpgrade) around the else clause, but delete the last bit from there that sets create_storage. Maybe we'd still want a comment saying that it's intentional that create_storage = true even though it will be overwritten later, but then, I think, we wouldn't need to set create_storage in two different places. Maybe I'm wrong. - If we're not going to do that, then I think you should swap the if and else clauses and reverse the sense of the test. In createdb(), CreateTableSpace(), and a bunch of existing places, we do if (IsBinaryUpgrade) { ... } else { ... } so I don't think it makes sense for this one to instead do if (!IsBinaryUpgrade) { ... } else { ... }. - I'm not sure that I'd bother renaming binary_upgrade_set_pg_class_oids_and_relfilenodes(). It's such a long name, and a relfilenode is kind of an OID, so the current name isn't even really wrong. I'd probably drop the header comment too, since it seems rather obvious. But both of these things are judgement calls. - Inside that function, there is a comment that says "Indexes cannot have toast tables, so we need not make this probe in the index code path." However, you have moved the code from someplace where it didn't happen for indexes to someplace where it happens for both tables and indexes. Therefore the comment, which was true when the code was where it was before, is now false. So you need to update it. - It is not clear to me why pg_upgrade's Makefile needs to be changed to include -DFRONTEND in CPPFLAGS. All of the .c files in this directory include postgres_fe.h rather than postgres.h, and that file has #define FRONTEND 1. Moreover, there are no actual code changes in this directory, so why should the Makefile need any change? - A couple of comment changes - and the commit message - mention data encryption, but that's not a feature that this patch implements, nor are we committed to adding it in the immediate future (or ever, really). So I think those places should be revised to say that we do this because we want the filenames to match between the old and new clusters, and leave the reasons why that might be a good thing up to the reader's imagination. Thanks, -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 16:12 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 1 reply; 78+ messages in thread From: Tom Lane @ 2021-08-17 16:12 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Robert Haas <[email protected]> writes: > To me, adding a separate TOC entry for a thing that is not really a > separate object seems like a scary hack that might come back to bite > us. Unfortunately, I don't know enough about pg_dump to say exactly > how it might come back to bite us, which leaves wide open the > possibility that I am completely wrong.... I just think it's the > intention that archive entries correspond to actual objects in the > database, not commands that we want executed in some particular order. I agree, this seems like a moderately bad idea. It could get broken either by executing only one of the TOC entries during restore, or by executing them in the wrong order. The latter possibility could be forestalled by adding a dependency, which I do not see this hunk doing, which is clearly a bug. The former possibility would require user intervention, so maybe it's in the category of "if you break this you get to keep both pieces". Still, it's ugly. > If that criticism is indeed correct, then my proposal would be to > instead add a WITH OID = nnn option to CREATE DATABASE and allow it to > be used only in binary upgrade mode. If it's not too complicated to implement, that seems like an OK idea from here. I don't have any great love for the way we handle OID preservation in binary upgrade mode, so not doing it exactly the same way for databases doesn't seem like a disadvantage. regards, tom lane ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 16:33 Stephen Frost <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-17 16:33 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Tom Lane ([email protected]) wrote: > Robert Haas <[email protected]> writes: > > To me, adding a separate TOC entry for a thing that is not really a > > separate object seems like a scary hack that might come back to bite > > us. Unfortunately, I don't know enough about pg_dump to say exactly > > how it might come back to bite us, which leaves wide open the > > possibility that I am completely wrong.... I just think it's the > > intention that archive entries correspond to actual objects in the > > database, not commands that we want executed in some particular order. > > I agree, this seems like a moderately bad idea. It could get broken > either by executing only one of the TOC entries during restore, or > by executing them in the wrong order. The latter possibility could > be forestalled by adding a dependency, which I do not see this hunk > doing, which is clearly a bug. The former possibility would require > user intervention, so maybe it's in the category of "if you break > this you get to keep both pieces". Still, it's ugly. Yeah, agreed. > > If that criticism is indeed correct, then my proposal would be to > > instead add a WITH OID = nnn option to CREATE DATABASE and allow it to > > be used only in binary upgrade mode. > > If it's not too complicated to implement, that seems like an OK idea > from here. I don't have any great love for the way we handle OID > preservation in binary upgrade mode, so not doing it exactly the same > way for databases doesn't seem like a disadvantage. Also agreed on this, though I wonder- do we actually need to explicitly make CREATE DATABASE q WITH OID = 1234; only work during binary upgrade mode in the backend? That strikes me as perhaps doing more work than we really need to while also preventing something that users might actually like to do. Either way, we'll need to check that the OID given to us can be used for the database, I'd think. Having pg_dump only include it in binary upgrade mode is fine though. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 16:42 Tom Lane <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Tom Lane @ 2021-08-17 16:42 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Stephen Frost <[email protected]> writes: > Also agreed on this, though I wonder- do we actually need to explicitly > make CREATE DATABASE q WITH OID = 1234; only work during binary upgrade > mode in the backend? That strikes me as perhaps doing more work than we > really need to while also preventing something that users might actually > like to do. There should be adequate defenses against a duplicate OID already, so +1 --- no reason to insist this only be used during binary upgrade. Actually though ... I've not read the patch, but what does it do about the fact that the postgres and template0 DBs do not have stable OIDs? I cannot imagine any way to force those to match across PG versions that would not be an unsustainable crock. regards, tom lane ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 17:37 Robert Haas <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Robert Haas @ 2021-08-17 17:37 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Tue, Aug 17, 2021 at 12:42 PM Tom Lane <[email protected]> wrote: > Actually though ... I've not read the patch, but what does it do about > the fact that the postgres and template0 DBs do not have stable OIDs? > I cannot imagine any way to force those to match across PG versions > that would not be an unsustainable crock. Well, it's interesting that you mention that, because there's a comment in the patch that probably has to do with this: + /* + * Make sure that pg_upgrade does not change database OID. Don't care + * about "postgres" database, backend will assign it fixed OID anyway. + * ("template1" has fixed OID too but the value 1 should not collide with + * any other OID so backend pays no attention to it.) + */ I wasn't able to properly understand that comment, and to be honest I'm not sure I precisely understand your concern either. I don't quite see why the template0 database matters. I think that database isn't going to be dumped, or restored, so as far as pg_upgrade is concerned it might as well not exist in either cluster, and I don't see why pg_upgrade can't therefore just ignore it completely. But template1 and postgres are another matter. If I understand correctly, those databases are going to be created in the new cluster by initdb, but then pg_upgrade is going to populate them with data - including relation files - from the old cluster. And, yeah, I don't see how we could make those database OIDs match, which is not great. To be honest, what I'd be inclined to do about that is just nail down those OIDs for future releases. In fact, I'd probably go so far as to hardcode that in such a way that even if you drop those databases and recreate them, they get recreated with the same hard-coded OID. Now that doesn't do anything to create stability when people upgrade from an old release to a current one, but I don't really see that as an enormous problem. The only hard requirement for this feature is if we use the database OID for some kind of encryption or integrity checking or checksum type feature. Then, you want to avoid having the database OID change when you upgrade, so that the encryption or integrity check or checksum in question does not have to be recomputed for every page as part of pg_upgrade. But, that only matters if you're going between two releases that support that feature, which will not be the case if you're upgrading from some old release. Apart from that kind of feature, it still seems like a nice-to-have to keep database OIDs the same, but if those cases end up as exceptions, oh well. Does that seem reasonable, or am I missing something big? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 17:54 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 2 replies; 78+ messages in thread From: Tom Lane @ 2021-08-17 17:54 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Robert Haas <[email protected]> writes: > I wasn't able to properly understand that comment, and to be honest > I'm not sure I precisely understand your concern either. I don't quite > see why the template0 database matters. I think that database isn't > going to be dumped, or restored, so as far as pg_upgrade is concerned > it might as well not exist in either cluster, and I don't see why > pg_upgrade can't therefore just ignore it completely. But template1 > and postgres are another matter. If I understand correctly, those > databases are going to be created in the new cluster by initdb, but > then pg_upgrade is going to populate them with data - including > relation files - from the old cluster. Right. If pg_upgrade explicitly ignores template0 then its OID need not be stable ... at least, not unless there's a chance it could conflict with some other database OID, which would become a live possibility if we let users get at "WITH OID = n". (Having said that, I'm not sure that pg_upgrade special-cases template0, or that it should do so.) > To be honest, what I'd be inclined to do about that is just nail down > those OIDs for future releases. Yeah, I was thinking along similar lines. > In fact, I'd probably go so far as to > hardcode that in such a way that even if you drop those databases and > recreate them, they get recreated with the same hard-coded OID. Less sure that this is a good idea, though. In particular, I do not think that you can make it work in the face of alter database template1 rename to oops; create database template1; > The only hard requirement for this feature is if we > use the database OID for some kind of encryption or integrity checking > or checksum type feature. It's fairly unclear to me why that is so important as to justify the amount of klugery that this line of thought seems to be bringing. regards, tom lane ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 17:57 Tom Lane <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Tom Lane @ 2021-08-17 17:57 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> I wrote: > Robert Haas <[email protected]> writes: >> The only hard requirement for this feature is if we >> use the database OID for some kind of encryption or integrity checking >> or checksum type feature. > It's fairly unclear to me why that is so important as to justify the > amount of klugery that this line of thought seems to be bringing. And, not to put too fine a point on it, how will you possibly do that without entirely breaking CREATE DATABASE? regards, tom lane ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 18:07 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Shruthi Gowda @ 2021-08-17 18:07 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Tue, Aug 17, 2021 at 11:07 PM Robert Haas <[email protected]> wrote: > > On Tue, Aug 17, 2021 at 12:42 PM Tom Lane <[email protected]> wrote: > > Actually though ... I've not read the patch, but what does it do about > > the fact that the postgres and template0 DBs do not have stable OIDs? > > I cannot imagine any way to force those to match across PG versions > > that would not be an unsustainable crock. > > Well, it's interesting that you mention that, because there's a > comment in the patch that probably has to do with this: > > + /* > + * Make sure that pg_upgrade does not change database OID. Don't care > + * about "postgres" database, backend will assign it fixed OID anyway. > + * ("template1" has fixed OID too but the value 1 should not collide with > + * any other OID so backend pays no attention to it.) > + */ > In the original patch the author intended to avoid dumping the postgres DB OID like below: + if (dopt->binary_upgrade && dbCatId.oid != PostgresDbOid) Since postgres OID is not hardcoded/fixed I removed the check. My bad I missed updating the comment section. Sorry for the confusion. Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 18:32 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-17 18:32 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 17, 2021 at 11:56:30AM -0400, Robert Haas wrote: > On Wed, Aug 11, 2021 at 3:41 AM Shruthi Gowda <[email protected]> wrote: > > I have fixed all the issues and now the patch is working as expected. > > Hi, > > I'm changing the subject line since the patch does something which was > discussed on that thread but isn't really related to the old email > subject. In general, I think this patch is uncontroversial and in > reasonably good shape. However, there's one part that I'm not too sure > about. If Tom Lane happens to be paying attention to this thread, I > think his feedback would be particularly useful, since he knows a lot > about the inner workings of pg_dump. Opinions from anybody else would > be great, too. Anyway, here's the hunk that worries me: What is the value of preserving db/ts/relfilenode OIDs? -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-17 18:50 Robert Haas <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-17 18:50 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Tue, Aug 17, 2021 at 1:54 PM Tom Lane <[email protected]> wrote: > Right. If pg_upgrade explicitly ignores template0 then its OID > need not be stable ... at least, not unless there's a chance it > could conflict with some other database OID, which would become > a live possibility if we let users get at "WITH OID = n". Well, that might be a good reason not to let them do that, then, at least for n<64k. > > In fact, I'd probably go so far as to > > hardcode that in such a way that even if you drop those databases and > > recreate them, they get recreated with the same hard-coded OID. > > Less sure that this is a good idea, though. In particular, I do not > think that you can make it work in the face of > alter database template1 rename to oops; > create database template1; That is a really good point. If we can't categorically force the OID of those databases to have a particular, fixed value, and based on this example that seems to be impossible, then there's always a possibility that we might find a value in the old cluster that doesn't happen to match what is present in the new cluster. Seen from that angle, the problem is really with databases that are pre-existent in the new cluster but whose contents still need to be dumped. Maybe we could (optionally? conditionally?) drop those databases from the new cluster and then recreate them with the OID that we want them to have. > > The only hard requirement for this feature is if we > > use the database OID for some kind of encryption or integrity checking > > or checksum type feature. > > It's fairly unclear to me why that is so important as to justify the > amount of klugery that this line of thought seems to be bringing. Well, I think it would make sense to figure out how small we can make the kludge first, and then decide whether it's larger than we can tolerate. From my point of view, I completely understand why people to whom those kinds of features are important want to include all the fields that make up a buffer tag in the checksum or other integrity check. Right now, if somebody copies a page from one place to another, or if the operating system fumbles things and switches some pages around, we have no direct way of detecting that anything bad has happened. This is not the only problem that would need to be solved in order to fix that, but it's one of them, and I don't particularly see why it's not a valid goal. It's not as if a 16-bit checksum that is computed in exactly the same way for every page in the cluster is such state-of-the-art technology that only fools question its surpassing excellence. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-20 17:36 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-08-20 17:36 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> > The rest of this email will be detailed review comments on the patch > as presented, and thus probably only interesting to someone actually > working on the patch. Feel free to skip if that's not you. > > - I suggest splitting the patch into one portion that deals with > database OID and another portion that deals with tablespace OID and > relfilenode OID, or maybe splitting it all the way into three separate > patches, one for each. This could allow the uncontroversial parts to > get committed first while we're wondering what to do about the problem > described above. Thanks Robert for your comments. I have split the patch into two portions. One that handles DB OID and the other that handles tablespace OID and relfilenode OID. > - There are two places in the patch, one in dumpDatabase() and one in > generate_old_dump() where blank lines are removed with no other > changes. Please avoid whitespace-only hunks. These changes are avoided. > - If possible, please try to pgindent the new code. It's pretty good > what you did, but e.g. the declaration of > binary_upgrade_next_pg_tablespace_oid probably has less whitespace > than pgindent is going to want. Taken care in latest patches > - The comments in dumpDatabase() claim that "postgres" and "template1" > are handled specially in some way, but there seems to be no code that > matches those comments. The comment is removed. > - heap_create()'s logic around setting create_storage looks slightly > redundant. I'm not positive what would be better, but ... suppose you > just took the part that's currently gated by if (!IsBinaryUpgrade) and > did it unconditionally. Then put if (IsBinaryUpgrade) around the else > clause, but delete the last bit from there that sets create_storage. > Maybe we'd still want a comment saying that it's intentional that > create_storage = true even though it will be overwritten later, but > then, I think, we wouldn't need to set create_storage in two different > places. Maybe I'm wrong. > > - If we're not going to do that, then I think you should swap the if > and else clauses and reverse the sense of the test. In createdb(), > CreateTableSpace(), and a bunch of existing places, we do if > (IsBinaryUpgrade) { ... } else { ... } so I don't think it makes sense > for this one to instead do if (!IsBinaryUpgrade) { ... } else { ... }. I have avoided the redundant code and removed the comment as it does not make sense now that we are setting the create_storage conditionally. (In the original patch, create_storage was set to TRUE by default for binary upgrade case which was wrong and was hitting assert in the following flow). > - I'm not sure that I'd bother renaming > binary_upgrade_set_pg_class_oids_and_relfilenodes(). It's such a long > name, and a relfilenode is kind of an OID, so the current name isn't > even really wrong. I'd probably drop the header comment too, since it > seems rather obvious. But both of these things are judgement calls. I agree. I have retained the old function name. > - Inside that function, there is a comment that says "Indexes cannot > have toast tables, so we need not make this probe in the index code > path." However, you have moved the code from someplace where it didn't > happen for indexes to someplace where it happens for both tables and > indexes. Therefore the comment, which was true when the code was where > it was before, is now false. So you need to update it. The comment is updated. > - It is not clear to me why pg_upgrade's Makefile needs to be changed > to include -DFRONTEND in CPPFLAGS. All of the .c files in this > directory include postgres_fe.h rather than postgres.h, and that file > has #define FRONTEND 1. Moreover, there are no actual code changes in > this directory, so why should the Makefile need any change? Makefile change is removed. > - A couple of comment changes - and the commit message - mention data > encryption, but that's not a feature that this patch implements, nor > are we committed to adding it in the immediate future (or ever, > really). So I think those places should be revised to say that we do > this because we want the filenames to match between the old and new > clusters, and leave the reasons why that might be a good thing up to > the reader's imagination. Taken care. Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v2-0001-Preserve-database-OIDs-in-pg_upgrade.patch (8.6K, ../../CAASxf_N2EsZk8wU7wgWBzXpRZMC5dNxsZN21O5m3R3XTb5P9OQ@mail.gmail.com/2-v2-0001-Preserve-database-OIDs-in-pg_upgrade.patch) download | inline diff: From 6af195ee95bbde8e75fd6bc81350ede33256d866 Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Fri, 20 Aug 2021 22:07:42 +0530 Subject: [PATCH v2] Preserve database OIDs in pg_upgrade The patch aims to preserve the database OIDs during binary upgrade so that the database OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- src/backend/commands/dbcommands.c | 22 +++++++++++++--- src/backend/utils/adt/pg_upgrade_support.c | 11 ++++++++ src/bin/pg_dump/pg_dump.c | 30 ++++++++++++++++++++++ src/bin/pg_upgrade/pg_upgrade.c | 3 +++ src/include/catalog/binary_upgrade.h | 2 ++ src/include/catalog/pg_proc.dat | 4 +++ .../spgist_name_ops/expected/spgist_name_ops.out | 6 +++-- 7 files changed, 72 insertions(+), 6 deletions(-) diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 029fab4..a1e4a83 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -92,6 +92,7 @@ static void remove_dbtablespaces(Oid db_id); static bool check_db_file_conflict(Oid db_id); static int errdetail_busy_db(int notherbackends, int npreparedxacts); +Oid binary_upgrade_next_pg_database_oid = InvalidOid; /* * CREATE DATABASE @@ -504,11 +505,24 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + if (IsBinaryUpgrade) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + if (!OidIsValid(binary_upgrade_next_pg_database_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_database OID value not set when in binary upgrade mode"))); + + dboid = binary_upgrade_next_pg_database_oid; + binary_upgrade_next_pg_database_oid = InvalidOid; + } + else + { + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7..c499434 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,17 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_database_oid(PG_FUNCTION_ARGS) +{ + Oid dboid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_database_oid = dboid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 90ac445..0a938a2 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -47,6 +47,7 @@ #include "catalog/pg_authid_d.h" #include "catalog/pg_cast_d.h" #include "catalog/pg_class_d.h" +#include "catalog/pg_database.h" #include "catalog/pg_default_acl_d.h" #include "catalog/pg_largeobject_d.h" #include "catalog/pg_largeobject_metadata_d.h" @@ -2781,6 +2782,7 @@ dumpDatabase(Archive *fout) PQExpBuffer dbQry = createPQExpBuffer(); PQExpBuffer delQry = createPQExpBuffer(); PQExpBuffer creaQry = createPQExpBuffer(); + PQExpBuffer setDBIdQry = createPQExpBuffer(); PQExpBuffer labelq = createPQExpBuffer(); PGconn *conn = GetConnection(fout); PGresult *res; @@ -2950,6 +2952,34 @@ dumpDatabase(Archive *fout) qdatname = pg_strdup(fmtId(datname)); + /* Make sure that pg_upgrade does not change database OID. */ + if (dopt->binary_upgrade) + { + appendPQExpBufferStr(setDBIdQry, "\n-- For binary upgrade, must preserve pg_database oid\n"); + appendPQExpBuffer(setDBIdQry, + "SELECT pg_catalog.binary_upgrade_set_next_pg_database_oid('%u'::pg_catalog.oid);\n", + dbCatId.oid); + + dbDumpId = createDumpId(); + + /* + * Need a separate entry, otherwise the command will be run in the + * same transaction as the CREATE DATABASE command, which is not + * allowed. + */ + ArchiveEntry(fout, + dbCatId, /* catalog ID */ + dbDumpId, /* dump ID */ + ARCHIVE_OPTS(.tag = datname, + .owner = dba, + .description = "SET_DB_OID", + .section = SECTION_PRE_DATA, + .createStmt = setDBIdQry->data, + .dropStmt = NULL)); + + destroyPQExpBuffer(setDBIdQry); + } + /* * Prepare the CREATE DATABASE command. We must specify encoding, locale, * and tablespace since those can't be altered later. Other DB properties diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd7..ad25710 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -22,6 +22,9 @@ * this, old/new pg_class.relfilenode values will not match if CLUSTER, * REINDEX, or VACUUM FULL have been performed in the old cluster. * + * We control assignment of pg_database.oid because we want the oid to match + * between the old and new cluster. + * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. * diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7..3809521 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,6 +14,8 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_database_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b603700..1d7ae19 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11036,6 +11036,10 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4548', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_database_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_database_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddce..1dcdad4 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -56,10 +56,11 @@ select * from t binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid + binary_upgrade_set_next_pg_database_oid | | binary_upgrade_set_next_pg_database_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) +(10 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -101,10 +102,11 @@ select * from t binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid + binary_upgrade_set_next_pg_database_oid | | binary_upgrade_set_next_pg_database_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) +(10 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 1.8.3.1 [application/octet-stream] v2-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch (20.6K, ../../CAASxf_N2EsZk8wU7wgWBzXpRZMC5dNxsZN21O5m3R3XTb5P9OQ@mail.gmail.com/3-v2-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch) download | inline diff: From 216bafaf9b0d081bdc670d64d162de75d5b9155c Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Fri, 20 Aug 2021 19:26:12 +0530 Subject: [PATCH v2] Preserve relfilenode and tablespace OID in pg_upgrade The patch aims to preserve the OIDs of relfilenode and tablespace during binary upgrade so that the OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- src/backend/catalog/heap.c | 36 +++++++++ src/backend/catalog/index.c | 1 + src/backend/commands/tablespace.c | 14 +++- src/backend/utils/adt/pg_upgrade_support.c | 44 ++++++++++ src/bin/pg_dump/pg_dump.c | 94 ++++++++++++++-------- src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/pg_upgrade.c | 13 +-- src/include/catalog/binary_upgrade.h | 5 ++ src/include/catalog/pg_proc.dat | 16 ++++ .../spgist_name_ops/expected/spgist_name_ops.out | 12 ++- 10 files changed, 195 insertions(+), 43 deletions(-) diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 83746d3..f3c99f6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -379,6 +381,40 @@ heap_create(const char *relname, relfilenode = relid; } + if (IsBinaryUpgrade) + { + /* Override relfilenode? */ + if (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || + relkind == RELKIND_MATVIEW) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + } + else if (relkind == RELKIND_TOASTVALUE) + { + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; + } + else if (relkind == RELKIND_INDEX) + { + if (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_class index relfilenode value not set when in binary upgrade mode"))); + relfilenode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; + } + } + /* * Never allow a pg_class entry to explicitly specify the database's * default tablespace in reltablespace; force it to zero instead. This diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 26bfa74..a124617 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -86,6 +86,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index a54239a..34e5746 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -88,6 +88,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -335,7 +336,18 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + if (IsBinaryUpgrade) + { + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7..d4dc284 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,17 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); @@ -85,6 +96,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_heap_pg_class_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -96,6 +118,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_index_pg_class_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -107,6 +140,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_toast_pg_class_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { Oid enumoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 90ac445..6ced774 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4700,48 +4700,63 @@ binary_upgrade_set_pg_class_oids(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index) { + PQExpBuffer upgrade_query = createPQExpBuffer(); + PGresult *upgrade_res; + Oid pg_class_relfilenode; + Oid pg_class_reltoastrelid; + Oid pg_class_relfilenode_toast; + char pg_class_relkind; + Oid pg_index_indexrelid; + Oid pg_class_relfilenode_toast_idx; + + /* + * Preserve the OIDs of the table's toast table and index, if any. + * + * One complexity is that the current table definition might not require + * the creation of a TOAST table, but the old database might have a TOAST + * table that was created earlier, before some wide columns were dropped. + * By setting the TOAST oid we force creation of the TOAST heap and index + * by the new backend, so we can copy the files during binary upgrade + * without worrying about this case. + */ + appendPQExpBuffer(upgrade_query, + "SELECT c.relfilenode, c.relkind, c.reltoastrelid, ct.relfilenode AS relfilenode_toast, i.indexrelid, cti.relfilenode AS relfilenode_toast_index " + "FROM pg_catalog.pg_class c LEFT JOIN " + "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " + "WHERE c.oid = '%u'::pg_catalog.oid;", + pg_class_oid); + + upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + + pg_class_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode"))); + pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "reltoastrelid"))); + pg_class_relfilenode_toast = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode_toast"))); + pg_class_relkind = *PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relkind")); + pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "indexrelid"))); + pg_class_relfilenode_toast_idx = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode_toast_index"))); + appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { - PQExpBuffer upgrade_query = createPQExpBuffer(); - PGresult *upgrade_res; - Oid pg_class_reltoastrelid; - char pg_class_relkind; - Oid pg_index_indexrelid; - appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); - /* - * Preserve the OIDs of the table's toast table and index, if any. - * Indexes cannot have toast tables, so we need not make this probe in - * the index code path. - * - * One complexity is that the current table definition might not - * require the creation of a TOAST table, but the old database might - * have a TOAST table that was created earlier, before some wide - * columns were dropped. By setting the TOAST oid we force creation - * of the TOAST heap and index by the new backend, so we can copy the - * files during binary upgrade without worrying about this case. - */ - appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, c.relkind, i.indexrelid " - "FROM pg_catalog.pg_class c LEFT JOIN " - "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " - "WHERE c.oid = '%u'::pg_catalog.oid;", - pg_class_oid); - - upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); - - pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "reltoastrelid"))); - pg_class_relkind = *PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "relkind")); - pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "indexrelid"))); + /* Not every relation has storage. */ + if (OidIsValid(pg_class_relfilenode)) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode); /* * In a pre-v12 database, partitioned tables might be marked as having @@ -4753,20 +4768,31 @@ binary_upgrade_set_pg_class_oids(Archive *fout, appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_reltoastrelid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode_toast); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_index_indexrelid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode_toast_idx); } PQclear(upgrade_res); destroyPQExpBuffer(upgrade_query); } else + { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); } diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c291017..618187c 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1265,6 +1265,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd7..acaf343 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,13 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because we want the filenames + * to match between the old and new cluster. + * + * We control assignment of pg_tablespace.oid because we want the oid to match + * between the old and new cluster. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7..4ba5748 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,19 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b603700..0626525 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11036,6 +11036,22 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4544', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_pg_class_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_pg_class_relfilenode' }, +{ oid => '4545', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_pg_class_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_pg_class_relfilenode' }, +{ oid => '4546', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_pg_class_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_pg_class_relfilenode' }, +{ oid => '4547', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddce..8541b4a 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -52,14 +52,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_pg_class_relfilenode | | binary_upgrade_set_next_heap_pg_class_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_pg_class_relfilenode | 1 | binary_upgrade_set_next_index_pg_class_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_pg_class_relfilenode | 1 | binary_upgrade_set_next_toast_pg_class_relfilenode +(13 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -97,14 +101,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_pg_class_relfilenode | | binary_upgrade_set_next_heap_pg_class_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_pg_class_relfilenode | 1 | binary_upgrade_set_next_index_pg_class_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_pg_class_relfilenode | 1 | binary_upgrade_set_next_toast_pg_class_relfilenode +(13 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 1.8.3.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-23 20:57 Robert Haas <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 3 replies; 78+ messages in thread From: Robert Haas @ 2021-08-23 20:57 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Fri, Aug 20, 2021 at 1:36 PM Shruthi Gowda <[email protected]> wrote: > Thanks Robert for your comments. > I have split the patch into two portions. One that handles DB OID and > the other that > handles tablespace OID and relfilenode OID. It's pretty clear from the discussion, I think, that the database OID one is going to need rework to be considered. Regarding the other one: - The comment in binary_upgrade_set_pg_class_oids() is still not accurate. You removed the sentence which says "Indexes cannot have toast tables, so we need not make this probe in the index code path" but the immediately preceding sentence is still inaccurate in at least two ways. First, it only talks about tables, but the code now applies to indexes. Second, it only talks about OIDs, but now also deals with refilenodes. It's really important to fully update every comment that might be affected by your changes! - The SQL query in that function isn't completely correct. There is a left join from pg_class to pg_index whose ON clause includes "c.reltoastrelid = i.indrelid AND i.indisvalid." The reason it's likely that is because it is possible, in corner cases, for a TOAST table to have multiple TOAST indexes. I forget exactly how that happens, but I think it might be like if a REINDEX CONCURRENTLY on the TOAST table fails midway through, or something of that sort. Now if that happens, the LEFT JOIN you added is going to cause the output to contain multiple rows, because you didn't replicate the i.indisvalid condition into that ON clause. And then it will fail. Apparently we don't have a pg_upgrade test case for this scenario; we probably should. Actually what I think would be even better than putting i.indisvalid into that ON clause would be to join off of i.indrelid rather than c.reltoastrelid. - The code that decodes the various columns of this query does so in a slightly different order than the query itself. It would be better to make it match. Perhaps put relkind first in both cases. I might also think about trying to make the column naming a bit more consistent, e.g. relkind, relfilenode, toast_oid, toast_relfilenode, toast_index_oid, toast_index_relfilenode. - In heap_create(), the wording of the error messages is not quite consistent. You have "relfilenode value not set when in binary upgrade mode", "toast relfilenode value not set when in binary upgrade mode", and "pg_class index relfilenode value not set when in binary upgrade mode". Why does the last one mention pg_class when the other two don't? - The code in heap_create() now has no comments whatsoever, which is a shame, because it's actually kind of a tricky bit of logic. Someone might wonder why we override the relfilenode inside that function instead of doing it at the same places where we absorb binary_upgrade_next_{heap,index,toast}_pg_class_oid and the passing down the relfilenode. I think the answer is that passing down the relfilenode from the caller would result in storage not actually being created, whereas in this case we want it to be created but just with the value we specify, and the reason we want that is because we need later DDL that happens after these statements but before the old cluster's relations are moved over to execute successfully, which it won't if the storage is altogether absent. However, that raises the question of whether this patch has even got the basic design right. Maybe we ought to actually be absorbing the relfilenode setting at the same places where we're doing so for the OID, and then passing an additional parameter to heap_create() like bool suppress_storage or something like that. Maybe, taking it even further, we ought to be changing the signatures of binary_upgrade_next_heap_pg_class_oid and friends to be two-argument functions, and pass down the OID and the relfilenode in the same call, rather than calling two separate functions. I'm not so much concerned about the cost of calling two functions as the potential for confusion. I'm not honestly sure that either of these changes are the right thing to do, but I am pretty strongly inclined to do at least the first part - trying to absorb reloid and relfilenode in the same places. If we're not going to do that we certainly need to explain why we're doing it the way we are in the comments. It's not really this patch's fault, but it would sure be nice if we had some better testing for this area. Suppose this patch somehow changed nothing from the present behavior. How would we know? Or suppose it managed to somehow set all the relfilenodes in the new cluster to random values rather than the intended one? There's no automated testing that would catch any of that, and it's not obvious how it could be added to test.sh. I suppose what we really need to do at some point is rewrite that as a TAP test, but that seems like a separate project from this patch. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-23 21:12 Stephen Frost <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-23 21:12 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> Greetings, * Robert Haas ([email protected]) wrote: > On Fri, Aug 20, 2021 at 1:36 PM Shruthi Gowda <[email protected]> wrote: > > Thanks Robert for your comments. > > I have split the patch into two portions. One that handles DB OID and > > the other that > > handles tablespace OID and relfilenode OID. > > It's pretty clear from the discussion, I think, that the database OID > one is going to need rework to be considered. Regarding that ... I have to wonder just what promises we feel we've made when it comes to what a user is expected to be able to do with the new cluster *before* pg_upgrade is run on it. For my part, I sure feel like it's "nothing", in which case it seems like we can do things that we can't do with a running system, like literally just DROP and recreate with the correct OID of any databases we need to, or even push that back to the user to do that at initdb time with some kind of error thrown by pg_upgrade during the --check phase. "Initial databases have non-standard OIDs, recreate destination cluster with initdb --with-oid=12341" or something along those lines. Also open to the idea of simply forcing 'template1' to always being OID=1 even if it's dropped/recreated and then just dropping/recreating the template0 and postgres databases if they've got different OIDs than what the old cluster did- after all, they should be getting entirely re-populated as part of the pg_upgrade process itself. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 00:29 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 2 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-24 00:29 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Aug 23, 2021 at 04:57:31PM -0400, Robert Haas wrote: > On Fri, Aug 20, 2021 at 1:36 PM Shruthi Gowda <[email protected]> wrote: > > Thanks Robert for your comments. > > I have split the patch into two portions. One that handles DB OID and > > the other that > > handles tablespace OID and relfilenode OID. > > It's pretty clear from the discussion, I think, that the database OID > one is going to need rework to be considered. I assume this patch is not going to be applied until there is an actual use case for preserving these values. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 09:10 Shruthi Gowda <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Shruthi Gowda @ 2021-08-24 09:10 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 5:59 AM Bruce Momjian <[email protected]> wrote: > > On Mon, Aug 23, 2021 at 04:57:31PM -0400, Robert Haas wrote: > > On Fri, Aug 20, 2021 at 1:36 PM Shruthi Gowda <[email protected]> wrote: > > > Thanks Robert for your comments. > > > I have split the patch into two portions. One that handles DB OID and > > > the other that > > > handles tablespace OID and relfilenode OID. > > > > It's pretty clear from the discussion, I think, that the database OID > > one is going to need rework to be considered. > > I assume this patch is not going to be applied until there is an actual > use case for preserving these values. JFI, I added an entry into commit fest for this patch. link: https://commitfest.postgresql.org/34/3296/ ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 15:24 Robert Haas <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Robert Haas @ 2021-08-24 15:24 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Aug 23, 2021 at 5:12 PM Stephen Frost <[email protected]> wrote: > Regarding that ... I have to wonder just what promises we feel we've > made when it comes to what a user is expected to be able to do with the > new cluster *before* pg_upgrade is run on it. For my part, I sure feel > like it's "nothing", in which case it seems like we can do things that > we can't do with a running system, like literally just DROP and recreate > with the correct OID of any databases we need to, or even push that back > to the user to do that at initdb time with some kind of error thrown by > pg_upgrade during the --check phase. "Initial databases have > non-standard OIDs, recreate destination cluster with initdb > --with-oid=12341" or something along those lines. Yeah, possibly. Honestly, I find it weird that pg_upgrade expects the new cluster to already exist. It seems like it would be more sensible if it created the cluster itself. That's not entirely trivial, because for example you have to create it with the correct locale settings and stuff. But if you require the cluster to exist already, then you run into the kinds of questions that you're asking here, and whether the answer is "nothing" as you propose here or something more than that, it's clearly not "whatever you want" nor anything close to that. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 15:28 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 2 replies; 78+ messages in thread From: Robert Haas @ 2021-08-24 15:28 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Aug 23, 2021 at 8:29 PM Bruce Momjian <[email protected]> wrote: > I assume this patch is not going to be applied until there is an actual > use case for preserving these values. My interpretation of the preceding discussion was that several people thought this change was a good idea regardless of whether anything ever happens with TDE, so I wasn't seeing a reason to wait. Personally, I've always thought that it was quite odd that pg_upgrade didn't preserve the relfilenode values, so I'm in favor of the change. I bet we could even make some simplifications to that code if we got all of this sorted out, which seems like it would be nice. I think it was also mentioned that this might be nice for pgBackRest, which apparently permits incremental backups across major version upgrades but likes filenames to match. That being said, if you or somebody else thinks that this is a bad idea or that the reasons offered up until now are insufficient, feel free to make that argument. I just work here... -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 16:04 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 2 replies; 78+ messages in thread From: Tom Lane @ 2021-08-24 16:04 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Robert Haas <[email protected]> writes: > On Mon, Aug 23, 2021 at 8:29 PM Bruce Momjian <[email protected]> wrote: >> I assume this patch is not going to be applied until there is an actual >> use case for preserving these values. > ... > That being said, if you or somebody else thinks that this is a bad > idea or that the reasons offered up until now are insufficient, feel > free to make that argument. I just work here... Per upthread discussion, it seems impractical to fully guarantee that database OIDs match, which seems to mean that the whole premise collapses. Like Bruce, I want to see a plausible use case justifying any partial-guarantee scenario before we add more complication (= bugs) to pg_upgrade. regards, tom lane ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 16:39 Bruce Momjian <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-24 16:39 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Tue, Aug 24, 2021 at 12:04:00PM -0400, Tom Lane wrote: > Robert Haas <[email protected]> writes: > > On Mon, Aug 23, 2021 at 8:29 PM Bruce Momjian <[email protected]> wrote: > >> I assume this patch is not going to be applied until there is an actual > >> use case for preserving these values. > > > ... > > > That being said, if you or somebody else thinks that this is a bad > > idea or that the reasons offered up until now are insufficient, feel > > free to make that argument. I just work here... > > Per upthread discussion, it seems impractical to fully guarantee > that database OIDs match, which seems to mean that the whole premise > collapses. Like Bruce, I want to see a plausible use case justifying > any partial-guarantee scenario before we add more complication (= bugs) > to pg_upgrade. Yes, pg_upgrade is already complex enough, so why add more complexity for some cosmetic value. (I think "cosmetic" flew out the window with pg_upgrade long ago. ;-) ) I know that pgBackRest has asked for stable relfilenodes to make incremental file system backups after pg_upgrade smaller, but if we want to make relfilenodes stable, we had better understand that is _why_ we are adding this complexity. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 16:40 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-24 16:40 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 11:28:37AM -0400, Robert Haas wrote: > On Mon, Aug 23, 2021 at 8:29 PM Bruce Momjian <[email protected]> wrote: > > I assume this patch is not going to be applied until there is an actual > > use case for preserving these values. > > My interpretation of the preceding discussion was that several people > thought this change was a good idea regardless of whether anything > ever happens with TDE, so I wasn't seeing a reason to wait. > Personally, I've always thought that it was quite odd that pg_upgrade > didn't preserve the relfilenode values, so I'm in favor of the change. > I bet we could even make some simplifications to that code if we got > all of this sorted out, which seems like it would be nice. Yes, if this ends up being a cleanup with no added complexity, that would be nice, but I had not seen how that was possible in the psat. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 16:43 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 2 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-24 16:43 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 11:24:21AM -0400, Robert Haas wrote: > On Mon, Aug 23, 2021 at 5:12 PM Stephen Frost <[email protected]> wrote: > > Regarding that ... I have to wonder just what promises we feel we've > > made when it comes to what a user is expected to be able to do with the > > new cluster *before* pg_upgrade is run on it. For my part, I sure feel > > like it's "nothing", in which case it seems like we can do things that > > we can't do with a running system, like literally just DROP and recreate > > with the correct OID of any databases we need to, or even push that back > > to the user to do that at initdb time with some kind of error thrown by > > pg_upgrade during the --check phase. "Initial databases have > > non-standard OIDs, recreate destination cluster with initdb > > --with-oid=12341" or something along those lines. > > Yeah, possibly. Honestly, I find it weird that pg_upgrade expects the > new cluster to already exist. It seems like it would be more sensible > if it created the cluster itself. That's not entirely trivial, because > for example you have to create it with the correct locale settings and > stuff. But if you require the cluster to exist already, then you run > into the kinds of questions that you're asking here, and whether the > answer is "nothing" as you propose here or something more than that, > it's clearly not "whatever you want" nor anything close to that. Yes, it is a trade-off. If we had pg_upgrade create the new cluster, the pg_upgrade instructions would be simpler, but pg_upgrade would be more complex since it has to adjust _everything_ properly so pg_upgrade works --- I never got to that point, but I am willing to explore what would be required. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 17:23 Robert Haas <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Robert Haas @ 2021-08-24 17:23 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Tue, Aug 24, 2021 at 12:04 PM Tom Lane <[email protected]> wrote: > Per upthread discussion, it seems impractical to fully guarantee > that database OIDs match, which seems to mean that the whole premise > collapses. Like Bruce, I want to see a plausible use case justifying > any partial-guarantee scenario before we add more complication (= bugs) > to pg_upgrade. I think you might be overlooking the emails from Stephen and I where we suggested how that could be made to work? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 17:25 Stephen Frost <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Stephen Frost @ 2021-08-24 17:25 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> Greetings, * Robert Haas ([email protected]) wrote: > On Mon, Aug 23, 2021 at 5:12 PM Stephen Frost <[email protected]> wrote: > > Regarding that ... I have to wonder just what promises we feel we've > > made when it comes to what a user is expected to be able to do with the > > new cluster *before* pg_upgrade is run on it. For my part, I sure feel > > like it's "nothing", in which case it seems like we can do things that > > we can't do with a running system, like literally just DROP and recreate > > with the correct OID of any databases we need to, or even push that back > > to the user to do that at initdb time with some kind of error thrown by > > pg_upgrade during the --check phase. "Initial databases have > > non-standard OIDs, recreate destination cluster with initdb > > --with-oid=12341" or something along those lines. > > Yeah, possibly. Honestly, I find it weird that pg_upgrade expects the > new cluster to already exist. It seems like it would be more sensible > if it created the cluster itself. That's not entirely trivial, because > for example you have to create it with the correct locale settings and > stuff. But if you require the cluster to exist already, then you run > into the kinds of questions that you're asking here, and whether the > answer is "nothing" as you propose here or something more than that, > it's clearly not "whatever you want" nor anything close to that. Yeah, I'd had a similar thought and also tend to agree that it'd make more sense for pg_upgrade to set up the new cluster too, and doing so in a way that makes sure that it matches the old cluster as that's rather important. Having the user do it also implies that there is some freedom for the user to mess around with the new cluster before running pg_upgrade, it seems to me anyway, and that's certainly not something that we've built anything into pg_upgrade to deal with cleanly.. It isn't like initdb takes all *that* long to run either, and reducing the number of steps that the user has to take to perform an upgrade sure seems like a good thing to do. Anyhow, just wanted to throw that out there as another way we might approach this. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 17:26 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-24 17:26 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 12:43 PM Bruce Momjian <[email protected]> wrote: > Yes, it is a trade-off. If we had pg_upgrade create the new cluster, > the pg_upgrade instructions would be simpler, but pg_upgrade would be > more complex since it has to adjust _everything_ properly so pg_upgrade > works --- I never got to that point, but I am willing to explore what > would be required. It's probably a topic for another thread, rather than this one, but I think that would be very cool. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 17:46 Stephen Frost <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Stephen Frost @ 2021-08-24 17:46 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Bruce Momjian <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> Greetings, * Robert Haas ([email protected]) wrote: > On Tue, Aug 24, 2021 at 12:43 PM Bruce Momjian <[email protected]> wrote: > > Yes, it is a trade-off. If we had pg_upgrade create the new cluster, > > the pg_upgrade instructions would be simpler, but pg_upgrade would be > > more complex since it has to adjust _everything_ properly so pg_upgrade > > works --- I never got to that point, but I am willing to explore what > > would be required. > > It's probably a topic for another thread, rather than this one, but I > think that would be very cool. Yes, definite +1 on this. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 18:16 Bruce Momjian <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Bruce Momjian @ 2021-08-24 18:16 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 12:43:20PM -0400, Bruce Momjian wrote: > Yes, it is a trade-off. If we had pg_upgrade create the new cluster, > the pg_upgrade instructions would be simpler, but pg_upgrade would be > more complex since it has to adjust _everything_ properly so pg_upgrade > works --- I never got to that point, but I am willing to explore what > would be required. One other issue --- the more that pg_upgrade preserves, the more likely pg_upgrade will break when some internal changes happen in Postgres. Therefore, if you want pg_upgrade to preserve something, you have to have a good reason --- even code simplicity might not be a sufficient reason. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 18:34 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-24 18:34 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 2:16 PM Bruce Momjian <[email protected]> wrote: > One other issue --- the more that pg_upgrade preserves, the more likely > pg_upgrade will break when some internal changes happen in Postgres. > Therefore, if you want pg_upgrade to preserve something, you have to > have a good reason --- even code simplicity might not be a sufficient > reason. While I accept that as a general principle, I don't think it's really applicable in this case. pg_upgrade already knows all about relfilenodes; it has a source file called relfilenode.c. I don't see that a pg_upgrade that preserves relfilenodes is any more or less likely to break in the future than a pg_upgrade that renumbers all the files so that the relation OID and the relfilenode are equal. You've got about the same amount of reliance on the on-disk layout either way. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-24 18:49 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-24 18:49 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 02:34:26PM -0400, Robert Haas wrote: > On Tue, Aug 24, 2021 at 2:16 PM Bruce Momjian <[email protected]> wrote: > > One other issue --- the more that pg_upgrade preserves, the more likely > > pg_upgrade will break when some internal changes happen in Postgres. > > Therefore, if you want pg_upgrade to preserve something, you have to > > have a good reason --- even code simplicity might not be a sufficient > > reason. > > While I accept that as a general principle, I don't think it's really > applicable in this case. pg_upgrade already knows all about > relfilenodes; it has a source file called relfilenode.c. I don't see > that a pg_upgrade that preserves relfilenodes is any more or less > likely to break in the future than a pg_upgrade that renumbers all the > files so that the relation OID and the relfilenode are equal. You've > got about the same amount of reliance on the on-disk layout either > way. I was making more of a general statement that preservation can be problematic and its impact must be researched. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:00 Robert Haas <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Robert Haas @ 2021-08-26 15:00 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Tue, Aug 17, 2021 at 2:50 PM Robert Haas <[email protected]> wrote: > > Less sure that this is a good idea, though. In particular, I do not > > think that you can make it work in the face of > > alter database template1 rename to oops; > > create database template1; > > That is a really good point. If we can't categorically force the OID > of those databases to have a particular, fixed value, and based on > this example that seems to be impossible, then there's always a > possibility that we might find a value in the old cluster that doesn't > happen to match what is present in the new cluster. Seen from that > angle, the problem is really with databases that are pre-existent in > the new cluster but whose contents still need to be dumped. Maybe we > could (optionally? conditionally?) drop those databases from the new > cluster and then recreate them with the OID that we want them to have. Actually, we do that already. create_new_objects() runs pg_restore with --create for most databases, but with --clean --create for template1 and postgres. This means that template1 and postgres will always be recreated in the new cluster, and other databases are assumed not to exist in the new cluster and the upgrade will fail if they unexpectedly do. And the reason why pg_upgrade does that is that it wants to "propagate [the] database-level properties" of postgres and template1. So suppose we just make the database OID one of the database-level properties that we want to propagate. That should mostly just work, but where can things go wrong? The only real failure mode is we try to create a database in the new cluster and find out that the OID is already in use. If the new OID that collides >64k, then the user has messed with the new cluster before doing that. And since pg_upgrade is pretty clearly already assuming that you shouldn't do that, it's fine to also make that assumption in this case. We can disregard such cases as user error. If the new OID that collides is <64k, then it must be colliding with template0, template1, or postgres in the new cluster, because those are the only databases that can have such OIDs since, currently, we don't allow users to specify an OID for a new database. And the problem cannot be with template1, because we hard-code its OID to 1. If there is a database with OID 1 in either cluster, it must be template1, and if there is a database with OID 1 in both clusters, it must be template1 in both cases, and we'll just drop and recreate it with OID 1 and everything is fine. So we need only consider template0 and postgres, which are created with system-generated OIDs. And, it would be no issue if either of those databases had the same OID in the old and new cluster, so the only possible OID collision is one where the same system-generated OID was assigned to one of those databases in the old cluster and to the other in the new cluster. First consider the case where template0 has OID, say, 13000, in the old cluster, and postgres has that OID in the new cluster. No problem occurs, because template0 isn't transferred anyway. The reverse direction is a problem, though. If postgres had been assigned OID 13000 in the old cluster and, by sheer chance, template0 had that OID in the new cluster, then the upgrade would fail, because it wouldn't be able to recreate the postgres database with the correct OID. But that doesn't seem very difficult to fix. I think all we need to do is have initdb assign a fixed OID to template0 at creation time. Then, in any new release to which someone might be trying to upgrade, the system-generated OID assigned to postgres in the old release can't match the fixed OID assigned to template0 in the new release, so the one problem case is ruled out. We do need, however, to make sure that the assign-my-database-a-fixed-OID syntax is either entirely restricted to initdb & pg_upgrade or at least that OIDS < 64k can only be assigned in one of those modes. Otherwise, some creative person could manufacture new problem cases by setting up the source database so that the OID of one of their databases matches the fixed OID we gave to template0 or template1, or the system-generated OID for postgres in the new cluster. In short, as far as I can see, all we need to do to preserve database OIDs across pg_upgrade is: 1. Add a new syntax for creating a database with a given OID, and use it in pg_dump --binary-upgrade. 2. Don't let users use it at least for OIDs <64k, or maybe just don't let them use it at all. 3. But let initdb use it, and have initdb set the initial OID for template0 to a fixed value < 10000. If the user changes it later, no problem; the cluster into which they are upgrading won't contain any databases with high-numbered OIDs. Anyone see a flaw in that analysis? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:24 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 2 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 15:24 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > Anyone see a flaw in that analysis? I am still waiting to hear the purpose of this preservation. As long as you don't apply the patch, I guess I will just stop asking. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:35 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-26 15:35 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Tom Lane <[email protected]>; Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 11:24 AM Bruce Momjian <[email protected]> wrote: > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > Anyone see a flaw in that analysis? > > I am still waiting to hear the purpose of this preservation. As long as > you don't apply the patch, I guess I will just stop asking. You make it sound like I didn't answer that question the last time you asked it, but I did.[1] I went back to the previous thread and found that, in fact, there's at least one email *from you* appearing to endorse that concept for reasons unrelated to TDE[2] and another where you appear to agree that it would be useful for TDE to do it.[3] Stephen Frost also wrote up his discussion during the Unconference and some of his reasons for liking the idea.[4] If you've changed your mind about this being a good idea, or if you no longer think it's useful without TDE, that's fine. Everyone is entitled to change their opinion. But then please say that straight out. It baffles me why you're now acting as if it hasn't been discussed when it clearly has been, and both you and I were participants in that discussion. [1] https://www.postgresql.org/message-id/[email protected]... [2] https://www.postgresql.org/message-id/[email protected] [3] https://www.postgresql.org/message-id/[email protected] [4] https://www.postgresql.org/message-id/[email protected] -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:36 Stephen Frost <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-26 15:36 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Bruce Momjian ([email protected]) wrote: > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > Anyone see a flaw in that analysis? > > I am still waiting to hear the purpose of this preservation. As long as > you don't apply the patch, I guess I will just stop asking. I'm a bit confused why this question keeps coming up as we've discussed multiple reasons (incremental backups, possible use for TDE which would make this required, general improved sanity when working with pg_upgrade is frankly a benefit in its own right too...). If the additional code was a huge burden or even a moderate one then that might be an argument against, but it hardly sounds like it will be given Robert's thorough analysis so far and the (admittedly not complete, but not that far from it based on the DB OID review) proposed patch. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:39 Stephen Frost <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-26 15:39 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Robert Haas ([email protected]) wrote: > On Tue, Aug 17, 2021 at 2:50 PM Robert Haas <[email protected]> wrote: > > > Less sure that this is a good idea, though. In particular, I do not > > > think that you can make it work in the face of > > > alter database template1 rename to oops; > > > create database template1; > > > > That is a really good point. If we can't categorically force the OID > > of those databases to have a particular, fixed value, and based on > > this example that seems to be impossible, then there's always a > > possibility that we might find a value in the old cluster that doesn't > > happen to match what is present in the new cluster. Seen from that > > angle, the problem is really with databases that are pre-existent in > > the new cluster but whose contents still need to be dumped. Maybe we > > could (optionally? conditionally?) drop those databases from the new > > cluster and then recreate them with the OID that we want them to have. > > Actually, we do that already. create_new_objects() runs pg_restore > with --create for most databases, but with --clean --create for > template1 and postgres. This means that template1 and postgres will > always be recreated in the new cluster, and other databases are > assumed not to exist in the new cluster and the upgrade will fail if > they unexpectedly do. And the reason why pg_upgrade does that is that > it wants to "propagate [the] database-level properties" of postgres > and template1. So suppose we just make the database OID one of the > database-level properties that we want to propagate. That should > mostly just work, but where can things go wrong? > > The only real failure mode is we try to create a database in the new > cluster and find out that the OID is already in use. If the new OID > that collides >64k, then the user has messed with the new cluster > before doing that. And since pg_upgrade is pretty clearly already > assuming that you shouldn't do that, it's fine to also make that > assumption in this case. We can disregard such cases as user error. > > If the new OID that collides is <64k, then it must be colliding with > template0, template1, or postgres in the new cluster, because those > are the only databases that can have such OIDs since, currently, we > don't allow users to specify an OID for a new database. And the > problem cannot be with template1, because we hard-code its OID to 1. > If there is a database with OID 1 in either cluster, it must be > template1, and if there is a database with OID 1 in both clusters, it > must be template1 in both cases, and we'll just drop and recreate it > with OID 1 and everything is fine. So we need only consider template0 > and postgres, which are created with system-generated OIDs. And, it > would be no issue if either of those databases had the same OID in the > old and new cluster, so the only possible OID collision is one where > the same system-generated OID was assigned to one of those databases > in the old cluster and to the other in the new cluster. > > First consider the case where template0 has OID, say, 13000, in the > old cluster, and postgres has that OID in the new cluster. No problem > occurs, because template0 isn't transferred anyway. The reverse > direction is a problem, though. If postgres had been assigned OID > 13000 in the old cluster and, by sheer chance, template0 had that OID > in the new cluster, then the upgrade would fail, because it wouldn't > be able to recreate the postgres database with the correct OID. > > But that doesn't seem very difficult to fix. I think all we need to do > is have initdb assign a fixed OID to template0 at creation time. Then, > in any new release to which someone might be trying to upgrade, the > system-generated OID assigned to postgres in the old release can't > match the fixed OID assigned to template0 in the new release, so the > one problem case is ruled out. We do need, however, to make sure that > the assign-my-database-a-fixed-OID syntax is either entirely > restricted to initdb & pg_upgrade or at least that OIDS < 64k can only > be assigned in one of those modes. Otherwise, some creative person > could manufacture new problem cases by setting up the source database > so that the OID of one of their databases matches the fixed OID we > gave to template0 or template1, or the system-generated OID for > postgres in the new cluster. > > In short, as far as I can see, all we need to do to preserve database > OIDs across pg_upgrade is: > > 1. Add a new syntax for creating a database with a given OID, and use > it in pg_dump --binary-upgrade. > 2. Don't let users use it at least for OIDs <64k, or maybe just don't > let them use it at all. > 3. But let initdb use it, and have initdb set the initial OID for > template0 to a fixed value < 10000. If the user changes it later, no > problem; the cluster into which they are upgrading won't contain any > databases with high-numbered OIDs. > > Anyone see a flaw in that analysis? This looks like a pretty good analysis to me. As it relates to the question about allowing users to specify an OID, I'd be inclined to allow it but only for OIDs >64k. We've certainly reserved things in the past and I don't see any issue with having that reservation here, but if we're going to build the capability to specify the OID into CREATE DATABASE then it seems a bit odd to disallow users from using it, as long as we're preventing them from causing problems with it. Are there issues that you see with allowing users to specify the OID even with the >64k restriction..? I can't think of one offhand but perhaps I'm missing something. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:44 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 15:44 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Stephen Frost <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 11:35:01AM -0400, Robert Haas wrote: > On Thu, Aug 26, 2021 at 11:24 AM Bruce Momjian <[email protected]> wrote: > > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > > Anyone see a flaw in that analysis? > > > > I am still waiting to hear the purpose of this preservation. As long as > > you don't apply the patch, I guess I will just stop asking. > > You make it sound like I didn't answer that question the last time you > asked it, but I did.[1] I went back to the previous thread and found > that, in fact, there's at least one email *from you* appearing to > endorse that concept for reasons unrelated to TDE[2] and another where > you appear to agree that it would be useful for TDE to do it.[3] > Stephen Frost also wrote up his discussion during the Unconference and > some of his reasons for liking the idea.[4] > > If you've changed your mind about this being a good idea, or if you no > longer think it's useful without TDE, that's fine. Everyone is > entitled to change their opinion. But then please say that straight > out. It baffles me why you're now acting as if it hasn't been > discussed when it clearly has been, and both you and I were > participants in that discussion. > > [1] https://www.postgresql.org/message-id/[email protected]... > [2] https://www.postgresql.org/message-id/[email protected] > [3] https://www.postgresql.org/message-id/[email protected] > [4] https://www.postgresql.org/message-id/[email protected] Yes, it would help incremental backup of pgBackRest, as reported by the developers. However, I have seen no discussion if this is useful enough reason to add the complexity to preserve this. The TODO list shows "Desirability" as the first item to be discussed, so I expected that to be discussed first. Also, with TDE not progressing (and my approach not even needing this), I have not seen a full discussion if this item is desirable based on its complexity. What I did see is this patch appear with no context of why it is useful given our current plans, except for pgBackRest, which I think I mentioned. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:48 Bruce Momjian <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 15:48 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 11:36:51AM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian ([email protected]) wrote: > > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > > Anyone see a flaw in that analysis? > > > > I am still waiting to hear the purpose of this preservation. As long as > > you don't apply the patch, I guess I will just stop asking. > > I'm a bit confused why this question keeps coming up as we've discussed > multiple reasons (incremental backups, possible use for TDE which would I have not seen much explaination on pgBackRest, except me mentioning it. Is this something really useful? As far as TDE, I haven't seen any concrete plan for that, so why add this code for that reason? > make this required, general improved sanity when working with pg_upgrade > is frankly a benefit in its own right too...). If the additional code How? I am not aware of any advantage except cosmetic. > was a huge burden or even a moderate one then that might be an argument > against, but it hardly sounds like it will be given Robert's thorough > analysis so far and the (admittedly not complete, but not that far from > it based on the DB OID review) proposed patch. I am find to add it if it is minor, but I want to see the calculus of its value vs complexity, which I have not seen spelled out. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 15:52 Robert Haas <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-26 15:52 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 11:39 AM Stephen Frost <[email protected]> wrote: > This looks like a pretty good analysis to me. As it relates to the > question about allowing users to specify an OID, I'd be inclined to > allow it but only for OIDs >64k. We've certainly reserved things in the > past and I don't see any issue with having that reservation here, but if > we're going to build the capability to specify the OID into CREATE > DATABASE then it seems a bit odd to disallow users from using it, as > long as we're preventing them from causing problems with it. > > Are there issues that you see with allowing users to specify the OID > even with the >64k restriction..? I can't think of one offhand but > perhaps I'm missing something. So I actually should have said 16k here, not 64k, as somebody already pointed out to me off-list. Whee! I don't know of a reason not to let people do that, other than that it seems like an attractive nuisance. People will do it and it will fail because they chose a duplicate OID, or they'll complain that a regular dump and restore didn't preserve their database OIDs, or maybe they'll expect that they can copy a database from one cluster to another because they gave it the same OID! That said, I don't see a great harm in it. It just seems to me like exposing knobs to users that don't seem to have any legitimate use may be borrowing trouble. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 16:34 Stephen Frost <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-26 16:34 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Bruce Momjian ([email protected]) wrote: > On Thu, Aug 26, 2021 at 11:36:51AM -0400, Stephen Frost wrote: > > * Bruce Momjian ([email protected]) wrote: > > > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > > > Anyone see a flaw in that analysis? > > > > > > I am still waiting to hear the purpose of this preservation. As long as > > > you don't apply the patch, I guess I will just stop asking. > > > > I'm a bit confused why this question keeps coming up as we've discussed > > multiple reasons (incremental backups, possible use for TDE which would > > I have not seen much explaination on pgBackRest, except me mentioning > it. Is this something really useful? Being able to quickly perform a backup on a newly upgraded cluster would certainly be valuable and that's definitely not possible today due to all of the filenames changing. > As far as TDE, I haven't seen any concrete plan for that, so why add > this code for that reason? That this would help with TDE (of which there seems little doubt...) is an additional benefit to this. Specifically, taking the existing work that's already been done to allow block-by-block encryption and adjusting it for AES-XTS and then using the db-dir+relfileno+block number as the IV, just like many disk encryption systems do, avoids the concerns that were brought up about using LSN for the IV with CTR and it's certainly not difficult to do, but it does depend on this change. This was all discussed previously and it sure looks like a sensible approach to use that mirrors what many other systems already do successfully. > > make this required, general improved sanity when working with pg_upgrade > > is frankly a benefit in its own right too...). If the additional code > > How? I am not aware of any advantage except cosmetic. Having to resort to matching up inode numbers between the two clusters after a pg_upgrade to figure out what files are actually the same underneath is a pain that goes beyond just cosmetics imv. Removing that additional level that admins, and developers for that matter, have to go through would be a nice improvement on its own. > > was a huge burden or even a moderate one then that might be an argument > > against, but it hardly sounds like it will be given Robert's thorough > > analysis so far and the (admittedly not complete, but not that far from > > it based on the DB OID review) proposed patch. > > I am find to add it if it is minor, but I want to see the calculus of > its value vs complexity, which I have not seen spelled out. I feel that this, along with the prior discussions, spells it out sufficiently given the patch's complexity looks to be reasonably minor and very similar to the existing things that pg_upgrade already does. Had pg_upgrade done this in the first place, I don't think there would have been nearly this amount of discussion about it. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 16:37 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-26 16:37 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Stephen Frost <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 11:48 AM Bruce Momjian <[email protected]> wrote: > I am find to add it if it is minor, but I want to see the calculus of > its value vs complexity, which I have not seen spelled out. I don't think it's going to be all that complicated, but we're going to have to wait until we have something closer to a final patch before we can really evaluate that. I am honestly a little puzzled about why you think complexity is such a big issue for this patch in particular. I feel we do probably several hundred things every release cycle that are more complicated than this, so it doesn't seem like this is particularly extraordinary or needs a lot of extra scrutiny. I do think there is some risk that there are messy cases we can't handle cleanly, but if that becomes an issue then I'll abandon the effort until a solution can be found. I'm not trying to relentlessly drive something through that is a bad idea on principle. I agree with all Stephen's comments, too. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 16:42 Stephen Frost <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Stephen Frost @ 2021-08-26 16:42 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Robert Haas ([email protected]) wrote: > On Thu, Aug 26, 2021 at 11:39 AM Stephen Frost <[email protected]> wrote: > > This looks like a pretty good analysis to me. As it relates to the > > question about allowing users to specify an OID, I'd be inclined to > > allow it but only for OIDs >64k. We've certainly reserved things in the > > past and I don't see any issue with having that reservation here, but if > > we're going to build the capability to specify the OID into CREATE > > DATABASE then it seems a bit odd to disallow users from using it, as > > long as we're preventing them from causing problems with it. > > > > Are there issues that you see with allowing users to specify the OID > > even with the >64k restriction..? I can't think of one offhand but > > perhaps I'm missing something. > > So I actually should have said 16k here, not 64k, as somebody already > pointed out to me off-list. Whee! Hah, yes, of course. > I don't know of a reason not to let people do that, other than that it > seems like an attractive nuisance. People will do it and it will fail > because they chose a duplicate OID, or they'll complain that a regular > dump and restore didn't preserve their database OIDs, or maybe they'll > expect that they can copy a database from one cluster to another > because they gave it the same OID! That said, I don't see a great harm > in it. It just seems to me like exposing knobs to users that don't > seem to have any legitimate use may be borrowing trouble. We're going to have to gate this somehow to allow the OIDs under 16k to be used, so it seems like what you're suggesting is that we have that gate in place but then allow any OID to be used if you've crossed that gate? That is, if we do something like: SELECT pg_catalog.binary_upgrade_allow_setting_db_oid(); CREATE DATABASE blah WITH OID 1234; for pg_upgrade, well, users who are interested may well figure out how to do that themselves if they decide they want to set the OID, whereas if it 'just works' provided they don't try to use an OID too low then maybe they won't try to bypass the restriction against using system OIDs..? Ok, I'll give you that this is a stretch and I'm on the fence about if it's worthwhile or not to include and document and if, as you say, it's inviting trouble to allow users to set it. Users do seem to have a knack for finding things even when they aren't documented and then we get to deal with those complaints too. :) Perhaps others have some stronger feelings one way or another. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 16:49 Bruce Momjian <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 16:49 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 12:34:56PM -0400, Stephen Frost wrote: > * Bruce Momjian ([email protected]) wrote: > > On Thu, Aug 26, 2021 at 11:36:51AM -0400, Stephen Frost wrote: > > > * Bruce Momjian ([email protected]) wrote: > > > > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > > > > Anyone see a flaw in that analysis? > > > > > > > > I am still waiting to hear the purpose of this preservation. As long as > > > > you don't apply the patch, I guess I will just stop asking. > > > > > > I'm a bit confused why this question keeps coming up as we've discussed > > > multiple reasons (incremental backups, possible use for TDE which would > > > > I have not seen much explaination on pgBackRest, except me mentioning > > it. Is this something really useful? > > Being able to quickly perform a backup on a newly upgraded cluster would > certainly be valuable and that's definitely not possible today due to > all of the filenames changing. You mean incremental backup, right? I was told this by the pgBackRest developers during PGCon, but I have not heard that stated publicly, so I hate to go just on what I heard rather than seeing that stated publicly. > > As far as TDE, I haven't seen any concrete plan for that, so why add > > this code for that reason? > > That this would help with TDE (of which there seems little doubt...) is > an additional benefit to this. Specifically, taking the existing work > that's already been done to allow block-by-block encryption and > adjusting it for AES-XTS and then using the db-dir+relfileno+block > number as the IV, just like many disk encryption systems do, avoids the > concerns that were brought up about using LSN for the IV with CTR and > it's certainly not difficult to do, but it does depend on this change. > This was all discussed previously and it sure looks like a sensible > approach to use that mirrors what many other systems already do > successfully. Well, I would think we would not add this for TDE until we were sure someone was working on adding TDE. > > > make this required, general improved sanity when working with pg_upgrade > > > is frankly a benefit in its own right too...). If the additional code > > > > How? I am not aware of any advantage except cosmetic. > > Having to resort to matching up inode numbers between the two clusters > after a pg_upgrade to figure out what files are actually the same > underneath is a pain that goes beyond just cosmetics imv. Removing that > additional level that admins, and developers for that matter, have to go > through would be a nice improvement on its own. OK, I was just not aware anyone did that, since I have never hard anyone complain about it before. > > > was a huge burden or even a moderate one then that might be an argument > > > against, but it hardly sounds like it will be given Robert's thorough > > > analysis so far and the (admittedly not complete, but not that far from > > > it based on the DB OID review) proposed patch. > > > > I am find to add it if it is minor, but I want to see the calculus of > > its value vs complexity, which I have not seen spelled out. > > I feel that this, along with the prior discussions, spells it out > sufficiently given the patch's complexity looks to be reasonably minor > and very similar to the existing things that pg_upgrade already does. > Had pg_upgrade done this in the first place, I don't think there would > have been nearly this amount of discussion about it. Well, there is a reason pg_upgrade didn't initially do this --- because it adds complexity, and potentially makes future changes to pg_upgrade necessary if the server behavior changes. I am not saying this change is wrong, but I think the reasons need to be stated in this thread, rather than just moving forward. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 16:51 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 16:51 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 12:37:19PM -0400, Robert Haas wrote: > On Thu, Aug 26, 2021 at 11:48 AM Bruce Momjian <[email protected]> wrote: > > I am find to add it if it is minor, but I want to see the calculus of > > its value vs complexity, which I have not seen spelled out. > > I don't think it's going to be all that complicated, but we're going > to have to wait until we have something closer to a final patch before > we can really evaluate that. I am honestly a little puzzled about why > you think complexity is such a big issue for this patch in particular. > I feel we do probably several hundred things every release cycle that > are more complicated than this, so it doesn't seem like this is > particularly extraordinary or needs a lot of extra scrutiny. I do > think there is some risk that there are messy cases we can't handle > cleanly, but if that becomes an issue then I'll abandon the effort > until a solution can be found. I'm not trying to relentlessly drive > something through that is a bad idea on principle. > > I agree with all Stephen's comments, too. I just don't want to add requirements/complexity to pg_upgrade without clearly stated reasons because future database changes will need to honor this new preservation behavior. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 17:03 Stephen Frost <[email protected]> parent: Bruce Momjian <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-26 17:03 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Bruce Momjian ([email protected]) wrote: > On Thu, Aug 26, 2021 at 12:34:56PM -0400, Stephen Frost wrote: > > * Bruce Momjian ([email protected]) wrote: > > > On Thu, Aug 26, 2021 at 11:36:51AM -0400, Stephen Frost wrote: > > > > * Bruce Momjian ([email protected]) wrote: > > > > > On Thu, Aug 26, 2021 at 11:00:47AM -0400, Robert Haas wrote: > > > > > > Anyone see a flaw in that analysis? > > > > > > > > > > I am still waiting to hear the purpose of this preservation. As long as > > > > > you don't apply the patch, I guess I will just stop asking. > > > > > > > > I'm a bit confused why this question keeps coming up as we've discussed > > > > multiple reasons (incremental backups, possible use for TDE which would > > > > > > I have not seen much explaination on pgBackRest, except me mentioning > > > it. Is this something really useful? > > > > Being able to quickly perform a backup on a newly upgraded cluster would > > certainly be valuable and that's definitely not possible today due to > > all of the filenames changing. > > You mean incremental backup, right? I was told this by the pgBackRest > developers during PGCon, but I have not heard that stated publicly, so I > hate to go just on what I heard rather than seeing that stated publicly. Yes, we're talking about either incremental (or perhaps differential) backup where only the files which are actually different would be backed up. Just like with PG, I can't provide any complete guarantees that we'd be able to actually make this possible after a major version with pgBackRest with this change, but it definitely isn't possible *without* this change. I can't see any reason why we wouldn't be able to do a checksum-based incremental backup though (which would be *much* faster than a regular backup) once this change is made and have that be a reliable and trustworthy backup. I'd want to think about it more and discuss it with David in some detail before saying if we could maybe perform a timestamp-based incremental backup (without checksum'ing the files, as we do in normal situations), but that would really just be a bonus. > > > As far as TDE, I haven't seen any concrete plan for that, so why add > > > this code for that reason? > > > > That this would help with TDE (of which there seems little doubt...) is > > an additional benefit to this. Specifically, taking the existing work > > that's already been done to allow block-by-block encryption and > > adjusting it for AES-XTS and then using the db-dir+relfileno+block > > number as the IV, just like many disk encryption systems do, avoids the > > concerns that were brought up about using LSN for the IV with CTR and > > it's certainly not difficult to do, but it does depend on this change. > > This was all discussed previously and it sure looks like a sensible > > approach to use that mirrors what many other systems already do > > successfully. > > Well, I would think we would not add this for TDE until we were sure > someone was working on adding TDE. That this would help with TDE is what I'd consider an added bonus. > > > > make this required, general improved sanity when working with pg_upgrade > > > > is frankly a benefit in its own right too...). If the additional code > > > > > > How? I am not aware of any advantage except cosmetic. > > > > Having to resort to matching up inode numbers between the two clusters > > after a pg_upgrade to figure out what files are actually the same > > underneath is a pain that goes beyond just cosmetics imv. Removing that > > additional level that admins, and developers for that matter, have to go > > through would be a nice improvement on its own. > > OK, I was just not aware anyone did that, since I have never hard anyone > complain about it before. I've certainly done it and I'd be kind of surprised if others haven't, but I've also played a lot with pg_dump in various modes, so perhaps that's not a great representation. I've definitely had to explain to clients why there's a whole different set of filenames after a pg_upgrade and why that is the case for an 'in place' upgrade before too. > > > > was a huge burden or even a moderate one then that might be an argument > > > > against, but it hardly sounds like it will be given Robert's thorough > > > > analysis so far and the (admittedly not complete, but not that far from > > > > it based on the DB OID review) proposed patch. > > > > > > I am find to add it if it is minor, but I want to see the calculus of > > > its value vs complexity, which I have not seen spelled out. > > > > I feel that this, along with the prior discussions, spells it out > > sufficiently given the patch's complexity looks to be reasonably minor > > and very similar to the existing things that pg_upgrade already does. > > Had pg_upgrade done this in the first place, I don't think there would > > have been nearly this amount of discussion about it. > > Well, there is a reason pg_upgrade didn't initially do this --- because > it adds complexity, and potentially makes future changes to pg_upgrade > necessary if the server behavior changes. I have a very hard time seeing what changes might happen in the server in this space that wouldn't have an impact on pg_upgrade, with or without this. > I am not saying this change is wrong, but I think the reasons need to be > stated in this thread, rather than just moving forward. Ok, they've been stated and it seems to at least Robert and myself that this is worthwhile to at least continue through to a concluded patch, after which we can contemplate that patch's complexity against these reasons. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 17:16 Bruce Momjian <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 17:16 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 01:03:54PM -0400, Stephen Frost wrote: > Yes, we're talking about either incremental (or perhaps differential) > backup where only the files which are actually different would be backed > up. Just like with PG, I can't provide any complete guarantees that > we'd be able to actually make this possible after a major version with > pgBackRest with this change, but it definitely isn't possible *without* > this change. I can't see any reason why we wouldn't be able to do a > checksum-based incremental backup though (which would be *much* faster > than a regular backup) once this change is made and have that be a > reliable and trustworthy backup. I'd want to think about it more and > discuss it with David in some detail before saying if we could maybe > perform a timestamp-based incremental backup (without checksum'ing the > files, as we do in normal situations), but that would really just be a > bonus. Well, it would be nice to know exactly how it would help pgBackRest if that is one of the reasons we are adding this feature. > > > > As far as TDE, I haven't seen any concrete plan for that, so why add > > > > this code for that reason? > > > > > > That this would help with TDE (of which there seems little doubt...) is > > > an additional benefit to this. Specifically, taking the existing work > > > that's already been done to allow block-by-block encryption and > > > adjusting it for AES-XTS and then using the db-dir+relfileno+block > > > number as the IV, just like many disk encryption systems do, avoids the > > > concerns that were brought up about using LSN for the IV with CTR and > > > it's certainly not difficult to do, but it does depend on this change. > > > This was all discussed previously and it sure looks like a sensible > > > approach to use that mirrors what many other systems already do > > > successfully. > > > > Well, I would think we would not add this for TDE until we were sure > > someone was working on adding TDE. > > That this would help with TDE is what I'd consider an added bonus. Not if we have no plans to implement TDE, which was my point. Why not wait to see if we are actually going to implement TDE rather than adding it now. It is just so obvious, why do I have to state this? > > > > > make this required, general improved sanity when working with pg_upgrade > > > > > is frankly a benefit in its own right too...). If the additional code > > > > > > > > How? I am not aware of any advantage except cosmetic. > > > > > > Having to resort to matching up inode numbers between the two clusters > > > after a pg_upgrade to figure out what files are actually the same > > > underneath is a pain that goes beyond just cosmetics imv. Removing that > > > additional level that admins, and developers for that matter, have to go > > > through would be a nice improvement on its own. > > > > OK, I was just not aware anyone did that, since I have never hard anyone > > complain about it before. > > I've certainly done it and I'd be kind of surprised if others haven't, > but I've also played a lot with pg_dump in various modes, so perhaps > that's not a great representation. I've definitely had to explain to > clients why there's a whole different set of filenames after a > pg_upgrade and why that is the case for an 'in place' upgrade before > too. Uh, so I guess I am right that few people have mentioned this in the past. Why were users caring about the file names? > > > > > was a huge burden or even a moderate one then that might be an argument > > > > > against, but it hardly sounds like it will be given Robert's thorough > > > > > analysis so far and the (admittedly not complete, but not that far from > > > > > it based on the DB OID review) proposed patch. > > > > > > > > I am find to add it if it is minor, but I want to see the calculus of > > > > its value vs complexity, which I have not seen spelled out. > > > > > > I feel that this, along with the prior discussions, spells it out > > > sufficiently given the patch's complexity looks to be reasonably minor > > > and very similar to the existing things that pg_upgrade already does. > > > Had pg_upgrade done this in the first place, I don't think there would > > > have been nearly this amount of discussion about it. > > > > Well, there is a reason pg_upgrade didn't initially do this --- because > > it adds complexity, and potentially makes future changes to pg_upgrade > > necessary if the server behavior changes. > > I have a very hard time seeing what changes might happen in the server > in this space that wouldn't have an impact on pg_upgrade, with or > without this. I don't know, but I have to ask since I can't know the future, so any "preseration" has to be studied. > > I am not saying this change is wrong, but I think the reasons need to be > > stated in this thread, rather than just moving forward. > > Ok, they've been stated and it seems to at least Robert and myself that > this is worthwhile to at least continue through to a concluded patch, > after which we can contemplate that patch's complexity against these > reasons. OK, that works for me. What bothers me is that the Desirability of this changes has not be clearly stated in this thread. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 17:20 Robert Haas <[email protected]> parent: Bruce Momjian <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-08-26 17:20 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Stephen Frost <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 12:51 PM Bruce Momjian <[email protected]> wrote: > I just don't want to add requirements/complexity to pg_upgrade without > clearly stated reasons because future database changes will need to > honor this new preservation behavior. Well, I agree that it's good to have reasons clearly stated and I hope that at this point you agree that they have been. Whether you agree with them is another question, but I hope you at least agree that they have been stated. As far as the other part of your concern, what I think makes this change pretty safe is that we are preserving more things rather than fewer. I can imagine some server behavior depending on something being the same between the old and the new clusters, but it is harder to imagine a dependency on something not being preserved. For example, we know that the OIDs of pg_type rows have to be the same in the old and new cluster because arrays are stored on disk with the type OIDs included. Therefore those need to be preserved. If in the future we changed things so that arrays - and other container types - did not include the type OIDs in the on-disk representation, then perhaps it would no longer be necessary to preserve the OIDs of pg_type rows across a pg_upgrade. However, it would not be harmful to do so. It just might not be required. So I think this proposed change is in the safe direction. If relfilenodes were currently preserved and we wanted to make them not be preserved, then I think you would be quite right to say "whoa, whoa, that could be a problem." Indeed it could. If anyone then in the future wanted to introduce a dependency on them staying the same, they would have a problem. However, nothing in the server itself can care about relfilenodes - or anything else - being *different* across a pg_upgrade. The whole point of pg_upgrade is to make it feel like you have the same database after you run it as you did before you ran it, even though under the hood a lot of surgery has been done. Barring bugs, you can never be sad about there being too LITTLE difference between the post-upgrade database and the pre-upgrade database. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 17:24 Stephen Frost <[email protected]> parent: Bruce Momjian <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Stephen Frost @ 2021-08-26 17:24 UTC (permalink / raw) To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> Greetings, * Bruce Momjian ([email protected]) wrote: > On Thu, Aug 26, 2021 at 01:03:54PM -0400, Stephen Frost wrote: > > Yes, we're talking about either incremental (or perhaps differential) > > backup where only the files which are actually different would be backed > > up. Just like with PG, I can't provide any complete guarantees that > > we'd be able to actually make this possible after a major version with > > pgBackRest with this change, but it definitely isn't possible *without* > > this change. I can't see any reason why we wouldn't be able to do a > > checksum-based incremental backup though (which would be *much* faster > > than a regular backup) once this change is made and have that be a > > reliable and trustworthy backup. I'd want to think about it more and > > discuss it with David in some detail before saying if we could maybe > > perform a timestamp-based incremental backup (without checksum'ing the > > files, as we do in normal situations), but that would really just be a > > bonus. > > Well, it would be nice to know exactly how it would help pgBackRest if > that is one of the reasons we are adding this feature. pgBackRest keeps a manifest for every file in the PG data directory that is backed up and we identify that file by the filename. Further, we calculate a checksum for every file. If the filenames didn't change then we'd be able to compare the file in the new cluster against the file and checksum in the manifest in order to be able to perform the incremental/differential backup. We don't store the inodes in the manifest though, and we don't have any concept of looking at multiple data directories at the same time or anything like that (which would also mean that the old data directory would have to be kept around for that to even work, which seems like a good bit of additional complication and risk that someone might start up the old cluster by accident..). That's how it'd be very helpful to pgBackRest for the filenames to be preserved across pg_upgrade's. > > > > > As far as TDE, I haven't seen any concrete plan for that, so why add > > > > > this code for that reason? > > > > > > > > That this would help with TDE (of which there seems little doubt...) is > > > > an additional benefit to this. Specifically, taking the existing work > > > > that's already been done to allow block-by-block encryption and > > > > adjusting it for AES-XTS and then using the db-dir+relfileno+block > > > > number as the IV, just like many disk encryption systems do, avoids the > > > > concerns that were brought up about using LSN for the IV with CTR and > > > > it's certainly not difficult to do, but it does depend on this change. > > > > This was all discussed previously and it sure looks like a sensible > > > > approach to use that mirrors what many other systems already do > > > > successfully. > > > > > > Well, I would think we would not add this for TDE until we were sure > > > someone was working on adding TDE. > > > > That this would help with TDE is what I'd consider an added bonus. > > Not if we have no plans to implement TDE, which was my point. Why not > wait to see if we are actually going to implement TDE rather than adding > it now. It is just so obvious, why do I have to state this? There's been multiple years of effort put into implementing TDE and I'm sure hopeful that it continues as I'm trying to put effort into moving it forward myself. I'm a bit baffled by the idea that we're just suddenly going to stop putting effort into TDE as it is brought up time and time again by clients that I've talked to as one of the few reasons they haven't moved to PG yet- I can't believe that hasn't been experienced by folks at other organizations too, I mean, there's people maintaining forks of PG specifically for TDE ... Seems like maybe we were both seeing something as obvious to the other that wasn't actually the case. > > > > > > make this required, general improved sanity when working with pg_upgrade > > > > > > is frankly a benefit in its own right too...). If the additional code > > > > > > > > > > How? I am not aware of any advantage except cosmetic. > > > > > > > > Having to resort to matching up inode numbers between the two clusters > > > > after a pg_upgrade to figure out what files are actually the same > > > > underneath is a pain that goes beyond just cosmetics imv. Removing that > > > > additional level that admins, and developers for that matter, have to go > > > > through would be a nice improvement on its own. > > > > > > OK, I was just not aware anyone did that, since I have never hard anyone > > > complain about it before. > > > > I've certainly done it and I'd be kind of surprised if others haven't, > > but I've also played a lot with pg_dump in various modes, so perhaps > > that's not a great representation. I've definitely had to explain to > > clients why there's a whole different set of filenames after a > > pg_upgrade and why that is the case for an 'in place' upgrade before > > too. > > Uh, so I guess I am right that few people have mentioned this in the > past. Why were users caring about the file names? This is a bit baffling to me. Users and admins certainly care about what files their data is stored in and knowing how to find them. Covering the data directory structure is a commonly asked for part of the training that I regularly do for clients. > > > > > > was a huge burden or even a moderate one then that might be an argument > > > > > > against, but it hardly sounds like it will be given Robert's thorough > > > > > > analysis so far and the (admittedly not complete, but not that far from > > > > > > it based on the DB OID review) proposed patch. > > > > > > > > > > I am find to add it if it is minor, but I want to see the calculus of > > > > > its value vs complexity, which I have not seen spelled out. > > > > > > > > I feel that this, along with the prior discussions, spells it out > > > > sufficiently given the patch's complexity looks to be reasonably minor > > > > and very similar to the existing things that pg_upgrade already does. > > > > Had pg_upgrade done this in the first place, I don't think there would > > > > have been nearly this amount of discussion about it. > > > > > > Well, there is a reason pg_upgrade didn't initially do this --- because > > > it adds complexity, and potentially makes future changes to pg_upgrade > > > necessary if the server behavior changes. > > > > I have a very hard time seeing what changes might happen in the server > > in this space that wouldn't have an impact on pg_upgrade, with or > > without this. > > I don't know, but I have to ask since I can't know the future, so any > "preseration" has to be studied. We can gain, perhaps, some insight looking into the past and that seems to indicate that this is certainly a very stable part of the server code in the first place, which would imply that it's unlikely that there'll be much need to adjust this code in the future in the first place. > > > I am not saying this change is wrong, but I think the reasons need to be > > > stated in this thread, rather than just moving forward. > > > > Ok, they've been stated and it seems to at least Robert and myself that > > this is worthwhile to at least continue through to a concluded patch, > > after which we can contemplate that patch's complexity against these > > reasons. > > OK, that works for me. What bothers me is that the Desirability of this > changes has not be clearly stated in this thread. I hope that this email and the many many prior ones have gotten across the desirability of the change. Thanks, Stephen Attachments: [application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 17:34 Bruce Momjian <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 17:34 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 01:20:38PM -0400, Robert Haas wrote: > So I think this proposed change is in the safe direction. If > relfilenodes were currently preserved and we wanted to make them not > be preserved, then I think you would be quite right to say "whoa, > whoa, that could be a problem." Indeed it could. If anyone then in the > future wanted to introduce a dependency on them staying the same, they > would have a problem. However, nothing in the server itself can care > about relfilenodes - or anything else - being *different* across a > pg_upgrade. The whole point of pg_upgrade is to make it feel like you > have the same database after you run it as you did before you ran it, > even though under the hood a lot of surgery has been done. Barring > bugs, you can never be sad about there being too LITTLE difference > between the post-upgrade database and the pre-upgrade database. Yes, this makes sense, and it is good we have stated the possible benefits now: * pgBackRest * pg_upgrade diagnostics * TDE (maybe) We can eventually evaluate the value of this based on those items. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-08-26 17:37 Bruce Momjian <[email protected]> parent: Stephen Frost <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bruce Momjian @ 2021-08-26 17:37 UTC (permalink / raw) To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Shruthi Gowda <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]> On Thu, Aug 26, 2021 at 01:24:46PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian ([email protected]) wrote: > > On Thu, Aug 26, 2021 at 01:03:54PM -0400, Stephen Frost wrote: > > > Yes, we're talking about either incremental (or perhaps differential) > > > backup where only the files which are actually different would be backed > > > up. Just like with PG, I can't provide any complete guarantees that > > > we'd be able to actually make this possible after a major version with > > > pgBackRest with this change, but it definitely isn't possible *without* > > > this change. I can't see any reason why we wouldn't be able to do a > > > checksum-based incremental backup though (which would be *much* faster > > > than a regular backup) once this change is made and have that be a > > > reliable and trustworthy backup. I'd want to think about it more and > > > discuss it with David in some detail before saying if we could maybe > > > perform a timestamp-based incremental backup (without checksum'ing the > > > files, as we do in normal situations), but that would really just be a > > > bonus. > > > > Well, it would be nice to know exactly how it would help pgBackRest if > > that is one of the reasons we are adding this feature. > > pgBackRest keeps a manifest for every file in the PG data directory that > is backed up and we identify that file by the filename. Further, we > calculate a checksum for every file. If the filenames didn't change > then we'd be able to compare the file in the new cluster against the > file and checksum in the manifest in order to be able to perform the > incremental/differential backup. We don't store the inodes in the > manifest though, and we don't have any concept of looking at multiple > data directories at the same time or anything like that (which would > also mean that the old data directory would have to be kept around for > that to even work, which seems like a good bit of additional > complication and risk that someone might start up the old cluster by > accident..). > > That's how it'd be very helpful to pgBackRest for the filenames to be > preserved across pg_upgrade's. OK, that is clear. > > > > > > As far as TDE, I haven't seen any concrete plan for that, so why add > > > > > > this code for that reason? > > > > > > > > > > That this would help with TDE (of which there seems little doubt...) is > > > > > an additional benefit to this. Specifically, taking the existing work > > > > > that's already been done to allow block-by-block encryption and > > > > > adjusting it for AES-XTS and then using the db-dir+relfileno+block > > > > > number as the IV, just like many disk encryption systems do, avoids the > > > > > concerns that were brought up about using LSN for the IV with CTR and > > > > > it's certainly not difficult to do, but it does depend on this change. > > > > > This was all discussed previously and it sure looks like a sensible > > > > > approach to use that mirrors what many other systems already do > > > > > successfully. > > > > > > > > Well, I would think we would not add this for TDE until we were sure > > > > someone was working on adding TDE. > > > > > > That this would help with TDE is what I'd consider an added bonus. > > > > Not if we have no plans to implement TDE, which was my point. Why not > > wait to see if we are actually going to implement TDE rather than adding > > it now. It is just so obvious, why do I have to state this? > > There's been multiple years of effort put into implementing TDE and I'm > sure hopeful that it continues as I'm trying to put effort into moving > it forward myself. I'm a bit baffled by the idea that we're just Well, this is the first time I am hearing this publicly. > suddenly going to stop putting effort into TDE as it is brought up time > and time again by clients that I've talked to as one of the few reasons > they haven't moved to PG yet- I can't believe that hasn't been > experienced by folks at other organizations too, I mean, there's people > maintaining forks of PG specifically for TDE ... Agreed. > > > I've certainly done it and I'd be kind of surprised if others haven't, > > > but I've also played a lot with pg_dump in various modes, so perhaps > > > that's not a great representation. I've definitely had to explain to > > > clients why there's a whole different set of filenames after a > > > pg_upgrade and why that is the case for an 'in place' upgrade before > > > too. > > > > Uh, so I guess I am right that few people have mentioned this in the > > past. Why were users caring about the file names? > > This is a bit baffling to me. Users and admins certainly care about > what files their data is stored in and knowing how to find them. > Covering the data directory structure is a commonly asked for part of > the training that I regularly do for clients. I just never thought people cared about the file names, since I have never heard a complaint about how pg_upgrade works all these years. > > > I have a very hard time seeing what changes might happen in the server > > > in this space that wouldn't have an impact on pg_upgrade, with or > > > without this. > > > > I don't know, but I have to ask since I can't know the future, so any > > "preseration" has to be studied. > > We can gain, perhaps, some insight looking into the past and that seems > to indicate that this is certainly a very stable part of the server code > in the first place, which would imply that it's unlikely that there'll > be much need to adjust this code in the future in the first place. Good, it have to ask. > > > > I am not saying this change is wrong, but I think the reasons need to be > > > > stated in this thread, rather than just moving forward. > > > > > > Ok, they've been stated and it seems to at least Robert and myself that > > > this is worthwhile to at least continue through to a concluded patch, > > > after which we can contemplate that patch's complexity against these > > > reasons. > > > > OK, that works for me. What bothers me is that the Desirability of this > > changes has not be clearly stated in this thread. > > I hope that this email and the many many prior ones have gotten across > the desirability of the change. Yes, I think we are in a better position now to evaluate this. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion. ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-09-22 19:06 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 2 siblings, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-09-22 19:06 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Aug 24, 2021 at 2:27 AM Robert Haas <[email protected]> wrote: > It's pretty clear from the discussion, I think, that the database OID > one is going to need rework to be considered. > > Regarding the other one: > > - The comment in binary_upgrade_set_pg_class_oids() is still not > accurate. You removed the sentence which says "Indexes cannot have > toast tables, so we need not make this probe in the index code path" > but the immediately preceding sentence is still inaccurate in at least > two ways. First, it only talks about tables, but the code now applies > to indexes. Second, it only talks about OIDs, but now also deals with > refilenodes. It's really important to fully update every comment that > might be affected by your changes! The comment is updated. > - The SQL query in that function isn't completely correct. There is a > left join from pg_class to pg_index whose ON clause includes > "c.reltoastrelid = i.indrelid AND i.indisvalid." The reason it's > likely that is because it is possible, in corner cases, for a TOAST > table to have multiple TOAST indexes. I forget exactly how that > happens, but I think it might be like if a REINDEX CONCURRENTLY on the > TOAST table fails midway through, or something of that sort. Now if > that happens, the LEFT JOIN you added is going to cause the output to > contain multiple rows, because you didn't replicate the i.indisvalid > condition into that ON clause. And then it will fail. Apparently we > don't have a pg_upgrade test case for this scenario; we probably > should. Actually what I think would be even better than putting > i.indisvalid into that ON clause would be to join off of i.indrelid > rather than c.reltoastrelid. The SQL query will not result in duplicate rows because the first join filters the duplicate rows if any with the on clause ' i.indisvalid' on it. The result of the first join is further left joined with pg_class and pg_class will not have duplicate rows for a given oid. > - The code that decodes the various columns of this query does so in a > slightly different order than the query itself. It would be better to > make it match. Perhaps put relkind first in both cases. I might also > think about trying to make the column naming a bit more consistent, > e.g. relkind, relfilenode, toast_oid, toast_relfilenode, > toast_index_oid, toast_index_relfilenode. Fixed. > - In heap_create(), the wording of the error messages is not quite > consistent. You have "relfilenode value not set when in binary upgrade > mode", "toast relfilenode value not set when in binary upgrade mode", > and "pg_class index relfilenode value not set when in binary upgrade > mode". Why does the last one mention pg_class when the other two > don't? The error message is made consistent. This code chuck is moved to a different place as a part of another review comment fix. > - The code in heap_create() now has no comments whatsoever, which is a > shame, because it's actually kind of a tricky bit of logic. Someone > might wonder why we override the relfilenode inside that function > instead of doing it at the same places where we absorb > binary_upgrade_next_{heap,index,toast}_pg_class_oid and the passing > down the relfilenode. I think the answer is that passing down the > relfilenode from the caller would result in storage not actually being > created, whereas in this case we want it to be created but just with > the value we specify, and the reason we want that is because we need > later DDL that happens after these statements but before the old > cluster's relations are moved over to execute successfully, which it > won't if the storage is altogether absent. > However, that raises the question of whether this patch has even got > the basic design right. Maybe we ought to actually be absorbing the > relfilenode setting at the same places where we're doing so for the > OID, and then passing an additional parameter to heap_create() like > bool suppress_storage or something like that. Maybe, taking it even > further, we ought to be changing the signatures of > binary_upgrade_next_heap_pg_class_oid and friends to be two-argument > functions, and pass down the OID and the relfilenode in the same call, > rather than calling two separate functions. I'm not so much concerned > about the cost of calling two functions as the potential for > confusion. I'm not honestly sure that either of these changes are the > right thing to do, but I am pretty strongly inclined to do at least > the first part - trying to absorb reloid and relfilenode in the same > places. If we're not going to do that we certainly need to explain why > we're doing it the way we are in the comments. As per your suggestion, reloid and relfilenode are absorbed in the same place. An additional parameter called 'suppress_storage' is passed to heap_create() which indicates whether or not to create the storage when the caller passed a valid relfilenode. I did not make the changes to set the oid and relfilenode in the same call. I feel the uniformity w.r.t the other function signatures in pg_upgrade_support.c will be lost because currently each function sets only one attribute. Also, renaming the applicable function names to represent that they set both oid and relfilenode will make the function name even longer. We may opt to not include the relfilenode in the function name instead use a generic name like binary_upgrade_set_next_xxx_pg_class_oid() but then we will end up with some functions that set two attributes and some functions that set one attribute. > It's not really this patch's fault, but it would sure be nice if we > had some better testing for this area. Suppose this patch somehow > changed nothing from the present behavior. How would we know? Or > suppose it managed to somehow set all the relfilenodes in the new > cluster to random values rather than the intended one? There's no > automated testing that would catch any of that, and it's not obvious > how it could be added to test.sh. I suppose what we really need to do > at some point is rewrite that as a TAP test, but that seems like a > separate project from this patch. I have verified the table, index, toast table and toast index relfilenode and DB oid in old and new cluster manually and it is working as expected. I have also attached the patch to preserve the DB oid. As discussed, template0 will be created with a fixed oid during initdb. I am using OID 2 for template0. Even though oid 2 is already in use for the 'pg_am' catalog I see no harm in using it for template0 DB because oid doesn’t have to be unique across the database - it has to be unique for the particular catalog table. Kindly let me know if I am missing something? Apparently, if we did decide to pick an unused oid for template0 then I see a challenge in removing that oid from the unused oid list. I could not come up with a feasible solution for handling it. Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v3-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch (26.9K, ../../CAASxf_OjBEVrkWCvwKkoT8nyYQ6Uk1MHDhhVSyZ_EXCQG7J8Dg@mail.gmail.com/2-v3-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch) download | inline diff: From 2c00a37c4a3a00bb765232fea64e2bc02bfc8129 Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Wed, 22 Sep 2021 23:33:41 +0530 Subject: [PATCH v3 1/2] Preserve relfilenode and tablespace OID in pg_upgrade The patch aims to preserve the OIDs of relfilenode and tablespace during binary upgrade so that the OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- src/backend/bootstrap/bootparse.y | 3 +- src/backend/catalog/heap.c | 72 ++++++++++++-- src/backend/catalog/index.c | 22 ++++- src/backend/commands/tablespace.c | 17 +++- src/backend/utils/adt/pg_upgrade_support.c | 44 +++++++++ src/bin/pg_dump/pg_dump.c | 104 +++++++++++++-------- src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/pg_upgrade.c | 13 +-- src/include/catalog/binary_upgrade.h | 5 + src/include/catalog/heap.h | 3 +- src/include/catalog/pg_proc.dat | 16 ++++ .../spgist_name_ops/expected/spgist_name_ops.out | 12 ++- 12 files changed, 252 insertions(+), 62 deletions(-) diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y index 5fcd004..6560b19 100644 --- a/src/backend/bootstrap/bootparse.y +++ b/src/backend/bootstrap/bootparse.y @@ -212,7 +212,8 @@ Boot_CreateStmt: mapped_relation, true, &relfrozenxid, - &relminmxid); + &relminmxid, + true); elog(DEBUG4, "bootstrap relation created"); } else diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 83746d3..851a4e2 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -286,7 +288,10 @@ SystemAttributeByName(const char *attname) * * Note API change: the caller must now always provide the OID * to use for the relation. The relfilenode may (and, normally, - * should) be left unspecified. + * should) be left unspecified unless explicitly set in binary-upgrade mode. + * + * suppress_storage indicates whether or not to create the storage if the + * caller passed a valid relfilenode. * * rel->rd_rel is initialized by RelationBuildLocalRelation, * and is mostly zeroes at return. @@ -306,7 +311,8 @@ heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid) + MultiXactId *relminmxid, + bool suppress_storage) { bool create_storage; Relation rel; @@ -367,16 +373,22 @@ heap_create(const char *relname, } /* - * Decide whether to create storage. If caller passed a valid relfilenode, - * storage is already created, so don't do it here. Also don't create it - * for relkinds without physical storage. + * Decide whether to create storage. If suppress_storage is true and a + * valid relfilenode is passed, storage is already created, so don't do it + * here. If suppress_storage is false and relfilenode is valid, create the + * storage with the specified relfilenode. If relfilenode is unspecified + * by the caller then create the storage with oid same as relid. Also, + * don't create it for relkinds without physical storage. */ - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + if (!RELKIND_HAS_STORAGE(relkind) || (OidIsValid(relfilenode) && suppress_storage)) create_storage = false; else { create_storage = true; - relfilenode = relid; + + /* Initialize relfilenode to relid */ + if (!OidIsValid(relfilenode)) + relfilenode = relid; } /* @@ -1168,8 +1180,12 @@ heap_create_with_catalog(const char *relname, Oid existing_relid; Oid old_type_oid; Oid new_type_oid; + + /* By default set to InvalidOid unless overridden by binary-upgrade */ + Oid relfilenode = InvalidOid; TransactionId relfrozenxid; MultiXactId relminmxid; + bool suppress_storage = true; pg_class_desc = table_open(RelationRelationId, RowExclusiveLock); @@ -1230,7 +1246,7 @@ heap_create_with_catalog(const char *relname, */ if (!OidIsValid(relid)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade && (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || relkind == RELKIND_VIEW || relkind == RELKIND_MATVIEW || @@ -1244,7 +1260,27 @@ heap_create_with_catalog(const char *relname, relid = binary_upgrade_next_heap_pg_class_oid; binary_upgrade_next_heap_pg_class_oid = InvalidOid; + + /* + * Override the relfilenode and set the suppress_storage flag to + * false so that the storage gets created with the specified + * relfilenode during heap_create(). + */ + if (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || + relkind == RELKIND_MATVIEW) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + + suppress_storage = false; + } } + /* There might be no TOAST table, so we have to test for it. */ else if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_toast_pg_class_oid) && @@ -1252,6 +1288,21 @@ heap_create_with_catalog(const char *relname, { relid = binary_upgrade_next_toast_pg_class_oid; binary_upgrade_next_toast_pg_class_oid = InvalidOid; + + /* + * Override the toast relfilenode and set the suppress_storage + * flag to false so that the storage gets created with the + * specified relfilenode during heap_create(). + */ + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; + + suppress_storage = false; } else relid = GetNewRelFileNode(reltablespace, pg_class_desc, @@ -1294,7 +1345,7 @@ heap_create_with_catalog(const char *relname, relnamespace, reltablespace, relid, - InvalidOid, + relfilenode, accessmtd, tupdesc, relkind, @@ -1303,7 +1354,8 @@ heap_create_with_catalog(const char *relname, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + suppress_storage); Assert(relid == RelationGetRelid(new_rel_desc)); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 26bfa74..56a4ef9 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -86,6 +86,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system @@ -732,6 +733,7 @@ index_create(Relation heapRelation, char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; + bool suppress_storage = true; /* constraint flags can only be set when a constraint is requested */ Assert((constr_flags == 0) || @@ -903,7 +905,7 @@ index_create(Relation heapRelation, */ if (!OidIsValid(indexRelationId)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { if (!OidIsValid(binary_upgrade_next_index_pg_class_oid)) @@ -913,6 +915,21 @@ index_create(Relation heapRelation, indexRelationId = binary_upgrade_next_index_pg_class_oid; binary_upgrade_next_index_pg_class_oid = InvalidOid; + + /* + * Overide the relfilenode and set the suppress_storage to false + * so that the storage gets created with the specified relfilenode + * during heap_create(). + */ + if ((relkind == RELKIND_INDEX) && + (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode))) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("index relfilenode value not set when in binary upgrade mode"))); + relFileNode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; + + suppress_storage = false; } else { @@ -939,7 +956,8 @@ index_create(Relation heapRelation, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + suppress_storage); Assert(relfrozenxid == InvalidTransactionId); Assert(relminmxid == InvalidMultiXactId); diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 4b96eec..51ffa97 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -88,6 +88,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -335,8 +336,20 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, - Anum_pg_tablespace_oid); + if (IsBinaryUpgrade) + { + /* Use binary-upgrade override for tablespace oid */ + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->tablespacename)); diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7..135c382 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,17 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); @@ -85,6 +96,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_heap_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -96,6 +118,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_index_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -107,6 +140,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { Oid enumoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index a485fb2..b148919 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4693,73 +4693,101 @@ binary_upgrade_set_pg_class_oids(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index) { + PQExpBuffer upgrade_query = createPQExpBuffer(); + PGresult *upgrade_res; + Oid relfilenode; + Oid toast_oid; + Oid toast_relfilenode; + char relkind; + Oid toast_index_oid; + Oid toast_index_relfilenode; + + /* + * Preserve the OID and relfilenode of the table, table's index, table's + * toast table and toast table's index if any. + * + * One complexity is that the current table definition might not require + * the creation of a TOAST table, but the old database might have a TOAST + * table that was created earlier, before some wide columns were dropped. + * By setting the TOAST oid we force creation of the TOAST heap and index + * by the new backend, so we can copy the files during binary upgrade + * without worrying about this case. + */ + appendPQExpBuffer(upgrade_query, + "SELECT c.relkind, c.relfilenode, c.reltoastrelid, ct.relfilenode AS toast_relfilenode, i.indexrelid, cti.relfilenode AS toast_index_relfilenode " + "FROM pg_catalog.pg_class c LEFT JOIN " + "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " + "WHERE c.oid = '%u'::pg_catalog.oid;", + pg_class_oid); + + upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + + relkind = *PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relkind")); + + relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode"))); + toast_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "reltoastrelid"))); + toast_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_relfilenode"))); + toast_index_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "indexrelid"))); + toast_index_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_index_relfilenode"))); + appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { - PQExpBuffer upgrade_query = createPQExpBuffer(); - PGresult *upgrade_res; - Oid pg_class_reltoastrelid; - char pg_class_relkind; - Oid pg_index_indexrelid; - appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); - /* - * Preserve the OIDs of the table's toast table and index, if any. - * Indexes cannot have toast tables, so we need not make this probe in - * the index code path. - * - * One complexity is that the current table definition might not - * require the creation of a TOAST table, but the old database might - * have a TOAST table that was created earlier, before some wide - * columns were dropped. By setting the TOAST oid we force creation - * of the TOAST heap and index by the new backend, so we can copy the - * files during binary upgrade without worrying about this case. - */ - appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, c.relkind, i.indexrelid " - "FROM pg_catalog.pg_class c LEFT JOIN " - "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " - "WHERE c.oid = '%u'::pg_catalog.oid;", - pg_class_oid); - - upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); - - pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "reltoastrelid"))); - pg_class_relkind = *PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "relkind")); - pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "indexrelid"))); + /* Not every relation has storage. */ + if (OidIsValid(relfilenode)) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); /* * In a pre-v12 database, partitioned tables might be marked as having * toast tables, but we should ignore them if so. */ - if (OidIsValid(pg_class_reltoastrelid) && - pg_class_relkind != RELKIND_PARTITIONED_TABLE) + if (OidIsValid(toast_oid) && + relkind != RELKIND_PARTITIONED_TABLE) { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_class_reltoastrelid); + toast_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n", + toast_relfilenode); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_index_indexrelid); + toast_index_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + toast_index_relfilenode); } PQclear(upgrade_res); destroyPQExpBuffer(upgrade_query); } else + { + /* Preserve the OID and relfilenode of the index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); } diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c291017..618187c 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1265,6 +1265,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd7..acaf343 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,13 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because we want the filenames + * to match between the old and new cluster. + * + * We control assignment of pg_tablespace.oid because we want the oid to match + * between the old and new cluster. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7..4ba5748 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,19 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index 6ce480b..2008758 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -59,7 +59,8 @@ extern Relation heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid); + MultiXactId *relminmxid, + bool suppress_storage); extern Oid heap_create_with_catalog(const char *relname, Oid relnamespace, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index d068d65..cd0a20d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11036,6 +11036,22 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4544', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_relfilenode' }, +{ oid => '4545', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_relfilenode' }, +{ oid => '4546', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_relfilenode' }, +{ oid => '4547', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddce..1ee65ed 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -52,14 +52,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -97,14 +101,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 1.8.3.1 [application/octet-stream] v3-0002-Preserve-database-OIDs-in-pg_upgrade.patch (7.9K, ../../CAASxf_OjBEVrkWCvwKkoT8nyYQ6Uk1MHDhhVSyZ_EXCQG7J8Dg@mail.gmail.com/3-v3-0002-Preserve-database-OIDs-in-pg_upgrade.patch) download | inline diff: From dc0a50b5513c38bfa0d1356c370475edec0b49cf Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Wed, 22 Sep 2021 23:36:41 +0530 Subject: [PATCH v3 2/2] Preserve database OIDs in pg_upgrade The patch aims to preserve the database OIDs during binary upgrade so that the database OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- doc/src/sgml/ref/create_database.sgml | 15 ++++++++++++++- src/backend/commands/dbcommands.c | 35 ++++++++++++++++++++++++++++++----- src/backend/parser/gram.y | 3 ++- src/bin/initdb/initdb.c | 21 +++++++++++++-------- src/bin/pg_dump/pg_dump.c | 16 ++++++++++++++-- src/bin/psql/tab-complete.c | 2 +- 6 files changed, 74 insertions(+), 18 deletions(-) diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml index 41cb406..3865a96 100644 --- a/doc/src/sgml/ref/create_database.sgml +++ b/doc/src/sgml/ref/create_database.sgml @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> [ TABLESPACE [=] <replaceable class="parameter">tablespace_name</replaceable> ] [ ALLOW_CONNECTIONS [=] <replaceable class="parameter">allowconn</replaceable> ] [ CONNECTION LIMIT [=] <replaceable class="parameter">connlimit</replaceable> ] - [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] ] + [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] + [ OID [=] <replaceable class="parameter">db_oid</replaceable> ] ] </synopsis> </refsynopsisdiv> @@ -203,6 +204,18 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> </para> </listitem> </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">oid</replaceable></term> + <listitem> + <para> + The object identifier with which the database gets created. + The oid value should be greater than 16383. CREATE DATABASE fails if a + database with specified oid already exists. + </para> + </listitem> + </varlistentry> + </variablelist> <para> diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 029fab4..0251157 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -117,7 +117,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) HeapTuple tuple; Datum new_record[Natts_pg_database]; bool new_record_nulls[Natts_pg_database]; - Oid dboid; + Oid dboid = InvalidOid; Oid datdba; ListCell *option; DefElem *dtablespacename = NULL; @@ -217,6 +217,27 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) errhint("Consider using tablespaces instead."), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "oid") == 0) + { + dboid = defGetInt32(defel); + + /* + * Throw an error if the user specified oid < FirstNormalObjectId for + * creating the database. However, we need to allow creating database + * with oid < FirstNormalObjectId for below cases: + * 1. creating template0 with fixed oid during initdb + * 2. creating databases with oids from the old cluster during binary + * upgrade. + */ + if ((dboid < FirstNormalObjectId) && + (strcmp(dbname, "template0") != 0) && + (!IsBinaryUpgrade)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("Invalid value for option \"%s\"", defel->defname), + errhint("Consider using a value greater than %u.", + (FirstNormalObjectId - 1))); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* Select an OID for the new database if is not explicitly configured. */ + if (!OidIsValid(dboid)) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e3068a3..b265959 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -687,7 +687,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OF OFF OFFSET OID OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -10254,6 +10254,7 @@ createdb_opt_name: | OWNER { $$ = pstrdup($1); } | TABLESPACE { $$ = pstrdup($1); } | TEMPLATE { $$ = pstrdup($1); } + | OID { $$ = pstrdup($1); } ; /* diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index 1ed4808..a5d88f1 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -1836,15 +1836,12 @@ static void make_template0(FILE *cmdfd) { const char *const *line; - static const char *const template0_setup[] = { - "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false;\n\n", - /* - * We use the OID of template0 to determine datlastsysoid - */ - "UPDATE pg_database SET datlastsysoid = " - " (SELECT oid FROM pg_database " - " WHERE datname = 'template0');\n\n", + /* + * Create template0 database with fixed OID 2 + */ + static const char *const template0_setup[] = { + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID 2;\n\n", /* * Explicitly revoke public create-schema and create-temp-table @@ -1877,6 +1874,14 @@ make_postgres(FILE *cmdfd) static const char *const postgres_setup[] = { "CREATE DATABASE postgres;\n\n", "COMMENT ON DATABASE postgres IS 'default administrative connection database';\n\n", + + /* + * We use the OID of postgres to determine datlastsysoid + */ + "UPDATE pg_database SET datlastsysoid = " + " (SELECT oid FROM pg_database " + " WHERE datname = 'postgres');\n\n", + NULL }; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b148919..47c657e 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -2957,8 +2957,20 @@ dumpDatabase(Archive *fout) * are left to the DATABASE PROPERTIES entry, so that they can be applied * after reconnecting to the target DB. */ - appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", - qdatname); + if (dopt->binary_upgrade) + { + /* + * Make sure that binary upgrade propogate the database OID to the new + * cluster + */ + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0 OID = %u", + qdatname, dbCatId.oid); + } + else + { + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", + qdatname); + } if (strlen(encoding) > 0) { appendPQExpBufferStr(creaQry, " ENCODING = "); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 5cd5838..396b4c5 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2516,7 +2516,7 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("OWNER", "TEMPLATE", "ENCODING", "TABLESPACE", "IS_TEMPLATE", "ALLOW_CONNECTIONS", "CONNECTION LIMIT", - "LC_COLLATE", "LC_CTYPE", "LOCALE"); + "LC_COLLATE", "LC_CTYPE", "LOCALE", "OID"); else if (Matches("CREATE", "DATABASE", MatchAny, "TEMPLATE")) COMPLETE_WITH_QUERY(Query_for_list_of_template_databases); -- 1.8.3.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-09-23 19:13 Robert Haas <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-09-23 19:13 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Wed, Sep 22, 2021 at 3:07 PM Shruthi Gowda <[email protected]> wrote: > > - The comment in binary_upgrade_set_pg_class_oids() is still not > > accurate. You removed the sentence which says "Indexes cannot have > > toast tables, so we need not make this probe in the index code path" > > but the immediately preceding sentence is still inaccurate in at least > > two ways. First, it only talks about tables, but the code now applies > > to indexes. Second, it only talks about OIDs, but now also deals with > > refilenodes. It's really important to fully update every comment that > > might be affected by your changes! > > The comment is updated. Looks good. > The SQL query will not result in duplicate rows because the first join > filters the duplicate rows if any with the on clause ' i.indisvalid' > on it. The result of the first join is further left joined with pg_class and > pg_class will not have duplicate rows for a given oid. Oh, you're right. My mistake. > As per your suggestion, reloid and relfilenode are absorbed in the same place. > An additional parameter called 'suppress_storage' is passed to heap_create() > which indicates whether or not to create the storage when the caller > passed a valid relfilenode. I find it confusing to have both suppress_storage and create_storage with one basically as the negation of the other. To avoid that sort of thing I generally have a policy that variables and options should say whether something should happen, rather than whether it should be prevented from happening. Sometimes there are good reasons - such as strong existing precedent - to deviate from this practice but I think it's good to follow when possible. So my proposal is to always have create_storage and never suppress_storage, and if some function needs to adjust the value of create_storage that was passed to it then OK. > I did not make the changes to set the oid and relfilenode in the same call. > I feel the uniformity w.r.t the other function signatures in > pg_upgrade_support.c will be lost because currently each function sets > only one attribute. > Also, renaming the applicable function names to represent that they > set both oid and relfilenode will make the function name even longer. > We may opt to not include the relfilenode in the function name instead > use a generic name like binary_upgrade_set_next_xxx_pg_class_oid() but > then > we will end up with some functions that set two attributes and some > functions that set one attribute. OK. > I have also attached the patch to preserve the DB oid. As discussed, > template0 will be created with a fixed oid during initdb. I am using > OID 2 for template0. Even though oid 2 is already in use for the > 'pg_am' catalog I see no harm in using it for template0 DB because oid > doesn’t have to be unique across the database - it has to be unique > for the particular catalog table. Kindly let me know if I am missing > something? > Apparently, if we did decide to pick an unused oid for template0 then > I see a challenge in removing that oid from the unused oid list. I > could not come up with a feasible solution for handling it. You are correct that there is no intrinsic reason why the same OID can't be used in various different catalogs. We have a policy of not doing that, though; I'm not clear on the reason. Maybe it'd be OK to deviate from that policy here, but another option would be to simply change the unused_oids script (and maybe some of the others). It already has: my $FirstGenbkiObjectId = Catalog::FindDefinedSymbol('access/transam.h', '..', 'FirstGenbkiObjectId'); push @{$oids}, $FirstGenbkiObjectId; Presumably it could be easily adapted to push the value of some other defined symbol into @{$oids} also, thus making that OID in effect used. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-10-04 16:44 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-10-04 16:44 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Fri, Sep 24, 2021 at 12:44 AM Robert Haas <[email protected]> wrote: > > On Wed, Sep 22, 2021 at 3:07 PM Shruthi Gowda <[email protected]> wrote: > > > - The comment in binary_upgrade_set_pg_class_oids() is still not > > > accurate. You removed the sentence which says "Indexes cannot have > > > toast tables, so we need not make this probe in the index code path" > > > but the immediately preceding sentence is still inaccurate in at least > > > two ways. First, it only talks about tables, but the code now applies > > > to indexes. Second, it only talks about OIDs, but now also deals with > > > refilenodes. It's really important to fully update every comment that > > > might be affected by your changes! > > > > The comment is updated. > > Looks good. > > > The SQL query will not result in duplicate rows because the first join > > filters the duplicate rows if any with the on clause ' i.indisvalid' > > on it. The result of the first join is further left joined with pg_class and > > pg_class will not have duplicate rows for a given oid. > > Oh, you're right. My mistake. > > > As per your suggestion, reloid and relfilenode are absorbed in the same place. > > An additional parameter called 'suppress_storage' is passed to heap_create() > > which indicates whether or not to create the storage when the caller > > passed a valid relfilenode. > > I find it confusing to have both suppress_storage and create_storage > with one basically as the negation of the other. To avoid that sort of > thing I generally have a policy that variables and options should say > whether something should happen, rather than whether it should be > prevented from happening. Sometimes there are good reasons - such as > strong existing precedent - to deviate from this practice but I think > it's good to follow when possible. So my proposal is to always have > create_storage and never suppress_storage, and if some function needs > to adjust the value of create_storage that was passed to it then OK. Sure, I agree. In the latest patch, only 'create_storage' is used. > > I did not make the changes to set the oid and relfilenode in the same call. > > I feel the uniformity w.r.t the other function signatures in > > pg_upgrade_support.c will be lost because currently each function sets > > only one attribute. > > Also, renaming the applicable function names to represent that they > > set both oid and relfilenode will make the function name even longer. > > We may opt to not include the relfilenode in the function name instead > > use a generic name like binary_upgrade_set_next_xxx_pg_class_oid() but > > then > > we will end up with some functions that set two attributes and some > > functions that set one attribute. > > OK. > > > I have also attached the patch to preserve the DB oid. As discussed, > > template0 will be created with a fixed oid during initdb. I am using > > OID 2 for template0. Even though oid 2 is already in use for the > > 'pg_am' catalog I see no harm in using it for template0 DB because oid > > doesn’t have to be unique across the database - it has to be unique > > for the particular catalog table. Kindly let me know if I am missing > > something? > > Apparently, if we did decide to pick an unused oid for template0 then > > I see a challenge in removing that oid from the unused oid list. I > > could not come up with a feasible solution for handling it. > > You are correct that there is no intrinsic reason why the same OID > can't be used in various different catalogs. We have a policy of not > doing that, though; I'm not clear on the reason. Maybe it'd be OK to > deviate from that policy here, but another option would be to simply > change the unused_oids script (and maybe some of the others). It > already has: > > my $FirstGenbkiObjectId = > Catalog::FindDefinedSymbol('access/transam.h', '..', 'FirstGenbkiObjectId'); > push @{$oids}, $FirstGenbkiObjectId; > > Presumably it could be easily adapted to push the value of some other > defined symbol into @{$oids} also, thus making that OID in effect > used. Thanks for the inputs, Robert. In the v4 patch, an unused OID (i.e, 4) is fixed for the template0 and the same is removed from unused oid list. In addition to the review comment fixes, I have removed some code that is no longer needed/doesn't make sense since we preserve the OIDs. Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v4-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch (30.6K, ../../CAASxf_Mj3SSXFmtd9O6sf-+y4PJz+oZ57RAi0bRKKLAOBDcorA@mail.gmail.com/2-v4-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch) download | inline diff: From 7dd616333bd650c01c078885f3a8078c12fcd94c Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Mon, 4 Oct 2021 21:06:41 +0530 Subject: [PATCH v4 1/2] Preserve relfilenode and tablespace OID in pg_upgrade The patch aims to preserve the tablespace, relfilenode OIDs during binary upgrade so that the OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- src/backend/bootstrap/bootparse.y | 3 +- src/backend/catalog/heap.c | 76 ++++++++++++--- src/backend/catalog/index.c | 22 ++++- src/backend/commands/tablespace.c | 17 +++- src/backend/utils/adt/pg_upgrade_support.c | 44 +++++++++ src/bin/pg_dump/pg_dump.c | 104 +++++++++++++-------- src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/info.c | 31 +----- src/bin/pg_upgrade/pg_upgrade.c | 13 +-- src/bin/pg_upgrade/pg_upgrade.h | 10 +- src/bin/pg_upgrade/relfilenode.c | 6 +- src/include/catalog/binary_upgrade.h | 5 + src/include/catalog/heap.h | 3 +- src/include/catalog/pg_proc.dat | 16 ++++ .../spgist_name_ops/expected/spgist_name_ops.out | 12 ++- 15 files changed, 260 insertions(+), 105 deletions(-) diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y index 5fcd004..58b994c 100644 --- a/src/backend/bootstrap/bootparse.y +++ b/src/backend/bootstrap/bootparse.y @@ -212,7 +212,8 @@ Boot_CreateStmt: mapped_relation, true, &relfrozenxid, - &relminmxid); + &relminmxid, + false); elog(DEBUG4, "bootstrap relation created"); } else diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 83746d3..e45c61d 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -286,7 +288,10 @@ SystemAttributeByName(const char *attname) * * Note API change: the caller must now always provide the OID * to use for the relation. The relfilenode may (and, normally, - * should) be left unspecified. + * should) be left unspecified unless explicitly set in binary-upgrade mode. + * + * create_storage indicates whether or not to create the storage if the + * caller passed a valid relfilenode. * * rel->rd_rel is initialized by RelationBuildLocalRelation, * and is mostly zeroes at return. @@ -306,9 +311,9 @@ heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid) + MultiXactId *relminmxid, + bool create_storage) { - bool create_storage; Relation rel; /* The caller must have provided an OID for the relation. */ @@ -367,16 +372,25 @@ heap_create(const char *relname, } /* - * Decide whether to create storage. If caller passed a valid relfilenode, - * storage is already created, so don't do it here. Also don't create it - * for relkinds without physical storage. + * Decide whether to create storage. If relfilenode is valid and + * create_storage is false, storage is already created, so don't do it + * here. If relfilenode is valid and create_storage is true, create the + * storage with the specified relfilenode. If relfilenode is unspecified + * by the caller then create the storage with oid same as relid. Also, + * don't create it for relkinds without physical storage. */ - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + if (!RELKIND_HAS_STORAGE(relkind) || (OidIsValid(relfilenode) && !create_storage)) create_storage = false; else { create_storage = true; - relfilenode = relid; + + /* + * Create the storage with oid same as relid if relfilenode is + * unspecified by the caller + */ + if (!OidIsValid(relfilenode)) + relfilenode = relid; } /* @@ -1168,8 +1182,12 @@ heap_create_with_catalog(const char *relname, Oid existing_relid; Oid old_type_oid; Oid new_type_oid; + + /* By default set to InvalidOid unless overridden by binary-upgrade */ + Oid relfilenode = InvalidOid; TransactionId relfrozenxid; MultiXactId relminmxid; + bool create_storage = false; pg_class_desc = table_open(RelationRelationId, RowExclusiveLock); @@ -1230,7 +1248,7 @@ heap_create_with_catalog(const char *relname, */ if (!OidIsValid(relid)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade && (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || relkind == RELKIND_VIEW || relkind == RELKIND_MATVIEW || @@ -1244,7 +1262,27 @@ heap_create_with_catalog(const char *relname, relid = binary_upgrade_next_heap_pg_class_oid; binary_upgrade_next_heap_pg_class_oid = InvalidOid; + + /* + * Override the relfilenode and set the create_storage flag to + * true so that the storage gets created with the specified + * relfilenode during heap_create(). + */ + if (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || + relkind == RELKIND_MATVIEW) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + + create_storage = true; + } } + /* There might be no TOAST table, so we have to test for it. */ else if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_toast_pg_class_oid) && @@ -1252,6 +1290,21 @@ heap_create_with_catalog(const char *relname, { relid = binary_upgrade_next_toast_pg_class_oid; binary_upgrade_next_toast_pg_class_oid = InvalidOid; + + /* + * Override the toast relfilenode and set the create_storage flag + * to true so that the storage gets created with the specified + * relfilenode during heap_create(). + */ + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; + + create_storage = true; } else relid = GetNewRelFileNode(reltablespace, pg_class_desc, @@ -1294,7 +1347,7 @@ heap_create_with_catalog(const char *relname, relnamespace, reltablespace, relid, - InvalidOid, + relfilenode, accessmtd, tupdesc, relkind, @@ -1303,7 +1356,8 @@ heap_create_with_catalog(const char *relname, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + create_storage); Assert(relid == RelationGetRelid(new_rel_desc)); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 26bfa74..f167b10 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -86,6 +86,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system @@ -732,6 +733,7 @@ index_create(Relation heapRelation, char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; + bool create_storage = false; /* constraint flags can only be set when a constraint is requested */ Assert((constr_flags == 0) || @@ -903,7 +905,7 @@ index_create(Relation heapRelation, */ if (!OidIsValid(indexRelationId)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { if (!OidIsValid(binary_upgrade_next_index_pg_class_oid)) @@ -913,6 +915,21 @@ index_create(Relation heapRelation, indexRelationId = binary_upgrade_next_index_pg_class_oid; binary_upgrade_next_index_pg_class_oid = InvalidOid; + + /* + * Overide the relfilenode and set the create_storage to true so + * that the storage gets created with the specified relfilenode + * during heap_create(). + */ + if ((relkind == RELKIND_INDEX) && + (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode))) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("index relfilenode value not set when in binary upgrade mode"))); + relFileNode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; + + create_storage = true; } else { @@ -939,7 +956,8 @@ index_create(Relation heapRelation, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + create_storage); Assert(relfrozenxid == InvalidTransactionId); Assert(relminmxid == InvalidMultiXactId); diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 4b96eec..51ffa97 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -88,6 +88,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -335,8 +336,20 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, - Anum_pg_tablespace_oid); + if (IsBinaryUpgrade) + { + /* Use binary-upgrade override for tablespace oid */ + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->tablespacename)); diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7..135c382 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,17 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); @@ -85,6 +96,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_heap_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -96,6 +118,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_index_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -107,6 +140,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { Oid enumoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index a485fb2..b148919 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4693,73 +4693,101 @@ binary_upgrade_set_pg_class_oids(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index) { + PQExpBuffer upgrade_query = createPQExpBuffer(); + PGresult *upgrade_res; + Oid relfilenode; + Oid toast_oid; + Oid toast_relfilenode; + char relkind; + Oid toast_index_oid; + Oid toast_index_relfilenode; + + /* + * Preserve the OID and relfilenode of the table, table's index, table's + * toast table and toast table's index if any. + * + * One complexity is that the current table definition might not require + * the creation of a TOAST table, but the old database might have a TOAST + * table that was created earlier, before some wide columns were dropped. + * By setting the TOAST oid we force creation of the TOAST heap and index + * by the new backend, so we can copy the files during binary upgrade + * without worrying about this case. + */ + appendPQExpBuffer(upgrade_query, + "SELECT c.relkind, c.relfilenode, c.reltoastrelid, ct.relfilenode AS toast_relfilenode, i.indexrelid, cti.relfilenode AS toast_index_relfilenode " + "FROM pg_catalog.pg_class c LEFT JOIN " + "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " + "WHERE c.oid = '%u'::pg_catalog.oid;", + pg_class_oid); + + upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + + relkind = *PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relkind")); + + relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode"))); + toast_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "reltoastrelid"))); + toast_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_relfilenode"))); + toast_index_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "indexrelid"))); + toast_index_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_index_relfilenode"))); + appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { - PQExpBuffer upgrade_query = createPQExpBuffer(); - PGresult *upgrade_res; - Oid pg_class_reltoastrelid; - char pg_class_relkind; - Oid pg_index_indexrelid; - appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); - /* - * Preserve the OIDs of the table's toast table and index, if any. - * Indexes cannot have toast tables, so we need not make this probe in - * the index code path. - * - * One complexity is that the current table definition might not - * require the creation of a TOAST table, but the old database might - * have a TOAST table that was created earlier, before some wide - * columns were dropped. By setting the TOAST oid we force creation - * of the TOAST heap and index by the new backend, so we can copy the - * files during binary upgrade without worrying about this case. - */ - appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, c.relkind, i.indexrelid " - "FROM pg_catalog.pg_class c LEFT JOIN " - "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " - "WHERE c.oid = '%u'::pg_catalog.oid;", - pg_class_oid); - - upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); - - pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "reltoastrelid"))); - pg_class_relkind = *PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "relkind")); - pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "indexrelid"))); + /* Not every relation has storage. */ + if (OidIsValid(relfilenode)) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); /* * In a pre-v12 database, partitioned tables might be marked as having * toast tables, but we should ignore them if so. */ - if (OidIsValid(pg_class_reltoastrelid) && - pg_class_relkind != RELKIND_PARTITIONED_TABLE) + if (OidIsValid(toast_oid) && + relkind != RELKIND_PARTITIONED_TABLE) { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_class_reltoastrelid); + toast_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n", + toast_relfilenode); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_index_indexrelid); + toast_index_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + toast_index_relfilenode); } PQclear(upgrade_res); destroyPQExpBuffer(upgrade_query); } else + { + /* Preserve the OID and relfilenode of the index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); } diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c291017..618187c 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1265,6 +1265,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 5d9a26c..775564e 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -199,14 +199,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->old_db_oid = old_db->db_oid; map->new_db_oid = new_db->db_oid; - /* - * old_relfilenode might differ from pg_class.oid (and hence - * new_relfilenode) because of CLUSTER, REINDEX, or VACUUM FULL. - */ - map->old_relfilenode = old_rel->relfilenode; - - /* new_relfilenode will match old and new pg_class.oid */ - map->new_relfilenode = new_rel->relfilenode; + /* relfilenode is preserved across old and new cluster */ + map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ map->nspname = old_rel->nspname; @@ -278,27 +272,6 @@ report_unmatched_relation(const RelInfo *rel, const DbInfo *db, bool is_new_db) reloid, db->db_name, reldesc); } - -void -print_maps(FileNameMap *maps, int n_maps, const char *db_name) -{ - if (log_opts.verbose) - { - int mapnum; - - pg_log(PG_VERBOSE, "mappings for database \"%s\":\n", db_name); - - for (mapnum = 0; mapnum < n_maps; mapnum++) - pg_log(PG_VERBOSE, "%s.%s: %u to %u\n", - maps[mapnum].nspname, maps[mapnum].relname, - maps[mapnum].old_relfilenode, - maps[mapnum].new_relfilenode); - - pg_log(PG_VERBOSE, "\n\n"); - } -} - - /* * get_db_and_rel_infos() * diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd7..acaf343 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,13 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because we want the filenames + * to match between the old and new cluster. + * + * We control assignment of pg_tablespace.oid because we want the oid to match + * between the old and new cluster. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index ca0795f..c79ddfc 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -159,13 +159,7 @@ typedef struct const char *new_tablespace_suffix; Oid old_db_oid; Oid new_db_oid; - - /* - * old/new relfilenodes might differ for pg_largeobject(_metadata) indexes - * due to VACUUM FULL or REINDEX. Other relfilenodes are preserved. - */ - Oid old_relfilenode; - Oid new_relfilenode; + Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ char *relname; @@ -390,8 +384,6 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db, DbInfo *new_db, int *nmaps, const char *old_pgdata, const char *new_pgdata); void get_db_and_rel_infos(ClusterInfo *cluster); -void print_maps(FileNameMap *maps, int n, - const char *db_name); /* option.c */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 4deae7d..3bc5b44 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -119,8 +119,6 @@ transfer_all_new_dbs(DbInfoArr *old_db_arr, DbInfoArr *new_db_arr, new_pgdata); if (n_maps) { - print_maps(mappings, n_maps, new_db->db_name); - transfer_single_new_db(mappings, n_maps, old_tablespace); } /* We allocate something even for n_maps == 0 */ @@ -206,14 +204,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro map->old_tablespace, map->old_tablespace_suffix, map->old_db_oid, - map->old_relfilenode, + map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, map->new_db_oid, - map->new_relfilenode, + map->relfilenode, type_suffix, extent_suffix); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7..4ba5748 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,19 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index 6ce480b..7cf5402 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -59,7 +59,8 @@ extern Relation heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid); + MultiXactId *relminmxid, + bool create_storage); extern Oid heap_create_with_catalog(const char *relname, Oid relnamespace, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index d068d65..cd0a20d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11036,6 +11036,22 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4544', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_relfilenode' }, +{ oid => '4545', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_relfilenode' }, +{ oid => '4546', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_relfilenode' }, +{ oid => '4547', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddce..1ee65ed 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -52,14 +52,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -97,14 +101,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 1.8.3.1 [application/octet-stream] v4-0002-Preserve-database-OIDs-in-pg_upgrade.patch (12.4K, ../../CAASxf_Mj3SSXFmtd9O6sf-+y4PJz+oZ57RAi0bRKKLAOBDcorA@mail.gmail.com/3-v4-0002-Preserve-database-OIDs-in-pg_upgrade.patch) download | inline diff: From c2d00b36558957782f5f4f75b6613a4c971d8ef0 Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Mon, 4 Oct 2021 21:38:33 +0530 Subject: [PATCH v4 2/2] Preserve database OIDs in pg_upgrade The patch aims to preserve the database OIDs across binary upgrade Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- doc/src/sgml/ref/create_database.sgml | 15 ++++++++++++++- src/backend/commands/dbcommands.c | 35 ++++++++++++++++++++++++++++++----- src/backend/parser/gram.y | 3 ++- src/bin/initdb/initdb.c | 21 +++++++++++++-------- src/bin/pg_dump/pg_dump.c | 16 ++++++++++++++-- src/bin/pg_upgrade/IMPLEMENTATION | 2 +- src/bin/pg_upgrade/info.c | 9 +++------ src/bin/pg_upgrade/pg_upgrade.h | 3 +-- src/bin/pg_upgrade/relfilenode.c | 4 ++-- src/bin/psql/tab-complete.c | 2 +- src/include/access/transam.h | 3 +++ src/include/catalog/unused_oids | 5 +++++ 12 files changed, 89 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml index 41cb406..3865a96 100644 --- a/doc/src/sgml/ref/create_database.sgml +++ b/doc/src/sgml/ref/create_database.sgml @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> [ TABLESPACE [=] <replaceable class="parameter">tablespace_name</replaceable> ] [ ALLOW_CONNECTIONS [=] <replaceable class="parameter">allowconn</replaceable> ] [ CONNECTION LIMIT [=] <replaceable class="parameter">connlimit</replaceable> ] - [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] ] + [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] + [ OID [=] <replaceable class="parameter">db_oid</replaceable> ] ] </synopsis> </refsynopsisdiv> @@ -203,6 +204,18 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> </para> </listitem> </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">oid</replaceable></term> + <listitem> + <para> + The object identifier with which the database gets created. + The oid value should be greater than 16383. CREATE DATABASE fails if a + database with specified oid already exists. + </para> + </listitem> + </varlistentry> + </variablelist> <para> diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 029fab4..0251157 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -117,7 +117,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) HeapTuple tuple; Datum new_record[Natts_pg_database]; bool new_record_nulls[Natts_pg_database]; - Oid dboid; + Oid dboid = InvalidOid; Oid datdba; ListCell *option; DefElem *dtablespacename = NULL; @@ -217,6 +217,27 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) errhint("Consider using tablespaces instead."), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "oid") == 0) + { + dboid = defGetInt32(defel); + + /* + * Throw an error if the user specified oid < FirstNormalObjectId for + * creating the database. However, we need to allow creating database + * with oid < FirstNormalObjectId for below cases: + * 1. creating template0 with fixed oid during initdb + * 2. creating databases with oids from the old cluster during binary + * upgrade. + */ + if ((dboid < FirstNormalObjectId) && + (strcmp(dbname, "template0") != 0) && + (!IsBinaryUpgrade)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("Invalid value for option \"%s\"", defel->defname), + errhint("Consider using a value greater than %u.", + (FirstNormalObjectId - 1))); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* Select an OID for the new database if is not explicitly configured. */ + if (!OidIsValid(dboid)) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 08f1bf1..a8cdd43 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -687,7 +687,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OF OFF OFFSET OID OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -10254,6 +10254,7 @@ createdb_opt_name: | OWNER { $$ = pstrdup($1); } | TABLESPACE { $$ = pstrdup($1); } | TEMPLATE { $$ = pstrdup($1); } + | OID { $$ = pstrdup($1); } ; /* diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index 1ed4808..3ce3c59 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -1836,15 +1836,12 @@ static void make_template0(FILE *cmdfd) { const char *const *line; - static const char *const template0_setup[] = { - "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false;\n\n", - /* - * We use the OID of template0 to determine datlastsysoid - */ - "UPDATE pg_database SET datlastsysoid = " - " (SELECT oid FROM pg_database " - " WHERE datname = 'template0');\n\n", + /* + * Create template0 database with oid Template0ObjectId i.e, 4 + */ + static const char *const template0_setup[] = { + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID 4;\n\n", /* * Explicitly revoke public create-schema and create-temp-table @@ -1877,6 +1874,14 @@ make_postgres(FILE *cmdfd) static const char *const postgres_setup[] = { "CREATE DATABASE postgres;\n\n", "COMMENT ON DATABASE postgres IS 'default administrative connection database';\n\n", + + /* + * We use the OID of postgres to determine datlastsysoid + */ + "UPDATE pg_database SET datlastsysoid = " + " (SELECT oid FROM pg_database " + " WHERE datname = 'postgres');\n\n", + NULL }; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b148919..47c657e 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -2957,8 +2957,20 @@ dumpDatabase(Archive *fout) * are left to the DATABASE PROPERTIES entry, so that they can be applied * after reconnecting to the target DB. */ - appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", - qdatname); + if (dopt->binary_upgrade) + { + /* + * Make sure that binary upgrade propogate the database OID to the new + * cluster + */ + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0 OID = %u", + qdatname, dbCatId.oid); + } + else + { + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", + qdatname); + } if (strlen(encoding) > 0) { appendPQExpBufferStr(creaQry, " ENCODING = "); diff --git a/src/bin/pg_upgrade/IMPLEMENTATION b/src/bin/pg_upgrade/IMPLEMENTATION index 69fcd70..384834a 100644 --- a/src/bin/pg_upgrade/IMPLEMENTATION +++ b/src/bin/pg_upgrade/IMPLEMENTATION @@ -84,7 +84,7 @@ cluster using the first part of the pg_dumpall output. Next, pg_upgrade executes the remainder of the script produced earlier by pg_dumpall --- this script effectively creates the complete user-defined metadata from the old cluster to the new cluster. It -preserves the relfilenode numbers so TOAST and other references +preserves the DB, tablespace, relfilenode OIDs so TOAST and other references to relfilenodes in user data is preserved. (See binary-upgrade usage in pg_dump). diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 775564e..fe1357d 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -196,10 +196,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->new_tablespace_suffix = new_cluster.tablespace_suffix; } - map->old_db_oid = old_db->db_oid; - map->new_db_oid = new_db->db_oid; - - /* relfilenode is preserved across old and new cluster */ + /* DB oid and relfilenodes are preserved between old and new cluster */ + map->db_oid = old_db->db_oid; map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ @@ -330,8 +328,7 @@ get_db_infos(ClusterInfo *cluster) " LEFT OUTER JOIN pg_catalog.pg_tablespace t " " ON d.dattablespace = t.oid " "WHERE d.datallowconn = true " - /* we don't preserve pg_database.oid so we sort by name */ - "ORDER BY 2", + "ORDER BY 1", /* 9.2 removed the spclocation column */ (GET_MAJOR_VERSION(cluster->major_version) <= 901) ? "t.spclocation" : "pg_catalog.pg_tablespace_location(t.oid)"); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index c79ddfc..937c45b 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -157,8 +157,7 @@ typedef struct const char *new_tablespace; const char *old_tablespace_suffix; const char *new_tablespace_suffix; - Oid old_db_oid; - Oid new_db_oid; + Oid db_oid; Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 3bc5b44..a4fc298 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -203,14 +203,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro snprintf(old_file, sizeof(old_file), "%s%s/%u/%u%s%s", map->old_tablespace, map->old_tablespace_suffix, - map->old_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, - map->new_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 5cd5838..396b4c5 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2516,7 +2516,7 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("OWNER", "TEMPLATE", "ENCODING", "TABLESPACE", "IS_TEMPLATE", "ALLOW_CONNECTIONS", "CONNECTION LIMIT", - "LC_COLLATE", "LC_CTYPE", "LOCALE"); + "LC_COLLATE", "LC_CTYPE", "LOCALE", "OID"); else if (Matches("CREATE", "DATABASE", MatchAny, "TEMPLATE")) COMPLETE_WITH_QUERY(Query_for_list_of_template_databases); diff --git a/src/include/access/transam.h b/src/include/access/transam.h index d22de19..716a319 100644 --- a/src/include/access/transam.h +++ b/src/include/access/transam.h @@ -196,6 +196,9 @@ FullTransactionIdAdvance(FullTransactionId *dest) #define FirstUnpinnedObjectId 12000 #define FirstNormalObjectId 16384 +/* OID 4 is reserved for Templete0 database */ +#define Template0ObjectId 4 + /* * VariableCache is a data structure in shared memory that is used to track * OID and XID assignment state. For largely historical reasons, there is diff --git a/src/include/catalog/unused_oids b/src/include/catalog/unused_oids index 5b7ce5f..2750913 100755 --- a/src/include/catalog/unused_oids +++ b/src/include/catalog/unused_oids @@ -32,6 +32,11 @@ my @input_files = glob("pg_*.h"); my $oids = Catalog::FindAllOidsFromHeaders(@input_files); +# Push the OID that is reserved for template0 database. +my $Template0ObjectId = + Catalog::FindDefinedSymbol('access/transam.h', '..', 'Template0ObjectId'); +push @{$oids}, $Template0ObjectId; + # Also push FirstGenbkiObjectId to serve as a terminator for the last gap. my $FirstGenbkiObjectId = Catalog::FindDefinedSymbol('access/transam.h', '..', 'FirstGenbkiObjectId'); -- 1.8.3.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-10-06 20:35 Robert Haas <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-10-06 20:35 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Oct 4, 2021 at 12:44 PM Shruthi Gowda <[email protected]> wrote: > Thanks for the inputs, Robert. In the v4 patch, an unused OID (i.e, 4) > is fixed for the template0 and the same is removed from unused oid > list. > > In addition to the review comment fixes, I have removed some code that > is no longer needed/doesn't make sense since we preserve the OIDs. This is not a full review, but I'm wondering about this bit of code: - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + if (!RELKIND_HAS_STORAGE(relkind) || (OidIsValid(relfilenode) && !create_storage)) create_storage = false; else { create_storage = true; - relfilenode = relid; + + /* + * Create the storage with oid same as relid if relfilenode is + * unspecified by the caller + */ + if (!OidIsValid(relfilenode)) + relfilenode = relid; } This seems hard to understand, and I wonder if perhaps it can be simplified. If !RELKIND_HAS_STORAGE(relkind), then we're going to set create_storage to false if it was previously true, and otherwise just do nothing. Otherwise, if !create_storage, we'll enter the create_storage = false branch which effectively does nothing. Otherwise, if !OidIsValid(relfilenode), we'll set relfilenode = relid. So couldn't we express that like this? if (!RELKIND_HAS_STORAGE(relkind)) create_storage = false; else if (create_storage && !OidIsValid(relfilenode)) relfilenode = relid; If so, that seems more clear. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-10-07 07:24 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-10-07 07:24 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Thu, Oct 7, 2021 at 2:05 AM Robert Haas <[email protected]> wrote: > > On Mon, Oct 4, 2021 at 12:44 PM Shruthi Gowda <[email protected]> wrote: > > Thanks for the inputs, Robert. In the v4 patch, an unused OID (i.e, 4) > > is fixed for the template0 and the same is removed from unused oid > > list. > > > > In addition to the review comment fixes, I have removed some code that > > is no longer needed/doesn't make sense since we preserve the OIDs. > > This is not a full review, but I'm wondering about this bit of code: > > - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) > + if (!RELKIND_HAS_STORAGE(relkind) || (OidIsValid(relfilenode) > && !create_storage)) > create_storage = false; > else > { > create_storage = true; > - relfilenode = relid; > + > + /* > + * Create the storage with oid same as relid if relfilenode is > + * unspecified by the caller > + */ > + if (!OidIsValid(relfilenode)) > + relfilenode = relid; > } > > This seems hard to understand, and I wonder if perhaps it can be > simplified. If !RELKIND_HAS_STORAGE(relkind), then we're going to set > create_storage to false if it was previously true, and otherwise just > do nothing. Otherwise, if !create_storage, we'll enter the > create_storage = false branch which effectively does nothing. > Otherwise, if !OidIsValid(relfilenode), we'll set relfilenode = relid. > So couldn't we express that like this? > > if (!RELKIND_HAS_STORAGE(relkind)) > create_storage = false; > else if (create_storage && !OidIsValid(relfilenode)) > relfilenode = relid; > > If so, that seems more clear. 'create_storage' flag says whether or not to create the storage when a valid relfilenode is passed. 'create_storage' flag alone cannot make the storage creation decision in heap_create(). Only binary upgrade flow sets the 'create_storage' flag to true and expects storage gets created with specified relfilenode. Every other caller/flow passes false for 'create_storage' and we still need to create storage in heap_create() if relkind has storage. That's why I have explicitly set 'create_storage = true' in the else flow and initialize relfilenode on need basis. Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-10-07 14:02 Robert Haas <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-10-07 14:02 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Thu, Oct 7, 2021 at 3:24 AM Shruthi Gowda <[email protected]> wrote: > Every other > caller/flow passes false for 'create_storage' and we still need to > create storage in heap_create() if relkind has storage. That seems surprising. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-10-26 13:24 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-10-26 13:24 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Thu, Oct 7, 2021 at 7:33 PM Robert Haas <[email protected]> wrote: > > On Thu, Oct 7, 2021 at 3:24 AM Shruthi Gowda <[email protected]> wrote: > > Every other > > caller/flow passes false for 'create_storage' and we still need to > > create storage in heap_create() if relkind has storage. > > That seems surprising. I have revised the patch w.r.t the way 'create_storage' is interpreted in heap_create() along with some minor changes to preserve the DBOID patch. Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch (29.9K, ../../CAASxf_Nzsprv-JxO=8QDzK226iYMu0fspgRD4RoYQ=ddcO0Xqw@mail.gmail.com/2-v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch) download | inline diff: From da9a1c117cb299543e4468a082518f778ec52dff Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Tue, 26 Oct 2021 18:16:00 +0530 Subject: [PATCH v5 1/2] Preserve relfilenode and tablespace OID in pg_upgrade The patch aims to preserve the tablespace, relfilenode OIDs during binary upgrade so that the OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- src/backend/bootstrap/bootparse.y | 3 +- src/backend/catalog/heap.c | 59 +++++++++--- src/backend/catalog/index.c | 19 +++- src/backend/commands/tablespace.c | 17 +++- src/backend/utils/adt/pg_upgrade_support.c | 44 +++++++++ src/bin/pg_dump/pg_dump.c | 104 +++++++++++++-------- src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/info.c | 31 +----- src/bin/pg_upgrade/pg_upgrade.c | 13 +-- src/bin/pg_upgrade/pg_upgrade.h | 10 +- src/bin/pg_upgrade/relfilenode.c | 6 +- src/include/catalog/binary_upgrade.h | 5 + src/include/catalog/heap.h | 3 +- src/include/catalog/pg_proc.dat | 16 ++++ .../spgist_name_ops/expected/spgist_name_ops.out | 12 ++- 15 files changed, 237 insertions(+), 108 deletions(-) diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y index 5fcd004..6560b19 100644 --- a/src/backend/bootstrap/bootparse.y +++ b/src/backend/bootstrap/bootparse.y @@ -212,7 +212,8 @@ Boot_CreateStmt: mapped_relation, true, &relfrozenxid, - &relminmxid); + &relminmxid, + true); elog(DEBUG4, "bootstrap relation created"); } else diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 81cc39f..ebd8454 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -286,7 +288,9 @@ SystemAttributeByName(const char *attname) * * Note API change: the caller must now always provide the OID * to use for the relation. The relfilenode may (and, normally, - * should) be left unspecified. + * should) be left unspecified unless explicitly set in binary-upgrade mode. + * + * create_storage indicates whether or not to create the storage. * * rel->rd_rel is initialized by RelationBuildLocalRelation, * and is mostly zeroes at return. @@ -306,9 +310,9 @@ heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid) + MultiXactId *relminmxid, + bool create_storage) { - bool create_storage; Relation rel; /* The caller must have provided an OID for the relation. */ @@ -366,17 +370,17 @@ heap_create(const char *relname, break; } - /* - * Decide whether to create storage. If caller passed a valid relfilenode, - * storage is already created, so don't do it here. Also don't create it - * for relkinds without physical storage. - */ - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + /* Don't create storage for relkinds without physical storage. */ + if (!RELKIND_HAS_STORAGE(relkind)) create_storage = false; else { - create_storage = true; - relfilenode = relid; + /* + * If relfilenode is unspecified by the caller then create storage + * with oid same as relid. + */ + if (!OidIsValid(relfilenode)) + relfilenode = relid; } /* @@ -1171,6 +1175,9 @@ heap_create_with_catalog(const char *relname, Oid existing_relid; Oid old_type_oid; Oid new_type_oid; + + /* By default set to InvalidOid unless overridden by binary-upgrade */ + Oid relfilenode = InvalidOid; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1233,7 +1240,7 @@ heap_create_with_catalog(const char *relname, */ if (!OidIsValid(relid)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade && (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || relkind == RELKIND_VIEW || relkind == RELKIND_MATVIEW || @@ -1247,7 +1254,21 @@ heap_create_with_catalog(const char *relname, relid = binary_upgrade_next_heap_pg_class_oid; binary_upgrade_next_heap_pg_class_oid = InvalidOid; + + /* Override the relfilenode */ + if (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || + relkind == RELKIND_MATVIEW) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + } } + /* There might be no TOAST table, so we have to test for it. */ else if (IsBinaryUpgrade && OidIsValid(binary_upgrade_next_toast_pg_class_oid) && @@ -1255,6 +1276,15 @@ heap_create_with_catalog(const char *relname, { relid = binary_upgrade_next_toast_pg_class_oid; binary_upgrade_next_toast_pg_class_oid = InvalidOid; + + /* Override the toast relfilenode */ + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; } else relid = GetNewRelFileNode(reltablespace, pg_class_desc, @@ -1297,7 +1327,7 @@ heap_create_with_catalog(const char *relname, relnamespace, reltablespace, relid, - InvalidOid, + relfilenode, accessmtd, tupdesc, relkind, @@ -1306,7 +1336,8 @@ heap_create_with_catalog(const char *relname, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + RELKIND_HAS_STORAGE(relkind)); Assert(relid == RelationGetRelid(new_rel_desc)); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 26bfa74..88c82f9 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -86,6 +86,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system @@ -732,6 +733,7 @@ index_create(Relation heapRelation, char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; + bool create_storage; /* constraint flags can only be set when a constraint is requested */ Assert((constr_flags == 0) || @@ -739,6 +741,9 @@ index_create(Relation heapRelation, /* partitioned indexes must never be "built" by themselves */ Assert(!partitioned || (flags & INDEX_CREATE_SKIP_BUILD)); + /* Set the create_storage flag */ + create_storage = !partitioned && !OidIsValid(relFileNode); + relkind = partitioned ? RELKIND_PARTITIONED_INDEX : RELKIND_INDEX; is_exclusion = (indexInfo->ii_ExclusionOps != NULL); @@ -903,7 +908,7 @@ index_create(Relation heapRelation, */ if (!OidIsValid(indexRelationId)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { if (!OidIsValid(binary_upgrade_next_index_pg_class_oid)) @@ -913,6 +918,15 @@ index_create(Relation heapRelation, indexRelationId = binary_upgrade_next_index_pg_class_oid; binary_upgrade_next_index_pg_class_oid = InvalidOid; + + /* Overide the index relfilenode */ + if ((relkind == RELKIND_INDEX) && + (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode))) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("index relfilenode value not set when in binary upgrade mode"))); + relFileNode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; } else { @@ -939,7 +953,8 @@ index_create(Relation heapRelation, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + create_storage); Assert(relfrozenxid == InvalidTransactionId); Assert(relminmxid == InvalidMultiXactId); diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 4b96eec..51ffa97 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -88,6 +88,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -335,8 +336,20 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, - Anum_pg_tablespace_oid); + if (IsBinaryUpgrade) + { + /* Use binary-upgrade override for tablespace oid */ + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->tablespacename)); diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7..135c382 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,17 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); @@ -85,6 +96,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_heap_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -96,6 +118,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_index_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -107,6 +140,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { Oid enumoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index e9f68e8..13213ff 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4693,73 +4693,101 @@ binary_upgrade_set_pg_class_oids(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index) { + PQExpBuffer upgrade_query = createPQExpBuffer(); + PGresult *upgrade_res; + Oid relfilenode; + Oid toast_oid; + Oid toast_relfilenode; + char relkind; + Oid toast_index_oid; + Oid toast_index_relfilenode; + + /* + * Preserve the OID and relfilenode of the table, table's index, table's + * toast table and toast table's index if any. + * + * One complexity is that the current table definition might not require + * the creation of a TOAST table, but the old database might have a TOAST + * table that was created earlier, before some wide columns were dropped. + * By setting the TOAST oid we force creation of the TOAST heap and index + * by the new backend, so we can copy the files during binary upgrade + * without worrying about this case. + */ + appendPQExpBuffer(upgrade_query, + "SELECT c.relkind, c.relfilenode, c.reltoastrelid, ct.relfilenode AS toast_relfilenode, i.indexrelid, cti.relfilenode AS toast_index_relfilenode " + "FROM pg_catalog.pg_class c LEFT JOIN " + "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " + "WHERE c.oid = '%u'::pg_catalog.oid;", + pg_class_oid); + + upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + + relkind = *PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relkind")); + + relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode"))); + toast_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "reltoastrelid"))); + toast_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_relfilenode"))); + toast_index_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "indexrelid"))); + toast_index_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_index_relfilenode"))); + appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { - PQExpBuffer upgrade_query = createPQExpBuffer(); - PGresult *upgrade_res; - Oid pg_class_reltoastrelid; - char pg_class_relkind; - Oid pg_index_indexrelid; - appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); - /* - * Preserve the OIDs of the table's toast table and index, if any. - * Indexes cannot have toast tables, so we need not make this probe in - * the index code path. - * - * One complexity is that the current table definition might not - * require the creation of a TOAST table, but the old database might - * have a TOAST table that was created earlier, before some wide - * columns were dropped. By setting the TOAST oid we force creation - * of the TOAST heap and index by the new backend, so we can copy the - * files during binary upgrade without worrying about this case. - */ - appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, c.relkind, i.indexrelid " - "FROM pg_catalog.pg_class c LEFT JOIN " - "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " - "WHERE c.oid = '%u'::pg_catalog.oid;", - pg_class_oid); - - upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); - - pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "reltoastrelid"))); - pg_class_relkind = *PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "relkind")); - pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "indexrelid"))); + /* Not every relation has storage. */ + if (OidIsValid(relfilenode)) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); /* * In a pre-v12 database, partitioned tables might be marked as having * toast tables, but we should ignore them if so. */ - if (OidIsValid(pg_class_reltoastrelid) && - pg_class_relkind != RELKIND_PARTITIONED_TABLE) + if (OidIsValid(toast_oid) && + relkind != RELKIND_PARTITIONED_TABLE) { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_class_reltoastrelid); + toast_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n", + toast_relfilenode); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_index_indexrelid); + toast_index_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + toast_index_relfilenode); } PQclear(upgrade_res); destroyPQExpBuffer(upgrade_query); } else + { + /* Preserve the OID and relfilenode of the index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); } diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c291017..618187c 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1265,6 +1265,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 5d9a26c..775564e 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -199,14 +199,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->old_db_oid = old_db->db_oid; map->new_db_oid = new_db->db_oid; - /* - * old_relfilenode might differ from pg_class.oid (and hence - * new_relfilenode) because of CLUSTER, REINDEX, or VACUUM FULL. - */ - map->old_relfilenode = old_rel->relfilenode; - - /* new_relfilenode will match old and new pg_class.oid */ - map->new_relfilenode = new_rel->relfilenode; + /* relfilenode is preserved across old and new cluster */ + map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ map->nspname = old_rel->nspname; @@ -278,27 +272,6 @@ report_unmatched_relation(const RelInfo *rel, const DbInfo *db, bool is_new_db) reloid, db->db_name, reldesc); } - -void -print_maps(FileNameMap *maps, int n_maps, const char *db_name) -{ - if (log_opts.verbose) - { - int mapnum; - - pg_log(PG_VERBOSE, "mappings for database \"%s\":\n", db_name); - - for (mapnum = 0; mapnum < n_maps; mapnum++) - pg_log(PG_VERBOSE, "%s.%s: %u to %u\n", - maps[mapnum].nspname, maps[mapnum].relname, - maps[mapnum].old_relfilenode, - maps[mapnum].new_relfilenode); - - pg_log(PG_VERBOSE, "\n\n"); - } -} - - /* * get_db_and_rel_infos() * diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd7..acaf343 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,13 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because we want the filenames + * to match between the old and new cluster. + * + * We control assignment of pg_tablespace.oid because we want the oid to match + * between the old and new cluster. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index ca0795f..c79ddfc 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -159,13 +159,7 @@ typedef struct const char *new_tablespace_suffix; Oid old_db_oid; Oid new_db_oid; - - /* - * old/new relfilenodes might differ for pg_largeobject(_metadata) indexes - * due to VACUUM FULL or REINDEX. Other relfilenodes are preserved. - */ - Oid old_relfilenode; - Oid new_relfilenode; + Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ char *relname; @@ -390,8 +384,6 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db, DbInfo *new_db, int *nmaps, const char *old_pgdata, const char *new_pgdata); void get_db_and_rel_infos(ClusterInfo *cluster); -void print_maps(FileNameMap *maps, int n, - const char *db_name); /* option.c */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 4deae7d..3bc5b44 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -119,8 +119,6 @@ transfer_all_new_dbs(DbInfoArr *old_db_arr, DbInfoArr *new_db_arr, new_pgdata); if (n_maps) { - print_maps(mappings, n_maps, new_db->db_name); - transfer_single_new_db(mappings, n_maps, old_tablespace); } /* We allocate something even for n_maps == 0 */ @@ -206,14 +204,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro map->old_tablespace, map->old_tablespace_suffix, map->old_db_oid, - map->old_relfilenode, + map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, map->new_db_oid, - map->new_relfilenode, + map->relfilenode, type_suffix, extent_suffix); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7..4ba5748 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,19 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index 6ce480b..7cf5402 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -59,7 +59,8 @@ extern Relation heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid); + MultiXactId *relminmxid, + bool create_storage); extern Oid heap_create_with_catalog(const char *relname, Oid relnamespace, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index d068d65..cd0a20d 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11036,6 +11036,22 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4544', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_relfilenode' }, +{ oid => '4545', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_relfilenode' }, +{ oid => '4546', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_relfilenode' }, +{ oid => '4547', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddce..1ee65ed 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -52,14 +52,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -97,14 +101,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 1.8.3.1 [application/octet-stream] v5-0002-Preserve-database-OIDs-in-pg_upgrade.patch (12.7K, ../../CAASxf_Nzsprv-JxO=8QDzK226iYMu0fspgRD4RoYQ=ddcO0Xqw@mail.gmail.com/3-v5-0002-Preserve-database-OIDs-in-pg_upgrade.patch) download | inline diff: From 554d92617025bb33d97a68b60697009c2ad2c030 Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Tue, 26 Oct 2021 18:40:51 +0530 Subject: [PATCH v5 2/2] Preserve database OIDs in pg_upgrade The patch aims to preserve the database OIDs across binary upgrade Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- doc/src/sgml/ref/create_database.sgml | 15 ++++++++++++++- src/backend/commands/dbcommands.c | 35 ++++++++++++++++++++++++++++++----- src/backend/parser/gram.y | 3 ++- src/bin/initdb/initdb.c | 23 +++++++++++++++-------- src/bin/pg_dump/pg_dump.c | 16 ++++++++++++++-- src/bin/pg_upgrade/IMPLEMENTATION | 2 +- src/bin/pg_upgrade/info.c | 9 +++------ src/bin/pg_upgrade/pg_upgrade.h | 3 +-- src/bin/pg_upgrade/relfilenode.c | 4 ++-- src/bin/psql/tab-complete.c | 2 +- src/include/access/transam.h | 3 +++ src/include/catalog/unused_oids | 5 +++++ 12 files changed, 91 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml index 41cb406..fc108a8 100644 --- a/doc/src/sgml/ref/create_database.sgml +++ b/doc/src/sgml/ref/create_database.sgml @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> [ TABLESPACE [=] <replaceable class="parameter">tablespace_name</replaceable> ] [ ALLOW_CONNECTIONS [=] <replaceable class="parameter">allowconn</replaceable> ] [ CONNECTION LIMIT [=] <replaceable class="parameter">connlimit</replaceable> ] - [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] ] + [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] + [ OID [=] <replaceable class="parameter">db_oid</replaceable> ] ] </synopsis> </refsynopsisdiv> @@ -203,6 +204,18 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> </para> </listitem> </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">oid</replaceable></term> + <listitem> + <para> + The object identifier with which the database gets created. + The OID range for user objects starts from 16384. CREATE DATABASE fails if a + database with specified oid already exists. + </para> + </listitem> + </varlistentry> + </variablelist> <para> diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 029fab4..e8cdb7f 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -117,7 +117,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) HeapTuple tuple; Datum new_record[Natts_pg_database]; bool new_record_nulls[Natts_pg_database]; - Oid dboid; + Oid dboid = InvalidOid; Oid datdba; ListCell *option; DefElem *dtablespacename = NULL; @@ -217,6 +217,27 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) errhint("Consider using tablespaces instead."), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "oid") == 0) + { + dboid = defGetInt32(defel); + + /* + * Throw an error if the user specified oid < FirstNormalObjectId for + * creating the database. However, we need to allow creating database + * with oid < FirstNormalObjectId for below cases: + * 1. creating template0 with fixed oid during initdb + * 2. creating databases with oids from the old cluster during binary + * upgrade. + */ + if ((dboid < FirstNormalObjectId) && + (strcmp(dbname, "template0") != 0) && + (!IsBinaryUpgrade)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("Invalid value for option \"%s\"", defel->defname), + errhint("The specified OID %u is less than the minimum OID for user objects %u.", + dboid, FirstNormalObjectId)); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* Select an OID for the new database if is not explicitly configured. */ + if (!OidIsValid(dboid)) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 08f1bf1..a8cdd43 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -687,7 +687,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OF OFF OFFSET OID OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -10254,6 +10254,7 @@ createdb_opt_name: | OWNER { $$ = pstrdup($1); } | TABLESPACE { $$ = pstrdup($1); } | TEMPLATE { $$ = pstrdup($1); } + | OID { $$ = pstrdup($1); } ; /* diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index 1ed4808..61cac6e 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -59,6 +59,7 @@ #include "sys/mman.h" #endif +#include "access/transam.h" #include "access/xlog_internal.h" #include "catalog/pg_authid_d.h" #include "catalog/pg_class_d.h" /* pgrminclude ignore */ @@ -1836,15 +1837,13 @@ static void make_template0(FILE *cmdfd) { const char *const *line; - static const char *const template0_setup[] = { - "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false;\n\n", - /* - * We use the OID of template0 to determine datlastsysoid - */ - "UPDATE pg_database SET datlastsysoid = " - " (SELECT oid FROM pg_database " - " WHERE datname = 'template0');\n\n", + /* + * Create template0 database with oid Template0ObjectId i.e, 4 + */ + static const char *const template0_setup[] = { + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID " + CppAsString2(Template0ObjectId) ";\n\n", /* * Explicitly revoke public create-schema and create-temp-table @@ -1877,6 +1876,14 @@ make_postgres(FILE *cmdfd) static const char *const postgres_setup[] = { "CREATE DATABASE postgres;\n\n", "COMMENT ON DATABASE postgres IS 'default administrative connection database';\n\n", + + /* + * We use the OID of postgres to determine datlastsysoid + */ + "UPDATE pg_database SET datlastsysoid = " + " (SELECT oid FROM pg_database " + " WHERE datname = 'postgres');\n\n", + NULL }; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 13213ff..ce37bf4 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -2957,8 +2957,20 @@ dumpDatabase(Archive *fout) * are left to the DATABASE PROPERTIES entry, so that they can be applied * after reconnecting to the target DB. */ - appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", - qdatname); + if (dopt->binary_upgrade) + { + /* + * Make sure that binary upgrade propogate the database OID to the new + * cluster + */ + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0 OID = %u", + qdatname, dbCatId.oid); + } + else + { + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", + qdatname); + } if (strlen(encoding) > 0) { appendPQExpBufferStr(creaQry, " ENCODING = "); diff --git a/src/bin/pg_upgrade/IMPLEMENTATION b/src/bin/pg_upgrade/IMPLEMENTATION index 69fcd70..384834a 100644 --- a/src/bin/pg_upgrade/IMPLEMENTATION +++ b/src/bin/pg_upgrade/IMPLEMENTATION @@ -84,7 +84,7 @@ cluster using the first part of the pg_dumpall output. Next, pg_upgrade executes the remainder of the script produced earlier by pg_dumpall --- this script effectively creates the complete user-defined metadata from the old cluster to the new cluster. It -preserves the relfilenode numbers so TOAST and other references +preserves the DB, tablespace, relfilenode OIDs so TOAST and other references to relfilenodes in user data is preserved. (See binary-upgrade usage in pg_dump). diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 775564e..fe1357d 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -196,10 +196,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->new_tablespace_suffix = new_cluster.tablespace_suffix; } - map->old_db_oid = old_db->db_oid; - map->new_db_oid = new_db->db_oid; - - /* relfilenode is preserved across old and new cluster */ + /* DB oid and relfilenodes are preserved between old and new cluster */ + map->db_oid = old_db->db_oid; map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ @@ -330,8 +328,7 @@ get_db_infos(ClusterInfo *cluster) " LEFT OUTER JOIN pg_catalog.pg_tablespace t " " ON d.dattablespace = t.oid " "WHERE d.datallowconn = true " - /* we don't preserve pg_database.oid so we sort by name */ - "ORDER BY 2", + "ORDER BY 1", /* 9.2 removed the spclocation column */ (GET_MAJOR_VERSION(cluster->major_version) <= 901) ? "t.spclocation" : "pg_catalog.pg_tablespace_location(t.oid)"); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index c79ddfc..937c45b 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -157,8 +157,7 @@ typedef struct const char *new_tablespace; const char *old_tablespace_suffix; const char *new_tablespace_suffix; - Oid old_db_oid; - Oid new_db_oid; + Oid db_oid; Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 3bc5b44..a4fc298 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -203,14 +203,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro snprintf(old_file, sizeof(old_file), "%s%s/%u/%u%s%s", map->old_tablespace, map->old_tablespace_suffix, - map->old_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, - map->new_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index ecae9df..b812f61 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2516,7 +2516,7 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("OWNER", "TEMPLATE", "ENCODING", "TABLESPACE", "IS_TEMPLATE", "ALLOW_CONNECTIONS", "CONNECTION LIMIT", - "LC_COLLATE", "LC_CTYPE", "LOCALE"); + "LC_COLLATE", "LC_CTYPE", "LOCALE", "OID"); else if (Matches("CREATE", "DATABASE", MatchAny, "TEMPLATE")) COMPLETE_WITH_QUERY(Query_for_list_of_template_databases); diff --git a/src/include/access/transam.h b/src/include/access/transam.h index d22de19..716a319 100644 --- a/src/include/access/transam.h +++ b/src/include/access/transam.h @@ -196,6 +196,9 @@ FullTransactionIdAdvance(FullTransactionId *dest) #define FirstUnpinnedObjectId 12000 #define FirstNormalObjectId 16384 +/* OID 4 is reserved for Templete0 database */ +#define Template0ObjectId 4 + /* * VariableCache is a data structure in shared memory that is used to track * OID and XID assignment state. For largely historical reasons, there is diff --git a/src/include/catalog/unused_oids b/src/include/catalog/unused_oids index 5b7ce5f..2750913 100755 --- a/src/include/catalog/unused_oids +++ b/src/include/catalog/unused_oids @@ -32,6 +32,11 @@ my @input_files = glob("pg_*.h"); my $oids = Catalog::FindAllOidsFromHeaders(@input_files); +# Push the OID that is reserved for template0 database. +my $Template0ObjectId = + Catalog::FindDefinedSymbol('access/transam.h', '..', 'Template0ObjectId'); +push @{$oids}, $Template0ObjectId; + # Also push FirstGenbkiObjectId to serve as a terminator for the last gap. my $FirstGenbkiObjectId = Catalog::FindDefinedSymbol('access/transam.h', '..', 'FirstGenbkiObjectId'); -- 1.8.3.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-06 04:44 Sadhuprasad Patro <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Sadhuprasad Patro @ 2021-12-06 04:44 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Robert Haas <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Oct 26, 2021 at 6:55 PM Shruthi Gowda <[email protected]> wrote: > > > I have revised the patch w.r.t the way 'create_storage' is interpreted > in heap_create() along with some minor changes to preserve the DBOID > patch. > Hi Shruthi, I am reviewing the attached patches and providing a few comments here below for patch "v5-0002-Preserve-database-OIDs-in-pg_upgrade.patch" 1. --- a/doc/src/sgml/ref/create_database.sgml +++ b/doc/src/sgml/ref/create_database.sgml @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> - [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] ] + [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] + [ OID [=] <replaceable class="parameter">db_oid</replaceable> ] ] Replace "db_oid" with 'oid'. Below in the listitem, we have mentioned 'oid'. 2. --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c + if ((dboid < FirstNormalObjectId) && + (strcmp(dbname, "template0") != 0) && + (!IsBinaryUpgrade)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("Invalid value for option \"%s\"", defel->defname), + errhint("The specified OID %u is less than the minimum OID for user objects %u.", + dboid, FirstNormalObjectId)); + } Are we sure that 'IsBinaryUpgrade' will be set properly, before the createdb function is called? Can we recheck once ? 3. @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* Select an OID for the new database if is not explicitly configured. */ + if (!OidIsValid(dboid)) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); I think we need to do 'check_db_file_conflict' for the USER given OID also.. right? It may already be present. 4. --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c /* + * Create template0 database with oid Template0ObjectId i.e, 4 + */ + Better to mention here, why OID 4 is reserved for template0 database?. 5. + /* + * Create template0 database with oid Template0ObjectId i.e, 4 + */ + static const char *const template0_setup[] = { + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID " + CppAsString2(Template0ObjectId) ";\n\n", Can we write something like, 'OID = CppAsString2(Template0ObjectId)'? mention "=". 6. + + /* + * We use the OID of postgres to determine datlastsysoid + */ + "UPDATE pg_database SET datlastsysoid = " + " (SELECT oid FROM pg_database " + " WHERE datname = 'postgres');\n\n", + Make the above comment a single line comment. 7. There are some spelling mistakes in the comments as below, please correct the same + /* + * Make sure that binary upgrade propogate the database OID to the new =====> correct spelling + * cluster + */ +/* OID 4 is reserved for Templete0 database */ ====> Correct spelling +#define Template0ObjectId 4 I am reviewing another patch "v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg" as well and will provide the comments soon if any... Thanks & Regards SadhuPrasad EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-06 17:55 Robert Haas <[email protected]> parent: Sadhuprasad Patro <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Robert Haas @ 2021-12-06 17:55 UTC (permalink / raw) To: Sadhuprasad Patro <[email protected]>; +Cc: Shruthi Gowda <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Sun, Dec 5, 2021 at 11:44 PM Sadhuprasad Patro <[email protected]> wrote: > 1. > --- a/doc/src/sgml/ref/create_database.sgml > +++ b/doc/src/sgml/ref/create_database.sgml > @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable > class="parameter">name</replaceable> > - [ IS_TEMPLATE [=] <replaceable > class="parameter">istemplate</replaceable> ] ] > + [ IS_TEMPLATE [=] <replaceable > class="parameter">istemplate</replaceable> ] > + [ OID [=] <replaceable > class="parameter">db_oid</replaceable> ] ] > > Replace "db_oid" with 'oid'. Below in the listitem, we have mentioned 'oid'. I agree that the listitem and the synopsis need to be consistent, but it could be made consistent either by changing that one to db_oid or this one to oid. > 2. > --- a/src/backend/commands/dbcommands.c > +++ b/src/backend/commands/dbcommands.c > + if ((dboid < FirstNormalObjectId) && > + (strcmp(dbname, "template0") != 0) && > + (!IsBinaryUpgrade)) > + ereport(ERROR, > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), > + errmsg("Invalid value for option \"%s\"", defel->defname), > + errhint("The specified OID %u is less than the minimum OID for user > objects %u.", > + dboid, FirstNormalObjectId)); > + } > > Are we sure that 'IsBinaryUpgrade' will be set properly, before the > createdb function is called? Can we recheck once ? How could it be set incorrectly, and how could we recheck this? > 3. > @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > */ > pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); > > - do > + /* Select an OID for the new database if is not explicitly configured. */ > + if (!OidIsValid(dboid)) > { > - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, > - Anum_pg_database_oid); > - } while (check_db_file_conflict(dboid)); > > I think we need to do 'check_db_file_conflict' for the USER given OID > also.. right? It may already be present. Hopefully, if that happens, we straight up fail later on. > 4. > --- a/src/bin/initdb/initdb.c > +++ b/src/bin/initdb/initdb.c > > /* > + * Create template0 database with oid Template0ObjectId i.e, 4 > + */ > + > > Better to mention here, why OID 4 is reserved for template0 database?. I'm not sure how we would give a reason for selecting an arbitrary constant? We could perhaps explain why we use a fixed OID. But there's no reason it has to be 4, I think. > 5. > + /* > + * Create template0 database with oid Template0ObjectId i.e, 4 > + */ > + static const char *const template0_setup[] = { > + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID " > + CppAsString2(Template0ObjectId) ";\n\n", > > Can we write something like, 'OID = CppAsString2(Template0ObjectId)'? > mention "=". That seems like a good idea, because it would be more consistent. > 6. > + > + /* > + * We use the OID of postgres to determine datlastsysoid > + */ > + "UPDATE pg_database SET datlastsysoid = " > + " (SELECT oid FROM pg_database " > + " WHERE datname = 'postgres');\n\n", > + > > Make the above comment a single line comment. I think what Shruthi did is more correct. It doesn't have to be done as a single-line comment just because it can fit on one line. And Shruthi didn't write this comment anyway, it's only moved slightly from where it was before. > 7. > There are some spelling mistakes in the comments as below, please > correct the same > + /* > + * Make sure that binary upgrade propogate the database OID to the > new =====> correct spelling > + * cluster > + */ > > +/* OID 4 is reserved for Templete0 database */ > ====> Correct spelling > +#define Template0ObjectId 4 Yes, those would be good to fix. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-13 14:40 Shruthi Gowda <[email protected]> parent: Sadhuprasad Patro <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-12-13 14:40 UTC (permalink / raw) To: Sadhuprasad Patro <[email protected]>; +Cc: Robert Haas <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Dec 6, 2021 at 10:14 AM Sadhuprasad Patro <[email protected]> wrote: > > On Tue, Oct 26, 2021 at 6:55 PM Shruthi Gowda <[email protected]> wrote: > > > > > > I have revised the patch w.r.t the way 'create_storage' is interpreted > > in heap_create() along with some minor changes to preserve the DBOID > > patch. > > > > Hi Shruthi, > > I am reviewing the attached patches and providing a few comments here > below for patch "v5-0002-Preserve-database-OIDs-in-pg_upgrade.patch" > > 1. > --- a/doc/src/sgml/ref/create_database.sgml > +++ b/doc/src/sgml/ref/create_database.sgml > @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable > class="parameter">name</replaceable> > - [ IS_TEMPLATE [=] <replaceable > class="parameter">istemplate</replaceable> ] ] > + [ IS_TEMPLATE [=] <replaceable > class="parameter">istemplate</replaceable> ] > + [ OID [=] <replaceable > class="parameter">db_oid</replaceable> ] ] > > Replace "db_oid" with 'oid'. Below in the listitem, we have mentioned 'oid'. Replaced "db_oid" with "oid" > > 2. > --- a/src/backend/commands/dbcommands.c > +++ b/src/backend/commands/dbcommands.c > + if ((dboid < FirstNormalObjectId) && > + (strcmp(dbname, "template0") != 0) && > + (!IsBinaryUpgrade)) > + ereport(ERROR, > + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), > + errmsg("Invalid value for option \"%s\"", defel->defname), > + errhint("The specified OID %u is less than the minimum OID for user > objects %u.", > + dboid, FirstNormalObjectId)); > + } > > Are we sure that 'IsBinaryUpgrade' will be set properly, before the > createdb function is called? Can we recheck once ? I believe 'IsBinaryUpgrade' will be set to true when pg_upgrade is invoked. pg_ugrade internally does pg_dump and pg_restore for every database in the cluster. > 3. > @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > */ > pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); > > - do > + /* Select an OID for the new database if is not explicitly configured. */ > + if (!OidIsValid(dboid)) > { > - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, > - Anum_pg_database_oid); > - } while (check_db_file_conflict(dboid)); > > I think we need to do 'check_db_file_conflict' for the USER given OID > also.. right? It may already be present. If a datafile with user-specified OID exists, the create database fails with the below error. postgres=# create database d2 oid 16452; ERROR: could not create directory "base/16452": File exists > 4. > --- a/src/bin/initdb/initdb.c > +++ b/src/bin/initdb/initdb.c > > /* > + * Create template0 database with oid Template0ObjectId i.e, 4 > + */ > + > > Better to mention here, why OID 4 is reserved for template0 database?. The comment is updated to explain why template0 oid is fixed. > 5. > + /* > + * Create template0 database with oid Template0ObjectId i.e, 4 > + */ > + static const char *const template0_setup[] = { > + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID " > + CppAsString2(Template0ObjectId) ";\n\n", > > Can we write something like, 'OID = CppAsString2(Template0ObjectId)'? > mention "=". Fixed > 6. > + > + /* > + * We use the OID of postgres to determine datlastsysoid > + */ > + "UPDATE pg_database SET datlastsysoid = " > + " (SELECT oid FROM pg_database " > + " WHERE datname = 'postgres');\n\n", > + > > Make the above comment a single line comment. As Robert confirmed, this part of the code is moved from a different place. > 7. > There are some spelling mistakes in the comments as below, please > correct the same > + /* > + * Make sure that binary upgrade propogate the database OID to the > new =====> correct spelling > + * cluster > + */ > > +/* OID 4 is reserved for Templete0 database */ > ====> Correct spelling > +#define Template0ObjectId 4 > Fixed. > I am reviewing another patch > "v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg" as well > and will provide the comments soon if any... Thanks. I have rebased relfilenode oid preserve patch. You may use the rebased patch for review. Thanks & Regards Shruthi K C EnterpriseDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v6-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch (29.7K, ../../CAASxf_NUPgnNUfn82i0xa--Ws0p0iP95_zmH2Mw6xyhZNDUbcg@mail.gmail.com/2-v6-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg.patch) download | inline diff: From 9bbddded62ab81499bd8dabf351258cff32fe733 Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Sun, 12 Dec 2021 23:26:31 +0530 Subject: [PATCH v6 1/2] Preserve relfilenode and tablespace OID in pg_upgrade The patch aims to preserve the tablespace, relfilenode OIDs during binary upgrade so that the OIDs are same across old and new cluster. Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- src/backend/bootstrap/bootparse.y | 3 +- src/backend/catalog/heap.c | 58 +++++++++--- src/backend/catalog/index.c | 19 +++- src/backend/commands/tablespace.c | 17 +++- src/backend/utils/adt/pg_upgrade_support.c | 44 +++++++++ src/bin/pg_dump/pg_dump.c | 104 +++++++++++++-------- src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/info.c | 31 +----- src/bin/pg_upgrade/pg_upgrade.c | 13 +-- src/bin/pg_upgrade/pg_upgrade.h | 10 +- src/bin/pg_upgrade/relfilenode.c | 6 +- src/include/catalog/binary_upgrade.h | 5 + src/include/catalog/heap.h | 3 +- src/include/catalog/pg_proc.dat | 16 ++++ .../spgist_name_ops/expected/spgist_name_ops.out | 12 ++- 15 files changed, 236 insertions(+), 108 deletions(-) diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y index 5fcd004..6560b19 100644 --- a/src/backend/bootstrap/bootparse.y +++ b/src/backend/bootstrap/bootparse.y @@ -212,7 +212,8 @@ Boot_CreateStmt: mapped_relation, true, &relfrozenxid, - &relminmxid); + &relminmxid, + true); elog(DEBUG4, "bootstrap relation created"); } else diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 6780ec5..6c1abd4 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -286,7 +288,9 @@ SystemAttributeByName(const char *attname) * * Note API change: the caller must now always provide the OID * to use for the relation. The relfilenode may (and, normally, - * should) be left unspecified. + * should) be left unspecified unless explicitly set in binary-upgrade mode. + * + * create_storage indicates whether or not to create the storage. * * rel->rd_rel is initialized by RelationBuildLocalRelation, * and is mostly zeroes at return. @@ -306,9 +310,9 @@ heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid) + MultiXactId *relminmxid, + bool create_storage) { - bool create_storage; Relation rel; /* The caller must have provided an OID for the relation. */ @@ -343,17 +347,17 @@ heap_create(const char *relname, if (!RELKIND_HAS_TABLESPACE(relkind)) reltablespace = InvalidOid; - /* - * Decide whether to create storage. If caller passed a valid relfilenode, - * storage is already created, so don't do it here. Also don't create it - * for relkinds without physical storage. - */ - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + /* Don't create storage for relkinds without physical storage. */ + if (!RELKIND_HAS_STORAGE(relkind)) create_storage = false; else { - create_storage = true; - relfilenode = relid; + /* + * If relfilenode is unspecified by the caller then create storage + * with oid same as relid. + */ + if (!OidIsValid(relfilenode)) + relfilenode = relid; } /* @@ -1121,6 +1125,9 @@ heap_create_with_catalog(const char *relname, Oid existing_relid; Oid old_type_oid; Oid new_type_oid; + + /* By default set to InvalidOid unless overridden by binary-upgrade */ + Oid relfilenode = InvalidOid; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1183,7 +1190,7 @@ heap_create_with_catalog(const char *relname, */ if (!OidIsValid(relid)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { /* @@ -1200,6 +1207,15 @@ heap_create_with_catalog(const char *relname, { relid = binary_upgrade_next_toast_pg_class_oid; binary_upgrade_next_toast_pg_class_oid = InvalidOid; + + /* Override the toast relfilenode */ + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; } } else @@ -1211,6 +1227,19 @@ heap_create_with_catalog(const char *relname, relid = binary_upgrade_next_heap_pg_class_oid; binary_upgrade_next_heap_pg_class_oid = InvalidOid; + + /* Override the relfilenode */ + if (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || + relkind == RELKIND_MATVIEW) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + } } } @@ -1255,7 +1284,7 @@ heap_create_with_catalog(const char *relname, relnamespace, reltablespace, relid, - InvalidOid, + relfilenode, accessmtd, tupdesc, relkind, @@ -1264,7 +1293,8 @@ heap_create_with_catalog(const char *relname, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + RELKIND_HAS_STORAGE(relkind)); Assert(relid == RelationGetRelid(new_rel_desc)); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1757cd3..a4d1f9c 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -87,6 +87,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system @@ -733,6 +734,7 @@ index_create(Relation heapRelation, char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; + bool create_storage; /* constraint flags can only be set when a constraint is requested */ Assert((constr_flags == 0) || @@ -740,6 +742,9 @@ index_create(Relation heapRelation, /* partitioned indexes must never be "built" by themselves */ Assert(!partitioned || (flags & INDEX_CREATE_SKIP_BUILD)); + /* Set the create_storage flag */ + create_storage = !partitioned && !OidIsValid(relFileNode); + relkind = partitioned ? RELKIND_PARTITIONED_INDEX : RELKIND_INDEX; is_exclusion = (indexInfo->ii_ExclusionOps != NULL); @@ -904,7 +909,7 @@ index_create(Relation heapRelation, */ if (!OidIsValid(indexRelationId)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { if (!OidIsValid(binary_upgrade_next_index_pg_class_oid)) @@ -914,6 +919,15 @@ index_create(Relation heapRelation, indexRelationId = binary_upgrade_next_index_pg_class_oid; binary_upgrade_next_index_pg_class_oid = InvalidOid; + + /* Overide the index relfilenode */ + if ((relkind == RELKIND_INDEX) && + (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode))) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("index relfilenode value not set when in binary upgrade mode"))); + relFileNode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; } else { @@ -940,7 +954,8 @@ index_create(Relation heapRelation, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + create_storage); Assert(relfrozenxid == InvalidTransactionId); Assert(relminmxid == InvalidMultiXactId); diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 4b96eec..51ffa97 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -88,6 +88,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -335,8 +336,20 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, - Anum_pg_tablespace_oid); + if (IsBinaryUpgrade) + { + /* Use binary-upgrade override for tablespace oid */ + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->tablespacename)); diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7..135c382 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,17 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); @@ -85,6 +96,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_heap_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -96,6 +118,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_index_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -107,6 +140,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { Oid enumoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 10a86f9..b41d86b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4836,73 +4836,101 @@ binary_upgrade_set_pg_class_oids(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index) { + PQExpBuffer upgrade_query = createPQExpBuffer(); + PGresult *upgrade_res; + Oid relfilenode; + Oid toast_oid; + Oid toast_relfilenode; + char relkind; + Oid toast_index_oid; + Oid toast_index_relfilenode; + + /* + * Preserve the OID and relfilenode of the table, table's index, table's + * toast table and toast table's index if any. + * + * One complexity is that the current table definition might not require + * the creation of a TOAST table, but the old database might have a TOAST + * table that was created earlier, before some wide columns were dropped. + * By setting the TOAST oid we force creation of the TOAST heap and index + * by the new backend, so we can copy the files during binary upgrade + * without worrying about this case. + */ + appendPQExpBuffer(upgrade_query, + "SELECT c.relkind, c.relfilenode, c.reltoastrelid, ct.relfilenode AS toast_relfilenode, i.indexrelid, cti.relfilenode AS toast_index_relfilenode " + "FROM pg_catalog.pg_class c LEFT JOIN " + "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " + "WHERE c.oid = '%u'::pg_catalog.oid;", + pg_class_oid); + + upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + + relkind = *PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relkind")); + + relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode"))); + toast_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "reltoastrelid"))); + toast_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_relfilenode"))); + toast_index_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "indexrelid"))); + toast_index_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_index_relfilenode"))); + appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { - PQExpBuffer upgrade_query = createPQExpBuffer(); - PGresult *upgrade_res; - Oid pg_class_reltoastrelid; - char pg_class_relkind; - Oid pg_index_indexrelid; - appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); - /* - * Preserve the OIDs of the table's toast table and index, if any. - * Indexes cannot have toast tables, so we need not make this probe in - * the index code path. - * - * One complexity is that the current table definition might not - * require the creation of a TOAST table, but the old database might - * have a TOAST table that was created earlier, before some wide - * columns were dropped. By setting the TOAST oid we force creation - * of the TOAST heap and index by the new backend, so we can copy the - * files during binary upgrade without worrying about this case. - */ - appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, c.relkind, i.indexrelid " - "FROM pg_catalog.pg_class c LEFT JOIN " - "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " - "WHERE c.oid = '%u'::pg_catalog.oid;", - pg_class_oid); - - upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); - - pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "reltoastrelid"))); - pg_class_relkind = *PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "relkind")); - pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "indexrelid"))); + /* Not every relation has storage. */ + if (OidIsValid(relfilenode)) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); /* * In a pre-v12 database, partitioned tables might be marked as having * toast tables, but we should ignore them if so. */ - if (OidIsValid(pg_class_reltoastrelid) && - pg_class_relkind != RELKIND_PARTITIONED_TABLE) + if (OidIsValid(toast_oid) && + relkind != RELKIND_PARTITIONED_TABLE) { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_class_reltoastrelid); + toast_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n", + toast_relfilenode); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_index_indexrelid); + toast_index_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + toast_index_relfilenode); } PQclear(upgrade_res); destroyPQExpBuffer(upgrade_query); } else + { + /* Preserve the OID and relfilenode of the index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); } diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 27b9573..3f42316 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1222,6 +1222,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 5d9a26c..775564e 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -199,14 +199,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->old_db_oid = old_db->db_oid; map->new_db_oid = new_db->db_oid; - /* - * old_relfilenode might differ from pg_class.oid (and hence - * new_relfilenode) because of CLUSTER, REINDEX, or VACUUM FULL. - */ - map->old_relfilenode = old_rel->relfilenode; - - /* new_relfilenode will match old and new pg_class.oid */ - map->new_relfilenode = new_rel->relfilenode; + /* relfilenode is preserved across old and new cluster */ + map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ map->nspname = old_rel->nspname; @@ -278,27 +272,6 @@ report_unmatched_relation(const RelInfo *rel, const DbInfo *db, bool is_new_db) reloid, db->db_name, reldesc); } - -void -print_maps(FileNameMap *maps, int n_maps, const char *db_name) -{ - if (log_opts.verbose) - { - int mapnum; - - pg_log(PG_VERBOSE, "mappings for database \"%s\":\n", db_name); - - for (mapnum = 0; mapnum < n_maps; mapnum++) - pg_log(PG_VERBOSE, "%s.%s: %u to %u\n", - maps[mapnum].nspname, maps[mapnum].relname, - maps[mapnum].old_relfilenode, - maps[mapnum].new_relfilenode); - - pg_log(PG_VERBOSE, "\n\n"); - } -} - - /* * get_db_and_rel_infos() * diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd7..acaf343 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,13 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because we want the filenames + * to match between the old and new cluster. + * + * We control assignment of pg_tablespace.oid because we want the oid to match + * between the old and new cluster. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index ca0795f..c79ddfc 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -159,13 +159,7 @@ typedef struct const char *new_tablespace_suffix; Oid old_db_oid; Oid new_db_oid; - - /* - * old/new relfilenodes might differ for pg_largeobject(_metadata) indexes - * due to VACUUM FULL or REINDEX. Other relfilenodes are preserved. - */ - Oid old_relfilenode; - Oid new_relfilenode; + Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ char *relname; @@ -390,8 +384,6 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db, DbInfo *new_db, int *nmaps, const char *old_pgdata, const char *new_pgdata); void get_db_and_rel_infos(ClusterInfo *cluster); -void print_maps(FileNameMap *maps, int n, - const char *db_name); /* option.c */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 4deae7d..3bc5b44 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -119,8 +119,6 @@ transfer_all_new_dbs(DbInfoArr *old_db_arr, DbInfoArr *new_db_arr, new_pgdata); if (n_maps) { - print_maps(mappings, n_maps, new_db->db_name); - transfer_single_new_db(mappings, n_maps, old_tablespace); } /* We allocate something even for n_maps == 0 */ @@ -206,14 +204,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro map->old_tablespace, map->old_tablespace_suffix, map->old_db_oid, - map->old_relfilenode, + map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, map->new_db_oid, - map->new_relfilenode, + map->relfilenode, type_suffix, extent_suffix); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7..4ba5748 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,19 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index 6ce480b..7cf5402 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -59,7 +59,8 @@ extern Relation heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid); + MultiXactId *relminmxid, + bool create_storage); extern Oid heap_create_with_catalog(const char *relname, Oid relnamespace, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 79d787c..92727d8 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11058,6 +11058,22 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4545', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_relfilenode' }, +{ oid => '4546', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_relfilenode' }, +{ oid => '4547', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_relfilenode' }, +{ oid => '4548', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddce..1ee65ed 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -52,14 +52,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -97,14 +101,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 1.8.3.1 [application/octet-stream] v6-0002-Preserve-database-OIDs-in-pg_upgrade.patch (12.7K, ../../CAASxf_NUPgnNUfn82i0xa--Ws0p0iP95_zmH2Mw6xyhZNDUbcg@mail.gmail.com/3-v6-0002-Preserve-database-OIDs-in-pg_upgrade.patch) download | inline diff: From b0694cc31feb241eb82fda6f83897df784c66cfd Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Mon, 13 Dec 2021 19:30:28 +0530 Subject: [PATCH v6 2/2] Preserve database OIDs in pg_upgrade The patch aims to preserve the database OIDs across binary upgrade Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- doc/src/sgml/ref/create_database.sgml | 15 ++++++++++++++- src/backend/commands/dbcommands.c | 35 ++++++++++++++++++++++++++++++----- src/backend/parser/gram.y | 3 ++- src/bin/initdb/initdb.c | 24 ++++++++++++++++-------- src/bin/pg_dump/pg_dump.c | 16 ++++++++++++++-- src/bin/pg_upgrade/IMPLEMENTATION | 2 +- src/bin/pg_upgrade/info.c | 9 +++------ src/bin/pg_upgrade/pg_upgrade.h | 3 +-- src/bin/pg_upgrade/relfilenode.c | 4 ++-- src/bin/psql/tab-complete.c | 2 +- src/include/access/transam.h | 3 +++ src/include/catalog/unused_oids | 5 +++++ 12 files changed, 92 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml index 41cb406..94d4cfb 100644 --- a/doc/src/sgml/ref/create_database.sgml +++ b/doc/src/sgml/ref/create_database.sgml @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> [ TABLESPACE [=] <replaceable class="parameter">tablespace_name</replaceable> ] [ ALLOW_CONNECTIONS [=] <replaceable class="parameter">allowconn</replaceable> ] [ CONNECTION LIMIT [=] <replaceable class="parameter">connlimit</replaceable> ] - [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] ] + [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] + [ OID [=] <replaceable class="parameter">oid</replaceable> ] ] </synopsis> </refsynopsisdiv> @@ -203,6 +204,18 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> </para> </listitem> </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">oid</replaceable></term> + <listitem> + <para> + The object identifier with which the database gets created. + The OID range for user objects starts from 16384. CREATE DATABASE fails if a + database with specified oid already exists. + </para> + </listitem> + </varlistentry> + </variablelist> <para> diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 029fab4..e8cdb7f 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -117,7 +117,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) HeapTuple tuple; Datum new_record[Natts_pg_database]; bool new_record_nulls[Natts_pg_database]; - Oid dboid; + Oid dboid = InvalidOid; Oid datdba; ListCell *option; DefElem *dtablespacename = NULL; @@ -217,6 +217,27 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) errhint("Consider using tablespaces instead."), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "oid") == 0) + { + dboid = defGetInt32(defel); + + /* + * Throw an error if the user specified oid < FirstNormalObjectId for + * creating the database. However, we need to allow creating database + * with oid < FirstNormalObjectId for below cases: + * 1. creating template0 with fixed oid during initdb + * 2. creating databases with oids from the old cluster during binary + * upgrade. + */ + if ((dboid < FirstNormalObjectId) && + (strcmp(dbname, "template0") != 0) && + (!IsBinaryUpgrade)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("Invalid value for option \"%s\"", defel->defname), + errhint("The specified OID %u is less than the minimum OID for user objects %u.", + dboid, FirstNormalObjectId)); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* Select an OID for the new database if is not explicitly configured. */ + if (!OidIsValid(dboid)) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 3d4dd43..9572f9a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -711,7 +711,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OF OFF OFFSET OID OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -10416,6 +10416,7 @@ createdb_opt_name: | OWNER { $$ = pstrdup($1); } | TABLESPACE { $$ = pstrdup($1); } | TEMPLATE { $$ = pstrdup($1); } + | OID { $$ = pstrdup($1); } ; /* diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index 403adb0..31aa21a 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -59,6 +59,7 @@ #include "sys/mman.h" #endif +#include "access/transam.h" #include "access/xlog_internal.h" #include "catalog/pg_authid_d.h" #include "catalog/pg_class_d.h" /* pgrminclude ignore */ @@ -1838,15 +1839,14 @@ static void make_template0(FILE *cmdfd) { const char *const *line; - static const char *const template0_setup[] = { - "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false;\n\n", - /* - * We use the OID of template0 to determine datlastsysoid - */ - "UPDATE pg_database SET datlastsysoid = " - " (SELECT oid FROM pg_database " - " WHERE datname = 'template0');\n\n", + /* + * Template0 oid is fixed during initdb to avoid oid conflict across versions + * during binary upgrade. + */ + static const char *const template0_setup[] = { + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID = " + CppAsString2(Template0ObjectId) ";\n\n", /* * Explicitly revoke public create-schema and create-temp-table @@ -1879,6 +1879,14 @@ make_postgres(FILE *cmdfd) static const char *const postgres_setup[] = { "CREATE DATABASE postgres;\n\n", "COMMENT ON DATABASE postgres IS 'default administrative connection database';\n\n", + + /* + * We use the OID of postgres to determine datlastsysoid + */ + "UPDATE pg_database SET datlastsysoid = " + " (SELECT oid FROM pg_database " + " WHERE datname = 'postgres');\n\n", + NULL }; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b41d86b..62e418c 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -3018,8 +3018,20 @@ dumpDatabase(Archive *fout) * are left to the DATABASE PROPERTIES entry, so that they can be applied * after reconnecting to the target DB. */ - appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", - qdatname); + if (dopt->binary_upgrade) + { + /* + * Make sure that binary upgrade propagate the database OID to the new + * cluster + */ + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0 OID = %u", + qdatname, dbCatId.oid); + } + else + { + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", + qdatname); + } if (strlen(encoding) > 0) { appendPQExpBufferStr(creaQry, " ENCODING = "); diff --git a/src/bin/pg_upgrade/IMPLEMENTATION b/src/bin/pg_upgrade/IMPLEMENTATION index 69fcd70..384834a 100644 --- a/src/bin/pg_upgrade/IMPLEMENTATION +++ b/src/bin/pg_upgrade/IMPLEMENTATION @@ -84,7 +84,7 @@ cluster using the first part of the pg_dumpall output. Next, pg_upgrade executes the remainder of the script produced earlier by pg_dumpall --- this script effectively creates the complete user-defined metadata from the old cluster to the new cluster. It -preserves the relfilenode numbers so TOAST and other references +preserves the DB, tablespace, relfilenode OIDs so TOAST and other references to relfilenodes in user data is preserved. (See binary-upgrade usage in pg_dump). diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 775564e..fe1357d 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -196,10 +196,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->new_tablespace_suffix = new_cluster.tablespace_suffix; } - map->old_db_oid = old_db->db_oid; - map->new_db_oid = new_db->db_oid; - - /* relfilenode is preserved across old and new cluster */ + /* DB oid and relfilenodes are preserved between old and new cluster */ + map->db_oid = old_db->db_oid; map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ @@ -330,8 +328,7 @@ get_db_infos(ClusterInfo *cluster) " LEFT OUTER JOIN pg_catalog.pg_tablespace t " " ON d.dattablespace = t.oid " "WHERE d.datallowconn = true " - /* we don't preserve pg_database.oid so we sort by name */ - "ORDER BY 2", + "ORDER BY 1", /* 9.2 removed the spclocation column */ (GET_MAJOR_VERSION(cluster->major_version) <= 901) ? "t.spclocation" : "pg_catalog.pg_tablespace_location(t.oid)"); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index c79ddfc..937c45b 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -157,8 +157,7 @@ typedef struct const char *new_tablespace; const char *old_tablespace_suffix; const char *new_tablespace_suffix; - Oid old_db_oid; - Oid new_db_oid; + Oid db_oid; Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 3bc5b44..a4fc298 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -203,14 +203,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro snprintf(old_file, sizeof(old_file), "%s%s/%u/%u%s%s", map->old_tablespace, map->old_tablespace_suffix, - map->old_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, - map->new_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 2f412ca..414d709 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2587,7 +2587,7 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("OWNER", "TEMPLATE", "ENCODING", "TABLESPACE", "IS_TEMPLATE", "ALLOW_CONNECTIONS", "CONNECTION LIMIT", - "LC_COLLATE", "LC_CTYPE", "LOCALE"); + "LC_COLLATE", "LC_CTYPE", "LOCALE", "OID"); else if (Matches("CREATE", "DATABASE", MatchAny, "TEMPLATE")) COMPLETE_WITH_QUERY(Query_for_list_of_template_databases); diff --git a/src/include/access/transam.h b/src/include/access/transam.h index d22de19..d06e3a9 100644 --- a/src/include/access/transam.h +++ b/src/include/access/transam.h @@ -196,6 +196,9 @@ FullTransactionIdAdvance(FullTransactionId *dest) #define FirstUnpinnedObjectId 12000 #define FirstNormalObjectId 16384 +/* OID 4 is reserved for Template0 database */ +#define Template0ObjectId 4 + /* * VariableCache is a data structure in shared memory that is used to track * OID and XID assignment state. For largely historical reasons, there is diff --git a/src/include/catalog/unused_oids b/src/include/catalog/unused_oids index 5b7ce5f..2750913 100755 --- a/src/include/catalog/unused_oids +++ b/src/include/catalog/unused_oids @@ -32,6 +32,11 @@ my @input_files = glob("pg_*.h"); my $oids = Catalog::FindAllOidsFromHeaders(@input_files); +# Push the OID that is reserved for template0 database. +my $Template0ObjectId = + Catalog::FindDefinedSymbol('access/transam.h', '..', 'Template0ObjectId'); +push @{$oids}, $Template0ObjectId; + # Also push FirstGenbkiObjectId to serve as a terminator for the last gap. my $FirstGenbkiObjectId = Catalog::FindDefinedSymbol('access/transam.h', '..', 'FirstGenbkiObjectId'); -- 1.8.3.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-13 15:13 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Shruthi Gowda @ 2021-12-13 15:13 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Sadhuprasad Patro <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Dec 6, 2021 at 11:25 PM Robert Haas <[email protected]> wrote: > > On Sun, Dec 5, 2021 at 11:44 PM Sadhuprasad Patro <[email protected]> wrote: > > 3. > > @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > > */ > > pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); > > > > - do > > + /* Select an OID for the new database if is not explicitly configured. */ > > + if (!OidIsValid(dboid)) > > { > > - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, > > - Anum_pg_database_oid); > > - } while (check_db_file_conflict(dboid)); > > > > I think we need to do 'check_db_file_conflict' for the USER given OID > > also.. right? It may already be present. > > Hopefully, if that happens, we straight up fail later on. That's right. If a database with user-specified OID exists, the createdb fails with a "duplicate key value" error. If just a data directory with user-specified OID exists, MakePGDirectory() fails to create the directory and the cleanup callback createdb_failure_callback() removes the directory that was not created by 'createdb()' function. The subsequent create database call with the same OID will succeed. Should we handle the case where a data directory exists and the corresponding DB with that oid does not exist? I presume this situation doesn't arise unless the user tries to create directories in the data path. Any thoughts? Thanks & Regards Shruthi KC EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-13 21:05 Robert Haas <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Robert Haas @ 2021-12-13 21:05 UTC (permalink / raw) To: Shruthi Gowda <[email protected]>; +Cc: Sadhuprasad Patro <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Dec 13, 2021 at 9:40 AM Shruthi Gowda <[email protected]> wrote: > > I am reviewing another patch > > "v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg" as well > > and will provide the comments soon if any... I spent much of today reviewing 0001. Here's an updated version, so far only lightly tested. Please check whether I've broken anything. Here are the changes: - I adjusted the function header comment for heap_create. Your proposed comment seemed like it was pretty detailed but not 100% correct. It also made one of the lines kind of long because you didn't wrap the text in the surrounding style. I decided to make it simpler and shorter instead of longer still and 100% correct. - I removed a one-line comment that said /* Override the toast relfilenode */ because it preceded an error check, not a line of code that would have done what the comment claimed. - I removed a one-line comment that said /* Override the relfilenode */ because the following code would only sometimes override the relfilenode. The code didn't seem complex enough to justify a a longer and more accurate comment, so I just took it out. - I changed a test for (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || relkind == RELKIND_MATVIEW) to use RELKIND_HAS_STORAGE(). It's true that not all of the storage types that RELKIND_HAS_STORAGE() tests are possible here, but that's not a reason to avoiding using the macro. If somebody adds a new relkind with storage in the future, they might miss the need to manually update this place, but they will not likely miss the need to update RELKIND_HAS_STORAGE() since, if they did, their code probably wouldn't work at all. - I changed the way that you were passing create_storage down to heap_create. I think I said before that you should EITHER fix it so that we set create_storage = true only when the relation actually has storage OR ELSE have heap_create() itself override the value to false when there is no storage. You did both. There are times when it's reasonable to ensure the same thing in multiple places, but this doesn't seem to be one of them. So I took that out. I chose to retain the code in heap_create() that overrides the value to false, added a comment explaining that it does that, and then adjusted the callers to ignore the storage type. I then added comments, and in one place an assertion, to make it clearer what is happening. - In pg_dump.c, I adjusted the comment that says "Not every relation has storage." and the test that immediately follows, to ignore the relfilenode when relkind says it's a partitioned table. Really, partitioned tables should never have had relfilenodes, but as it turns out, they used to have them. Let me know your thoughts. -- Robert Haas EDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v7-0001-pg_upgrade-Preserve-relfilenodes-and-tablespace-O.patch (30.7K, ../../CA+TgmoZW+yuY=K6J2ts18goQh=GqPUZo4q6d8ucJmnKR=hEabw@mail.gmail.com/2-v7-0001-pg_upgrade-Preserve-relfilenodes-and-tablespace-O.patch) download | inline diff: From e24c58382415ceefe247b6961cd6995b3e413214 Mon Sep 17 00:00:00 2001 From: Robert Haas <[email protected]> Date: Mon, 13 Dec 2021 15:40:13 -0500 Subject: [PATCH v7] pg_upgrade: Preserve relfilenodes and tablespace OIDs. The fact that relfilenodes and tablespace OIDs change when a cluster is upgraded can be confusing for people trying to troubleshoot problems with pg_upgrade. It also isn't much fun if you're trying to use 'rsync' or similar to update a standby after upgrading the master. And if in the future we ever encrypt blocks, we might want to use a salt or nonce that depends on the relfilenode and tablespace OID. For all of these reasons, arrange for pg_dump to preserve them as it already does for relation OIDs. Database OIDs have a similar issue, but there are some tricky points in that case that do not apply to these cases, so that problem is left for another patch. Shruthi KC, based on an earlier patch from Antonin Houska, reviewed and with some adjustments by me. --- src/backend/bootstrap/bootparse.y | 3 +- src/backend/catalog/heap.c | 63 ++++++++--- src/backend/catalog/index.c | 23 +++- src/backend/commands/tablespace.c | 17 ++- src/backend/utils/adt/pg_upgrade_support.c | 44 ++++++++ src/bin/pg_dump/pg_dump.c | 104 ++++++++++++------ src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/info.c | 31 +----- src/bin/pg_upgrade/pg_upgrade.c | 13 ++- src/bin/pg_upgrade/pg_upgrade.h | 10 +- src/bin/pg_upgrade/relfilenode.c | 6 +- src/include/catalog/binary_upgrade.h | 5 + src/include/catalog/heap.h | 3 +- src/include/catalog/pg_proc.dat | 16 +++ .../expected/spgist_name_ops.out | 12 +- 15 files changed, 246 insertions(+), 107 deletions(-) diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y index 5fcd004e1b..6560b19e14 100644 --- a/src/backend/bootstrap/bootparse.y +++ b/src/backend/bootstrap/bootparse.y @@ -212,7 +212,8 @@ Boot_CreateStmt: mapped_relation, true, &relfrozenxid, - &relminmxid); + &relminmxid, + true); elog(DEBUG4, "bootstrap relation created"); } else diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 6780ec53b7..11bbb9a768 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -285,8 +287,12 @@ SystemAttributeByName(const char *attname) * heap_create - Create an uncataloged heap relation * * Note API change: the caller must now always provide the OID - * to use for the relation. The relfilenode may (and, normally, - * should) be left unspecified. + * to use for the relation. The relfilenode may be (and in + * the simplest cases is) left unspecified. + * + * create_storage indicates whether or not to create the storage. + * However, even if create_storage is true, no storage will be + * created if the relkind is one that doesn't have storage. * * rel->rd_rel is initialized by RelationBuildLocalRelation, * and is mostly zeroes at return. @@ -306,9 +312,9 @@ heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid) + MultiXactId *relminmxid, + bool create_storage) { - bool create_storage; Relation rel; /* The caller must have provided an OID for the relation. */ @@ -343,17 +349,17 @@ heap_create(const char *relname, if (!RELKIND_HAS_TABLESPACE(relkind)) reltablespace = InvalidOid; - /* - * Decide whether to create storage. If caller passed a valid relfilenode, - * storage is already created, so don't do it here. Also don't create it - * for relkinds without physical storage. - */ - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + /* Don't create storage for relkinds without physical storage. */ + if (!RELKIND_HAS_STORAGE(relkind)) create_storage = false; else { - create_storage = true; - relfilenode = relid; + /* + * If relfilenode is unspecified by the caller then create storage + * with oid same as relid. + */ + if (!OidIsValid(relfilenode)) + relfilenode = relid; } /* @@ -1121,6 +1127,9 @@ heap_create_with_catalog(const char *relname, Oid existing_relid; Oid old_type_oid; Oid new_type_oid; + + /* By default set to InvalidOid unless overridden by binary-upgrade */ + Oid relfilenode = InvalidOid; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1183,7 +1192,7 @@ heap_create_with_catalog(const char *relname, */ if (!OidIsValid(relid)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { /* @@ -1200,6 +1209,14 @@ heap_create_with_catalog(const char *relname, { relid = binary_upgrade_next_toast_pg_class_oid; binary_upgrade_next_toast_pg_class_oid = InvalidOid; + + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; } } else @@ -1211,6 +1228,17 @@ heap_create_with_catalog(const char *relname, relid = binary_upgrade_next_heap_pg_class_oid; binary_upgrade_next_heap_pg_class_oid = InvalidOid; + + if (RELKIND_HAS_STORAGE(relkind)) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + } } } @@ -1250,12 +1278,16 @@ heap_create_with_catalog(const char *relname, * Create the relcache entry (mostly dummy at this point) and the physical * disk file. (If we fail further down, it's the smgr's responsibility to * remove the disk file again.) + * + * NB: Note that passing create_storage = true is correct even for binary + * upgrade. The storage we create here will be replaced later, but we need + * to have something on disk in the meanwhile. */ new_rel_desc = heap_create(relname, relnamespace, reltablespace, relid, - InvalidOid, + relfilenode, accessmtd, tupdesc, relkind, @@ -1264,7 +1296,8 @@ heap_create_with_catalog(const char *relname, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + true); Assert(relid == RelationGetRelid(new_rel_desc)); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1757cd3446..88f2d60f01 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -87,6 +87,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system @@ -733,6 +734,7 @@ index_create(Relation heapRelation, char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; + bool create_storage = !OidIsValid(relFileNode); /* constraint flags can only be set when a constraint is requested */ Assert((constr_flags == 0) || @@ -904,7 +906,7 @@ index_create(Relation heapRelation, */ if (!OidIsValid(indexRelationId)) { - /* Use binary-upgrade override for pg_class.oid/relfilenode? */ + /* Use binary-upgrade override for pg_class.oid and relfilenode */ if (IsBinaryUpgrade) { if (!OidIsValid(binary_upgrade_next_index_pg_class_oid)) @@ -914,6 +916,22 @@ index_create(Relation heapRelation, indexRelationId = binary_upgrade_next_index_pg_class_oid; binary_upgrade_next_index_pg_class_oid = InvalidOid; + + /* Overide the index relfilenode */ + if ((relkind == RELKIND_INDEX) && + (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode))) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("index relfilenode value not set when in binary upgrade mode"))); + relFileNode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; + + /* + * Note that we want create_storage = true for binary upgrade. + * The storage we create here will be replaced later, but we need + * to have something on disk in the meanwhile. + */ + Assert(create_storage); } else { @@ -940,7 +958,8 @@ index_create(Relation heapRelation, mapped_relation, allow_system_table_mods, &relfrozenxid, - &relminmxid); + &relminmxid, + create_storage); Assert(relfrozenxid == InvalidTransactionId); Assert(relminmxid == InvalidMultiXactId); diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 4b96eec9df..51ffa97576 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -88,6 +88,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -335,8 +336,20 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, - Anum_pg_tablespace_oid); + if (IsBinaryUpgrade) + { + /* Use binary-upgrade override for tablespace oid */ + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = DirectFunctionCall1(namein, CStringGetDatum(stmt->tablespacename)); diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index b5b46d7231..135c382371 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -29,6 +29,17 @@ do { \ errmsg("function can only be called when server is in binary upgrade mode"))); \ } while (0) +Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { @@ -84,6 +95,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +Datum +binary_upgrade_set_next_heap_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { @@ -95,6 +117,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +Datum +binary_upgrade_set_next_index_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { @@ -106,6 +139,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } +Datum +binary_upgrade_set_next_toast_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 10a86f9810..8a8ea596bf 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -4836,73 +4836,105 @@ binary_upgrade_set_pg_class_oids(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index) { + PQExpBuffer upgrade_query = createPQExpBuffer(); + PGresult *upgrade_res; + Oid relfilenode; + Oid toast_oid; + Oid toast_relfilenode; + char relkind; + Oid toast_index_oid; + Oid toast_index_relfilenode; + + /* + * Preserve the OID and relfilenode of the table, table's index, table's + * toast table and toast table's index if any. + * + * One complexity is that the current table definition might not require + * the creation of a TOAST table, but the old database might have a TOAST + * table that was created earlier, before some wide columns were dropped. + * By setting the TOAST oid we force creation of the TOAST heap and index + * by the new backend, so we can copy the files during binary upgrade + * without worrying about this case. + */ + appendPQExpBuffer(upgrade_query, + "SELECT c.relkind, c.relfilenode, c.reltoastrelid, ct.relfilenode AS toast_relfilenode, i.indexrelid, cti.relfilenode AS toast_index_relfilenode " + "FROM pg_catalog.pg_class c LEFT JOIN " + "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " + "WHERE c.oid = '%u'::pg_catalog.oid;", + pg_class_oid); + + upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + + relkind = *PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relkind")); + + relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "relfilenode"))); + toast_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "reltoastrelid"))); + toast_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_relfilenode"))); + toast_index_oid = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "indexrelid"))); + toast_index_relfilenode = atooid(PQgetvalue(upgrade_res, 0, + PQfnumber(upgrade_res, "toast_index_relfilenode"))); + appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { - PQExpBuffer upgrade_query = createPQExpBuffer(); - PGresult *upgrade_res; - Oid pg_class_reltoastrelid; - char pg_class_relkind; - Oid pg_index_indexrelid; - appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); /* - * Preserve the OIDs of the table's toast table and index, if any. - * Indexes cannot have toast tables, so we need not make this probe in - * the index code path. - * - * One complexity is that the current table definition might not - * require the creation of a TOAST table, but the old database might - * have a TOAST table that was created earlier, before some wide - * columns were dropped. By setting the TOAST oid we force creation - * of the TOAST heap and index by the new backend, so we can copy the - * files during binary upgrade without worrying about this case. + * Not every relation has storage. Also, in a pre-v12 database, + * partitioned tables have a relfilenode, which should not be preserved + * when upgrading. */ - appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, c.relkind, i.indexrelid " - "FROM pg_catalog.pg_class c LEFT JOIN " - "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " - "WHERE c.oid = '%u'::pg_catalog.oid;", - pg_class_oid); - - upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); - - pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "reltoastrelid"))); - pg_class_relkind = *PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "relkind")); - pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, - PQfnumber(upgrade_res, "indexrelid"))); + if (OidIsValid(relfilenode) && relkind != RELKIND_PARTITIONED_TABLE) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); /* * In a pre-v12 database, partitioned tables might be marked as having * toast tables, but we should ignore them if so. */ - if (OidIsValid(pg_class_reltoastrelid) && - pg_class_relkind != RELKIND_PARTITIONED_TABLE) + if (OidIsValid(toast_oid) && + relkind != RELKIND_PARTITIONED_TABLE) { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_class_reltoastrelid); + toast_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_relfilenode('%u'::pg_catalog.oid);\n", + toast_relfilenode); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", - pg_index_indexrelid); + toast_index_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + toast_index_relfilenode); } PQclear(upgrade_res); destroyPQExpBuffer(upgrade_query); } else + { + /* Preserve the OID and relfilenode of the index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_relfilenode('%u'::pg_catalog.oid);\n", + relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); } diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 27b95732c8..3f42316a84 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1222,6 +1222,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 5d9a26cf82..775564e580 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -199,14 +199,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->old_db_oid = old_db->db_oid; map->new_db_oid = new_db->db_oid; - /* - * old_relfilenode might differ from pg_class.oid (and hence - * new_relfilenode) because of CLUSTER, REINDEX, or VACUUM FULL. - */ - map->old_relfilenode = old_rel->relfilenode; - - /* new_relfilenode will match old and new pg_class.oid */ - map->new_relfilenode = new_rel->relfilenode; + /* relfilenode is preserved across old and new cluster */ + map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ map->nspname = old_rel->nspname; @@ -278,27 +272,6 @@ report_unmatched_relation(const RelInfo *rel, const DbInfo *db, bool is_new_db) reloid, db->db_name, reldesc); } - -void -print_maps(FileNameMap *maps, int n_maps, const char *db_name) -{ - if (log_opts.verbose) - { - int mapnum; - - pg_log(PG_VERBOSE, "mappings for database \"%s\":\n", db_name); - - for (mapnum = 0; mapnum < n_maps; mapnum++) - pg_log(PG_VERBOSE, "%s.%s: %u to %u\n", - maps[mapnum].nspname, maps[mapnum].relname, - maps[mapnum].old_relfilenode, - maps[mapnum].new_relfilenode); - - pg_log(PG_VERBOSE, "\n\n"); - } -} - - /* * get_db_and_rel_infos() * diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 3628bd74a7..acaf343fb9 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,13 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because we want the filenames + * to match between the old and new cluster. + * + * We control assignment of pg_tablespace.oid because we want the oid to match + * between the old and new cluster. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index ca0795f68f..c79ddfc6e4 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -159,13 +159,7 @@ typedef struct const char *new_tablespace_suffix; Oid old_db_oid; Oid new_db_oid; - - /* - * old/new relfilenodes might differ for pg_largeobject(_metadata) indexes - * due to VACUUM FULL or REINDEX. Other relfilenodes are preserved. - */ - Oid old_relfilenode; - Oid new_relfilenode; + Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ char *relname; @@ -390,8 +384,6 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db, DbInfo *new_db, int *nmaps, const char *old_pgdata, const char *new_pgdata); void get_db_and_rel_infos(ClusterInfo *cluster); -void print_maps(FileNameMap *maps, int n, - const char *db_name); /* option.c */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index 4deae7d985..3bc5b44199 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -119,8 +119,6 @@ transfer_all_new_dbs(DbInfoArr *old_db_arr, DbInfoArr *new_db_arr, new_pgdata); if (n_maps) { - print_maps(mappings, n_maps, new_db->db_name); - transfer_single_new_db(mappings, n_maps, old_tablespace); } /* We allocate something even for n_maps == 0 */ @@ -206,14 +204,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro map->old_tablespace, map->old_tablespace_suffix, map->old_db_oid, - map->old_relfilenode, + map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, map->new_db_oid, - map->new_relfilenode, + map->relfilenode, type_suffix, extent_suffix); diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index f6e82e7ac5..4ba5748e0a 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,19 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_mrng_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h index 6ce480b49c..7cf5402e5a 100644 --- a/src/include/catalog/heap.h +++ b/src/include/catalog/heap.h @@ -59,7 +59,8 @@ extern Relation heap_create(const char *relname, bool mapped_relation, bool allow_system_table_mods, TransactionId *relfrozenxid, - MultiXactId *relminmxid); + MultiXactId *relminmxid, + bool create_storage); extern Oid heap_create_with_catalog(const char *relname, Oid relnamespace, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 79d787cd26..92727d893f 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11058,6 +11058,22 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4545', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_relfilenode' }, +{ oid => '4546', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_relfilenode' }, +{ oid => '4547', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_relfilenode' }, +{ oid => '4548', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' }, # conversion functions { oid => '4302', diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out index ac0ddcecea..1ee65ede24 100644 --- a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out +++ b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out @@ -52,14 +52,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) -- Verify clean failure when INCLUDE'd columns result in overlength tuple -- The error message details are platform-dependent, so show only SQLSTATE @@ -97,14 +101,18 @@ select * from t ------------------------------------------------------+----+------------------------------------------------------ binary_upgrade_set_next_array_pg_type_oid | | binary_upgrade_set_next_array_pg_type_oid binary_upgrade_set_next_heap_pg_class_oid | | binary_upgrade_set_next_heap_pg_class_oid + binary_upgrade_set_next_heap_relfilenode | 1 | binary_upgrade_set_next_heap_relfilenode binary_upgrade_set_next_index_pg_class_oid | 1 | binary_upgrade_set_next_index_pg_class_oid + binary_upgrade_set_next_index_relfilenode | | binary_upgrade_set_next_index_relfilenode binary_upgrade_set_next_multirange_array_pg_type_oid | 1 | binary_upgrade_set_next_multirange_array_pg_type_oid binary_upgrade_set_next_multirange_pg_type_oid | 1 | binary_upgrade_set_next_multirange_pg_type_oid binary_upgrade_set_next_pg_authid_oid | | binary_upgrade_set_next_pg_authid_oid binary_upgrade_set_next_pg_enum_oid | | binary_upgrade_set_next_pg_enum_oid + binary_upgrade_set_next_pg_tablespace_oid | | binary_upgrade_set_next_pg_tablespace_oid binary_upgrade_set_next_pg_type_oid | | binary_upgrade_set_next_pg_type_oid binary_upgrade_set_next_toast_pg_class_oid | 1 | binary_upgrade_set_next_toast_pg_class_oid -(9 rows) + binary_upgrade_set_next_toast_relfilenode | | binary_upgrade_set_next_toast_relfilenode +(13 rows) \set VERBOSITY sqlstate insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000)); -- 2.24.3 (Apple Git-128) ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-14 18:21 Shruthi Gowda <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Shruthi Gowda @ 2021-12-14 18:21 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Sadhuprasad Patro <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Tue, Dec 14, 2021 at 2:35 AM Robert Haas <[email protected]> wrote: > > On Mon, Dec 13, 2021 at 9:40 AM Shruthi Gowda <[email protected]> wrote: > > > I am reviewing another patch > > > "v5-0001-Preserve-relfilenode-and-tablespace-OID-in-pg_upg" as well > > > and will provide the comments soon if any... > > I spent much of today reviewing 0001. Here's an updated version, so > far only lightly tested. Please check whether I've broken anything. > Here are the changes: Thanks, Robert for the updated version. I reviewed the changes and it looks fine. I also tested the patch. The patch works as expected. > - I adjusted the function header comment for heap_create. Your > proposed comment seemed like it was pretty detailed but not 100% > correct. It also made one of the lines kind of long because you didn't > wrap the text in the surrounding style. I decided to make it simpler > and shorter instead of longer still and 100% correct. The comment update looks fine. However, I still feel it would be good to mention on which (rare) circumstance a valid relfilenode can get passed. > - I removed a one-line comment that said /* Override the toast > relfilenode */ because it preceded an error check, not a line of code > that would have done what the comment claimed. > > - I removed a one-line comment that said /* Override the relfilenode > */ because the following code would only sometimes override the > relfilenode. The code didn't seem complex enough to justify a a longer > and more accurate comment, so I just took it out. Fine > - I changed a test for (relkind == RELKIND_RELATION || relkind == > RELKIND_SEQUENCE || relkind == RELKIND_MATVIEW) to use > RELKIND_HAS_STORAGE(). It's true that not all of the storage types > that RELKIND_HAS_STORAGE() tests are possible here, but that's not a > reason to avoiding using the macro. If somebody adds a new relkind > with storage in the future, they might miss the need to manually > update this place, but they will not likely miss the need to update > RELKIND_HAS_STORAGE() since, if they did, their code probably wouldn't > work at all. I agree. > - I changed the way that you were passing create_storage down to > heap_create. I think I said before that you should EITHER fix it so > that we set create_storage = true only when the relation actually has > storage OR ELSE have heap_create() itself override the value to false > when there is no storage. You did both. There are times when it's > reasonable to ensure the same thing in multiple places, but this > doesn't seem to be one of them. So I took that out. I chose to retain > the code in heap_create() that overrides the value to false, added a > comment explaining that it does that, and then adjusted the callers to > ignore the storage type. I then added comments, and in one place an > assertion, to make it clearer what is happening. The changes are fine. Thanks for the fine-tuning. > - In pg_dump.c, I adjusted the comment that says "Not every relation > has storage." and the test that immediately follows, to ignore the > relfilenode when relkind says it's a partitioned table. Really, > partitioned tables should never have had relfilenodes, but as it turns > out, they used to have them. > Fine. Understood. Thanks & Regards, Shruthi KC EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-14 18:39 tushar <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: tushar @ 2021-12-14 18:39 UTC (permalink / raw) To: Robert Haas <[email protected]>; Shruthi Gowda <[email protected]>; +Cc: Sadhuprasad Patro <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On 12/14/21 2:35 AM, Robert Haas wrote: > I spent much of today reviewing 0001. Here's an updated version, so > far only lightly tested. Please check whether I've broken anything. Thanks Robert, I tested from v96/12/13/v14 -> v15( with patch) things are working fine i.e table /index relfilenode is preserved, not changing after pg_upgrade. -- regards,tushar EnterpriseDB https://www.enterprisedb.com/ The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-15 17:03 tushar <[email protected]> parent: tushar <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: tushar @ 2021-12-15 17:03 UTC (permalink / raw) To: Robert Haas <[email protected]>; Shruthi Gowda <[email protected]>; +Cc: Sadhuprasad Patro <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On 12/15/21 12:09 AM, tushar wrote: >> I spent much of today reviewing 0001. Here's an updated version, so >> far only lightly tested. Please check whether I've broken anything. > Thanks Robert, I tested from v96/12/13/v14 -> v15( with patch) > things are working fine i.e table /index relfilenode is preserved, > not changing after pg_upgrade. I covered tablespace OIDs testing scenarios and that is also preserved after pg_upgrade. -- regards,tushar EnterpriseDB https://www.enterprisedb.com/ The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) @ 2021-12-17 07:33 Shruthi Gowda <[email protected]> parent: Shruthi Gowda <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Shruthi Gowda @ 2021-12-17 07:33 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Sadhuprasad Patro <[email protected]>; Stephen Frost <[email protected]>; Bruce Momjian <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Masahiko Sawada <[email protected]>; Tom Kincaid <[email protected]>; Amit Kapila <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]> On Mon, Dec 13, 2021 at 8:43 PM Shruthi Gowda <[email protected]> wrote: > > On Mon, Dec 6, 2021 at 11:25 PM Robert Haas <[email protected]> wrote: > > > > On Sun, Dec 5, 2021 at 11:44 PM Sadhuprasad Patro <[email protected]> wrote: > > > 3. > > > @@ -504,11 +525,15 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) > > > */ > > > pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); > > > > > > - do > > > + /* Select an OID for the new database if is not explicitly configured. */ > > > + if (!OidIsValid(dboid)) > > > { > > > - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, > > > - Anum_pg_database_oid); > > > - } while (check_db_file_conflict(dboid)); > > > > > > I think we need to do 'check_db_file_conflict' for the USER given OID > > > also.. right? It may already be present. > > > > Hopefully, if that happens, we straight up fail later on. > > That's right. If a database with user-specified OID exists, the > createdb fails with a "duplicate key value" error. > If just a data directory with user-specified OID exists, > MakePGDirectory() fails to create the directory and the cleanup > callback createdb_failure_callback() removes the directory that was > not created by 'createdb()' function. > The subsequent create database call with the same OID will succeed. > Should we handle the case where a data directory exists and the > corresponding DB with that oid does not exist? I presume this > situation doesn't arise unless the user tries to create directories in > the data path. Any thoughts? I have updated the DBOID preserve patch to handle this case and generated the latest patch on top of your v7-001-preserve-relfilenode patch. Thanks & Regards Shruthi KC EnterpriseDB: http://www.enterprisedb.com Attachments: [application/octet-stream] v7-0002-Preserve-database-OIDs-in-pg_upgrade.patch (13.1K, ../../CAASxf_Nmjh-Do92L6RXjjYGMnUZKx7RK1D266xRWhKjPA_7Egw@mail.gmail.com/2-v7-0002-Preserve-database-OIDs-in-pg_upgrade.patch) download | inline diff: From 84cbe8cc7202fbb643d5ee9204cddab6d7ef856c Mon Sep 17 00:00:00 2001 From: shruthikc-gowda <[email protected]> Date: Fri, 17 Dec 2021 12:55:27 +0530 Subject: [PATCH v7 2/2] Preserve database OIDs in pg_upgrade The patch aims to preserve the database OIDs across binary upgrade Author: Shruthi KC, based on an earlier patch from Antonin Houska Discussion: https://www.postgresql.org/message-id/7082.1562337694@localhost --- doc/src/sgml/ref/create_database.sgml | 15 +++++++++- src/backend/commands/dbcommands.c | 54 +++++++++++++++++++++++++++++++---- src/backend/parser/gram.y | 3 +- src/bin/initdb/initdb.c | 24 ++++++++++------ src/bin/pg_dump/pg_dump.c | 16 +++++++++-- src/bin/pg_upgrade/IMPLEMENTATION | 2 +- src/bin/pg_upgrade/info.c | 9 ++---- src/bin/pg_upgrade/pg_upgrade.h | 3 +- src/bin/pg_upgrade/relfilenode.c | 4 +-- src/bin/psql/tab-complete.c | 2 +- src/include/access/transam.h | 3 ++ src/include/catalog/unused_oids | 5 ++++ 12 files changed, 111 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/ref/create_database.sgml b/doc/src/sgml/ref/create_database.sgml index 41cb406..94d4cfb 100644 --- a/doc/src/sgml/ref/create_database.sgml +++ b/doc/src/sgml/ref/create_database.sgml @@ -31,7 +31,8 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> [ TABLESPACE [=] <replaceable class="parameter">tablespace_name</replaceable> ] [ ALLOW_CONNECTIONS [=] <replaceable class="parameter">allowconn</replaceable> ] [ CONNECTION LIMIT [=] <replaceable class="parameter">connlimit</replaceable> ] - [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] ] + [ IS_TEMPLATE [=] <replaceable class="parameter">istemplate</replaceable> ] + [ OID [=] <replaceable class="parameter">oid</replaceable> ] ] </synopsis> </refsynopsisdiv> @@ -203,6 +204,18 @@ CREATE DATABASE <replaceable class="parameter">name</replaceable> </para> </listitem> </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">oid</replaceable></term> + <listitem> + <para> + The object identifier with which the database gets created. + The OID range for user objects starts from 16384. CREATE DATABASE fails if a + database with specified oid already exists. + </para> + </listitem> + </varlistentry> + </variablelist> <para> diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 029fab4..2852cfd 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -117,7 +117,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) HeapTuple tuple; Datum new_record[Natts_pg_database]; bool new_record_nulls[Natts_pg_database]; - Oid dboid; + Oid dboid = InvalidOid; Oid datdba; ListCell *option; DefElem *dtablespacename = NULL; @@ -217,6 +217,27 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) errhint("Consider using tablespaces instead."), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "oid") == 0) + { + dboid = defGetInt32(defel); + + /* + * Throw an error if the user specified oid < FirstNormalObjectId for + * creating the database. However, we need to allow creating database + * with oid < FirstNormalObjectId for below cases: + * 1. creating template0 with fixed oid during initdb + * 2. creating databases with oids from the old cluster during binary + * upgrade. + */ + if ((dboid < FirstNormalObjectId) && + (strcmp(dbname, "template0") != 0) && + (!IsBinaryUpgrade)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("Invalid value for option \"%s\"", defel->defname), + errhint("The specified OID %u is less than the minimum OID for user objects %u.", + dboid, FirstNormalObjectId)); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -504,11 +525,34 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* + * If database OID is configured, check if the OID is already in use or + * data directory already exists. + */ + if (OidIsValid(dboid)) + { + char *existing_dbname = get_database_name(dboid); + + if (existing_dbname != NULL) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("database oid %u is already used by database %s", + dboid, existing_dbname)); + + if (check_db_file_conflict(dboid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE)), + errmsg("data directory exists for database oid %u", dboid)); + } + else { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + /* Select an OID for the new database if is not explicitly configured. */ + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 3d4dd43..9572f9a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -711,7 +711,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OF OFF OFFSET OID OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -10416,6 +10416,7 @@ createdb_opt_name: | OWNER { $$ = pstrdup($1); } | TABLESPACE { $$ = pstrdup($1); } | TEMPLATE { $$ = pstrdup($1); } + | OID { $$ = pstrdup($1); } ; /* diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index 03b80f9..34abb30 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -59,6 +59,7 @@ #include "sys/mman.h" #endif +#include "access/transam.h" #include "access/xlog_internal.h" #include "catalog/pg_authid_d.h" #include "catalog/pg_class_d.h" /* pgrminclude ignore */ @@ -1838,15 +1839,14 @@ static void make_template0(FILE *cmdfd) { const char *const *line; - static const char *const template0_setup[] = { - "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false;\n\n", - /* - * We use the OID of template0 to determine datlastsysoid - */ - "UPDATE pg_database SET datlastsysoid = " - " (SELECT oid FROM pg_database " - " WHERE datname = 'template0');\n\n", + /* + * Template0 oid is fixed during initdb to avoid oid conflict across versions + * during binary upgrade. + */ + static const char *const template0_setup[] = { + "CREATE DATABASE template0 IS_TEMPLATE = true ALLOW_CONNECTIONS = false OID = " + CppAsString2(Template0ObjectId) ";\n\n", /* * Explicitly revoke public create-schema and create-temp-table @@ -1879,6 +1879,14 @@ make_postgres(FILE *cmdfd) static const char *const postgres_setup[] = { "CREATE DATABASE postgres;\n\n", "COMMENT ON DATABASE postgres IS 'default administrative connection database';\n\n", + + /* + * We use the OID of postgres to determine datlastsysoid + */ + "UPDATE pg_database SET datlastsysoid = " + " (SELECT oid FROM pg_database " + " WHERE datname = 'postgres');\n\n", + NULL }; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index c3609af..97193cf 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -2840,8 +2840,20 @@ dumpDatabase(Archive *fout) * are left to the DATABASE PROPERTIES entry, so that they can be applied * after reconnecting to the target DB. */ - appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", - qdatname); + if (dopt->binary_upgrade) + { + /* + * Make sure that binary upgrade propagate the database OID to the new + * cluster + */ + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0 OID = %u", + qdatname, dbCatId.oid); + } + else + { + appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0", + qdatname); + } if (strlen(encoding) > 0) { appendPQExpBufferStr(creaQry, " ENCODING = "); diff --git a/src/bin/pg_upgrade/IMPLEMENTATION b/src/bin/pg_upgrade/IMPLEMENTATION index 69fcd70..384834a 100644 --- a/src/bin/pg_upgrade/IMPLEMENTATION +++ b/src/bin/pg_upgrade/IMPLEMENTATION @@ -84,7 +84,7 @@ cluster using the first part of the pg_dumpall output. Next, pg_upgrade executes the remainder of the script produced earlier by pg_dumpall --- this script effectively creates the complete user-defined metadata from the old cluster to the new cluster. It -preserves the relfilenode numbers so TOAST and other references +preserves the DB, tablespace, relfilenode OIDs so TOAST and other references to relfilenodes in user data is preserved. (See binary-upgrade usage in pg_dump). diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index ded8cbe..e74ee12 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -190,10 +190,8 @@ create_rel_filename_map(const char *old_data, const char *new_data, map->new_tablespace_suffix = new_cluster.tablespace_suffix; } - map->old_db_oid = old_db->db_oid; - map->new_db_oid = new_db->db_oid; - - /* relfilenode is preserved across old and new cluster */ + /* DB oid and relfilenodes are preserved between old and new cluster */ + map->db_oid = old_db->db_oid; map->relfilenode = old_rel->relfilenode; /* used only for logging and error reporting, old/new are identical */ @@ -324,8 +322,7 @@ get_db_infos(ClusterInfo *cluster) " LEFT OUTER JOIN pg_catalog.pg_tablespace t " " ON d.dattablespace = t.oid " "WHERE d.datallowconn = true " - /* we don't preserve pg_database.oid so we sort by name */ - "ORDER BY 2"); + "ORDER BY 1"); res = executeQueryOrDie(conn, "%s", query); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index b59252b..26ef6fc 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -145,8 +145,7 @@ typedef struct const char *new_tablespace; const char *old_tablespace_suffix; const char *new_tablespace_suffix; - Oid old_db_oid; - Oid new_db_oid; + Oid db_oid; Oid relfilenode; /* the rest are used only for logging and error reporting */ char *nspname; /* namespaces */ diff --git a/src/bin/pg_upgrade/relfilenode.c b/src/bin/pg_upgrade/relfilenode.c index ea3f3d4..cfb2347 100644 --- a/src/bin/pg_upgrade/relfilenode.c +++ b/src/bin/pg_upgrade/relfilenode.c @@ -193,14 +193,14 @@ transfer_relfile(FileNameMap *map, const char *type_suffix, bool vm_must_add_fro snprintf(old_file, sizeof(old_file), "%s%s/%u/%u%s%s", map->old_tablespace, map->old_tablespace_suffix, - map->old_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); snprintf(new_file, sizeof(new_file), "%s%s/%u/%u%s%s", map->new_tablespace, map->new_tablespace_suffix, - map->new_db_oid, + map->db_oid, map->relfilenode, type_suffix, extent_suffix); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index b524dc8..9c8b0b0 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2587,7 +2587,7 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH("OWNER", "TEMPLATE", "ENCODING", "TABLESPACE", "IS_TEMPLATE", "ALLOW_CONNECTIONS", "CONNECTION LIMIT", - "LC_COLLATE", "LC_CTYPE", "LOCALE"); + "LC_COLLATE", "LC_CTYPE", "LOCALE", "OID"); else if (Matches("CREATE", "DATABASE", MatchAny, "TEMPLATE")) COMPLETE_WITH_QUERY(Query_for_list_of_template_databases); diff --git a/src/include/access/transam.h b/src/include/access/transam.h index d22de19..d06e3a9 100644 --- a/src/include/access/transam.h +++ b/src/include/access/transam.h @@ -196,6 +196,9 @@ FullTransactionIdAdvance(FullTransactionId *dest) #define FirstUnpinnedObjectId 12000 #define FirstNormalObjectId 16384 +/* OID 4 is reserved for Template0 database */ +#define Template0ObjectId 4 + /* * VariableCache is a data structure in shared memory that is used to track * OID and XID assignment state. For largely historical reasons, there is diff --git a/src/include/catalog/unused_oids b/src/include/catalog/unused_oids index 5b7ce5f..2750913 100755 --- a/src/include/catalog/unused_oids +++ b/src/include/catalog/unused_oids @@ -32,6 +32,11 @@ my @input_files = glob("pg_*.h"); my $oids = Catalog::FindAllOidsFromHeaders(@input_files); +# Push the OID that is reserved for template0 database. +my $Template0ObjectId = + Catalog::FindDefinedSymbol('access/transam.h', '..', 'Template0ObjectId'); +push @{$oids}, $Template0ObjectId; + # Also push FirstGenbkiObjectId to serve as a terminator for the last gap. my $FirstGenbkiObjectId = Catalog::FindDefinedSymbol('access/transam.h', '..', 'FirstGenbkiObjectId'); -- 1.8.3.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-04-13 06:47 Pavel Luzanov <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Pavel Luzanov @ 2025-04-13 06:47 UTC (permalink / raw) To: Tom Lane <[email protected]>; Abhishek Chanda <[email protected]>; +Cc: pgsql-hackers On 13.04.2025 08:29, Tom Lane wrote: > Abhishek Chanda<[email protected]> writes: >> Currently, some slash commands in psql return an error saying "Did not >> find any XXXX named YYYY" while some return an empty table. This patch >> changes a few of the slash commands to return a similar error message. +1 for this patch. > Personally, if I were trying to make these things consistent, I'd have > gone in the other direction (ie return empty tables). +1 Returning empty tables is a more appropriate behavior. -- Pavel Luzanov Postgres Professional:https://postgrespro.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-04-13 16:40 Abhishek Chanda <[email protected]> parent: Pavel Luzanov <[email protected]> 0 siblings, 2 replies; 78+ messages in thread From: Abhishek Chanda @ 2025-04-13 16:40 UTC (permalink / raw) To: Pavel Luzanov <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers Thanks for the feedback, attached an updated patch that changes most of those "Did not find" messages to empty tables. I did not change two sets, listDbRoleSettings and listTables both have comments describing that these are deliberately this way. I wanted to post this update to close this loop for now. I will follow up once the merge window opens again. Thanks On Sun, Apr 13, 2025 at 1:47 AM Pavel Luzanov <[email protected]> wrote: > > On 13.04.2025 08:29, Tom Lane wrote: > > Abhishek Chanda <[email protected]> writes: > > Currently, some slash commands in psql return an error saying "Did not > find any XXXX named YYYY" while some return an empty table. This patch > changes a few of the slash commands to return a similar error message. > > > +1 for this patch. > > Personally, if I were trying to make these things consistent, I'd have > gone in the other direction (ie return empty tables). > > > +1 > Returning empty tables is a more appropriate behavior. > > -- > Pavel Luzanov > Postgres Professional: https://postgrespro.com -- Thanks and regards Abhishek Chanda Attachments: [application/octet-stream] v2-0001-Print-empty-table-when-a-given-object-is-not-foun.patch (3.6K, ../../CAKiP-K-QgovuxqunKJHHmL+i7px2Jc30gX8-rvAj7ug9E1f7zw@mail.gmail.com/2-v2-0001-Print-empty-table-when-a-given-object-is-not-foun.patch) download | inline diff: From 36377dd05b7743a879606bc36c10a9d6c12a7fc5 Mon Sep 17 00:00:00 2001 From: Abhishek Chanda <[email protected]> Date: Sat, 12 Apr 2025 22:30:17 -0500 Subject: [PATCH v2] Print empty table when a given object is not found in a slash command Currently, in a few cases we return an error of the form Did not find any XXXX named YYYY while in some cases we print a table with no rows. This patch changes a few of the former to the later so that we have an uniform interface. --- src/bin/psql/describe.c | 81 +---------------------------------------- 1 file changed, 1 insertion(+), 80 deletions(-) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 1d08268393e..ebb5d3b0c69 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1519,20 +1519,6 @@ describeTableDetails(const char *pattern, bool verbose, bool showSystem) if (!res) return false; - if (PQntuples(res) == 0) - { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any relation named \"%s\".", - pattern); - else - pg_log_error("Did not find any relations."); - } - PQclear(res); - return false; - } - for (i = 0; i < PQntuples(res); i++) { const char *oid; @@ -1719,14 +1705,6 @@ describeOneTableDetails(const char *schemaname, if (!res) goto error_return; - /* Did we get anything? */ - if (PQntuples(res) == 0) - { - if (!pset.quiet) - pg_log_error("Did not find any relation with OID %s.", oid); - goto error_return; - } - tableinfo.checks = atoi(PQgetvalue(res, 0, 0)); tableinfo.relkind = *(PQgetvalue(res, 0, 1)); tableinfo.hasindex = strcmp(PQgetvalue(res, 0, 2), "t") == 0; @@ -3762,6 +3740,7 @@ describeRoles(const char *pattern, bool verbose, bool showSystem) return false; nrows = PQntuples(res); + attr = pg_malloc0((nrows + 1) * sizeof(*attr)); printTableInit(&cont, &myopt, _("List of roles"), ncols, nrows); @@ -5403,20 +5382,6 @@ listTSParsersVerbose(const char *pattern) if (!res) return false; - if (PQntuples(res) == 0) - { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any text search parser named \"%s\".", - pattern); - else - pg_log_error("Did not find any text search parsers."); - } - PQclear(res); - return false; - } - for (i = 0; i < PQntuples(res); i++) { const char *oid; @@ -5781,20 +5746,6 @@ listTSConfigsVerbose(const char *pattern) if (!res) return false; - if (PQntuples(res) == 0) - { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any text search configuration named \"%s\".", - pattern); - else - pg_log_error("Did not find any text search configurations."); - } - PQclear(res); - return false; - } - for (i = 0; i < PQntuples(res); i++) { const char *oid; @@ -6256,20 +6207,6 @@ listExtensionContents(const char *pattern) if (!res) return false; - if (PQntuples(res) == 0) - { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any extension named \"%s\".", - pattern); - else - pg_log_error("Did not find any extensions."); - } - PQclear(res); - return false; - } - for (i = 0; i < PQntuples(res); i++) { const char *extname; @@ -6603,22 +6540,6 @@ describePublications(const char *pattern) return false; } - if (PQntuples(res) == 0) - { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any publication named \"%s\".", - pattern); - else - pg_log_error("Did not find any publications."); - } - - termPQExpBuffer(&buf); - PQclear(res); - return false; - } - for (i = 0; i < PQntuples(res); i++) { const char align = 'l'; -- 2.49.0 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-04-14 09:27 Pavel Luzanov <[email protected]> parent: Abhishek Chanda <[email protected]> 1 sibling, 2 replies; 78+ messages in thread From: Pavel Luzanov @ 2025-04-14 09:27 UTC (permalink / raw) To: Abhishek Chanda <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers On 13.04.2025 19:40, Abhishek Chanda wrote: > Thanks for the feedback, attached an updated patch that changes most > of those "Did not find" messages to empty tables. I did not change two > sets, listDbRoleSettings and listTables both have comments describing > that these are deliberately this way. Thanks. > I wanted to post this update to close this loop for now. I will follow > up once the merge window opens again. I recommend to create a new entry for this patch in the open July commitfest <https://commitfest.postgresql.org/53/;. I will be able to review this patch in a couple months later in June, if no one wants to review it earlier. -- Pavel Luzanov Postgres Professional:https://postgrespro.com ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-04-14 15:05 Abhishek Chanda <[email protected]> parent: Pavel Luzanov <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Abhishek Chanda @ 2025-04-14 15:05 UTC (permalink / raw) To: Pavel Luzanov <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers Thanks Pavel, done now https://commitfest.postgresql.org/patch/5699/ Thanks On Mon, Apr 14, 2025 at 4:27 AM Pavel Luzanov <[email protected]> wrote: > > On 13.04.2025 19:40, Abhishek Chanda wrote: > > Thanks for the feedback, attached an updated patch that changes most > of those "Did not find" messages to empty tables. I did not change two > sets, listDbRoleSettings and listTables both have comments describing > that these are deliberately this way. > > > Thanks. > > I wanted to post this update to close this loop for now. I will follow > up once the merge window opens again. > > > I recommend to create a new entry for this patch in the open July commitfest. > I will be able to review this patch in a couple months later in June, > if no one wants to review it earlier. > > -- > Pavel Luzanov > Postgres Professional: https://postgrespro.com -- Thanks and regards Abhishek Chanda ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-05-12 21:00 Robin Haberkorn <[email protected]> parent: Pavel Luzanov <[email protected]> 1 sibling, 1 reply; 78+ messages in thread From: Robin Haberkorn @ 2025-05-12 21:00 UTC (permalink / raw) To: Pavel Luzanov <[email protected]>; Abhishek Chanda <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers On Mon Apr 14, 2025 at 12:27:53 GMT +03, Pavel Luzanov wrote: > I recommend to create a new entry for this patch in the open July > commitfest <https://commitfest.postgresql.org/53/;. > I will be able to review this patch in a couple months later in June, > if no one wants to review it earlier. I could review it right now, i.e. do a pre-commit review according to [1]. Still have to "pay off" my own Commitfest entry. This would be my first PG review. But is it even desirable to write reviews before the beginning of the Commitfest? An important part -- especially in simple patches like this one -- would be to apply it to HEAD. And shouldn't that be better done as late as possible? [1] https://wiki.postgresql.org/wiki/Reviewing_a_Patch -- Robin Haberkorn Senior Software Engineer B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / https://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt, HRB 3537 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-07-01 11:03 Dean Rasheed <[email protected]> parent: Abhishek Chanda <[email protected]> 1 sibling, 0 replies; 78+ messages in thread From: Dean Rasheed @ 2025-07-01 11:03 UTC (permalink / raw) To: Abhishek Chanda <[email protected]>; +Cc: Pavel Luzanov <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Sun, 13 Apr 2025 at 17:40, Abhishek Chanda <[email protected]> wrote: > > Thanks for the feedback, attached an updated patch that changes most > of those "Did not find" messages to empty tables. I did not change two > sets, listDbRoleSettings and listTables both have comments describing > that these are deliberately this way. > +1 for always displaying an empty table if no objects are found. The reason given for listDbRoleSettings() being an exception could be solved by making the title depend on the number of arguments, e.g., "List of settings for role \"%s\" and database \"%s\".". It's not clear what the "historical reasons" are for listTables() being an exception, but if we're changing any of these, I'd say we should make them all consistent. Regards, Dean ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-07-11 09:15 Robin Haberkorn <[email protected]> parent: Robin Haberkorn <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Robin Haberkorn @ 2025-07-11 09:15 UTC (permalink / raw) To: Pavel Luzanov <[email protected]>; Abhishek Chanda <[email protected]>; +Cc: Tom Lane <[email protected]>; pgsql-hackers Hello Abhishek, I have reviewed your patch after there was no response to my initial proposal and nobody else wrote a review yet. The patch is in the correct unified-diff format (apparently generated via git format-patch). It applies cleanly to the current master branch. The patch does not however add any tests (e.g. to src/bin/psql/t/001_basic.pl). On the other hand, there weren't any tests for the affected slash-commands, so the patch doesn't break the test suite. The patch tries to remove special "Did not find any XXXX named YYYY" error messages for `\d` commands (`\d`, `\dx+`, `\dRp+`, `\dFp` and `\dF`) in psql, printing empty tables instead. This would make the behavior of the `\d` commands more consistent with the the behavior of ordinary user queries. On the other hand, the change in question could well break existing scripts calling `psql -c '\d ...'`, especially since the process return code changes. For instance, before the proposed change, psql would fail with return code 1: # psql -c '\d foo'; echo $? Did not find any relation named "foo". 1 After applying the patch, the return code will be 0 (success) instead: # psql -c '\d foo'; echo $? 0 I would suggest to rework the patch, so that 1 is still returned. This is also in line with how UNIX tools like `grep` behave in case they report an "empty" response (i.e. if `grep` does not produce any match in the given files). On the other hand returning 1 without printing any error message might create new inconsistencies in psql. More feedback is required from the community on that question. If the return code is considered important by the community, it would be a good reason for adding a test case, so that psql behavior doesn't change unexpectedly in the future. It is noteworthy, that both before and after the patch, a plain `\d` does output an error message while the psql process still returns with code 0: # psql -c '\d'; echo $? Did not find any relations. 0 You clarified you didn't remove the messages because of code comments in listTables() and listDbRoleSettings(). A failure return code however should probably still be returned (if we decide that this is what all the other \d commands should do). The examples above also raise another possible issue. The commands `\d`, `\dx+` and `\dRp+` actually do not print empty tables (instead of "Did not find any XXXX named YYYY") when invoked with a pattern, but produce no output at all. This probably makes sense considering that they could also print output for multiple items (e.g. tables). The remaining affected commands (`\dFp` and `\dF`) will actually print empty tables now. Another point of criticism is that the patch needlessly adds an empty line in describeRoles() - moreover a line with trailing whitespace. I would suggest to remove that change before committing. To summarize, in my opinion you should: * Get feedback on the desired return codes of psql in case of empty responses. * Fix the return codes if others agree that psql should return failure codes and add a test case. * Remove the unnecessary change in describeRoles(). Best regards, Robin Haberkorn On Tue May 13, 2025 at 00:00:17 GMT +03, Robin Haberkorn wrote: > On Mon Apr 14, 2025 at 12:27:53 GMT +03, Pavel Luzanov wrote: >> I recommend to create a new entry for this patch in the open July >> commitfest <https://commitfest.postgresql.org/53/;. >> I will be able to review this patch in a couple months later in June, >> if no one wants to review it earlier. > > I could review it right now, i.e. do a pre-commit review according to [1]. > Still have to "pay off" my own Commitfest entry. This would be my first PG > review. But is it even desirable to write reviews before the beginning of the > Commitfest? An important part -- especially in simple patches like this one > -- would be to apply it to HEAD. And shouldn't that be better done as late as > possible? > > [1] https://wiki.postgresql.org/wiki/Reviewing_a_Patch -- Robin Haberkorn Software Engineer B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / https://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt, HRB 3537 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-10-16 03:56 Abhishek Chanda <[email protected]> parent: Robin Haberkorn <[email protected]> 0 siblings, 1 reply; 78+ messages in thread From: Abhishek Chanda @ 2025-10-16 03:56 UTC (permalink / raw) To: Robin Haberkorn <[email protected]>; +Cc: Pavel Luzanov <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers Hi all, Thanks a lot for the review, Robin. And I am terribly sorry for the slow reply, it just took me a while to get to this. But I think I have addressed all your feedback, I have changed everything to set an error code of 1 if anything is not empty. Also added tests for the return code as requested, and cleaned up the change in describeRoles(). Please let me know if I missed anything. Attached an updated and rebased version of the patch. Thanks On Fri, Jul 11, 2025 at 4:15 AM Robin Haberkorn <[email protected]> wrote: > > Hello Abhishek, > > I have reviewed your patch after there was no response to my initial proposal > and nobody else wrote a review yet. > > The patch is in the correct unified-diff format (apparently generated via git > format-patch). It applies cleanly to the current master branch. The patch does > not however add any tests (e.g. to src/bin/psql/t/001_basic.pl). On the other > hand, there weren't any tests for the affected slash-commands, so the patch > doesn't break the test suite. > > The patch tries to remove special "Did not find any XXXX named YYYY" error > messages for `\d` commands (`\d`, `\dx+`, `\dRp+`, `\dFp` and `\dF`) in psql, > printing empty tables instead. This would make the behavior of the `\d` > commands more consistent with the the behavior of ordinary user queries. On the > other hand, the change in question could well break existing scripts calling > `psql -c '\d ...'`, especially since the process return code changes. For > instance, before the proposed change, psql would fail with return code 1: > > # psql -c '\d foo'; echo $? > Did not find any relation named "foo". > 1 > > After applying the patch, the return code will be 0 (success) instead: > > # psql -c '\d foo'; echo $? > 0 > > I would suggest to rework the patch, so that 1 is still returned. This is also > in line with how UNIX tools like `grep` behave in case they report an "empty" > response (i.e. if `grep` does not produce any match in the given files). On the > other hand returning 1 without printing any error message might create new > inconsistencies in psql. More feedback is required from the community on that > question. If the return code is considered important by the community, it would > be a good reason for adding a test case, so that psql behavior doesn't change > unexpectedly in the future. > > It is noteworthy, that both before and after the patch, a plain `\d` does > output an error message while the psql process still returns with code 0: > > # psql -c '\d'; echo $? > Did not find any relations. > 0 > > You clarified you didn't remove the messages because of code comments in > listTables() and listDbRoleSettings(). A failure return code however should > probably still be returned (if we decide that this is what all the other \d > commands should do). > > The examples above also raise another possible issue. The commands `\d`, `\dx+` > and `\dRp+` actually do not print empty tables (instead of "Did not find any > XXXX named YYYY") when invoked with a pattern, but produce no output at all. > This probably makes sense considering that they could also print output for > multiple items (e.g. tables). The remaining affected commands (`\dFp` and > `\dF`) will actually print empty tables now. > > Another point of criticism is that the patch needlessly adds an empty line in > describeRoles() - moreover a line with trailing whitespace. I would suggest to > remove that change before committing. > > To summarize, in my opinion you should: > > * Get feedback on the desired return codes of psql in case of > empty responses. > * Fix the return codes if others agree that psql should return failure codes > and add a test case. > * Remove the unnecessary change in describeRoles(). > > Best regards, > Robin Haberkorn > > On Tue May 13, 2025 at 00:00:17 GMT +03, Robin Haberkorn wrote: > > On Mon Apr 14, 2025 at 12:27:53 GMT +03, Pavel Luzanov wrote: > >> I recommend to create a new entry for this patch in the open July > >> commitfest <https://commitfest.postgresql.org/53/;. > >> I will be able to review this patch in a couple months later in June, > >> if no one wants to review it earlier. > > > > I could review it right now, i.e. do a pre-commit review according to [1]. > > Still have to "pay off" my own Commitfest entry. This would be my first PG > > review. But is it even desirable to write reviews before the beginning of the > > Commitfest? An important part -- especially in simple patches like this one > > -- would be to apply it to HEAD. And shouldn't that be better done as late as > > possible? > > > > [1] https://wiki.postgresql.org/wiki/Reviewing_a_Patch > > -- > Robin Haberkorn > Software Engineer > > B1 Systems GmbH > Osterfeldstraße 7 / 85088 Vohburg / https://www.b1-systems.de > GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt, HRB 3537 -- Thanks and regards Abhishek Chanda Attachments: [application/octet-stream] v3-0001-Print-empty-table-when-a-given-object-is-not-foun.patch (17.6K, ../../CAKiP-K-MNi9h4xwaykp7XFv+xuG7=Eq7wV_cRY8aBPd0Pa_sdQ@mail.gmail.com/2-v3-0001-Print-empty-table-when-a-given-object-is-not-foun.patch) download | inline diff: From 86ea629bc4ede9c51f9c000caddf1969b593b569 Mon Sep 17 00:00:00 2001 From: Abhishek Chanda <[email protected]> Date: Sat, 12 Apr 2025 22:30:17 -0500 Subject: [PATCH v3] Print empty table when a given object is not found in a slash command Currently, in a few cases we return an error of the form Did not find any XXXX named YYYY while in some cases we print a table with no rows. This patch changes a few of the former to the later so that we have an uniform interface. --- src/bin/psql/describe.c | 377 ++++++++++++++++++++++-------------- src/bin/psql/t/001_basic.pl | 26 +++ 2 files changed, 258 insertions(+), 145 deletions(-) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 36f24502842..de1bb31b231 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -136,6 +136,12 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -210,6 +216,12 @@ describeAccessMethods(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -272,6 +284,12 @@ describeTablespaces(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -621,6 +639,12 @@ describeFunctions(const char *functypes, const char *func_pattern, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (func_pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; @@ -727,6 +751,12 @@ describeTypes(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -928,6 +958,12 @@ describeOperators(const char *oper_pattern, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (oper_pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; @@ -1037,6 +1073,12 @@ listAllDbs(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -1270,6 +1312,13 @@ listDefaultACLs(const char *pattern) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + termPQExpBuffer(&buf); + PQclear(res); + return false; + } + termPQExpBuffer(&buf); PQclear(res); return true; @@ -1522,16 +1571,14 @@ describeTableDetails(const char *pattern, bool verbose, bool showSystem) if (PQntuples(res) == 0) { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any relation named \"%s\".", - pattern); - else - pg_log_error("Did not find any relations."); - } PQclear(res); - return false; + /* + * Print an empty table and return failure when no objects match + */ + if (pattern) + return listTables("tvmsE", pattern, verbose, showSystem); + else + return true; } for (i = 0; i < PQntuples(res); i++) @@ -1720,14 +1767,6 @@ describeOneTableDetails(const char *schemaname, if (!res) goto error_return; - /* Did we get anything? */ - if (PQntuples(res) == 0) - { - if (!pset.quiet) - pg_log_error("Did not find any relation with OID %s.", oid); - goto error_return; - } - tableinfo.checks = atoi(PQgetvalue(res, 0, 0)); tableinfo.relkind = *(PQgetvalue(res, 0, 1)); tableinfo.hasindex = strcmp(PQgetvalue(res, 0, 2), "t") == 0; @@ -3797,6 +3836,7 @@ describeRoles(const char *pattern, bool verbose, bool showSystem) return false; nrows = PQntuples(res); + attr = pg_malloc0((nrows + 1) * sizeof(*attr)); printTableInit(&cont, &myopt, _("List of roles"), ncols, nrows); @@ -3873,6 +3913,12 @@ describeRoles(const char *pattern, bool verbose, bool showSystem) free(attr[i]); free(attr); + if (pattern && nrows == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -3921,29 +3967,15 @@ listDbRoleSettings(const char *pattern, const char *pattern2) if (!res) return false; - /* - * Most functions in this file are content to print an empty table when - * there are no matching objects. We intentionally deviate from that - * here, but only in !quiet mode, because of the possibility that the user - * is confused about what the two pattern arguments mean. - */ - if (PQntuples(res) == 0 && !pset.quiet) - { - if (pattern && pattern2) - pg_log_error("Did not find any settings for role \"%s\" and database \"%s\".", - pattern, pattern2); - else if (pattern) - pg_log_error("Did not find any settings for role \"%s\".", - pattern); - else - pg_log_error("Did not find any settings."); - } - else - { - myopt.title = _("List of settings"); - myopt.translate_header = true; + myopt.title = _("List of settings"); + myopt.translate_header = true; - printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + + if ((pattern || pattern2) && PQntuples(res) == 0) + { + PQclear(res); + return false; } PQclear(res); @@ -4018,6 +4050,12 @@ describeRoleGrants(const char *pattern, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -4201,76 +4239,25 @@ listTables(const char *tabtypes, const char *pattern, bool verbose, bool showSys if (!res) return false; - /* - * Most functions in this file are content to print an empty table when - * there are no matching objects. We intentionally deviate from that - * here, but only in !quiet mode, for historical reasons. - */ - if (PQntuples(res) == 0 && !pset.quiet) - { - if (pattern) - { - if (ntypes != 1) - pg_log_error("Did not find any relations named \"%s\".", - pattern); - else if (showTables) - pg_log_error("Did not find any tables named \"%s\".", - pattern); - else if (showIndexes) - pg_log_error("Did not find any indexes named \"%s\".", - pattern); - else if (showViews) - pg_log_error("Did not find any views named \"%s\".", - pattern); - else if (showMatViews) - pg_log_error("Did not find any materialized views named \"%s\".", - pattern); - else if (showSeq) - pg_log_error("Did not find any sequences named \"%s\".", - pattern); - else if (showForeign) - pg_log_error("Did not find any foreign tables named \"%s\".", - pattern); - else /* should not get here */ - pg_log_error_internal("Did not find any ??? named \"%s\".", - pattern); - } - else - { - if (ntypes != 1) - pg_log_error("Did not find any relations."); - else if (showTables) - pg_log_error("Did not find any tables."); - else if (showIndexes) - pg_log_error("Did not find any indexes."); - else if (showViews) - pg_log_error("Did not find any views."); - else if (showMatViews) - pg_log_error("Did not find any materialized views."); - else if (showSeq) - pg_log_error("Did not find any sequences."); - else if (showForeign) - pg_log_error("Did not find any foreign tables."); - else /* should not get here */ - pg_log_error_internal("Did not find any ??? relations."); - } - } - else - { - myopt.title = - (ntypes != 1) ? _("List of relations") : - (showTables) ? _("List of tables") : - (showIndexes) ? _("List of indexes") : - (showViews) ? _("List of views") : - (showMatViews) ? _("List of materialized views") : - (showSeq) ? _("List of sequences") : - (showForeign) ? _("List of foreign tables") : - "List of ???"; /* should not get here */ - myopt.translate_header = true; - myopt.translate_columns = translate_columns; - myopt.n_translate_columns = lengthof(translate_columns); + myopt.title = + (ntypes != 1) ? _("List of relations") : + (showTables) ? _("List of tables") : + (showIndexes) ? _("List of indexes") : + (showViews) ? _("List of views") : + (showMatViews) ? _("List of materialized views") : + (showSeq) ? _("List of sequences") : + (showForeign) ? _("List of foreign tables") : + "List of ???"; /* should not get here */ + myopt.translate_header = true; + myopt.translate_columns = translate_columns; + myopt.n_translate_columns = lengthof(translate_columns); - printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; } PQclear(res); @@ -4568,6 +4555,12 @@ listLanguages(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -4652,6 +4645,12 @@ listDomains(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -4732,6 +4731,12 @@ listConversions(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -4800,6 +4805,12 @@ describeConfigurationParameters(const char *pattern, bool verbose, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -4880,6 +4891,12 @@ listEventTriggers(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -4976,6 +4993,12 @@ listExtendedStats(const char *pattern) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -5223,6 +5246,12 @@ listCollations(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -5327,6 +5356,13 @@ listSchemas(const char *pattern, bool verbose, bool showSystem) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + termPQExpBuffer(&buf); + PQclear(res); + return false; + } + termPQExpBuffer(&buf); PQclear(res); @@ -5398,6 +5434,12 @@ listTSParsers(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -5440,16 +5482,11 @@ listTSParsersVerbose(const char *pattern) if (PQntuples(res) == 0) { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any text search parser named \"%s\".", - pattern); - else - pg_log_error("Did not find any text search parsers."); - } PQclear(res); - return false; + if (pattern) + return listTSParsers(pattern, false); + else + return true; } for (i = 0; i < PQntuples(res); i++) @@ -5775,6 +5812,12 @@ listTSConfigs(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -5818,16 +5861,11 @@ listTSConfigsVerbose(const char *pattern) if (PQntuples(res) == 0) { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any text search configuration named \"%s\".", - pattern); - else - pg_log_error("Did not find any text search configurations."); - } PQclear(res); - return false; + if (pattern) + return listTSConfigs(pattern, false); + else + return true; } for (i = 0; i < PQntuples(res); i++) @@ -5996,6 +6034,12 @@ listForeignDataWrappers(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -6072,6 +6116,12 @@ listForeignServers(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -6127,6 +6177,12 @@ listUserMappings(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -6199,6 +6255,12 @@ listForeignTables(const char *pattern, bool verbose) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -6253,6 +6315,12 @@ listExtensions(const char *pattern) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; } @@ -6293,16 +6361,11 @@ listExtensionContents(const char *pattern) if (PQntuples(res) == 0) { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any extension named \"%s\".", - pattern); - else - pg_log_error("Did not find any extensions."); - } PQclear(res); - return false; + if (pattern) + return listExtensions(pattern); + else + return true; } for (i = 0; i < PQntuples(res); i++) @@ -6510,6 +6573,12 @@ listPublications(const char *pattern) printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if (pattern && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; @@ -6660,18 +6729,12 @@ describePublications(const char *pattern) if (PQntuples(res) == 0) { - if (!pset.quiet) - { - if (pattern) - pg_log_error("Did not find any publication named \"%s\".", - pattern); - else - pg_log_error("Did not find any publications."); - } - termPQExpBuffer(&buf); PQclear(res); - return false; + if (pattern) + return listPublications(pattern); + else + return true; } for (i = 0; i < PQntuples(res); i++) @@ -7051,6 +7114,12 @@ listOperatorClasses(const char *access_method_pattern, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if ((access_method_pattern || type_pattern) && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; @@ -7139,6 +7208,12 @@ listOperatorFamilies(const char *access_method_pattern, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if ((access_method_pattern || type_pattern) && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; @@ -7246,6 +7321,12 @@ listOpFamilyOperators(const char *access_method_pattern, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if ((access_method_pattern || family_pattern) && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; @@ -7338,6 +7419,12 @@ listOpFamilyFunctions(const char *access_method_pattern, printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + if ((access_method_pattern || family_pattern) && PQntuples(res) == 0) + { + PQclear(res); + return false; + } + PQclear(res); return true; diff --git a/src/bin/psql/t/001_basic.pl b/src/bin/psql/t/001_basic.pl index cf07a9dbd5e..011a6508df0 100644 --- a/src/bin/psql/t/001_basic.pl +++ b/src/bin/psql/t/001_basic.pl @@ -537,4 +537,30 @@ psql_fails_like( qr/backslash commands are restricted; only \\unrestrict is allowed/, 'meta-command in restrict mode fails'); +# Test that \d commands return exit code 1 when pattern matches nothing +# and don't print "Did not find" error messages +{ + my ($ret, $stdout, $stderr); + + ($ret, $stdout, $stderr) = $node->psql('postgres', '\\d nonexistent_table'); + isnt($ret, 0, '\\d with non-existent pattern: exit code not 0'); + unlike($stderr, qr/Did not find/, '\\d with non-existent pattern: no error message'); + + ($ret, $stdout, $stderr) = $node->psql('postgres', '\\dt nonexistent_table'); + isnt($ret, 0, '\\dt with non-existent pattern: exit code not 0'); + unlike($stderr, qr/Did not find/, '\\dt with non-existent pattern: no error message'); + + ($ret, $stdout, $stderr) = $node->psql('postgres', '\\df nonexistent_func'); + isnt($ret, 0, '\\df with non-existent pattern: exit code not 0'); + unlike($stderr, qr/Did not find/, '\\df with non-existent pattern: no error message'); + + ($ret, $stdout, $stderr) = $node->psql('postgres', '\\dx+ nonexistent_ext'); + isnt($ret, 0, '\\dx+ with non-existent pattern: exit code not 0'); + unlike($stderr, qr/Did not find/, '\\dx+ with non-existent pattern: no error message'); + + # Test that \d commands without patterns succeed even with no results + ($ret, $stdout, $stderr) = $node->psql('postgres', '\\d'); + is($ret, 0, '\\d without pattern: exit code 0'); +} + done_testing(); -- 2.50.1 ^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: Adding error messages to a few slash commands @ 2025-10-16 04:20 Michael Paquier <[email protected]> parent: Abhishek Chanda <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Michael Paquier @ 2025-10-16 04:20 UTC (permalink / raw) To: Abhishek Chanda <[email protected]>; +Cc: Robin Haberkorn <[email protected]>; Pavel Luzanov <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Wed, Oct 15, 2025 at 10:56:47PM -0500, Abhishek Chanda wrote: > Thanks a lot for the review, Robin. And I am terribly sorry for the > slow reply, it just took me a while to get to this. But I think I have > addressed all your feedback, I have changed everything to set an error > code of 1 if anything is not empty. Also added tests for the return > code as requested, and cleaned up the change in describeRoles(). > Please let me know if I missed anything. > > Attached an updated and rebased version of the patch. There is something that I am not following here. The consensus of the thread seems to be that folks are OK to show the results as empty tables instead of an error, backtracking on the "historical" reasons documented in the code. That's OK by me. Why not. However, your patch does the opposite. For example: =# \dg "no.such.role" Did not find any role named ""no.such.role"". On HEAD, this returns: =# \dg "no.such.role" List of roles Role name | Attributes -----------+------------ So your patch is doing the opposite of the consensus I am reading. Note that `make check` complains immediately with that. You also may want to be careful that all the code paths you are patching are covered in the regression tests. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP @ 2026-07-06 08:28 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bertrand Drouvot @ 2026-07-06 08:28 UTC (permalink / raw) roleSpecsToIds() resolves role names to OIDs without acquiring any lock. A concurrent DROP ROLE that commits between this resolution and the caller's use of the OID leaves the caller operating on a stale OID, which can create orphaned pg_auth_members entries. Fix this by acquiring AccessShareLock on each resolved role within roleSpecsToIds(), ensuring the role cannot be dropped while any caller is using its OID. Author: Bertrand Drouvot <[email protected]> Reported-by: Virender Singla <[email protected]> Reviewed-by: Discussion: https://postgr.es/m/aki6fMNLUx6%2BBR8K%40bdtpg Discussion: https://postgr.es/m/CAM6Zo8woa62ZFHtMKox6a4jb8qQ%3Dw87R2L0K8347iE-juQL2EA%40mail.gmail.com --- src/backend/commands/user.c | 13 +++++- .../expected/role-membership-drop-member.out | 36 ++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/role-membership-drop-member.spec | 43 +++++++++++++++++++ 4 files changed, 92 insertions(+), 1 deletion(-) 13.8% src/backend/commands/ 42.9% src/test/isolation/expected/ 42.2% src/test/isolation/specs/ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index 5b869e91c17..05ec753f4f0 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -1778,6 +1778,8 @@ ReassignOwnedObjects(ReassignOwnedStmt *stmt) * roleSpecsToIds * * Given a list of RoleSpecs, generate a list of role OIDs in the same order. + * Each role is locked with AccessShareLock to prevent concurrent DROP ROLE + * from removing it between resolution and the caller's catalog update. * * ROLESPEC_PUBLIC is not allowed. */ @@ -1792,7 +1794,16 @@ roleSpecsToIds(List *memberNames) RoleSpec *rolespec = lfirst_node(RoleSpec, l); Oid roleid; - roleid = get_rolespec_oid(rolespec, false); + if (rolespec->roletype == ROLESPEC_CSTRING) + roleid = RoleNameGetOid(rolespec->rolename, + AccessShareLock, false, + NULL, NULL); + else + { + roleid = get_rolespec_oid(rolespec, false); + LockSharedObject(AuthIdRelationId, roleid, 0, + AccessShareLock); + } result = lappend_oid(result, roleid); } return result; diff --git a/src/test/isolation/expected/role-membership-drop-member.out b/src/test/isolation/expected/role-membership-drop-member.out new file mode 100644 index 00000000000..e6cae59d2c9 --- /dev/null +++ b/src/test/isolation/expected/role-membership-drop-member.out @@ -0,0 +1,36 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_grant s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_grant: GRANT regress_role_group TO regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_alter_add s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_alter_add: ALTER GROUP regress_role_group ADD USER regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_create_role s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_create_role: CREATE ROLE regress_role_new ROLE regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_drop_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_drop_owned: DROP OWNED BY regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_reassign_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_reassign_owned: REASSIGN OWNED BY regress_role_member TO regress_role_group; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index b8ebe92553c..8fb8b52b77f 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -128,3 +128,4 @@ test: matview-write-skew test: lock-nowait test: for-portion-of test: ddl-dependency-locking +test: role-membership-drop-member diff --git a/src/test/isolation/specs/role-membership-drop-member.spec b/src/test/isolation/specs/role-membership-drop-member.spec new file mode 100644 index 00000000000..e0140826e52 --- /dev/null +++ b/src/test/isolation/specs/role-membership-drop-member.spec @@ -0,0 +1,43 @@ +# Test that role membership commands properly lock the grantee/member +# role to prevent concurrent DROP ROLE from creating orphaned # pg_auth_members +# entries. + +setup +{ + CREATE ROLE regress_role_group; + CREATE ROLE regress_role_member; +} + +teardown +{ + DROP ROLE IF EXISTS regress_role_group; + DROP ROLE IF EXISTS regress_role_member; + DROP ROLE IF EXISTS regress_role_new; +} + +session s1 +step s1_begin { BEGIN; } +step s1_grant { GRANT regress_role_group TO regress_role_member; } +step s1_alter_add { ALTER GROUP regress_role_group ADD USER regress_role_member; } +step s1_create_role { CREATE ROLE regress_role_new ROLE regress_role_member; } +step s1_drop_owned { DROP OWNED BY regress_role_member; } +step s1_reassign_owned { REASSIGN OWNED BY regress_role_member TO regress_role_group; } +step s1_commit { COMMIT; } + +session s2 +step s2_drop_member { DROP ROLE regress_role_member; } + +# GRANT role TO member - concurrent DROP of the member +permutation s1_begin s1_grant s2_drop_member s1_commit + +# ALTER ROLE ADD USER - concurrent DROP of the member +permutation s1_begin s1_alter_add s2_drop_member s1_commit + +# CREATE ROLE ... ROLE member - concurrent DROP of the member +permutation s1_begin s1_create_role s2_drop_member s1_commit + +# DROP OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_drop_owned s2_drop_member s1_commit + +# REASSIGN OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_reassign_owned s2_drop_member s1_commit -- 2.34.1 --cOHldBwZEf1OqVbN-- ^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP @ 2026-07-06 08:28 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bertrand Drouvot @ 2026-07-06 08:28 UTC (permalink / raw) roleSpecsToIds() resolves role names to OIDs without acquiring any lock. A concurrent DROP ROLE that commits between this resolution and the caller's use of the OID leaves the caller operating on a stale OID, which can create orphaned pg_auth_members entries. Fix this by acquiring AccessShareLock on each resolved role within roleSpecsToIds(), ensuring the role cannot be dropped while any caller is using its OID. Author: Bertrand Drouvot <[email protected]> Reported-by: Virender Singla <[email protected]> Reviewed-by: Discussion: https://postgr.es/m/aki6fMNLUx6%2BBR8K%40bdtpg Discussion: https://postgr.es/m/CAM6Zo8woa62ZFHtMKox6a4jb8qQ%3Dw87R2L0K8347iE-juQL2EA%40mail.gmail.com --- src/backend/commands/user.c | 13 +++++- .../expected/role-membership-drop-member.out | 36 ++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/role-membership-drop-member.spec | 43 +++++++++++++++++++ 4 files changed, 92 insertions(+), 1 deletion(-) 13.8% src/backend/commands/ 42.9% src/test/isolation/expected/ 42.2% src/test/isolation/specs/ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index 5b869e91c17..05ec753f4f0 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -1778,6 +1778,8 @@ ReassignOwnedObjects(ReassignOwnedStmt *stmt) * roleSpecsToIds * * Given a list of RoleSpecs, generate a list of role OIDs in the same order. + * Each role is locked with AccessShareLock to prevent concurrent DROP ROLE + * from removing it between resolution and the caller's catalog update. * * ROLESPEC_PUBLIC is not allowed. */ @@ -1792,7 +1794,16 @@ roleSpecsToIds(List *memberNames) RoleSpec *rolespec = lfirst_node(RoleSpec, l); Oid roleid; - roleid = get_rolespec_oid(rolespec, false); + if (rolespec->roletype == ROLESPEC_CSTRING) + roleid = RoleNameGetOid(rolespec->rolename, + AccessShareLock, false, + NULL, NULL); + else + { + roleid = get_rolespec_oid(rolespec, false); + LockSharedObject(AuthIdRelationId, roleid, 0, + AccessShareLock); + } result = lappend_oid(result, roleid); } return result; diff --git a/src/test/isolation/expected/role-membership-drop-member.out b/src/test/isolation/expected/role-membership-drop-member.out new file mode 100644 index 00000000000..e6cae59d2c9 --- /dev/null +++ b/src/test/isolation/expected/role-membership-drop-member.out @@ -0,0 +1,36 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_grant s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_grant: GRANT regress_role_group TO regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_alter_add s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_alter_add: ALTER GROUP regress_role_group ADD USER regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_create_role s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_create_role: CREATE ROLE regress_role_new ROLE regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_drop_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_drop_owned: DROP OWNED BY regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_reassign_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_reassign_owned: REASSIGN OWNED BY regress_role_member TO regress_role_group; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index b8ebe92553c..8fb8b52b77f 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -128,3 +128,4 @@ test: matview-write-skew test: lock-nowait test: for-portion-of test: ddl-dependency-locking +test: role-membership-drop-member diff --git a/src/test/isolation/specs/role-membership-drop-member.spec b/src/test/isolation/specs/role-membership-drop-member.spec new file mode 100644 index 00000000000..e0140826e52 --- /dev/null +++ b/src/test/isolation/specs/role-membership-drop-member.spec @@ -0,0 +1,43 @@ +# Test that role membership commands properly lock the grantee/member +# role to prevent concurrent DROP ROLE from creating orphaned # pg_auth_members +# entries. + +setup +{ + CREATE ROLE regress_role_group; + CREATE ROLE regress_role_member; +} + +teardown +{ + DROP ROLE IF EXISTS regress_role_group; + DROP ROLE IF EXISTS regress_role_member; + DROP ROLE IF EXISTS regress_role_new; +} + +session s1 +step s1_begin { BEGIN; } +step s1_grant { GRANT regress_role_group TO regress_role_member; } +step s1_alter_add { ALTER GROUP regress_role_group ADD USER regress_role_member; } +step s1_create_role { CREATE ROLE regress_role_new ROLE regress_role_member; } +step s1_drop_owned { DROP OWNED BY regress_role_member; } +step s1_reassign_owned { REASSIGN OWNED BY regress_role_member TO regress_role_group; } +step s1_commit { COMMIT; } + +session s2 +step s2_drop_member { DROP ROLE regress_role_member; } + +# GRANT role TO member - concurrent DROP of the member +permutation s1_begin s1_grant s2_drop_member s1_commit + +# ALTER ROLE ADD USER - concurrent DROP of the member +permutation s1_begin s1_alter_add s2_drop_member s1_commit + +# CREATE ROLE ... ROLE member - concurrent DROP of the member +permutation s1_begin s1_create_role s2_drop_member s1_commit + +# DROP OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_drop_owned s2_drop_member s1_commit + +# REASSIGN OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_reassign_owned s2_drop_member s1_commit -- 2.34.1 --cOHldBwZEf1OqVbN-- ^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP @ 2026-07-06 08:28 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bertrand Drouvot @ 2026-07-06 08:28 UTC (permalink / raw) roleSpecsToIds() resolves role names to OIDs without acquiring any lock. A concurrent DROP ROLE that commits between this resolution and the caller's use of the OID leaves the caller operating on a stale OID, which can create orphaned pg_auth_members entries. Fix this by acquiring AccessShareLock on each resolved role within roleSpecsToIds(), ensuring the role cannot be dropped while any caller is using its OID. Author: Bertrand Drouvot <[email protected]> Reported-by: Virender Singla <[email protected]> Reviewed-by: Discussion: https://postgr.es/m/aki6fMNLUx6%2BBR8K%40bdtpg Discussion: https://postgr.es/m/CAM6Zo8woa62ZFHtMKox6a4jb8qQ%3Dw87R2L0K8347iE-juQL2EA%40mail.gmail.com --- src/backend/commands/user.c | 13 +++++- .../expected/role-membership-drop-member.out | 36 ++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/role-membership-drop-member.spec | 43 +++++++++++++++++++ 4 files changed, 92 insertions(+), 1 deletion(-) 13.8% src/backend/commands/ 42.9% src/test/isolation/expected/ 42.2% src/test/isolation/specs/ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index 5b869e91c17..05ec753f4f0 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -1778,6 +1778,8 @@ ReassignOwnedObjects(ReassignOwnedStmt *stmt) * roleSpecsToIds * * Given a list of RoleSpecs, generate a list of role OIDs in the same order. + * Each role is locked with AccessShareLock to prevent concurrent DROP ROLE + * from removing it between resolution and the caller's catalog update. * * ROLESPEC_PUBLIC is not allowed. */ @@ -1792,7 +1794,16 @@ roleSpecsToIds(List *memberNames) RoleSpec *rolespec = lfirst_node(RoleSpec, l); Oid roleid; - roleid = get_rolespec_oid(rolespec, false); + if (rolespec->roletype == ROLESPEC_CSTRING) + roleid = RoleNameGetOid(rolespec->rolename, + AccessShareLock, false, + NULL, NULL); + else + { + roleid = get_rolespec_oid(rolespec, false); + LockSharedObject(AuthIdRelationId, roleid, 0, + AccessShareLock); + } result = lappend_oid(result, roleid); } return result; diff --git a/src/test/isolation/expected/role-membership-drop-member.out b/src/test/isolation/expected/role-membership-drop-member.out new file mode 100644 index 00000000000..e6cae59d2c9 --- /dev/null +++ b/src/test/isolation/expected/role-membership-drop-member.out @@ -0,0 +1,36 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_grant s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_grant: GRANT regress_role_group TO regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_alter_add s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_alter_add: ALTER GROUP regress_role_group ADD USER regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_create_role s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_create_role: CREATE ROLE regress_role_new ROLE regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_drop_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_drop_owned: DROP OWNED BY regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_reassign_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_reassign_owned: REASSIGN OWNED BY regress_role_member TO regress_role_group; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index b8ebe92553c..8fb8b52b77f 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -128,3 +128,4 @@ test: matview-write-skew test: lock-nowait test: for-portion-of test: ddl-dependency-locking +test: role-membership-drop-member diff --git a/src/test/isolation/specs/role-membership-drop-member.spec b/src/test/isolation/specs/role-membership-drop-member.spec new file mode 100644 index 00000000000..e0140826e52 --- /dev/null +++ b/src/test/isolation/specs/role-membership-drop-member.spec @@ -0,0 +1,43 @@ +# Test that role membership commands properly lock the grantee/member +# role to prevent concurrent DROP ROLE from creating orphaned # pg_auth_members +# entries. + +setup +{ + CREATE ROLE regress_role_group; + CREATE ROLE regress_role_member; +} + +teardown +{ + DROP ROLE IF EXISTS regress_role_group; + DROP ROLE IF EXISTS regress_role_member; + DROP ROLE IF EXISTS regress_role_new; +} + +session s1 +step s1_begin { BEGIN; } +step s1_grant { GRANT regress_role_group TO regress_role_member; } +step s1_alter_add { ALTER GROUP regress_role_group ADD USER regress_role_member; } +step s1_create_role { CREATE ROLE regress_role_new ROLE regress_role_member; } +step s1_drop_owned { DROP OWNED BY regress_role_member; } +step s1_reassign_owned { REASSIGN OWNED BY regress_role_member TO regress_role_group; } +step s1_commit { COMMIT; } + +session s2 +step s2_drop_member { DROP ROLE regress_role_member; } + +# GRANT role TO member - concurrent DROP of the member +permutation s1_begin s1_grant s2_drop_member s1_commit + +# ALTER ROLE ADD USER - concurrent DROP of the member +permutation s1_begin s1_alter_add s2_drop_member s1_commit + +# CREATE ROLE ... ROLE member - concurrent DROP of the member +permutation s1_begin s1_create_role s2_drop_member s1_commit + +# DROP OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_drop_owned s2_drop_member s1_commit + +# REASSIGN OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_reassign_owned s2_drop_member s1_commit -- 2.34.1 --cOHldBwZEf1OqVbN-- ^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP @ 2026-07-06 08:28 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bertrand Drouvot @ 2026-07-06 08:28 UTC (permalink / raw) roleSpecsToIds() resolves role names to OIDs without acquiring any lock. A concurrent DROP ROLE that commits between this resolution and the caller's use of the OID leaves the caller operating on a stale OID, which can create orphaned pg_auth_members entries. Fix this by acquiring AccessShareLock on each resolved role within roleSpecsToIds(), ensuring the role cannot be dropped while any caller is using its OID. Author: Bertrand Drouvot <[email protected]> Reported-by: Virender Singla <[email protected]> Reviewed-by: Discussion: https://postgr.es/m/aki6fMNLUx6%2BBR8K%40bdtpg Discussion: https://postgr.es/m/CAM6Zo8woa62ZFHtMKox6a4jb8qQ%3Dw87R2L0K8347iE-juQL2EA%40mail.gmail.com --- src/backend/commands/user.c | 13 +++++- .../expected/role-membership-drop-member.out | 36 ++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/role-membership-drop-member.spec | 43 +++++++++++++++++++ 4 files changed, 92 insertions(+), 1 deletion(-) 13.8% src/backend/commands/ 42.9% src/test/isolation/expected/ 42.2% src/test/isolation/specs/ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index 5b869e91c17..05ec753f4f0 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -1778,6 +1778,8 @@ ReassignOwnedObjects(ReassignOwnedStmt *stmt) * roleSpecsToIds * * Given a list of RoleSpecs, generate a list of role OIDs in the same order. + * Each role is locked with AccessShareLock to prevent concurrent DROP ROLE + * from removing it between resolution and the caller's catalog update. * * ROLESPEC_PUBLIC is not allowed. */ @@ -1792,7 +1794,16 @@ roleSpecsToIds(List *memberNames) RoleSpec *rolespec = lfirst_node(RoleSpec, l); Oid roleid; - roleid = get_rolespec_oid(rolespec, false); + if (rolespec->roletype == ROLESPEC_CSTRING) + roleid = RoleNameGetOid(rolespec->rolename, + AccessShareLock, false, + NULL, NULL); + else + { + roleid = get_rolespec_oid(rolespec, false); + LockSharedObject(AuthIdRelationId, roleid, 0, + AccessShareLock); + } result = lappend_oid(result, roleid); } return result; diff --git a/src/test/isolation/expected/role-membership-drop-member.out b/src/test/isolation/expected/role-membership-drop-member.out new file mode 100644 index 00000000000..e6cae59d2c9 --- /dev/null +++ b/src/test/isolation/expected/role-membership-drop-member.out @@ -0,0 +1,36 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_grant s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_grant: GRANT regress_role_group TO regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_alter_add s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_alter_add: ALTER GROUP regress_role_group ADD USER regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_create_role s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_create_role: CREATE ROLE regress_role_new ROLE regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_drop_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_drop_owned: DROP OWNED BY regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_reassign_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_reassign_owned: REASSIGN OWNED BY regress_role_member TO regress_role_group; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index b8ebe92553c..8fb8b52b77f 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -128,3 +128,4 @@ test: matview-write-skew test: lock-nowait test: for-portion-of test: ddl-dependency-locking +test: role-membership-drop-member diff --git a/src/test/isolation/specs/role-membership-drop-member.spec b/src/test/isolation/specs/role-membership-drop-member.spec new file mode 100644 index 00000000000..e0140826e52 --- /dev/null +++ b/src/test/isolation/specs/role-membership-drop-member.spec @@ -0,0 +1,43 @@ +# Test that role membership commands properly lock the grantee/member +# role to prevent concurrent DROP ROLE from creating orphaned # pg_auth_members +# entries. + +setup +{ + CREATE ROLE regress_role_group; + CREATE ROLE regress_role_member; +} + +teardown +{ + DROP ROLE IF EXISTS regress_role_group; + DROP ROLE IF EXISTS regress_role_member; + DROP ROLE IF EXISTS regress_role_new; +} + +session s1 +step s1_begin { BEGIN; } +step s1_grant { GRANT regress_role_group TO regress_role_member; } +step s1_alter_add { ALTER GROUP regress_role_group ADD USER regress_role_member; } +step s1_create_role { CREATE ROLE regress_role_new ROLE regress_role_member; } +step s1_drop_owned { DROP OWNED BY regress_role_member; } +step s1_reassign_owned { REASSIGN OWNED BY regress_role_member TO regress_role_group; } +step s1_commit { COMMIT; } + +session s2 +step s2_drop_member { DROP ROLE regress_role_member; } + +# GRANT role TO member - concurrent DROP of the member +permutation s1_begin s1_grant s2_drop_member s1_commit + +# ALTER ROLE ADD USER - concurrent DROP of the member +permutation s1_begin s1_alter_add s2_drop_member s1_commit + +# CREATE ROLE ... ROLE member - concurrent DROP of the member +permutation s1_begin s1_create_role s2_drop_member s1_commit + +# DROP OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_drop_owned s2_drop_member s1_commit + +# REASSIGN OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_reassign_owned s2_drop_member s1_commit -- 2.34.1 --cOHldBwZEf1OqVbN-- ^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v3 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP @ 2026-07-06 08:28 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bertrand Drouvot @ 2026-07-06 08:28 UTC (permalink / raw) roleSpecsToIds() resolves role names to OIDs without acquiring any lock. A concurrent DROP ROLE that commits between this resolution and the caller's use of the OID leaves the caller operating on a stale OID, which can create orphaned pg_auth_members entries. Fix this by acquiring AccessShareLock on each resolved role within roleSpecsToIds(), ensuring the role cannot be dropped while any caller is using its OID. The AccessShareLock is held until end of transaction, so an open transaction that performed GRANT, CREATE ROLE ... ROLE, or REASSIGN OWNED BY will block a concurrent DROP ROLE on the same role until it commits. Note that this introduces a potential deadlock between GRANT and DROP ROLE when both target overlapping roles. The deadlock is detected and one session is aborted with an error, which is preferable to the pre-patch behavior of silently creating orphaned catalog entries. Author: Bertrand Drouvot <[email protected]> Reported-by: Virender Singla <[email protected]> Reviewed-by: Surya Poondla <[email protected]> Discussion: https://postgr.es/m/aki6fMNLUx6%2BBR8K%40bdtpg Discussion: https://postgr.es/m/CAM6Zo8woa62ZFHtMKox6a4jb8qQ%3Dw87R2L0K8347iE-juQL2EA%40mail.gmail.com --- src/backend/commands/user.c | 21 +++++- .../expected/role-membership-drop-member.out | 75 +++++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/role-membership-drop-member.spec | 51 +++++++++++++ 4 files changed, 147 insertions(+), 1 deletion(-) 14.9% src/backend/commands/ 48.1% src/test/isolation/expected/ 36.1% src/test/isolation/specs/ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index 5b869e91c17..375fb527af2 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -1778,6 +1778,8 @@ ReassignOwnedObjects(ReassignOwnedStmt *stmt) * roleSpecsToIds * * Given a list of RoleSpecs, generate a list of role OIDs in the same order. + * Each role is locked with AccessShareLock to prevent concurrent DROP ROLE + * from removing it between resolution and the caller's catalog update. * * ROLESPEC_PUBLIC is not allowed. */ @@ -1792,7 +1794,24 @@ roleSpecsToIds(List *memberNames) RoleSpec *rolespec = lfirst_node(RoleSpec, l); Oid roleid; - roleid = get_rolespec_oid(rolespec, false); + if (rolespec->roletype == ROLESPEC_CSTRING) + roleid = RoleNameGetOid(rolespec->rolename, + AccessShareLock, false, + NULL, NULL); + else + { + roleid = get_rolespec_oid(rolespec, false); + LockSharedObject(AuthIdRelationId, roleid, 0, + AccessShareLock); + + /* Recheck that the role still exists after locking. */ + if (!SearchSysCacheExists1(AUTHOID, ObjectIdGetDatum(roleid))) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("role \"%s\" does not exist", + get_rolespec_name(rolespec)))); + } + result = lappend_oid(result, roleid); } return result; diff --git a/src/test/isolation/expected/role-membership-drop-member.out b/src/test/isolation/expected/role-membership-drop-member.out new file mode 100644 index 00000000000..5b267ea32c2 --- /dev/null +++ b/src/test/isolation/expected/role-membership-drop-member.out @@ -0,0 +1,75 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_grant s2_drop_member s1_commit s2_check_orphans +step s1_begin: BEGIN; +step s1_grant: GRANT regress_role_group TO regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> +step s2_check_orphans: + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; + +count +----- + 0 +(1 row) + + +starting permutation: s1_begin s1_alter_add s2_drop_member s1_commit s2_check_orphans +step s1_begin: BEGIN; +step s1_alter_add: ALTER GROUP regress_role_group ADD USER regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> +step s2_check_orphans: + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; + +count +----- + 0 +(1 row) + + +starting permutation: s1_begin s1_create_role s2_drop_member s1_commit s2_check_orphans +step s1_begin: BEGIN; +step s1_create_role: CREATE ROLE regress_role_new ROLE regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> +step s2_check_orphans: + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; + +count +----- + 0 +(1 row) + + +starting permutation: s1_begin s1_drop_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_drop_owned: DROP OWNED BY regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_reassign_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_reassign_owned: REASSIGN OWNED BY regress_role_member TO regress_role_group; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index b8ebe92553c..8fb8b52b77f 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -128,3 +128,4 @@ test: matview-write-skew test: lock-nowait test: for-portion-of test: ddl-dependency-locking +test: role-membership-drop-member diff --git a/src/test/isolation/specs/role-membership-drop-member.spec b/src/test/isolation/specs/role-membership-drop-member.spec new file mode 100644 index 00000000000..989fe996249 --- /dev/null +++ b/src/test/isolation/specs/role-membership-drop-member.spec @@ -0,0 +1,51 @@ +# Test that role membership commands properly lock the grantee/member +# role to prevent concurrent DROP ROLE from creating orphaned pg_auth_members +# entries, or from operating on a stale OID. + +setup +{ + CREATE ROLE regress_role_group; + CREATE ROLE regress_role_member; +} + +teardown +{ + DROP ROLE IF EXISTS regress_role_group; + DROP ROLE IF EXISTS regress_role_member; + DROP ROLE IF EXISTS regress_role_new; +} + +session s1 +step s1_begin { BEGIN; } +step s1_grant { GRANT regress_role_group TO regress_role_member; } +step s1_alter_add { ALTER GROUP regress_role_group ADD USER regress_role_member; } +step s1_create_role { CREATE ROLE regress_role_new ROLE regress_role_member; } +step s1_drop_owned { DROP OWNED BY regress_role_member; } +step s1_reassign_owned { REASSIGN OWNED BY regress_role_member TO regress_role_group; } +step s1_commit { COMMIT; } + +session s2 +step s2_drop_member { DROP ROLE regress_role_member; } +step s2_check_orphans { + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; +} + +# GRANT role TO member - concurrent DROP of the member +permutation s1_begin s1_grant s2_drop_member s1_commit s2_check_orphans + +# ALTER GROUP ADD USER - concurrent DROP of the member +permutation s1_begin s1_alter_add s2_drop_member s1_commit s2_check_orphans + +# CREATE ROLE ... ROLE member - concurrent DROP of the member +permutation s1_begin s1_create_role s2_drop_member s1_commit s2_check_orphans + +# DROP OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_drop_owned s2_drop_member s1_commit + +# REASSIGN OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_reassign_owned s2_drop_member s1_commit -- 2.34.1 --Dvg/WemKV0I3T/Od-- ^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v3 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP @ 2026-07-06 08:28 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 78+ messages in thread From: Bertrand Drouvot @ 2026-07-06 08:28 UTC (permalink / raw) roleSpecsToIds() resolves role names to OIDs without acquiring any lock. A concurrent DROP ROLE that commits between this resolution and the caller's use of the OID leaves the caller operating on a stale OID, which can create orphaned pg_auth_members entries. Fix this by acquiring AccessShareLock on each resolved role within roleSpecsToIds(), ensuring the role cannot be dropped while any caller is using its OID. The AccessShareLock is held until end of transaction, so an open transaction that performed GRANT, CREATE ROLE ... ROLE, or REASSIGN OWNED BY will block a concurrent DROP ROLE on the same role until it commits. Note that this introduces a potential deadlock between GRANT and DROP ROLE when both target overlapping roles. The deadlock is detected and one session is aborted with an error, which is preferable to the pre-patch behavior of silently creating orphaned catalog entries. Author: Bertrand Drouvot <[email protected]> Reported-by: Virender Singla <[email protected]> Reviewed-by: Surya Poondla <[email protected]> Discussion: https://postgr.es/m/aki6fMNLUx6%2BBR8K%40bdtpg Discussion: https://postgr.es/m/CAM6Zo8woa62ZFHtMKox6a4jb8qQ%3Dw87R2L0K8347iE-juQL2EA%40mail.gmail.com --- src/backend/commands/user.c | 21 +++++- .../expected/role-membership-drop-member.out | 75 +++++++++++++++++++ src/test/isolation/isolation_schedule | 1 + .../specs/role-membership-drop-member.spec | 51 +++++++++++++ 4 files changed, 147 insertions(+), 1 deletion(-) 14.9% src/backend/commands/ 48.1% src/test/isolation/expected/ 36.1% src/test/isolation/specs/ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index 5b869e91c17..375fb527af2 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -1778,6 +1778,8 @@ ReassignOwnedObjects(ReassignOwnedStmt *stmt) * roleSpecsToIds * * Given a list of RoleSpecs, generate a list of role OIDs in the same order. + * Each role is locked with AccessShareLock to prevent concurrent DROP ROLE + * from removing it between resolution and the caller's catalog update. * * ROLESPEC_PUBLIC is not allowed. */ @@ -1792,7 +1794,24 @@ roleSpecsToIds(List *memberNames) RoleSpec *rolespec = lfirst_node(RoleSpec, l); Oid roleid; - roleid = get_rolespec_oid(rolespec, false); + if (rolespec->roletype == ROLESPEC_CSTRING) + roleid = RoleNameGetOid(rolespec->rolename, + AccessShareLock, false, + NULL, NULL); + else + { + roleid = get_rolespec_oid(rolespec, false); + LockSharedObject(AuthIdRelationId, roleid, 0, + AccessShareLock); + + /* Recheck that the role still exists after locking. */ + if (!SearchSysCacheExists1(AUTHOID, ObjectIdGetDatum(roleid))) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("role \"%s\" does not exist", + get_rolespec_name(rolespec)))); + } + result = lappend_oid(result, roleid); } return result; diff --git a/src/test/isolation/expected/role-membership-drop-member.out b/src/test/isolation/expected/role-membership-drop-member.out new file mode 100644 index 00000000000..5b267ea32c2 --- /dev/null +++ b/src/test/isolation/expected/role-membership-drop-member.out @@ -0,0 +1,75 @@ +Parsed test spec with 2 sessions + +starting permutation: s1_begin s1_grant s2_drop_member s1_commit s2_check_orphans +step s1_begin: BEGIN; +step s1_grant: GRANT regress_role_group TO regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> +step s2_check_orphans: + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; + +count +----- + 0 +(1 row) + + +starting permutation: s1_begin s1_alter_add s2_drop_member s1_commit s2_check_orphans +step s1_begin: BEGIN; +step s1_alter_add: ALTER GROUP regress_role_group ADD USER regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> +step s2_check_orphans: + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; + +count +----- + 0 +(1 row) + + +starting permutation: s1_begin s1_create_role s2_drop_member s1_commit s2_check_orphans +step s1_begin: BEGIN; +step s1_create_role: CREATE ROLE regress_role_new ROLE regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> +step s2_check_orphans: + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; + +count +----- + 0 +(1 row) + + +starting permutation: s1_begin s1_drop_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_drop_owned: DROP OWNED BY regress_role_member; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> + +starting permutation: s1_begin s1_reassign_owned s2_drop_member s1_commit +step s1_begin: BEGIN; +step s1_reassign_owned: REASSIGN OWNED BY regress_role_member TO regress_role_group; +step s2_drop_member: DROP ROLE regress_role_member; <waiting ...> +step s1_commit: COMMIT; +step s2_drop_member: <... completed> diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index b8ebe92553c..8fb8b52b77f 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -128,3 +128,4 @@ test: matview-write-skew test: lock-nowait test: for-portion-of test: ddl-dependency-locking +test: role-membership-drop-member diff --git a/src/test/isolation/specs/role-membership-drop-member.spec b/src/test/isolation/specs/role-membership-drop-member.spec new file mode 100644 index 00000000000..989fe996249 --- /dev/null +++ b/src/test/isolation/specs/role-membership-drop-member.spec @@ -0,0 +1,51 @@ +# Test that role membership commands properly lock the grantee/member +# role to prevent concurrent DROP ROLE from creating orphaned pg_auth_members +# entries, or from operating on a stale OID. + +setup +{ + CREATE ROLE regress_role_group; + CREATE ROLE regress_role_member; +} + +teardown +{ + DROP ROLE IF EXISTS regress_role_group; + DROP ROLE IF EXISTS regress_role_member; + DROP ROLE IF EXISTS regress_role_new; +} + +session s1 +step s1_begin { BEGIN; } +step s1_grant { GRANT regress_role_group TO regress_role_member; } +step s1_alter_add { ALTER GROUP regress_role_group ADD USER regress_role_member; } +step s1_create_role { CREATE ROLE regress_role_new ROLE regress_role_member; } +step s1_drop_owned { DROP OWNED BY regress_role_member; } +step s1_reassign_owned { REASSIGN OWNED BY regress_role_member TO regress_role_group; } +step s1_commit { COMMIT; } + +session s2 +step s2_drop_member { DROP ROLE regress_role_member; } +step s2_check_orphans { + SELECT count(*) + FROM pg_auth_members m + LEFT JOIN pg_authid ra ON m.roleid = ra.oid + LEFT JOIN pg_authid me ON m.member = me.oid + LEFT JOIN pg_authid gr ON m.grantor = gr.oid + WHERE ra.oid IS NULL OR me.oid IS NULL OR gr.oid IS NULL; +} + +# GRANT role TO member - concurrent DROP of the member +permutation s1_begin s1_grant s2_drop_member s1_commit s2_check_orphans + +# ALTER GROUP ADD USER - concurrent DROP of the member +permutation s1_begin s1_alter_add s2_drop_member s1_commit s2_check_orphans + +# CREATE ROLE ... ROLE member - concurrent DROP of the member +permutation s1_begin s1_create_role s2_drop_member s1_commit s2_check_orphans + +# DROP OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_drop_owned s2_drop_member s1_commit + +# REASSIGN OWNED BY role - concurrent DROP of the role +permutation s1_begin s1_reassign_owned s2_drop_member s1_commit -- 2.34.1 --Dvg/WemKV0I3T/Od-- ^ permalink raw reply [nested|flat] 78+ messages in thread
end of thread, other threads:[~2026-07-06 08:28 UTC | newest] Thread overview: 78+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-08-17 15:56 preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce) Robert Haas <[email protected]> 2021-08-17 16:12 ` Tom Lane <[email protected]> 2021-08-17 16:33 ` Stephen Frost <[email protected]> 2021-08-17 16:42 ` Tom Lane <[email protected]> 2021-08-17 17:37 ` Robert Haas <[email protected]> 2021-08-17 17:54 ` Tom Lane <[email protected]> 2021-08-17 17:57 ` Tom Lane <[email protected]> 2021-08-17 18:50 ` Robert Haas <[email protected]> 2021-08-26 15:00 ` Robert Haas <[email protected]> 2021-08-26 15:24 ` Bruce Momjian <[email protected]> 2021-08-26 15:35 ` Robert Haas <[email protected]> 2021-08-26 15:44 ` Bruce Momjian <[email protected]> 2021-08-26 15:36 ` Stephen Frost <[email protected]> 2021-08-26 15:48 ` Bruce Momjian <[email protected]> 2021-08-26 16:34 ` Stephen Frost <[email protected]> 2021-08-26 16:49 ` Bruce Momjian <[email protected]> 2021-08-26 17:03 ` Stephen Frost <[email protected]> 2021-08-26 17:16 ` Bruce Momjian <[email protected]> 2021-08-26 17:24 ` Stephen Frost <[email protected]> 2021-08-26 17:37 ` Bruce Momjian <[email protected]> 2021-08-26 16:37 ` Robert Haas <[email protected]> 2021-08-26 16:51 ` Bruce Momjian <[email protected]> 2021-08-26 17:20 ` Robert Haas <[email protected]> 2021-08-26 17:34 ` Bruce Momjian <[email protected]> 2021-08-26 15:39 ` Stephen Frost <[email protected]> 2021-08-26 15:52 ` Robert Haas <[email protected]> 2021-08-26 16:42 ` Stephen Frost <[email protected]> 2021-08-17 18:07 ` Shruthi Gowda <[email protected]> 2021-08-17 18:32 ` Bruce Momjian <[email protected]> 2021-08-20 17:36 ` Shruthi Gowda <[email protected]> 2021-08-23 20:57 ` Robert Haas <[email protected]> 2021-08-23 21:12 ` Stephen Frost <[email protected]> 2021-08-24 15:24 ` Robert Haas <[email protected]> 2021-08-24 16:43 ` Bruce Momjian <[email protected]> 2021-08-24 17:26 ` Robert Haas <[email protected]> 2021-08-24 17:46 ` Stephen Frost <[email protected]> 2021-08-24 18:16 ` Bruce Momjian <[email protected]> 2021-08-24 18:34 ` Robert Haas <[email protected]> 2021-08-24 18:49 ` Bruce Momjian <[email protected]> 2021-08-24 17:25 ` Stephen Frost <[email protected]> 2021-08-24 00:29 ` Bruce Momjian <[email protected]> 2021-08-24 09:10 ` Shruthi Gowda <[email protected]> 2021-08-24 15:28 ` Robert Haas <[email protected]> 2021-08-24 16:04 ` Tom Lane <[email protected]> 2021-08-24 16:39 ` Bruce Momjian <[email protected]> 2021-08-24 17:23 ` Robert Haas <[email protected]> 2021-08-24 16:40 ` Bruce Momjian <[email protected]> 2021-09-22 19:06 ` Shruthi Gowda <[email protected]> 2021-09-23 19:13 ` Robert Haas <[email protected]> 2021-10-04 16:44 ` Shruthi Gowda <[email protected]> 2021-10-06 20:35 ` Robert Haas <[email protected]> 2021-10-07 07:24 ` Shruthi Gowda <[email protected]> 2021-10-07 14:02 ` Robert Haas <[email protected]> 2021-10-26 13:24 ` Shruthi Gowda <[email protected]> 2021-12-06 04:44 ` Sadhuprasad Patro <[email protected]> 2021-12-06 17:55 ` Robert Haas <[email protected]> 2021-12-13 15:13 ` Shruthi Gowda <[email protected]> 2021-12-17 07:33 ` Shruthi Gowda <[email protected]> 2021-12-13 14:40 ` Shruthi Gowda <[email protected]> 2021-12-13 21:05 ` Robert Haas <[email protected]> 2021-12-14 18:21 ` Shruthi Gowda <[email protected]> 2021-12-14 18:39 ` tushar <[email protected]> 2021-12-15 17:03 ` tushar <[email protected]> 2025-04-13 06:47 Re: Adding error messages to a few slash commands Pavel Luzanov <[email protected]> 2025-04-13 16:40 ` Re: Adding error messages to a few slash commands Abhishek Chanda <[email protected]> 2025-04-14 09:27 ` Re: Adding error messages to a few slash commands Pavel Luzanov <[email protected]> 2025-04-14 15:05 ` Re: Adding error messages to a few slash commands Abhishek Chanda <[email protected]> 2025-05-12 21:00 ` Re: Adding error messages to a few slash commands Robin Haberkorn <[email protected]> 2025-07-11 09:15 ` Re: Adding error messages to a few slash commands Robin Haberkorn <[email protected]> 2025-10-16 03:56 ` Re: Adding error messages to a few slash commands Abhishek Chanda <[email protected]> 2025-10-16 04:20 ` Re: Adding error messages to a few slash commands Michael Paquier <[email protected]> 2025-07-01 11:03 ` Re: Adding error messages to a few slash commands Dean Rasheed <[email protected]> 2026-07-06 08:28 [PATCH v3 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP Bertrand Drouvot <[email protected]> 2026-07-06 08:28 [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP Bertrand Drouvot <[email protected]> 2026-07-06 08:28 [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP Bertrand Drouvot <[email protected]> 2026-07-06 08:28 [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP Bertrand Drouvot <[email protected]> 2026-07-06 08:28 [PATCH v2 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP Bertrand Drouvot <[email protected]> 2026-07-06 08:28 [PATCH v3 2/2] Protect role resolution in roleSpecsToIds() against concurrent DROP Bertrand Drouvot <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox