public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v5 11/14] hio: Use ExtendBufferedRelBy() 31+ messages / 7 participants [nested] [flat]
* [PATCH v5 11/14] hio: Use ExtendBufferedRelBy() @ 2022-10-26 21:14 Andres Freund <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Andres Freund @ 2022-10-26 21:14 UTC (permalink / raw) --- src/backend/access/heap/hio.c | 332 ++++++++++++++++++++-------------- 1 file changed, 194 insertions(+), 138 deletions(-) diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c index 65886839e70..40f53d7177e 100644 --- a/src/backend/access/heap/hio.c +++ b/src/backend/access/heap/hio.c @@ -186,89 +186,176 @@ GetVisibilityMapPins(Relation relation, Buffer buffer1, Buffer buffer2, } /* - * Extend a relation by multiple blocks to avoid future contention on the - * relation extension lock. Our goal is to pre-extend the relation by an - * amount which ramps up as the degree of contention ramps up, but limiting - * the result to some sane overall value. + * Extend the relation. By multiple pages, if beneficial. + * + * If the caller needs multiple pages (num_pages > 1), we always try to extend + * by at least that much. + * + * If there is contention on the extension lock, we don't just extend "for + * ourselves", but we try to help others. We can do so by adding empty pages + * into the FSM. Typically there is no contention when we can't use the FSM. + * + * We do have to limit the number of pages to extend by to some value, as the + * buffers for all the extended pages need to, temporarily, be pinned. For now + * we define MAX_BUFFERS_TO_EXTEND_BY to be 64 buffers, it's hard to see + * benefits with higher numbers. This partially is because copyfrom.c's + * MAX_BUFFERED_TUPLES / MAX_BUFFERED_BYTES prevents larger multi_inserts. */ -static void -RelationAddExtraBlocks(Relation relation, BulkInsertState bistate) +static Buffer +RelationAddBlocks(Relation relation, BulkInsertState bistate, + int num_pages, bool use_fsm) { - BlockNumber blockNum, - firstBlock = InvalidBlockNumber; - int extraBlocks; - int lockWaiters; - - /* Use the length of the lock wait queue to judge how much to extend. */ - lockWaiters = RelationExtensionLockWaiterCount(relation); - if (lockWaiters <= 0) - return; +#define MAX_BUFFERS_TO_EXTEND_BY 64 + Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY]; + BlockNumber firstBlock = InvalidBlockNumber; + BlockNumber firstBlockFSM = InvalidBlockNumber; + uint32 extend_by_pages; + uint32 not_in_fsm_pages; + BlockNumber curBlock; + uint32 waitcount; + Buffer buffer; /* - * It might seem like multiplying the number of lock waiters by as much as - * 20 is too aggressive, but benchmarking revealed that smaller numbers - * were insufficient. 512 is just an arbitrary cap to prevent - * pathological results. + * Determine by how many pages to try to extend by. */ - extraBlocks = Min(512, lockWaiters * 20); - - do + if (bistate == NULL && !use_fsm) { - Buffer buffer; - Page page; - Size freespace; - /* - * Extend by one page. This should generally match the main-line - * extension code in RelationGetBufferForTuple, except that we hold - * the relation extension lock throughout, and we don't immediately - * initialize the page (see below). + * If we have neither bistate, nor can use the FSM, we can't bulk + * extend - there'd be no way to find the additional pages. */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - page = BufferGetPage(buffer); - - if (!PageIsNew(page)) - elog(ERROR, "page %u of relation \"%s\" should be empty but is not", - BufferGetBlockNumber(buffer), - RelationGetRelationName(relation)); - - /* - * Add the page to the FSM without initializing. If we were to - * initialize here, the page would potentially get flushed out to disk - * before we add any useful content. There's no guarantee that that'd - * happen before a potential crash, so we need to deal with - * uninitialized pages anyway, thus avoid the potential for - * unnecessary writes. - */ - - /* we'll need this info below */ - blockNum = BufferGetBlockNumber(buffer); - freespace = BufferGetPageSize(buffer) - SizeOfPageHeaderData; - - UnlockReleaseBuffer(buffer); - - /* Remember first block number thus added. */ - if (firstBlock == InvalidBlockNumber) - firstBlock = blockNum; - - /* - * Immediately update the bottom level of the FSM. This has a good - * chance of making this page visible to other concurrently inserting - * backends, and we want that to happen without delay. - */ - RecordPageWithFreeSpace(relation, blockNum, freespace); + extend_by_pages = 1; + } + else + { + /* + * Try to extend at least by the number of pages the caller needs. We + * can remember the additional pages (either via FSM or bistate). + */ + extend_by_pages = num_pages; + + if (!RELATION_IS_LOCAL(relation)) + waitcount = RelationExtensionLockWaiterCount(relation); + else + waitcount = 0; + + /* + * Multiply the number of pages to extend by the number of waiters. Do + * this even if we're not using the FSM, as it still relieves + * contention, by deferring the next time this backend needs to + * extend. In that case the extended pages will be found via + * bistate->next_free. + */ + extend_by_pages += extend_by_pages * waitcount; + + /* + * Can't extend by more than MAX_BUFFERS, we need to pin them all + * concurrently. + */ + extend_by_pages = Min(extend_by_pages, MAX_BUFFERS_TO_EXTEND_BY); } - while (--extraBlocks > 0); /* - * Updating the upper levels of the free space map is too expensive to do - * for every block, but it's worth doing once at the end to make sure that - * subsequent insertion activity sees all of those nifty free pages we - * just inserted. + * How many of the extended pages should be entered into the FSM? + * + * If we have a bistate, only enter pages that we don't need ourselves + * into the FSM. Otherwise every other backend will immediately try to + * use the pages this backend neds itself, causing unnecessary + * contention. If we don't have a bistate, we can't avoid the FSM. + * + * Never enter the page returned into the FSM, we'll immediately use it. */ - FreeSpaceMapVacuumRange(relation, firstBlock, blockNum + 1); + if (num_pages > 1 && bistate == NULL) + not_in_fsm_pages = 1; + else + not_in_fsm_pages = num_pages; + + /* prepare to put another buffer into the bistate */ + if (bistate && bistate->current_buf != InvalidBuffer) + { + ReleaseBuffer(bistate->current_buf); + bistate->current_buf = InvalidBuffer; + } + + /* + * Extend the relation. We ask for the first returned page to be locked, + * so that we are sure that nobody has inserted into the page + * concurrently. + * + * With the current MAX_BUFFERS_TO_EXTEND_BY there's no danger of + * [auto]vacuum trying to truncate later pages as REL_TRUNCATE_MINIMUM is + * way larger. + */ + firstBlock = ExtendBufferedRelBy(EB_REL(relation), MAIN_FORKNUM, + bistate ? bistate->strategy : NULL, + EB_LOCK_FIRST, + extend_by_pages, + victim_buffers, + &extend_by_pages); + /* the buffer the function will return */ + buffer = victim_buffers[0]; + + /* + * Relation is now extended. Release pins on all buffers, except for the + * first (which we'll return). If we decided to put pages into the FSM, + * we can do that as part of the same loop. + * + * FIXME: Figure out how to better deal with doing this operation while + * holding a buffer lock. Likely we can just release the buffer lock (to + * reacquire it later) after initializing the page - the target page isn't + * in the FSM, so it's going to be very rare that it's found. + */ + curBlock = firstBlock; + for (uint32 i = 0; i < extend_by_pages; i++, curBlock++) + { + Assert(curBlock == BufferGetBlockNumber(victim_buffers[i])); + Assert(BlockNumberIsValid(curBlock)); + + /* don't release the pin on the page returned by this function */ + if (i > 0) + ReleaseBuffer(victim_buffers[i]); + + if (i >= not_in_fsm_pages && use_fsm) + { + if (firstBlockFSM == InvalidBlockNumber) + firstBlockFSM = curBlock; + + RecordPageWithFreeSpace(relation, + curBlock, + BufferGetPageSize(victim_buffers[i]) - SizeOfPageHeaderData); + } + } + + if (use_fsm && firstBlockFSM != InvalidBlockNumber) + FreeSpaceMapVacuumRange(relation, firstBlockFSM, firstBlock + num_pages); + + if (bistate) + { + /* + * Remember the pages we extended by so we can use them without + * looking into the FSM. + */ + if (extend_by_pages > 1) + { + bistate->next_free = firstBlock + 1; + bistate->last_free = firstBlock + extend_by_pages - 1; + } + else + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + + /* maintain bistate->current_buf */ + IncrBufferRefCount(buffer); + bistate->current_buf = buffer; + } + + return buffer; +#undef MAX_BUFFERS_TO_EXTEND_BY } + /* * RelationGetBufferForTuple * @@ -350,10 +437,12 @@ RelationGetBufferForTuple(Relation relation, Size len, targetFreeSpace = 0; BlockNumber targetBlock, otherBlock; - bool needLock; len = MAXALIGN(len); /* be conservative */ + if (num_pages <= 0) + num_pages = 1; + /* Bulk insert is not supported for updates, only inserts. */ Assert(otherBuffer == InvalidBuffer || !bistate); @@ -558,83 +647,50 @@ loop: ReleaseBuffer(buffer); } - /* Without FSM, always fall out of the loop and extend */ - if (!use_fsm) - break; - - /* - * Update FSM as to condition of this page, and ask for another page - * to try. - */ - targetBlock = RecordAndGetPageWithFreeSpace(relation, - targetBlock, - pageFreeSpace, - targetFreeSpace); - } - - /* - * Have to extend the relation. - * - * We have to use a lock to ensure no one else is extending the rel at the - * same time, else we will both try to initialize the same new page. We - * can skip locking for new or temp relations, however, since no one else - * could be accessing them. - */ - needLock = !RELATION_IS_LOCAL(relation); - - /* - * If we need the lock but are not able to acquire it immediately, we'll - * consider extending the relation by multiple blocks at a time to manage - * contention on the relation extension lock. However, this only makes - * sense if we're using the FSM; otherwise, there's no point. - */ - if (needLock) - { - if (!use_fsm) - LockRelationForExtension(relation, ExclusiveLock); - else if (!ConditionalLockRelationForExtension(relation, ExclusiveLock)) + if (bistate + && bistate->next_free != InvalidBlockNumber + && bistate->next_free <= bistate->last_free) { - /* Couldn't get the lock immediately; wait for it. */ - LockRelationForExtension(relation, ExclusiveLock); - /* - * Check if some other backend has extended a block for us while - * we were waiting on the lock. + * We bulk extended the relation before, and there are still some + * unused pages from that extension, so we don't need to look in + * the FSM for a new page. But do record the free space from the + * last page, somebody might insert narrower tuples later. */ - targetBlock = GetPageWithFreeSpace(relation, targetFreeSpace); + if (use_fsm) + RecordPageWithFreeSpace(relation, targetBlock, pageFreeSpace); - /* - * If some other waiter has already extended the relation, we - * don't need to do so; just use the existing freespace. - */ - if (targetBlock != InvalidBlockNumber) + Assert(bistate->last_free != InvalidBlockNumber && + bistate->next_free <= bistate->last_free); + targetBlock = bistate->next_free; + if (bistate->next_free >= bistate->last_free) { - UnlockRelationForExtension(relation, ExclusiveLock); - goto loop; + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; } - - /* Time to bulk-extend. */ - RelationAddExtraBlocks(relation, bistate); + else + bistate->next_free++; + } + else if (!use_fsm) + { + /* Without FSM, always fall out of the loop and extend */ + break; + } + else + { + /* + * Update FSM as to condition of this page, and ask for another + * page to try. + */ + targetBlock = RecordAndGetPageWithFreeSpace(relation, + targetBlock, + pageFreeSpace, + targetFreeSpace); } } - /* - * In addition to whatever extension we performed above, we always add at - * least one block to satisfy our own request. - * - * XXX This does an lseek - rather expensive - but at the moment it is the - * only way to accurately determine how many blocks are in a relation. Is - * it worth keeping an accurate file length in shared memory someplace, - * rather than relying on the kernel to do it for us? - */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - - /* - * Release the file-extension lock; it's now OK for someone else to extend - * the relation some more. - */ - if (needLock) - UnlockRelationForExtension(relation, ExclusiveLock); + /* Have to extend the relation */ + buffer = RelationAddBlocks(relation, bistate, num_pages, use_fsm); /* * We need to initialize the empty new page. Double-check that it really -- 2.38.0 --wwosng5vofi3dmcs Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0012-WIP-Don-t-initialize-page-in-vm-fsm-_extend-not-n.patch" ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v5 12/15] hio: Use ExtendBufferedRelBy() @ 2022-10-26 21:14 Andres Freund <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Andres Freund @ 2022-10-26 21:14 UTC (permalink / raw) --- src/backend/access/heap/hio.c | 285 +++++++++++++++++----------------- 1 file changed, 146 insertions(+), 139 deletions(-) diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c index 65886839e7..48cfcff975 100644 --- a/src/backend/access/heap/hio.c +++ b/src/backend/access/heap/hio.c @@ -354,6 +270,9 @@ RelationGetBufferForTuple(Relation relation, Size len, so in RelationGetBufferForTuple() up above where your changes start, there is this code /* * We first try to put the tuple on the same page we last inserted a tuple * on, as cached in the BulkInsertState or relcache entry. If that * doesn't work, we ask the Free Space Map to locate a suitable page. * Since the FSM's info might be out of date, we have to be prepared to * loop around and retry multiple times. (To insure this isn't an infinite * loop, we must update the FSM with the correct amount of free space on * each page that proves not to be suitable.) If the FSM has no record of * a page with enough free space, we give up and extend the relation. * * When use_fsm is false, we either put the tuple onto the existing target * page or extend the relation. */ if (bistate && bistate->current_buf != InvalidBuffer) { targetBlock = BufferGetBlockNumber(bistate->current_buf); } else targetBlock = RelationGetTargetBlock(relation); if (targetBlock == InvalidBlockNumber && use_fsm) { /* * We have no cached target page, so ask the FSM for an initial * target. */ targetBlock = GetPageWithFreeSpace(relation, targetFreeSpace); } And, I was thinking how, ReadBufferBI() only has one caller now (RelationGetBufferForTuple()) and, this caller basically already has checked for the case in the inside of ReadBufferBI() (the code I pasted above) /* If we have the desired block already pinned, re-pin and return it */ if (bistate->current_buf != InvalidBuffer) { if (BufferGetBlockNumber(bistate->current_buf) == targetBlock) { /* * Currently the LOCK variants are only used for extending * relation, which should never reach this branch. */ Assert(mode != RBM_ZERO_AND_LOCK && mode != RBM_ZERO_AND_CLEANUP_LOCK); IncrBufferRefCount(bistate->current_buf); return bistate->current_buf; } /* ... else drop the old buffer */ So, I was thinking maybe there is some way to inline the logic for ReadBufferBI(), because I think it would feel more streamlined to me. @@ -558,18 +477,46 @@ loop: ReleaseBuffer(buffer); } Oh, and I forget which commit introduced BulkInsertState->next_free and last_free, but I remember thinking that it didn't seem to fit with the other parts of that commit. - /* Without FSM, always fall out of the loop and extend */ - if (!use_fsm) - break; + if (bistate + && bistate->next_free != InvalidBlockNumber + && bistate->next_free <= bistate->last_free) + { + /* + * We bulk extended the relation before, and there are still some + * unused pages from that extension, so we don't need to look in + * the FSM for a new page. But do record the free space from the + * last page, somebody might insert narrower tuples later. + */ Why couldn't we have found out that we bulk-extended before and get the block from there up above the while loop? + if (use_fsm) + RecordPageWithFreeSpace(relation, targetBlock, pageFreeSpace); - /* - * Update FSM as to condition of this page, and ask for another page - * to try. - */ - targetBlock = RecordAndGetPageWithFreeSpace(relation, - targetBlock, - pageFreeSpace, - targetFreeSpace); + Assert(bistate->last_free != InvalidBlockNumber && You don't need the below half of the assert. + bistate->next_free <= bistate->last_free); + targetBlock = bistate->next_free; + if (bistate->next_free >= bistate->last_free) they can only be equal at this point + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + else + bistate->next_free++; + } + else if (!use_fsm) + { + /* Without FSM, always fall out of the loop and extend */ + break; + } It would be nice to have a comment explaining why this is in its own else if instead of breaking earlier (i.e. !use_fsm is still a valid case in the if branch above it) + else + { + /* + * Update FSM as to condition of this page, and ask for another + * page to try. + */ + targetBlock = RecordAndGetPageWithFreeSpace(relation, + targetBlock, + pageFreeSpace, + targetFreeSpace); + } we can get rid of needLock and waitcount variables like this +#define MAX_BUFFERS 64 + Buffer victim_buffers[MAX_BUFFERS]; + BlockNumber firstBlock = InvalidBlockNumber; + BlockNumber firstBlockFSM = InvalidBlockNumber; + BlockNumber curBlock; + uint32 extend_by_pages; + uint32 no_fsm_pages; + uint32 waitcount; + + extend_by_pages = num_pages; + + /* + * Multiply the number of pages to extend by the number of waiters. Do + * this even if we're not using the FSM, as it does relieve + * contention. Pages will be found via bistate->next_free. + */ + if (needLock) + waitcount = RelationExtensionLockWaiterCount(relation); + else + waitcount = 0; + extend_by_pages += extend_by_pages * waitcount; if (!RELATION_IS_LOCAL(relation)) extend_by_pages += extend_by_pages * RelationExtensionLockWaiterCount(relation); + + /* + * can't extend by more than MAX_BUFFERS, we need to pin them all + * concurrently. FIXME: Need an NBuffers / MaxBackends type limit + * here. + */ + extend_by_pages = Min(extend_by_pages, MAX_BUFFERS); + + /* + * How many of the extended pages not to enter into the FSM. + * + * Only enter pages that we don't need ourselves into the FSM. + * Otherwise every other backend will immediately try to use the pages + * this backend neds itself, causing unnecessary contention. + * + * Bulk extended pages are remembered in bistate->next_free_buffer. So + * without a bistate we can't directly make use of them. + * + * Never enter the page returned into the FSM, we'll immediately use + * it. + */ + if (num_pages > 1 && bistate == NULL) + no_fsm_pages = 1; + else + no_fsm_pages = num_pages; this is more clearly this: no_fsm_pages = bistate == NULL ? 1 : num_pages; - /* - * Release the file-extension lock; it's now OK for someone else to extend - * the relation some more. - */ - if (needLock) - UnlockRelationForExtension(relation, ExclusiveLock); + if (bistate) + { + if (extend_by_pages > 1) + { + bistate->next_free = firstBlock + 1; + bistate->last_free = firstBlock + extend_by_pages - 1; + } + else + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + } + + buffer = victim_buffers[0]; If we move buffer = up, we can have only one if (bistate) + if (bistate) + { + IncrBufferRefCount(buffer); + bistate->current_buf = buffer; + } + } like this: buffer = victim_buffers[0]; if (bistate) { if (extend_by_pages > 1) { bistate->next_free = firstBlock + 1; bistate->last_free = firstBlock + extend_by_pages - 1; } else { bistate->next_free = InvalidBlockNumber; bistate->last_free = InvalidBlockNumber; } IncrBufferRefCount(buffer); bistate->current_buf = buffer; } ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v4 12/15] hio: Use ExtendBufferedRelBy() @ 2022-10-26 21:14 Andres Freund <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Andres Freund @ 2022-10-26 21:14 UTC (permalink / raw) --- src/backend/access/heap/hio.c | 287 +++++++++++++++++----------------- 1 file changed, 147 insertions(+), 140 deletions(-) diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c index 65886839e70..48cfcff975f 100644 --- a/src/backend/access/heap/hio.c +++ b/src/backend/access/heap/hio.c @@ -185,90 +185,6 @@ GetVisibilityMapPins(Relation relation, Buffer buffer1, Buffer buffer2, } } -/* - * Extend a relation by multiple blocks to avoid future contention on the - * relation extension lock. Our goal is to pre-extend the relation by an - * amount which ramps up as the degree of contention ramps up, but limiting - * the result to some sane overall value. - */ -static void -RelationAddExtraBlocks(Relation relation, BulkInsertState bistate) -{ - BlockNumber blockNum, - firstBlock = InvalidBlockNumber; - int extraBlocks; - int lockWaiters; - - /* Use the length of the lock wait queue to judge how much to extend. */ - lockWaiters = RelationExtensionLockWaiterCount(relation); - if (lockWaiters <= 0) - return; - - /* - * It might seem like multiplying the number of lock waiters by as much as - * 20 is too aggressive, but benchmarking revealed that smaller numbers - * were insufficient. 512 is just an arbitrary cap to prevent - * pathological results. - */ - extraBlocks = Min(512, lockWaiters * 20); - - do - { - Buffer buffer; - Page page; - Size freespace; - - /* - * Extend by one page. This should generally match the main-line - * extension code in RelationGetBufferForTuple, except that we hold - * the relation extension lock throughout, and we don't immediately - * initialize the page (see below). - */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - page = BufferGetPage(buffer); - - if (!PageIsNew(page)) - elog(ERROR, "page %u of relation \"%s\" should be empty but is not", - BufferGetBlockNumber(buffer), - RelationGetRelationName(relation)); - - /* - * Add the page to the FSM without initializing. If we were to - * initialize here, the page would potentially get flushed out to disk - * before we add any useful content. There's no guarantee that that'd - * happen before a potential crash, so we need to deal with - * uninitialized pages anyway, thus avoid the potential for - * unnecessary writes. - */ - - /* we'll need this info below */ - blockNum = BufferGetBlockNumber(buffer); - freespace = BufferGetPageSize(buffer) - SizeOfPageHeaderData; - - UnlockReleaseBuffer(buffer); - - /* Remember first block number thus added. */ - if (firstBlock == InvalidBlockNumber) - firstBlock = blockNum; - - /* - * Immediately update the bottom level of the FSM. This has a good - * chance of making this page visible to other concurrently inserting - * backends, and we want that to happen without delay. - */ - RecordPageWithFreeSpace(relation, blockNum, freespace); - } - while (--extraBlocks > 0); - - /* - * Updating the upper levels of the free space map is too expensive to do - * for every block, but it's worth doing once at the end to make sure that - * subsequent insertion activity sees all of those nifty free pages we - * just inserted. - */ - FreeSpaceMapVacuumRange(relation, firstBlock, blockNum + 1); -} - /* * RelationGetBufferForTuple * @@ -354,6 +270,9 @@ RelationGetBufferForTuple(Relation relation, Size len, len = MAXALIGN(len); /* be conservative */ + if (num_pages <= 0) + num_pages = 1; + /* Bulk insert is not supported for updates, only inserts. */ Assert(otherBuffer == InvalidBuffer || !bistate); @@ -558,18 +477,46 @@ loop: ReleaseBuffer(buffer); } - /* Without FSM, always fall out of the loop and extend */ - if (!use_fsm) - break; + if (bistate + && bistate->next_free != InvalidBlockNumber + && bistate->next_free <= bistate->last_free) + { + /* + * We bulk extended the relation before, and there are still some + * unused pages from that extension, so we don't need to look in + * the FSM for a new page. But do record the free space from the + * last page, somebody might insert narrower tuples later. + */ + if (use_fsm) + RecordPageWithFreeSpace(relation, targetBlock, pageFreeSpace); - /* - * Update FSM as to condition of this page, and ask for another page - * to try. - */ - targetBlock = RecordAndGetPageWithFreeSpace(relation, - targetBlock, - pageFreeSpace, - targetFreeSpace); + Assert(bistate->last_free != InvalidBlockNumber && + bistate->next_free <= bistate->last_free); + targetBlock = bistate->next_free; + if (bistate->next_free >= bistate->last_free) + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + else + bistate->next_free++; + } + else if (!use_fsm) + { + /* Without FSM, always fall out of the loop and extend */ + break; + } + else + { + /* + * Update FSM as to condition of this page, and ask for another + * page to try. + */ + targetBlock = RecordAndGetPageWithFreeSpace(relation, + targetBlock, + pageFreeSpace, + targetFreeSpace); + } } /* @@ -582,60 +529,120 @@ loop: */ needLock = !RELATION_IS_LOCAL(relation); - /* - * If we need the lock but are not able to acquire it immediately, we'll - * consider extending the relation by multiple blocks at a time to manage - * contention on the relation extension lock. However, this only makes - * sense if we're using the FSM; otherwise, there's no point. - */ - if (needLock) { - if (!use_fsm) - LockRelationForExtension(relation, ExclusiveLock); - else if (!ConditionalLockRelationForExtension(relation, ExclusiveLock)) +#define MAX_BUFFERS 64 + Buffer victim_buffers[MAX_BUFFERS]; + BlockNumber firstBlock = InvalidBlockNumber; + BlockNumber firstBlockFSM = InvalidBlockNumber; + BlockNumber curBlock; + uint32 extend_by_pages; + uint32 no_fsm_pages; + uint32 waitcount; + + extend_by_pages = num_pages; + + /* + * Multiply the number of pages to extend by the number of waiters. Do + * this even if we're not using the FSM, as it does relieve + * contention. Pages will be found via bistate->next_free. + */ + if (needLock) + waitcount = RelationExtensionLockWaiterCount(relation); + else + waitcount = 0; + extend_by_pages += extend_by_pages * waitcount; + + /* + * can't extend by more than MAX_BUFFERS, we need to pin them all + * concurrently. FIXME: Need an NBuffers / MaxBackends type limit + * here. + */ + extend_by_pages = Min(extend_by_pages, MAX_BUFFERS); + + /* + * How many of the extended pages not to enter into the FSM. + * + * Only enter pages that we don't need ourselves into the FSM. + * Otherwise every other backend will immediately try to use the pages + * this backend neds itself, causing unnecessary contention. + * + * Bulk extended pages are remembered in bistate->next_free_buffer. So + * without a bistate we can't directly make use of them. + * + * Never enter the page returned into the FSM, we'll immediately use + * it. + */ + if (num_pages > 1 && bistate == NULL) + no_fsm_pages = 1; + else + no_fsm_pages = num_pages; + + if (bistate && bistate->current_buf != InvalidBuffer) { - /* Couldn't get the lock immediately; wait for it. */ - LockRelationForExtension(relation, ExclusiveLock); + ReleaseBuffer(bistate->current_buf); + bistate->current_buf = InvalidBuffer; + } - /* - * Check if some other backend has extended a block for us while - * we were waiting on the lock. - */ - targetBlock = GetPageWithFreeSpace(relation, targetFreeSpace); + firstBlock = ExtendBufferedRelBy(EB_REL(relation), MAIN_FORKNUM, + bistate ? bistate->strategy : NULL, + EB_LOCK_FIRST, + extend_by_pages, + victim_buffers, + &extend_by_pages); - /* - * If some other waiter has already extended the relation, we - * don't need to do so; just use the existing freespace. - */ - if (targetBlock != InvalidBlockNumber) + /* + * Relation is now extended. Make all but the first buffer available + * to other backends. + * + * XXX: We don't necessarily need to release pin / update FSM while + * holding the extension lock. But there are some advantages. + */ + curBlock = firstBlock; + for (uint32 i = 0; i < extend_by_pages; i++, curBlock++) + { + Assert(curBlock == BufferGetBlockNumber(victim_buffers[i])); + Assert(BlockNumberIsValid(curBlock)); + + /* don't release the pin on the page returned by this function */ + if (i > 0) + ReleaseBuffer(victim_buffers[i]); + + if (i >= no_fsm_pages && use_fsm) { - UnlockRelationForExtension(relation, ExclusiveLock); - goto loop; - } + if (firstBlockFSM == InvalidBlockNumber) + firstBlockFSM = curBlock; - /* Time to bulk-extend. */ - RelationAddExtraBlocks(relation, bistate); + RecordPageWithFreeSpace(relation, + curBlock, + BufferGetPageSize(victim_buffers[i]) - SizeOfPageHeaderData); + } + } + + if (use_fsm && firstBlockFSM != InvalidBlockNumber) + FreeSpaceMapVacuumRange(relation, firstBlockFSM, firstBlock + num_pages); + + if (bistate) + { + if (extend_by_pages > 1) + { + bistate->next_free = firstBlock + 1; + bistate->last_free = firstBlock + extend_by_pages - 1; + } + else + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + } + + buffer = victim_buffers[0]; + if (bistate) + { + IncrBufferRefCount(buffer); + bistate->current_buf = buffer; } } - /* - * In addition to whatever extension we performed above, we always add at - * least one block to satisfy our own request. - * - * XXX This does an lseek - rather expensive - but at the moment it is the - * only way to accurately determine how many blocks are in a relation. Is - * it worth keeping an accurate file length in shared memory someplace, - * rather than relying on the kernel to do it for us? - */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - - /* - * Release the file-extension lock; it's now OK for someone else to extend - * the relation some more. - */ - if (needLock) - UnlockRelationForExtension(relation, ExclusiveLock); - /* * We need to initialize the empty new page. Double-check that it really * is empty (this should never happen, but if it does we don't want to -- 2.38.0 --njltjzibte523gwd Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0013-bufmgr-debug-Add-PrintBuffer-Desc.patch" ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v6 14/17] hio: Use ExtendBufferedRelBy() @ 2023-03-29 01:39 Andres Freund <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Andres Freund @ 2023-03-29 01:39 UTC (permalink / raw) --- src/backend/access/heap/hio.c | 370 ++++++++++++++++++++-------------- 1 file changed, 217 insertions(+), 153 deletions(-) diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c index cc913a028e0..6d66826d951 100644 --- a/src/backend/access/heap/hio.c +++ b/src/backend/access/heap/hio.c @@ -192,87 +192,197 @@ GetVisibilityMapPins(Relation relation, Buffer buffer1, Buffer buffer2, } /* - * Extend a relation by multiple blocks to avoid future contention on the - * relation extension lock. Our goal is to pre-extend the relation by an - * amount which ramps up as the degree of contention ramps up, but limiting - * the result to some sane overall value. + * Extend the relation. By multiple pages, if beneficial. + * + * If the caller needs multiple pages (num_pages > 1), we always try to extend + * by at least that much. + * + * If there is contention on the extension lock, we don't just extend "for + * ourselves", but we try to help others. We can do so by adding empty pages + * into the FSM. Typically there is no contention when we can't use the FSM. + * + * We do have to limit the number of pages to extend by to some value, as the + * buffers for all the extended pages need to, temporarily, be pinned. For now + * we define MAX_BUFFERS_TO_EXTEND_BY to be 64 buffers, it's hard to see + * benefits with higher numbers. This partially is because copyfrom.c's + * MAX_BUFFERED_TUPLES / MAX_BUFFERED_BYTES prevents larger multi_inserts. */ -static void -RelationAddExtraBlocks(Relation relation, BulkInsertState bistate) +static Buffer +RelationAddBlocks(Relation relation, BulkInsertState bistate, + int num_pages, bool use_fsm, bool *did_unlock) { - BlockNumber blockNum, - firstBlock = InvalidBlockNumber; - int extraBlocks; - int lockWaiters; - - /* Use the length of the lock wait queue to judge how much to extend. */ - lockWaiters = RelationExtensionLockWaiterCount(relation); - if (lockWaiters <= 0) - return; +#define MAX_BUFFERS_TO_EXTEND_BY 64 + Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY]; + BlockNumber first_block = InvalidBlockNumber; + BlockNumber last_block = InvalidBlockNumber; + uint32 extend_by_pages; + uint32 not_in_fsm_pages; + Buffer buffer; + Page page; /* - * It might seem like multiplying the number of lock waiters by as much as - * 20 is too aggressive, but benchmarking revealed that smaller numbers - * were insufficient. 512 is just an arbitrary cap to prevent - * pathological results. + * Determine by how many pages to try to extend by. */ - extraBlocks = Min(512, lockWaiters * 20); - - do + if (bistate == NULL && !use_fsm) { - Buffer buffer; - Page page; - Size freespace; - /* - * Extend by one page. This should generally match the main-line - * extension code in RelationGetBufferForTuple, except that we hold - * the relation extension lock throughout, and we don't immediately - * initialize the page (see below). + * If we have neither bistate, nor can use the FSM, we can't bulk + * extend - there'd be no way to find the additional pages. */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - page = BufferGetPage(buffer); - - if (!PageIsNew(page)) - elog(ERROR, "page %u of relation \"%s\" should be empty but is not", - BufferGetBlockNumber(buffer), - RelationGetRelationName(relation)); - - /* - * Add the page to the FSM without initializing. If we were to - * initialize here, the page would potentially get flushed out to disk - * before we add any useful content. There's no guarantee that that'd - * happen before a potential crash, so we need to deal with - * uninitialized pages anyway, thus avoid the potential for - * unnecessary writes. - */ - - /* we'll need this info below */ - blockNum = BufferGetBlockNumber(buffer); - freespace = BufferGetPageSize(buffer) - SizeOfPageHeaderData; - - UnlockReleaseBuffer(buffer); - - /* Remember first block number thus added. */ - if (firstBlock == InvalidBlockNumber) - firstBlock = blockNum; - - /* - * Immediately update the bottom level of the FSM. This has a good - * chance of making this page visible to other concurrently inserting - * backends, and we want that to happen without delay. - */ - RecordPageWithFreeSpace(relation, blockNum, freespace); + extend_by_pages = 1; + } + else + { + uint32 waitcount; + + /* + * Try to extend at least by the number of pages the caller needs. We + * can remember the additional pages (either via FSM or bistate). + */ + extend_by_pages = num_pages; + + if (!RELATION_IS_LOCAL(relation)) + waitcount = RelationExtensionLockWaiterCount(relation); + else + waitcount = 0; + + /* + * Multiply the number of pages to extend by the number of waiters. Do + * this even if we're not using the FSM, as it still relieves + * contention, by deferring the next time this backend needs to + * extend. In that case the extended pages will be found via + * bistate->next_free. + */ + extend_by_pages += extend_by_pages * waitcount; + + /* + * Can't extend by more than MAX_BUFFERS_TO_EXTEND_BY, we need to pin + * them all concurrently. + */ + extend_by_pages = Min(extend_by_pages, MAX_BUFFERS_TO_EXTEND_BY); } - while (--extraBlocks > 0); /* - * Updating the upper levels of the free space map is too expensive to do - * for every block, but it's worth doing once at the end to make sure that - * subsequent insertion activity sees all of those nifty free pages we - * just inserted. + * How many of the extended pages should be entered into the FSM? + * + * If we have a bistate, only enter pages that we don't need ourselves + * into the FSM. Otherwise every other backend will immediately try to + * use the pages this backend needs for itself, causing unnecessary + * contention. If we don't have a bistate, we can't avoid the FSM. + * + * Never enter the page returned into the FSM, we'll immediately use it. */ - FreeSpaceMapVacuumRange(relation, firstBlock, blockNum + 1); + if (num_pages > 1 && bistate == NULL) + not_in_fsm_pages = 1; + else + not_in_fsm_pages = num_pages; + + /* prepare to put another buffer into the bistate */ + if (bistate && bistate->current_buf != InvalidBuffer) + { + ReleaseBuffer(bistate->current_buf); + bistate->current_buf = InvalidBuffer; + } + + /* + * Extend the relation. We ask for the first returned page to be locked, + * so that we are sure that nobody has inserted into the page + * concurrently. + * + * With the current MAX_BUFFERS_TO_EXTEND_BY there's no danger of + * [auto]vacuum trying to truncate later pages as REL_TRUNCATE_MINIMUM is + * way larger. + */ + first_block = ExtendBufferedRelBy(EB_REL(relation), MAIN_FORKNUM, + bistate ? bistate->strategy : NULL, + EB_LOCK_FIRST, + extend_by_pages, + victim_buffers, + &extend_by_pages); + buffer = victim_buffers[0]; /* the buffer the function will return */ + last_block = first_block + (extend_by_pages - 1); + Assert(first_block == BufferGetBlockNumber(buffer)); + + /* + * Relation is now extended. Initialize the page. We do this here, before + * potentially releasing the lock on the page, because it allows us to + * double check that the page contents are empty (this should never + * happen, but if it does we don't want to risk wiping out valid data). + */ + page = BufferGetPage(buffer); + if (!PageIsNew(page)) + elog(ERROR, "page %u of relation \"%s\" should be empty but is not", + first_block, + RelationGetRelationName(relation)); + + PageInit(page, BufferGetPageSize(buffer), 0); + MarkBufferDirty(buffer); + + /* + * If we decided to put pages into the FSM, release the buffer lock (but + * not pin), we don't want to do IO while holding a buffer lock. This will + * necessitate a bit more extensive checking in our caller. + */ + if (use_fsm && not_in_fsm_pages < extend_by_pages) + { + LockBuffer(buffer, BUFFER_LOCK_UNLOCK); + *did_unlock = true; + } + + /* + * Relation is now extended. Release pins on all buffers, except for the + * first (which we'll return). If we decided to put pages into the FSM, + * we can do that as part of the same loop. + */ + for (uint32 i = 1; i < extend_by_pages; i++) + { + BlockNumber curBlock = first_block + i; + + Assert(curBlock == BufferGetBlockNumber(victim_buffers[i])); + Assert(BlockNumberIsValid(curBlock)); + + ReleaseBuffer(victim_buffers[i]); + + if (use_fsm && i >= not_in_fsm_pages) + { + Size freespace = BufferGetPageSize(victim_buffers[i]) - + SizeOfPageHeaderData; + + RecordPageWithFreeSpace(relation, curBlock, freespace); + } + } + + if (use_fsm && not_in_fsm_pages < extend_by_pages) + { + BlockNumber first_fsm_block = first_block + not_in_fsm_pages; + + FreeSpaceMapVacuumRange(relation, first_fsm_block, last_block); + } + + if (bistate) + { + /* + * Remember the additionaly pages we extended by, so we later can use + * them without looking into the FSM. + */ + if (extend_by_pages > 1) + { + bistate->next_free = first_block + 1; + bistate->last_free = last_block; + } + else + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + + /* maintain bistate->current_buf */ + IncrBufferRefCount(buffer); + bistate->current_buf = buffer; + } + + return buffer; +#undef MAX_BUFFERS_TO_EXTEND_BY } /* @@ -356,11 +466,13 @@ RelationGetBufferForTuple(Relation relation, Size len, targetFreeSpace = 0; BlockNumber targetBlock, otherBlock; - bool needLock; bool unlockedTargetBuffer; len = MAXALIGN(len); /* be conservative */ + if (num_pages <= 0) + num_pages = 1; + /* Bulk insert is not supported for updates, only inserts. */ Assert(otherBuffer == InvalidBuffer || !bistate); @@ -565,102 +677,53 @@ loop: ReleaseBuffer(buffer); } - /* Without FSM, always fall out of the loop and extend */ - if (!use_fsm) - break; - - /* - * Update FSM as to condition of this page, and ask for another page - * to try. - */ - targetBlock = RecordAndGetPageWithFreeSpace(relation, - targetBlock, - pageFreeSpace, - targetFreeSpace); - } - - /* - * Have to extend the relation. - * - * We have to use a lock to ensure no one else is extending the rel at the - * same time, else we will both try to initialize the same new page. We - * can skip locking for new or temp relations, however, since no one else - * could be accessing them. - */ - needLock = !RELATION_IS_LOCAL(relation); - - /* - * If we need the lock but are not able to acquire it immediately, we'll - * consider extending the relation by multiple blocks at a time to manage - * contention on the relation extension lock. However, this only makes - * sense if we're using the FSM; otherwise, there's no point. - */ - if (needLock) - { - if (!use_fsm) - LockRelationForExtension(relation, ExclusiveLock); - else if (!ConditionalLockRelationForExtension(relation, ExclusiveLock)) + if (bistate && bistate->next_free != InvalidBlockNumber) { - /* Couldn't get the lock immediately; wait for it. */ - LockRelationForExtension(relation, ExclusiveLock); + Assert(bistate->next_free <= bistate->last_free); /* - * Check if some other backend has extended a block for us while - * we were waiting on the lock. + * We bulk extended the relation before, and there are still some + * unused pages from that extension, so we don't need to look in + * the FSM for a new page. But do record the free space from the + * last page, somebody might insert narrower tuples later. */ - targetBlock = GetPageWithFreeSpace(relation, targetFreeSpace); + if (use_fsm) + RecordPageWithFreeSpace(relation, targetBlock, pageFreeSpace); - /* - * If some other waiter has already extended the relation, we - * don't need to do so; just use the existing freespace. - */ - if (targetBlock != InvalidBlockNumber) + targetBlock = bistate->next_free; + if (bistate->next_free >= bistate->last_free) { - UnlockRelationForExtension(relation, ExclusiveLock); - goto loop; + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; } - - /* Time to bulk-extend. */ - RelationAddExtraBlocks(relation, bistate); + else + bistate->next_free++; + } + else if (!use_fsm) + { + /* Without FSM, always fall out of the loop and extend */ + break; + } + else + { + /* + * Update FSM as to condition of this page, and ask for another + * page to try. + */ + targetBlock = RecordAndGetPageWithFreeSpace(relation, + targetBlock, + pageFreeSpace, + targetFreeSpace); } } - /* - * In addition to whatever extension we performed above, we always add at - * least one block to satisfy our own request. - * - * XXX This does an lseek - rather expensive - but at the moment it is the - * only way to accurately determine how many blocks are in a relation. Is - * it worth keeping an accurate file length in shared memory someplace, - * rather than relying on the kernel to do it for us? - */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - - /* - * Release the file-extension lock; it's now OK for someone else to extend - * the relation some more. - */ - if (needLock) - UnlockRelationForExtension(relation, ExclusiveLock); - + /* Have to extend the relation */ unlockedTargetBuffer = false; + buffer = RelationAddBlocks(relation, bistate, num_pages, use_fsm, &unlockedTargetBuffer); + targetBlock = BufferGetBlockNumber(buffer); - - /* - * We need to initialize the empty new page. Double-check that it really - * is empty (this should never happen, but if it does we don't want to - * risk wiping out valid data). - */ page = BufferGetPage(buffer); - if (!PageIsNew(page)) - elog(ERROR, "page %u of relation \"%s\" should be empty but is not", - targetBlock, - RelationGetRelationName(relation)); - - PageInit(page, BufferGetPageSize(buffer), 0); - MarkBufferDirty(buffer); - /* * The page is empty, pin vmbuffer to set all_frozen bit. We don't want to * do IO while the buffer is locked, so we unlock the page first if IO is @@ -672,8 +735,9 @@ loop: if (!visibilitymap_pin_ok(targetBlock, *vmbuffer)) { + if (!unlockedTargetBuffer) + LockBuffer(buffer, BUFFER_LOCK_UNLOCK); unlockedTargetBuffer = true; - LockBuffer(buffer, BUFFER_LOCK_UNLOCK); visibilitymap_pin(relation, targetBlock, vmbuffer); } } -- 2.38.0 --cfn72vqnlbycypta Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0015-WIP-Don-t-initialize-page-in-vm-fsm-_extend-not-n.patch" ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v7 12/14] hio: Use ExtendBufferedRelBy() @ 2023-03-29 01:39 Andres Freund <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Andres Freund @ 2023-03-29 01:39 UTC (permalink / raw) Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/[email protected] --- src/backend/access/heap/hio.c | 372 ++++++++++++++++++++-------------- 1 file changed, 218 insertions(+), 154 deletions(-) diff --git a/src/backend/access/heap/hio.c b/src/backend/access/heap/hio.c index 561d7329058..574c56cfb5f 100644 --- a/src/backend/access/heap/hio.c +++ b/src/backend/access/heap/hio.c @@ -212,87 +212,197 @@ GetVisibilityMapPins(Relation relation, Buffer buffer1, Buffer buffer2, } /* - * Extend a relation by multiple blocks to avoid future contention on the - * relation extension lock. Our goal is to pre-extend the relation by an - * amount which ramps up as the degree of contention ramps up, but limiting - * the result to some sane overall value. + * Extend the relation. By multiple pages, if beneficial. + * + * If the caller needs multiple pages (num_pages > 1), we always try to extend + * by at least that much. + * + * If there is contention on the extension lock, we don't just extend "for + * ourselves", but we try to help others. We can do so by adding empty pages + * into the FSM. Typically there is no contention when we can't use the FSM. + * + * We do have to limit the number of pages to extend by to some value, as the + * buffers for all the extended pages need to, temporarily, be pinned. For now + * we define MAX_BUFFERS_TO_EXTEND_BY to be 64 buffers, it's hard to see + * benefits with higher numbers. This partially is because copyfrom.c's + * MAX_BUFFERED_TUPLES / MAX_BUFFERED_BYTES prevents larger multi_inserts. */ -static void -RelationAddExtraBlocks(Relation relation, BulkInsertState bistate) +static Buffer +RelationAddBlocks(Relation relation, BulkInsertState bistate, + int num_pages, bool use_fsm, bool *did_unlock) { - BlockNumber blockNum, - firstBlock = InvalidBlockNumber; - int extraBlocks; - int lockWaiters; - - /* Use the length of the lock wait queue to judge how much to extend. */ - lockWaiters = RelationExtensionLockWaiterCount(relation); - if (lockWaiters <= 0) - return; +#define MAX_BUFFERS_TO_EXTEND_BY 64 + Buffer victim_buffers[MAX_BUFFERS_TO_EXTEND_BY]; + BlockNumber first_block = InvalidBlockNumber; + BlockNumber last_block = InvalidBlockNumber; + uint32 extend_by_pages; + uint32 not_in_fsm_pages; + Buffer buffer; + Page page; /* - * It might seem like multiplying the number of lock waiters by as much as - * 20 is too aggressive, but benchmarking revealed that smaller numbers - * were insufficient. 512 is just an arbitrary cap to prevent - * pathological results. + * Determine by how many pages to try to extend by. */ - extraBlocks = Min(512, lockWaiters * 20); - - do + if (bistate == NULL && !use_fsm) { - Buffer buffer; - Page page; - Size freespace; - /* - * Extend by one page. This should generally match the main-line - * extension code in RelationGetBufferForTuple, except that we hold - * the relation extension lock throughout, and we don't immediately - * initialize the page (see below). + * If we have neither bistate, nor can use the FSM, we can't bulk + * extend - there'd be no way to find the additional pages. */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - page = BufferGetPage(buffer); - - if (!PageIsNew(page)) - elog(ERROR, "page %u of relation \"%s\" should be empty but is not", - BufferGetBlockNumber(buffer), - RelationGetRelationName(relation)); - - /* - * Add the page to the FSM without initializing. If we were to - * initialize here, the page would potentially get flushed out to disk - * before we add any useful content. There's no guarantee that that'd - * happen before a potential crash, so we need to deal with - * uninitialized pages anyway, thus avoid the potential for - * unnecessary writes. - */ - - /* we'll need this info below */ - blockNum = BufferGetBlockNumber(buffer); - freespace = BufferGetPageSize(buffer) - SizeOfPageHeaderData; - - UnlockReleaseBuffer(buffer); - - /* Remember first block number thus added. */ - if (firstBlock == InvalidBlockNumber) - firstBlock = blockNum; - - /* - * Immediately update the bottom level of the FSM. This has a good - * chance of making this page visible to other concurrently inserting - * backends, and we want that to happen without delay. - */ - RecordPageWithFreeSpace(relation, blockNum, freespace); + extend_by_pages = 1; + } + else + { + uint32 waitcount; + + /* + * Try to extend at least by the number of pages the caller needs. We + * can remember the additional pages (either via FSM or bistate). + */ + extend_by_pages = num_pages; + + if (!RELATION_IS_LOCAL(relation)) + waitcount = RelationExtensionLockWaiterCount(relation); + else + waitcount = 0; + + /* + * Multiply the number of pages to extend by the number of waiters. Do + * this even if we're not using the FSM, as it still relieves + * contention, by deferring the next time this backend needs to + * extend. In that case the extended pages will be found via + * bistate->next_free. + */ + extend_by_pages += extend_by_pages * waitcount; + + /* + * Can't extend by more than MAX_BUFFERS_TO_EXTEND_BY, we need to pin + * them all concurrently. + */ + extend_by_pages = Min(extend_by_pages, MAX_BUFFERS_TO_EXTEND_BY); } - while (--extraBlocks > 0); /* - * Updating the upper levels of the free space map is too expensive to do - * for every block, but it's worth doing once at the end to make sure that - * subsequent insertion activity sees all of those nifty free pages we - * just inserted. + * How many of the extended pages should be entered into the FSM? + * + * If we have a bistate, only enter pages that we don't need ourselves + * into the FSM. Otherwise every other backend will immediately try to + * use the pages this backend needs for itself, causing unnecessary + * contention. If we don't have a bistate, we can't avoid the FSM. + * + * Never enter the page returned into the FSM, we'll immediately use it. */ - FreeSpaceMapVacuumRange(relation, firstBlock, blockNum + 1); + if (num_pages > 1 && bistate == NULL) + not_in_fsm_pages = 1; + else + not_in_fsm_pages = num_pages; + + /* prepare to put another buffer into the bistate */ + if (bistate && bistate->current_buf != InvalidBuffer) + { + ReleaseBuffer(bistate->current_buf); + bistate->current_buf = InvalidBuffer; + } + + /* + * Extend the relation. We ask for the first returned page to be locked, + * so that we are sure that nobody has inserted into the page + * concurrently. + * + * With the current MAX_BUFFERS_TO_EXTEND_BY there's no danger of + * [auto]vacuum trying to truncate later pages as REL_TRUNCATE_MINIMUM is + * way larger. + */ + first_block = ExtendBufferedRelBy(EB_REL(relation), MAIN_FORKNUM, + bistate ? bistate->strategy : NULL, + EB_LOCK_FIRST, + extend_by_pages, + victim_buffers, + &extend_by_pages); + buffer = victim_buffers[0]; /* the buffer the function will return */ + last_block = first_block + (extend_by_pages - 1); + Assert(first_block == BufferGetBlockNumber(buffer)); + + /* + * Relation is now extended. Initialize the page. We do this here, before + * potentially releasing the lock on the page, because it allows us to + * double check that the page contents are empty (this should never + * happen, but if it does we don't want to risk wiping out valid data). + */ + page = BufferGetPage(buffer); + if (!PageIsNew(page)) + elog(ERROR, "page %u of relation \"%s\" should be empty but is not", + first_block, + RelationGetRelationName(relation)); + + PageInit(page, BufferGetPageSize(buffer), 0); + MarkBufferDirty(buffer); + + /* + * If we decided to put pages into the FSM, release the buffer lock (but + * not pin), we don't want to do IO while holding a buffer lock. This will + * necessitate a bit more extensive checking in our caller. + */ + if (use_fsm && not_in_fsm_pages < extend_by_pages) + { + LockBuffer(buffer, BUFFER_LOCK_UNLOCK); + *did_unlock = true; + } + + /* + * Relation is now extended. Release pins on all buffers, except for the + * first (which we'll return). If we decided to put pages into the FSM, + * we can do that as part of the same loop. + */ + for (uint32 i = 1; i < extend_by_pages; i++) + { + BlockNumber curBlock = first_block + i; + + Assert(curBlock == BufferGetBlockNumber(victim_buffers[i])); + Assert(BlockNumberIsValid(curBlock)); + + ReleaseBuffer(victim_buffers[i]); + + if (use_fsm && i >= not_in_fsm_pages) + { + Size freespace = BufferGetPageSize(victim_buffers[i]) - + SizeOfPageHeaderData; + + RecordPageWithFreeSpace(relation, curBlock, freespace); + } + } + + if (use_fsm && not_in_fsm_pages < extend_by_pages) + { + BlockNumber first_fsm_block = first_block + not_in_fsm_pages; + + FreeSpaceMapVacuumRange(relation, first_fsm_block, last_block); + } + + if (bistate) + { + /* + * Remember the additionaly pages we extended by, so we later can use + * them without looking into the FSM. + */ + if (extend_by_pages > 1) + { + bistate->next_free = first_block + 1; + bistate->last_free = last_block; + } + else + { + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; + } + + /* maintain bistate->current_buf */ + IncrBufferRefCount(buffer); + bistate->current_buf = buffer; + } + + return buffer; +#undef MAX_BUFFERS_TO_EXTEND_BY } /* @@ -376,12 +486,14 @@ RelationGetBufferForTuple(Relation relation, Size len, targetFreeSpace = 0; BlockNumber targetBlock, otherBlock; - bool needLock; bool unlockedTargetBuffer; bool recheckVmPins; len = MAXALIGN(len); /* be conservative */ + if (num_pages <= 0) + num_pages = 1; + /* Bulk insert is not supported for updates, only inserts. */ Assert(otherBuffer == InvalidBuffer || !bistate); @@ -581,102 +693,54 @@ loop: ReleaseBuffer(buffer); } - /* Without FSM, always fall out of the loop and extend */ - if (!use_fsm) - break; - - /* - * Update FSM as to condition of this page, and ask for another page - * to try. - */ - targetBlock = RecordAndGetPageWithFreeSpace(relation, - targetBlock, - pageFreeSpace, - targetFreeSpace); - } - - /* - * Have to extend the relation. - * - * We have to use a lock to ensure no one else is extending the rel at the - * same time, else we will both try to initialize the same new page. We - * can skip locking for new or temp relations, however, since no one else - * could be accessing them. - */ - needLock = !RELATION_IS_LOCAL(relation); - - /* - * If we need the lock but are not able to acquire it immediately, we'll - * consider extending the relation by multiple blocks at a time to manage - * contention on the relation extension lock. However, this only makes - * sense if we're using the FSM; otherwise, there's no point. - */ - if (needLock) - { - if (!use_fsm) - LockRelationForExtension(relation, ExclusiveLock); - else if (!ConditionalLockRelationForExtension(relation, ExclusiveLock)) + if (bistate && bistate->next_free != InvalidBlockNumber) { - /* Couldn't get the lock immediately; wait for it. */ - LockRelationForExtension(relation, ExclusiveLock); + Assert(bistate->next_free <= bistate->last_free); /* - * Check if some other backend has extended a block for us while - * we were waiting on the lock. + * We bulk extended the relation before, and there are still some + * unused pages from that extension, so we don't need to look in + * the FSM for a new page. But do record the free space from the + * last page, somebody might insert narrower tuples later. */ - targetBlock = GetPageWithFreeSpace(relation, targetFreeSpace); + if (use_fsm) + RecordPageWithFreeSpace(relation, targetBlock, pageFreeSpace); - /* - * If some other waiter has already extended the relation, we - * don't need to do so; just use the existing freespace. - */ - if (targetBlock != InvalidBlockNumber) + targetBlock = bistate->next_free; + if (bistate->next_free >= bistate->last_free) { - UnlockRelationForExtension(relation, ExclusiveLock); - goto loop; + bistate->next_free = InvalidBlockNumber; + bistate->last_free = InvalidBlockNumber; } - - /* Time to bulk-extend. */ - RelationAddExtraBlocks(relation, bistate); + else + bistate->next_free++; + } + else if (!use_fsm) + { + /* Without FSM, always fall out of the loop and extend */ + break; + } + else + { + /* + * Update FSM as to condition of this page, and ask for another + * page to try. + */ + targetBlock = RecordAndGetPageWithFreeSpace(relation, + targetBlock, + pageFreeSpace, + targetFreeSpace); } } - /* - * In addition to whatever extension we performed above, we always add at - * least one block to satisfy our own request. - * - * XXX This does an lseek - rather expensive - but at the moment it is the - * only way to accurately determine how many blocks are in a relation. Is - * it worth keeping an accurate file length in shared memory someplace, - * rather than relying on the kernel to do it for us? - */ - buffer = ReadBufferBI(relation, P_NEW, RBM_ZERO_AND_LOCK, bistate); - - /* - * Release the file-extension lock; it's now OK for someone else to extend - * the relation some more. - */ - if (needLock) - UnlockRelationForExtension(relation, ExclusiveLock); - + /* Have to extend the relation */ unlockedTargetBuffer = false; + buffer = RelationAddBlocks(relation, bistate, num_pages, use_fsm, &unlockedTargetBuffer); + recheckVmPins = unlockedTargetBuffer; + targetBlock = BufferGetBlockNumber(buffer); - - /* - * We need to initialize the empty new page. Double-check that it really - * is empty (this should never happen, but if it does we don't want to - * risk wiping out valid data). - */ page = BufferGetPage(buffer); - if (!PageIsNew(page)) - elog(ERROR, "page %u of relation \"%s\" should be empty but is not", - targetBlock, - RelationGetRelationName(relation)); - - PageInit(page, BufferGetPageSize(buffer), 0); - MarkBufferDirty(buffer); - /* * The page is empty, pin vmbuffer to set all_frozen bit. We don't want to * do IO while the buffer is locked, so we unlock the page first if IO is @@ -688,8 +752,9 @@ loop: if (!visibilitymap_pin_ok(targetBlock, *vmbuffer)) { + if (!unlockedTargetBuffer) + LockBuffer(buffer, BUFFER_LOCK_UNLOCK); unlockedTargetBuffer = true; - LockBuffer(buffer, BUFFER_LOCK_UNLOCK); visibilitymap_pin(relation, targetBlock, vmbuffer); } } @@ -702,7 +767,6 @@ loop: * that another backend used space on this page. We check for that below, * and retry if necessary. */ - recheckVmPins = false; if (unlockedTargetBuffer) { /* released lock on target buffer above */ -- 2.38.0 --ju7ntqqtbf66a3ug Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0013-Convert-a-few-places-to-ExtendBufferedRelTo.patch" ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v3] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Previously, ALTER DEFAULT PRIVILEGE didn't support large objects, so if we want to allow users other than the owner to use the large object, we need to grant a privilege on it every time a large object is created. Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 15 ++- src/backend/catalog/aclchk.c | 21 +++++ src/backend/catalog/objectaddress.c | 18 +++- src/backend/catalog/pg_largeobject.c | 18 +++- src/backend/parser/gram.y | 5 +- src/bin/pg_dump/dumputils.c | 3 +- src/bin/pg_dump/pg_dump.c | 3 + src/bin/psql/describe.c | 6 +- src/bin/psql/tab-complete.c | 2 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 94 ++++++++++++++++++- src/test/regress/sql/privileges.sql | 36 +++++++ 14 files changed, 215 insertions(+), 11 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b654fae1b2..c8cf56c666 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3322,7 +3322,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 89aacec4fa..3ab695892d 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -51,6 +51,11 @@ GRANT { { USAGE | CREATE } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -83,6 +88,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -161,7 +173,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index d2abc48fd8..6463ae921a 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1077,6 +1077,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1268,6 +1272,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1511,6 +1525,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4334,6 +4351,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 85a7b7e641..94f38c2333 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2000,16 +2000,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_LARGEOBJECT: + objtype_str = "large objects"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_LARGEOBJECT))); } /* @@ -3798,6 +3802,12 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_LARGEOBJECT: + Assert(!nspname); + appendStringInfo(&buffer, + _("default privileges on new large objects belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -5720,6 +5730,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_LARGEOBJECT: + appendStringInfoString(&buffer, + " on large objects"); + break; } if (objname) diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index 5d9fdfbd4c..ddadb3224a 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -20,6 +20,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -39,6 +40,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -55,11 +58,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -70,6 +80,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 84cef57a70..a77df8f028 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -756,7 +756,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8054,6 +8054,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17672,6 +17673,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18291,6 +18293,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 5649859aa1..0fad29cbaf 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -506,7 +506,8 @@ do { \ CONVERT_PRIV('s', "SET"); CONVERT_PRIV('A', "ALTER SYSTEM"); } - else if (strcmp(type, "LARGE OBJECT") == 0) + else if (strcmp(type, "LARGE OBJECT") == 0 || + strcmp(type, "LARGE OBJECTS") == 0) { CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 546e7e4ce1..97484a1b1b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -15009,6 +15009,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_LARGEOBJECT: + type = "LARGE OBJECTS"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 7c9a1f234c..ee1b96b0b9 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1184,7 +1184,9 @@ listDefaultACLs(const char *pattern) printfPQExpBuffer(&buf, "SELECT pg_catalog.pg_get_userbyid(d.defaclrole) AS \"%s\",\n" " n.nspname AS \"%s\",\n" - " CASE d.defaclobjtype WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" + " CASE d.defaclobjtype " + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s'" + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" " ", gettext_noop("Owner"), gettext_noop("Schema"), @@ -1198,6 +1200,8 @@ listDefaultACLs(const char *pattern) gettext_noop("type"), DEFACLOBJ_NAMESPACE, gettext_noop("schema"), + DEFACLOBJ_LARGEOBJECT, + gettext_noop("large object"), gettext_noop("Type")); printACLColumn(&buf, "d.defaclacl"); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index a7ccde6d7d..7979fd32c6 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -4032,7 +4032,7 @@ psql_completion(const char *text, int start, int end) * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "LARGE OBJECTS"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d272cdf08b..f9f002fa45 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f8659078ce..cbc7f2e870 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -307,6 +307,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index 1d903babd3..270d82112b 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2563,11 +2563,103 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +BEGIN; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'SELECT'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'UPDATE'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1008); + lo_create +----------- + 1008 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'SELECT'); -- yes + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user6', 1008, 'SELECT'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'UPDATE'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1009); + lo_create +----------- + 1009 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'SELECT'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'UPDATE'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_create(1010); + lo_create +----------- + 1010 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'SELECT'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'UPDATE'); -- false + has_largeobject_privilege +--------------------------- + f +(1 row) + +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2578,7 +2670,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index 3f54b0f8f0..5598410e44 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1535,11 +1535,47 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- + +BEGIN; + +SELECT lo_create(1007); +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'SELECT'); -- no +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'UPDATE'); -- no + +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO regress_priv_user2; + +SELECT lo_create(1008); +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'SELECT'); -- yes +SELECT has_largeobject_privilege('regress_priv_user6', 1008, 'SELECT'); -- no +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'UPDATE'); -- no + +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1009); +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'SELECT'); -- true +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'UPDATE'); -- true + +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_create(1010); +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'SELECT'); -- true +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'UPDATE'); -- false + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.34.1 --Multipart=_Fri__13_Sep_2024_16_18_01_+0900_KlUGdC57lGXrO9ov-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 15 ++- src/backend/catalog/aclchk.c | 21 ++++ src/backend/catalog/pg_largeobject.c | 18 ++- src/backend/parser/gram.y | 5 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 104 +++++++++++++++++- src/test/regress/sql/privileges.sql | 47 ++++++++ 9 files changed, 208 insertions(+), 7 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 2907079e2a..b8cc822aeb 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3330,7 +3330,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 1de4c5c1b4..3b358b7a88 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -50,6 +50,11 @@ GRANT { USAGE | CREATE | ALL [ PRIVILEGES ] } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -81,6 +86,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -159,7 +171,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 7abf3c2a74..41baf81a1d 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1077,6 +1077,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1268,6 +1272,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1499,6 +1513,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4324,6 +4341,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index e235f7c5e6..578589b457 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -22,6 +22,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -41,6 +42,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -57,11 +60,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -72,6 +82,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e8b619926e..35535ab390 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -760,7 +760,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8170,6 +8170,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17788,6 +17789,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18411,6 +18413,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d272cdf08b..f9f002fa45 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f9a4afd472..6299c29389 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -307,6 +307,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index eb4b762ea1..29f0775373 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2357,11 +2357,110 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +ERROR: permission denied for large object 1007 +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + lowrite +--------- + 4 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2372,7 +2471,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; @@ -2708,7 +2807,8 @@ SELECT lo_unlink(oid) FROM pg_largeobject_metadata WHERE oid >= 1000 AND oid < 3 1 1 1 -(5 rows) + 1 +(6 rows) DROP GROUP regress_priv_group1; DROP GROUP regress_priv_group2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index eeb4c00292..e8db20573c 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1487,11 +1487,58 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ROLLBACK; + +BEGIN; + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.25.1 --Multipart=_Wed__24_Apr_2024_11_52_42_+0900_jDJ5qZzcvgxN5www-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v5] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Previously, ALTER DEFAULT PRIVILEGE didn't support large objects, so if we want to allow users other than the owner to use the large object, we need to grant a privilege on it every time a large object is created. Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 19 +++- src/backend/catalog/aclchk.c | 21 +++++ src/backend/catalog/objectaddress.c | 18 +++- src/backend/catalog/pg_largeobject.c | 18 +++- src/backend/parser/gram.y | 5 +- src/bin/pg_dump/dumputils.c | 3 +- src/bin/pg_dump/pg_dump.c | 3 + src/bin/psql/describe.c | 6 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 94 ++++++++++++++++++- src/test/regress/sql/privileges.sql | 36 +++++++ 14 files changed, 217 insertions(+), 13 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 4558f940aaf..45ba9c5118f 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3360,7 +3360,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 89aacec4fab..6acd0f1df91 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -51,6 +51,11 @@ GRANT { { USAGE | CREATE } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -83,6 +88,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -117,8 +129,8 @@ REVOKE [ GRANT OPTION FOR ] <para> Currently, only the privileges for schemas, tables (including views and foreign - tables), sequences, functions, and types (including domains) can be - altered. For this command, functions include aggregates and procedures. + tables), sequences, functions, types (including domains), and large objects + can be altered. For this command, functions include aggregates and procedures. The words <literal>FUNCTIONS</literal> and <literal>ROUTINES</literal> are equivalent in this command. (<literal>ROUTINES</literal> is preferred going forward as the standard term for functions and procedures taken @@ -161,7 +173,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 02a754cc30a..9ca8a88dc91 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1005,6 +1005,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1196,6 +1200,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1439,6 +1453,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4250,6 +4267,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index d8eb8d3deaa..b63fd57dc04 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2005,16 +2005,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_LARGEOBJECT: + objtype_str = "large objects"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_LARGEOBJECT))); } /* @@ -3844,6 +3848,12 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_LARGEOBJECT: + Assert(!nspname); + appendStringInfo(&buffer, + _("default privileges on new large objects belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -5766,6 +5776,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_LARGEOBJECT: + appendStringInfoString(&buffer, + " on large objects"); + break; } if (objname) diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index 0a477a8e8a9..71a9cc134e1 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -20,6 +20,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -39,6 +40,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -55,11 +58,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -70,6 +80,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 6a094ecc54f..f1156e2fca3 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -752,7 +752,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8177,6 +8177,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17882,6 +17883,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18504,6 +18506,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 5ae77f76367..ab0e9e6da3c 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -506,7 +506,8 @@ do { \ CONVERT_PRIV('s', "SET"); CONVERT_PRIV('A', "ALTER SYSTEM"); } - else if (strcmp(type, "LARGE OBJECT") == 0) + else if (strcmp(type, "LARGE OBJECT") == 0 || + strcmp(type, "LARGE OBJECTS") == 0) { CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 04c87ba8854..817cedef32c 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -15679,6 +15679,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_LARGEOBJECT: + type = "LARGE OBJECTS"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index e038e9dc9e2..8970677ac64 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1222,7 +1222,9 @@ listDefaultACLs(const char *pattern) printfPQExpBuffer(&buf, "SELECT pg_catalog.pg_get_userbyid(d.defaclrole) AS \"%s\",\n" " n.nspname AS \"%s\",\n" - " CASE d.defaclobjtype WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" + " CASE d.defaclobjtype " + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s'" + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" " ", gettext_noop("Owner"), gettext_noop("Schema"), @@ -1236,6 +1238,8 @@ listDefaultACLs(const char *pattern) gettext_noop("type"), DEFACLOBJ_NAMESPACE, gettext_noop("schema"), + DEFACLOBJ_LARGEOBJECT, + gettext_noop("large object"), gettext_noop("Type")); printACLColumn(&buf, "d.defaclacl"); diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 98951aef82c..c916b9299a8 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -4457,7 +4457,7 @@ match_previous_words(int pattern_id, * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "LARGE OBJECTS"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index 728024b1fa7..ce6e5098eaf 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 40cf090ce61..a4af3f717a1 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -308,6 +308,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index 5588d83e1bf..1fddb13b6ae 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2667,11 +2667,103 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +BEGIN; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'SELECT'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'UPDATE'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1008); + lo_create +----------- + 1008 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'SELECT'); -- yes + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user6', 1008, 'SELECT'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'UPDATE'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1009); + lo_create +----------- + 1009 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'SELECT'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'UPDATE'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_create(1010); + lo_create +----------- + 1010 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'SELECT'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'UPDATE'); -- false + has_largeobject_privilege +--------------------------- + f +(1 row) + +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2682,7 +2774,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index 286b1d03756..85d7280f35f 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1586,11 +1586,47 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- + +BEGIN; + +SELECT lo_create(1007); +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'SELECT'); -- no +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'UPDATE'); -- no + +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO regress_priv_user2; + +SELECT lo_create(1008); +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'SELECT'); -- yes +SELECT has_largeobject_privilege('regress_priv_user6', 1008, 'SELECT'); -- no +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'UPDATE'); -- no + +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1009); +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'SELECT'); -- true +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'UPDATE'); -- true + +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_create(1010); +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'SELECT'); -- true +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'UPDATE'); -- false + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.34.1 --Multipart=_Thu__3_Apr_2025_23_04_18_+0900_OP_tfBIdDuoj6p3Q-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v2] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 15 ++- src/backend/catalog/aclchk.c | 21 ++++ src/backend/catalog/objectaddress.c | 18 ++- src/backend/catalog/pg_largeobject.c | 18 ++- src/backend/parser/gram.y | 5 +- src/bin/pg_dump/dumputils.c | 3 +- src/bin/pg_dump/pg_dump.c | 3 + src/bin/psql/describe.c | 6 +- src/bin/psql/tab-complete.c | 2 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 104 +++++++++++++++++- src/test/regress/sql/privileges.sql | 47 ++++++++ 14 files changed, 235 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 2907079e2a..b8cc822aeb 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3330,7 +3330,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 1de4c5c1b4..3b358b7a88 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -50,6 +50,11 @@ GRANT { USAGE | CREATE | ALL [ PRIVILEGES ] } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -81,6 +86,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -159,7 +171,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 7abf3c2a74..41baf81a1d 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1077,6 +1077,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1268,6 +1272,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1499,6 +1513,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4324,6 +4341,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 7b536ac6fd..5b330967a6 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2000,16 +2000,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_LARGEOBJECT: + objtype_str = "large objects"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_LARGEOBJECT))); } /* @@ -3798,6 +3802,12 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_LARGEOBJECT: + Assert(!nspname); + appendStringInfo(&buffer, + _("default privileges on new large objects belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -5651,6 +5661,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_LARGEOBJECT: + appendStringInfoString(&buffer, + " on large objects"); + break; } if (objname) diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index e235f7c5e6..578589b457 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -22,6 +22,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -41,6 +42,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -57,11 +60,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -72,6 +82,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e8b619926e..35535ab390 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -760,7 +760,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8170,6 +8170,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17788,6 +17789,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18411,6 +18413,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 5649859aa1..0fad29cbaf 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -506,7 +506,8 @@ do { \ CONVERT_PRIV('s', "SET"); CONVERT_PRIV('A', "ALTER SYSTEM"); } - else if (strcmp(type, "LARGE OBJECT") == 0) + else if (strcmp(type, "LARGE OBJECT") == 0 || + strcmp(type, "LARGE OBJECTS") == 0) { CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b8acdd7355..298cbb3d56 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -15224,6 +15224,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_LARGEOBJECT: + type = "LARGE OBJECTS"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 4a9ee4a54d..12d002b709 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1184,7 +1184,9 @@ listDefaultACLs(const char *pattern) printfPQExpBuffer(&buf, "SELECT pg_catalog.pg_get_userbyid(d.defaclrole) AS \"%s\",\n" " n.nspname AS \"%s\",\n" - " CASE d.defaclobjtype WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" + " CASE d.defaclobjtype " + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s'" + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" " ", gettext_noop("Owner"), gettext_noop("Schema"), @@ -1198,6 +1200,8 @@ listDefaultACLs(const char *pattern) gettext_noop("type"), DEFACLOBJ_NAMESPACE, gettext_noop("schema"), + DEFACLOBJ_LARGEOBJECT, + gettext_noop("large object"), gettext_noop("Type")); printACLColumn(&buf, "d.defaclacl"); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 6fee3160f0..03889fc52f 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -4035,7 +4035,7 @@ psql_completion(const char *text, int start, int end) * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "LARGE OBJECTS"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d272cdf08b..f9f002fa45 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f9a4afd472..6299c29389 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -307,6 +307,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index eb4b762ea1..29f0775373 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2357,11 +2357,110 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +ERROR: permission denied for large object 1007 +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + lowrite +--------- + 4 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2372,7 +2471,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; @@ -2708,7 +2807,8 @@ SELECT lo_unlink(oid) FROM pg_largeobject_metadata WHERE oid >= 1000 AND oid < 3 1 1 1 -(5 rows) + 1 +(6 rows) DROP GROUP regress_priv_group1; DROP GROUP regress_priv_group2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index eeb4c00292..e8db20573c 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1487,11 +1487,58 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ROLLBACK; + +BEGIN; + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.25.1 --Multipart=_Wed__24_Apr_2024_15_32_33_+0900_oAlXWx4kASmsfCdm-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 15 ++- src/backend/catalog/aclchk.c | 21 ++++ src/backend/catalog/pg_largeobject.c | 18 ++- src/backend/parser/gram.y | 5 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 104 +++++++++++++++++- src/test/regress/sql/privileges.sql | 47 ++++++++ 9 files changed, 208 insertions(+), 7 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 2907079e2a..b8cc822aeb 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3330,7 +3330,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 1de4c5c1b4..3b358b7a88 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -50,6 +50,11 @@ GRANT { USAGE | CREATE | ALL [ PRIVILEGES ] } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -81,6 +86,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -159,7 +171,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 7abf3c2a74..41baf81a1d 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1077,6 +1077,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1268,6 +1272,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1499,6 +1513,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4324,6 +4341,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index e235f7c5e6..578589b457 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -22,6 +22,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -41,6 +42,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -57,11 +60,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -72,6 +82,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e8b619926e..35535ab390 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -760,7 +760,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8170,6 +8170,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17788,6 +17789,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18411,6 +18413,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d272cdf08b..f9f002fa45 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f9a4afd472..6299c29389 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -307,6 +307,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index eb4b762ea1..29f0775373 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2357,11 +2357,110 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +ERROR: permission denied for large object 1007 +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + lowrite +--------- + 4 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2372,7 +2471,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; @@ -2708,7 +2807,8 @@ SELECT lo_unlink(oid) FROM pg_largeobject_metadata WHERE oid >= 1000 AND oid < 3 1 1 1 -(5 rows) + 1 +(6 rows) DROP GROUP regress_priv_group1; DROP GROUP regress_priv_group2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index eeb4c00292..e8db20573c 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1487,11 +1487,58 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ROLLBACK; + +BEGIN; + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.25.1 --Multipart=_Wed__24_Apr_2024_11_52_42_+0900_jDJ5qZzcvgxN5www-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v3] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Previously, ALTER DEFAULT PRIVILEGE didn't support large objects, so if we want to allow users other than the owner to use the large object, we need to grant a privilege on it every time a large object is created. Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 15 ++- src/backend/catalog/aclchk.c | 21 +++++ src/backend/catalog/objectaddress.c | 18 +++- src/backend/catalog/pg_largeobject.c | 18 +++- src/backend/parser/gram.y | 5 +- src/bin/pg_dump/dumputils.c | 3 +- src/bin/pg_dump/pg_dump.c | 3 + src/bin/psql/describe.c | 6 +- src/bin/psql/tab-complete.c | 2 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 94 ++++++++++++++++++- src/test/regress/sql/privileges.sql | 36 +++++++ 14 files changed, 215 insertions(+), 11 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b654fae1b2..c8cf56c666 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3322,7 +3322,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 89aacec4fa..3ab695892d 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -51,6 +51,11 @@ GRANT { { USAGE | CREATE } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -83,6 +88,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -161,7 +173,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index d2abc48fd8..6463ae921a 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1077,6 +1077,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1268,6 +1272,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1511,6 +1525,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4334,6 +4351,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 85a7b7e641..94f38c2333 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2000,16 +2000,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_LARGEOBJECT: + objtype_str = "large objects"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_LARGEOBJECT))); } /* @@ -3798,6 +3802,12 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_LARGEOBJECT: + Assert(!nspname); + appendStringInfo(&buffer, + _("default privileges on new large objects belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -5720,6 +5730,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_LARGEOBJECT: + appendStringInfoString(&buffer, + " on large objects"); + break; } if (objname) diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index 5d9fdfbd4c..ddadb3224a 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -20,6 +20,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -39,6 +40,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -55,11 +58,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -70,6 +80,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 84cef57a70..a77df8f028 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -756,7 +756,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8054,6 +8054,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17672,6 +17673,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18291,6 +18293,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 5649859aa1..0fad29cbaf 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -506,7 +506,8 @@ do { \ CONVERT_PRIV('s', "SET"); CONVERT_PRIV('A', "ALTER SYSTEM"); } - else if (strcmp(type, "LARGE OBJECT") == 0) + else if (strcmp(type, "LARGE OBJECT") == 0 || + strcmp(type, "LARGE OBJECTS") == 0) { CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 546e7e4ce1..97484a1b1b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -15009,6 +15009,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_LARGEOBJECT: + type = "LARGE OBJECTS"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 7c9a1f234c..ee1b96b0b9 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1184,7 +1184,9 @@ listDefaultACLs(const char *pattern) printfPQExpBuffer(&buf, "SELECT pg_catalog.pg_get_userbyid(d.defaclrole) AS \"%s\",\n" " n.nspname AS \"%s\",\n" - " CASE d.defaclobjtype WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" + " CASE d.defaclobjtype " + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s'" + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" " ", gettext_noop("Owner"), gettext_noop("Schema"), @@ -1198,6 +1200,8 @@ listDefaultACLs(const char *pattern) gettext_noop("type"), DEFACLOBJ_NAMESPACE, gettext_noop("schema"), + DEFACLOBJ_LARGEOBJECT, + gettext_noop("large object"), gettext_noop("Type")); printACLColumn(&buf, "d.defaclacl"); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index a7ccde6d7d..7979fd32c6 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -4032,7 +4032,7 @@ psql_completion(const char *text, int start, int end) * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "LARGE OBJECTS"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d272cdf08b..f9f002fa45 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f8659078ce..cbc7f2e870 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -307,6 +307,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index 1d903babd3..270d82112b 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2563,11 +2563,103 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +BEGIN; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'SELECT'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'UPDATE'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1008); + lo_create +----------- + 1008 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'SELECT'); -- yes + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user6', 1008, 'SELECT'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'UPDATE'); -- no + has_largeobject_privilege +--------------------------- + f +(1 row) + +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1009); + lo_create +----------- + 1009 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'SELECT'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'UPDATE'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_create(1010); + lo_create +----------- + 1010 +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'SELECT'); -- true + has_largeobject_privilege +--------------------------- + t +(1 row) + +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'UPDATE'); -- false + has_largeobject_privilege +--------------------------- + f +(1 row) + +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2578,7 +2670,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index 3f54b0f8f0..5598410e44 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1535,11 +1535,47 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- + +BEGIN; + +SELECT lo_create(1007); +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'SELECT'); -- no +SELECT has_largeobject_privilege('regress_priv_user2', 1007, 'UPDATE'); -- no + +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO regress_priv_user2; + +SELECT lo_create(1008); +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'SELECT'); -- yes +SELECT has_largeobject_privilege('regress_priv_user6', 1008, 'SELECT'); -- no +SELECT has_largeobject_privilege('regress_priv_user2', 1008, 'UPDATE'); -- no + +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_create(1009); +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'SELECT'); -- true +SELECT has_largeobject_privilege('regress_priv_user2', 1009, 'UPDATE'); -- true + +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_create(1010); +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'SELECT'); -- true +SELECT has_largeobject_privilege('regress_priv_user2', 1010, 'UPDATE'); -- false + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.34.1 --Multipart=_Fri__13_Sep_2024_16_18_01_+0900_KlUGdC57lGXrO9ov-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v2] Extend ALTER DEFAULT PRIVILEGES for large objects @ 2024-03-08 08:43 Yugo Nagata <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2024-03-08 08:43 UTC (permalink / raw) Original patch by Haruka Takatsuka, some fixes and tests by Yugo Nagata. --- doc/src/sgml/catalogs.sgml | 3 +- .../sgml/ref/alter_default_privileges.sgml | 15 ++- src/backend/catalog/aclchk.c | 21 ++++ src/backend/catalog/objectaddress.c | 18 ++- src/backend/catalog/pg_largeobject.c | 18 ++- src/backend/parser/gram.y | 5 +- src/bin/pg_dump/dumputils.c | 3 +- src/bin/pg_dump/pg_dump.c | 3 + src/bin/psql/describe.c | 6 +- src/bin/psql/tab-complete.c | 2 +- src/include/catalog/pg_default_acl.h | 1 + src/include/parser/kwlist.h | 1 + src/test/regress/expected/privileges.out | 104 +++++++++++++++++- src/test/regress/sql/privileges.sql | 47 ++++++++ 14 files changed, 235 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 2907079e2a..b8cc822aeb 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -3330,7 +3330,8 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <literal>S</literal> = sequence, <literal>f</literal> = function, <literal>T</literal> = type, - <literal>n</literal> = schema + <literal>n</literal> = schema, + <literal>L</literal> = large object </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index 1de4c5c1b4..3b358b7a88 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -50,6 +50,11 @@ GRANT { USAGE | CREATE | ALL [ PRIVILEGES ] } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -81,6 +86,13 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } + [, ...] | ALL [ PRIVILEGES ] } + ON LARGE OBJECTS + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -159,7 +171,8 @@ REVOKE [ GRANT OPTION FOR ] If <literal>IN SCHEMA</literal> is omitted, the global default privileges are altered. <literal>IN SCHEMA</literal> is not allowed when setting privileges - for schemas, since schemas can't be nested. + for schemas and large objects, since schemas can't be nested and + large objects don't belong to a schema. </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 7abf3c2a74..41baf81a1d 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -1077,6 +1077,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_LARGEOBJECT: + all_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + errormsg = gettext_noop("invalid privilege type %s for large object"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1268,6 +1272,16 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_LARGEOBJECT: + if (OidIsValid(iacls->nspid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_GRANT_OPERATION), + errmsg("cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS"))); + objtype = DEFACLOBJ_LARGEOBJECT; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_LARGEOBJECT; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1499,6 +1513,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_LARGEOBJECT: + iacls.objtype = OBJECT_LARGEOBJECT; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -4324,6 +4341,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_LARGEOBJECT: + defaclobjtype = DEFACLOBJ_LARGEOBJECT; + break; + default: return NULL; } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 7b536ac6fd..5b330967a6 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2000,16 +2000,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_LARGEOBJECT: + objtype_str = "large objects"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_LARGEOBJECT))); } /* @@ -3798,6 +3802,12 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_LARGEOBJECT: + Assert(!nspname); + appendStringInfo(&buffer, + _("default privileges on new large objects belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -5651,6 +5661,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_LARGEOBJECT: + appendStringInfoString(&buffer, + " on large objects"); + break; } if (objname) diff --git a/src/backend/catalog/pg_largeobject.c b/src/backend/catalog/pg_largeobject.c index e235f7c5e6..578589b457 100644 --- a/src/backend/catalog/pg_largeobject.c +++ b/src/backend/catalog/pg_largeobject.c @@ -22,6 +22,7 @@ #include "catalog/pg_largeobject.h" #include "catalog/pg_largeobject_metadata.h" #include "miscadmin.h" +#include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/rel.h" @@ -41,6 +42,8 @@ LargeObjectCreate(Oid loid) Oid loid_new; Datum values[Natts_pg_largeobject_metadata]; bool nulls[Natts_pg_largeobject_metadata]; + Oid ownerId; + Acl *lomacl; pg_lo_meta = table_open(LargeObjectMetadataRelationId, RowExclusiveLock); @@ -57,11 +60,18 @@ LargeObjectCreate(Oid loid) loid_new = GetNewOidWithIndex(pg_lo_meta, LargeObjectMetadataOidIndexId, Anum_pg_largeobject_metadata_oid); + ownerId = GetUserId(); + lomacl = get_user_default_acl(OBJECT_LARGEOBJECT, ownerId, InvalidOid); values[Anum_pg_largeobject_metadata_oid - 1] = ObjectIdGetDatum(loid_new); values[Anum_pg_largeobject_metadata_lomowner - 1] - = ObjectIdGetDatum(GetUserId()); - nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; + = ObjectIdGetDatum(ownerId); + + if (lomacl != NULL) + values[Anum_pg_largeobject_metadata_lomacl - 1] + = PointerGetDatum(lomacl); + else + nulls[Anum_pg_largeobject_metadata_lomacl - 1] = true; ntup = heap_form_tuple(RelationGetDescr(pg_lo_meta), values, nulls); @@ -72,6 +82,10 @@ LargeObjectCreate(Oid loid) table_close(pg_lo_meta, RowExclusiveLock); + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(LargeObjectRelationId, loid_new, 0, + ownerId, lomacl); + return loid_new; } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index e8b619926e..35535ab390 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -760,7 +760,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF NULLS_P NUMERIC - OBJECT_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR + OBJECT_P OBJECTS_P OF OFF OFFSET OIDS OLD OMIT ON ONLY OPERATOR OPTION OPTIONS OR ORDER ORDINALITY OTHERS OUT_P OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER @@ -8170,6 +8170,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | LARGE_P OBJECTS_P { $$ = OBJECT_LARGEOBJECT; } ; @@ -17788,6 +17789,7 @@ unreserved_keyword: | NOWAIT | NULLS_P | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS @@ -18411,6 +18413,7 @@ bare_label_keyword: | NULLS_P | NUMERIC | OBJECT_P + | OBJECTS_P | OF | OFF | OIDS diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 5649859aa1..0fad29cbaf 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -506,7 +506,8 @@ do { \ CONVERT_PRIV('s', "SET"); CONVERT_PRIV('A', "ALTER SYSTEM"); } - else if (strcmp(type, "LARGE OBJECT") == 0) + else if (strcmp(type, "LARGE OBJECT") == 0 || + strcmp(type, "LARGE OBJECTS") == 0) { CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b8acdd7355..298cbb3d56 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -15224,6 +15224,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_LARGEOBJECT: + type = "LARGE OBJECTS"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 4a9ee4a54d..12d002b709 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1184,7 +1184,9 @@ listDefaultACLs(const char *pattern) printfPQExpBuffer(&buf, "SELECT pg_catalog.pg_get_userbyid(d.defaclrole) AS \"%s\",\n" " n.nspname AS \"%s\",\n" - " CASE d.defaclobjtype WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" + " CASE d.defaclobjtype " + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s'" + " WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' WHEN '%c' THEN '%s' END AS \"%s\",\n" " ", gettext_noop("Owner"), gettext_noop("Schema"), @@ -1198,6 +1200,8 @@ listDefaultACLs(const char *pattern) gettext_noop("type"), DEFACLOBJ_NAMESPACE, gettext_noop("schema"), + DEFACLOBJ_LARGEOBJECT, + gettext_noop("large object"), gettext_noop("Type")); printACLColumn(&buf, "d.defaclacl"); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 6fee3160f0..03889fc52f 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -4035,7 +4035,7 @@ psql_completion(const char *text, int start, int end) * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "LARGE OBJECTS"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d272cdf08b..f9f002fa45 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -68,6 +68,7 @@ MAKE_SYSCACHE(DEFACLROLENSPOBJ, pg_default_acl_role_nsp_obj_index, 8); #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_LARGEOBJECT 'L' /* large object */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f9a4afd472..6299c29389 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -307,6 +307,7 @@ PG_KEYWORD("nullif", NULLIF, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("nulls", NULLS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("numeric", NUMERIC, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("object", OBJECT_P, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("objects", OBJECTS_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("of", OF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("off", OFF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("offset", OFFSET, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index eb4b762ea1..29f0775373 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -2357,11 +2357,110 @@ SELECT has_schema_privilege('regress_priv_user2', 'testns4', 'CREATE'); -- yes ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +ERROR: permission denied for large object 1007 +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + lowrite +--------- + 4 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); + lo_unlink +----------- + 1 +(1 row) + +SELECT lo_create(1007); + lo_create +----------- + 1007 +(1 row) + +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok + loread +-------- + \x +(1 row) + +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ERROR: permission denied for large object 1007 +ROLLBACK; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error +ERROR: cannot use IN SCHEMA clause when using GRANT/REVOKE ON LARGE OBJECTS +\c - -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; @@ -2372,7 +2471,7 @@ SELECT count(*) FROM pg_shdepend classid = 'pg_default_acl'::regclass; count ------- - 5 + 6 (1 row) DROP OWNED BY regress_priv_user2, regress_priv_user2; @@ -2708,7 +2807,8 @@ SELECT lo_unlink(oid) FROM pg_largeobject_metadata WHERE oid >= 1000 AND oid < 3 1 1 1 -(5 rows) + 1 +(6 rows) DROP GROUP regress_priv_group1; DROP GROUP regress_priv_group2; diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql index eeb4c00292..e8db20573c 100644 --- a/src/test/regress/sql/privileges.sql +++ b/src/test/regress/sql/privileges.sql @@ -1487,11 +1487,58 @@ ALTER DEFAULT PRIVILEGES REVOKE ALL ON SCHEMAS FROM regress_priv_user2; COMMIT; +-- +-- Test for default privileges on large objects. This is done in a +-- separate, rollbacked, transaction to avoid any trouble with other +-- regression sessions. +-- +\c - +SET SESSION AUTHORIZATION regress_priv_user1; +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- to be denied +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +BEGIN; +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT ON LARGE OBJECTS TO public; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied +ROLLBACK; + +BEGIN; + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES GRANT SELECT, UPDATE ON LARGE OBJECTS TO regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- ok + +SET SESSION AUTHORIZATION regress_priv_user1; +ALTER DEFAULT PRIVILEGES REVOKE UPDATE ON LARGE OBJECTS FROM regress_priv_user2; +SELECT lo_unlink(1007); +SELECT lo_create(1007); +SET SESSION AUTHORIZATION regress_priv_user2; +SELECT loread(lo_open(1007, x'40000'::int), 32); -- ok +SELECT lowrite(lo_open(1007, x'20000'::int), 'abcd'); -- to be denied + +ROLLBACK; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON LARGE OBJECTS TO public; -- error + +\c - + -- Test for DROP OWNED BY with shared dependencies. This is done in a -- separate, rollbacked, transaction to avoid any trouble with other -- regression sessions. BEGIN; ALTER DEFAULT PRIVILEGES GRANT ALL ON FUNCTIONS TO regress_priv_user2; +ALTER DEFAULT PRIVILEGES GRANT ALL ON LARGE OBJECTS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SCHEMAS TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON SEQUENCES TO regress_priv_user2; ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO regress_priv_user2; -- 2.25.1 --Multipart=_Wed__24_Apr_2024_15_32_33_+0900_oAlXWx4kASmsfCdm-- ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Proposal for Updating CRC32C with AVX-512 Algorithm. @ 2024-12-19 00:00 Sterrett, Matthew <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Sterrett, Matthew @ 2024-12-19 00:00 UTC (permalink / raw) To: Devulapalli, Raghuveer <[email protected]>; Nathan Bossart <[email protected]>; +Cc: Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; Shankaran, Akash <[email protected]> On 12/7/2024 12:42 AM, Devulapalli, Raghuveer wrote: >> [0] https://cirrus-ci.com/task/6023394207989760 >> [1] https://cirrus-ci.com/task/5460444254568448 >> [2] https://cirrus-ci.com/task/6586344161411072 > > I was able to fix [0] and [1], but I can't think of why [2] fails. When I tried to reproduce this locally, I get a different unrelated error. Any idea why I am seeing this? > > LINK : fatal error LNK1181: cannot open input file 'C:\Program Files\Git\nologo' > > Commands: meson setup build && cd build && meson compile Hello! I'm Matthew Sterrett and I'm a coworker of Raghuveer; he asked me to look into the Windows build failures related to pg_comp_crc32c. It seems that the only thing that was required to fix that is to mark pg_comp_crc32c as PGDLLIMPORT, so I added a patch that does just that. I'm new to working with mailing lists, so please tell me if I messed anything up! Matthew Sterrett From 74d085d44d41af8ffb01f7bf2377ac487c7d4cc1 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Mon, 6 May 2024 08:34:17 -0700 Subject: [PATCH v10 1/4] Add a Postgres SQL function for crc32c benchmarking. Add a drive_crc32c() function to use for benchmarking crc32c computation. The function takes 2 arguments: (1) count: num of times CRC32C is computed in a loop. (2) num: #bytes in the buffer to calculate crc over. Signed-off-by: Paul Amonson <[email protected]> Signed-off-by: Raghuveer Devulapalli <[email protected]> --- src/test/modules/meson.build | 1 + src/test/modules/test_crc32c/Makefile | 20 ++++++++ src/test/modules/test_crc32c/meson.build | 22 +++++++++ .../modules/test_crc32c/test_crc32c--1.0.sql | 1 + src/test/modules/test_crc32c/test_crc32c.c | 47 +++++++++++++++++++ .../modules/test_crc32c/test_crc32c.control | 4 ++ 6 files changed, 95 insertions(+) create mode 100644 src/test/modules/test_crc32c/Makefile create mode 100644 src/test/modules/test_crc32c/meson.build create mode 100644 src/test/modules/test_crc32c/test_crc32c--1.0.sql create mode 100644 src/test/modules/test_crc32c/test_crc32c.c create mode 100644 src/test/modules/test_crc32c/test_crc32c.control diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index c829b61953..68d8904dd0 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -15,6 +15,7 @@ subdir('ssl_passphrase_callback') subdir('test_bloomfilter') subdir('test_copy_callbacks') subdir('test_custom_rmgrs') +subdir('test_crc32c') subdir('test_ddl_deparse') subdir('test_dsa') subdir('test_dsm_registry') diff --git a/src/test/modules/test_crc32c/Makefile b/src/test/modules/test_crc32c/Makefile new file mode 100644 index 0000000000..5b747c6184 --- /dev/null +++ b/src/test/modules/test_crc32c/Makefile @@ -0,0 +1,20 @@ +MODULE_big = test_crc32c +OBJS = test_crc32c.o +PGFILEDESC = "test" +EXTENSION = test_crc32c +DATA = test_crc32c--1.0.sql + +first: all + +# test_crc32c.o: CFLAGS+=-g + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_crc32c +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_crc32c/meson.build b/src/test/modules/test_crc32c/meson.build new file mode 100644 index 0000000000..7021a6d6cf --- /dev/null +++ b/src/test/modules/test_crc32c/meson.build @@ -0,0 +1,22 @@ +# Copyright (c) 2022-2024, PostgreSQL Global Development Group + +test_crc32c_sources = files( + 'test_crc32c.c', +) + +if host_system == 'windows' + test_crc32c_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_crc32c', + '--FILEDESC', 'test_crc32c - test code for crc32c library',]) +endif + +test_crc32c = shared_module('test_crc32c', + test_crc32c_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_crc32c + +test_install_data += files( + 'test_crc32c.control', + 'test_crc32c--1.0.sql', +) diff --git a/src/test/modules/test_crc32c/test_crc32c--1.0.sql b/src/test/modules/test_crc32c/test_crc32c--1.0.sql new file mode 100644 index 0000000000..32f8f0fb2e --- /dev/null +++ b/src/test/modules/test_crc32c/test_crc32c--1.0.sql @@ -0,0 +1 @@ +CREATE FUNCTION drive_crc32c (count int, num int) RETURNS bigint AS 'test_crc32c.so' LANGUAGE C; diff --git a/src/test/modules/test_crc32c/test_crc32c.c b/src/test/modules/test_crc32c/test_crc32c.c new file mode 100644 index 0000000000..b350caf5ce --- /dev/null +++ b/src/test/modules/test_crc32c/test_crc32c.c @@ -0,0 +1,47 @@ +/* select drive_crc32c(1000000, 1024); */ + +#include "postgres.h" +#include "fmgr.h" +#include "port/pg_crc32c.h" +#include "common/pg_prng.h" + +PG_MODULE_MAGIC; + +/* + * drive_crc32c(count: int, num: int) returns bigint + * + * count is the nuimber of loops to perform + * + * num is the number byte in the buffer to calculate + * crc32c over. + */ +PG_FUNCTION_INFO_V1(drive_crc32c); +Datum +drive_crc32c(PG_FUNCTION_ARGS) +{ + int64 count = PG_GETARG_INT64(0); + int64 num = PG_GETARG_INT64(1); + char* data = malloc((size_t)num); + pg_crc32c crc; + pg_prng_state state; + uint64 seed = 42; + pg_prng_seed(&state, seed); + /* set random data */ + for (uint64 i = 0; i < num; i++) + { + data[i] = pg_prng_uint32(&state) % 255; + } + + INIT_CRC32C(crc); + + while(count--) + { + INIT_CRC32C(crc); + COMP_CRC32C(crc, data, num); + FIN_CRC32C(crc); + } + + free((void *)data); + + PG_RETURN_INT64((int64_t)crc); +} diff --git a/src/test/modules/test_crc32c/test_crc32c.control b/src/test/modules/test_crc32c/test_crc32c.control new file mode 100644 index 0000000000..878a077ee1 --- /dev/null +++ b/src/test/modules/test_crc32c/test_crc32c.control @@ -0,0 +1,4 @@ +comment = 'test' +default_version = '1.0' +module_pathname = '$libdir/test_crc32c' +relocatable = true -- 2.34.1 From 2542c6830d98e146d79844fb84fe3fb1b2945c25 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 23 Jul 2024 11:23:23 -0700 Subject: [PATCH v10 2/4] Refactor: consolidate x86 ISA and OS runtime checks Move all x86 ISA and OS runtime checks into a single file for improved modularity and easier future maintenance. Signed-off-by: Paul Amonson <[email protected]> Signed-off-by: Raghuveer Devulapalli <[email protected]> --- src/include/port/pg_bitutils.h | 1 - src/include/port/pg_hw_feat_check.h | 33 ++++++ src/port/Makefile | 1 + src/port/meson.build | 3 + src/port/pg_bitutils.c | 22 +--- src/port/pg_crc32c_sse42_choose.c | 21 +--- src/port/pg_hw_feat_check.c | 163 ++++++++++++++++++++++++++++ src/port/pg_popcount_avx512.c | 78 ------------- 8 files changed, 205 insertions(+), 117 deletions(-) create mode 100644 src/include/port/pg_hw_feat_check.h create mode 100644 src/port/pg_hw_feat_check.c diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index a3cad46afe..461c7c13cf 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -312,7 +312,6 @@ extern PGDLLIMPORT uint64 (*pg_popcount_masked_optimized) (const char *buf, int * files. */ #ifdef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK -extern bool pg_popcount_avx512_available(void); extern uint64 pg_popcount_avx512(const char *buf, int bytes); extern uint64 pg_popcount_masked_avx512(const char *buf, int bytes, bits8 mask); #endif diff --git a/src/include/port/pg_hw_feat_check.h b/src/include/port/pg_hw_feat_check.h new file mode 100644 index 0000000000..58be900b54 --- /dev/null +++ b/src/include/port/pg_hw_feat_check.h @@ -0,0 +1,33 @@ +/*------------------------------------------------------------------------- + * + * pg_hw_feat_check.h + * Miscellaneous functions for cheing for hardware features at runtime. + * + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * src/include/port/pg_hw_feat_check.h + * + *------------------------------------------------------------------------- + */ +#ifndef PG_HW_FEAT_CHECK_H +#define PG_HW_FEAT_CHECK_H + +/* + * Test to see if all hardware features required by SSE 4.2 crc32c (64 bit) + * are available. + */ +extern PGDLLIMPORT bool pg_crc32c_sse42_available(void); + +/* + * Test to see if all hardware features required by SSE 4.1 POPCNT (64 bit) + * are available. + */ +extern PGDLLIMPORT bool pg_popcount_available(void); + +/* + * Test to see if all hardware features required by AVX-512 POPCNT are + * available. + */ +extern PGDLLIMPORT bool pg_popcount_avx512_available(void); +#endif /* PG_HW_FEAT_CHECK_H */ diff --git a/src/port/Makefile b/src/port/Makefile index 4c22431951..6088b56b71 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -45,6 +45,7 @@ OBJS = \ path.o \ pg_bitutils.o \ pg_popcount_avx512.o \ + pg_hw_feat_check.o \ pg_strong_random.o \ pgcheckdir.o \ pgmkdirp.o \ diff --git a/src/port/meson.build b/src/port/meson.build index c5bceed9cd..ec28590473 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,6 +8,9 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_avx512.c', + 'pg_crc32c_sse42_choose.c', + 'pg_crc32c_sse42.c', + 'pg_hw_feat_check.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index c8399981ee..c11b13dca2 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -20,7 +20,7 @@ #endif #include "port/pg_bitutils.h" - +#include "port/pg_hw_feat_check.h" /* * Array giving the position of the left-most set bit for each possible @@ -109,7 +109,6 @@ static uint64 pg_popcount_slow(const char *buf, int bytes); static uint64 pg_popcount_masked_slow(const char *buf, int bytes, bits8 mask); #ifdef TRY_POPCNT_FAST -static bool pg_popcount_available(void); static int pg_popcount32_choose(uint32 word); static int pg_popcount64_choose(uint64 word); static uint64 pg_popcount_choose(const char *buf, int bytes); @@ -127,25 +126,6 @@ uint64 (*pg_popcount_masked_optimized) (const char *buf, int bytes, bits8 mask) #ifdef TRY_POPCNT_FAST -/* - * Return true if CPUID indicates that the POPCNT instruction is available. - */ -static bool -pg_popcount_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 23)) != 0; /* POPCNT */ -} - /* * These functions get called on the first call to pg_popcount32 etc. * They detect whether we can use the asm implementations, and replace diff --git a/src/port/pg_crc32c_sse42_choose.c b/src/port/pg_crc32c_sse42_choose.c index 56d600f3a9..c659917af0 100644 --- a/src/port/pg_crc32c_sse42_choose.c +++ b/src/port/pg_crc32c_sse42_choose.c @@ -20,6 +20,7 @@ #include "c.h" +#if defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) #ifdef HAVE__GET_CPUID #include <cpuid.h> #endif @@ -29,22 +30,7 @@ #endif #include "port/pg_crc32c.h" - -static bool -pg_crc32c_sse42_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 20)) != 0; /* SSE 4.2 */ -} +#include "port/pg_hw_feat_check.h" /* * This gets called on the first call. It replaces the function pointer @@ -61,4 +47,5 @@ pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len) return pg_comp_crc32c(crc, data, len); } -pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; +pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; +#endif diff --git a/src/port/pg_hw_feat_check.c b/src/port/pg_hw_feat_check.c new file mode 100644 index 0000000000..260aa60502 --- /dev/null +++ b/src/port/pg_hw_feat_check.c @@ -0,0 +1,163 @@ +/*------------------------------------------------------------------------- + * + * pg_hw_feat_check.c + * Test for hardware features at runtime on x86_64 platforms. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_hw_feat_check.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" + +#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) +#include <cpuid.h> +#endif + +#include <immintrin.h> + +#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX) +#include <intrin.h> +#endif + +#include "port/pg_hw_feat_check.h" + +/* Define names for EXX registers to avoid hard to see bugs in code below. */ +typedef unsigned int exx_t; +typedef enum +{ + EAX = 0, + EBX = 1, + ECX = 2, + EDX = 3 +} reg_name; + +/* + * Helper function. + * Test for a bit being set in a exx_t register. + */ +inline static bool is_bit_set_in_exx(exx_t* regs, reg_name ex, int bit) +{ + return ((regs[ex] & (1 << bit)) != 0); +} + +/* + * x86_64 Platform CPUID check for Linux and Visual Studio platforms. + */ +inline static void +pg_getcpuid(unsigned int leaf, exx_t *exx) +{ +#if defined(HAVE__GET_CPUID) + __get_cpuid(leaf, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUID) + __cpuid(exx, 1); +#else +#error cpuid instruction not available +#endif +} + +/* + * x86_64 Platform CPUIDEX check for Linux and Visual Studio platforms. + */ +inline static void +pg_getcpuidex(unsigned int leaf, unsigned int subleaf, exx_t *exx) +{ +#if defined(HAVE__GET_CPUID_COUNT) + __get_cpuid_count(leaf, subleaf, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUIDEX) + __cpuidex(exx, 7, 0); +#else +#error cpuid instruction not available +#endif +} + +/* + * Check for CPU support for CPUID: osxsave + */ +inline static bool +osxsave_available(void) +{ +#if defined(HAVE_XSAVE_INTRINSICS) + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + + return is_bit_set_in_exx(exx, ECX, 27); /* osxsave */ +#else + return false; +#endif +} + +/* + * Does XGETBV say the ZMM registers are enabled? + * + * NB: Caller is responsible for verifying that osxsave_available() returns true + * before calling this. + */ +#ifdef HAVE_XSAVE_INTRINSICS +pg_attribute_target("xsave") +#endif +inline static bool +zmm_regs_available(void) +{ +#if defined(HAVE_XSAVE_INTRINSICS) + return (_xgetbv(0) & 0xe6) == 0xe6; +#else + return false; +#endif +} + +/* + * Does CPUID say there's support for AVX-512 popcount and byte-and-word + * instructions? + */ +inline static bool +avx512_popcnt_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + + return is_bit_set_in_exx(exx, ECX, 14) && is_bit_set_in_exx(exx, EBX, 30); +} + +/* + * Return true if CPUID indicates that the POPCNT instruction is available. + */ +bool PGDLLIMPORT pg_popcount_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + + return is_bit_set_in_exx(exx, ECX, 23); + } + + /* + * Returns true if the CPU supports the instructions required for the AVX-512 + * pg_popcount() implementation. + * + * PA: The call to 'osxsave_available' MUST preceed the call to + * 'zmm_regs_available' function per NB above. + */ +bool PGDLLIMPORT pg_popcount_avx512_available(void) +{ + return osxsave_available() && + zmm_regs_available() && + avx512_popcnt_available(); +} + +/* + * Does CPUID say there's support for SSE 4.2? + */ +bool PGDLLIMPORT pg_crc32c_sse42_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + + return is_bit_set_in_exx(exx, ECX, 20); +} + diff --git a/src/port/pg_popcount_avx512.c b/src/port/pg_popcount_avx512.c index c8a4f2b19f..1123a1a634 100644 --- a/src/port/pg_popcount_avx512.c +++ b/src/port/pg_popcount_avx512.c @@ -14,16 +14,7 @@ #ifdef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK -#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) -#include <cpuid.h> -#endif - #include <immintrin.h> - -#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX) -#include <intrin.h> -#endif - #include "port/pg_bitutils.h" /* @@ -33,75 +24,6 @@ */ #ifdef TRY_POPCNT_FAST -/* - * Does CPUID say there's support for XSAVE instructions? - */ -static inline bool -xsave_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - return (exx[2] & (1 << 27)) != 0; /* osxsave */ -} - -/* - * Does XGETBV say the ZMM registers are enabled? - * - * NB: Caller is responsible for verifying that xsave_available() returns true - * before calling this. - */ -#ifdef HAVE_XSAVE_INTRINSICS -pg_attribute_target("xsave") -#endif -static inline bool -zmm_regs_available(void) -{ -#ifdef HAVE_XSAVE_INTRINSICS - return (_xgetbv(0) & 0xe6) == 0xe6; -#else - return false; -#endif -} - -/* - * Does CPUID say there's support for AVX-512 popcount and byte-and-word - * instructions? - */ -static inline bool -avx512_popcnt_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID_COUNT) - __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUIDEX) - __cpuidex(exx, 7, 0); -#else -#error cpuid instruction not available -#endif - return (exx[2] & (1 << 14)) != 0 && /* avx512-vpopcntdq */ - (exx[1] & (1 << 30)) != 0; /* avx512-bw */ -} - -/* - * Returns true if the CPU supports the instructions required for the AVX-512 - * pg_popcount() implementation. - */ -bool -pg_popcount_avx512_available(void) -{ - return xsave_available() && - zmm_regs_available() && - avx512_popcnt_available(); -} - /* * pg_popcount_avx512 * Returns the number of 1-bits in buf -- 2.34.1 From f08e15c0834616c636d1cb949ed140926265847e Mon Sep 17 00:00:00 2001 From: Raghuveer Devulapalli <[email protected]> Date: Thu, 21 Nov 2024 12:42:09 -0800 Subject: [PATCH v10 3/4] Add AVX-512 CRC32C algorithm with a runtime check Adds pg_crc32c_avx512(): compute the crc32c of the buffer, where the buffer length must be at least 256, and a multiple of 64. Based on: "Fast CRC Computation for Generic Polynomials Using PCLMULQDQ Instruction" V. Gopal, E. Ozturk, et al., 2009" Benchmark numbers to compare against the SSE4.2 CRC32C algorithm was generated by using the drive_crc32c() function added in src/test/modules/test_crc32c/test_crc32c.c. +------------------+----------------+----------------+------------------+-------+------+ | Rate in bytes/us | SDP (SPR) | m6i | m7i | | | +------------------+----------------+----------------+------------------+ Multi-| | | higher is better | SSE42 | AVX512 | SSE42 | AVX512 | SSE42 | AVX512 | plier | % | +==================+=================+=======+========+========+========+=======+======+ | AVG Rate 64-8192 | 10,095 | 82,101 | 8,591 | 38,652 | 11,867 | 83,194 | 6.68 | 568% | +------------------+--------+--------+-------+--------+--------+--------+-------+------+ | AVG Rate 64-255 | 9,034 | 9,136 | 7,619 | 7,437 | 9,030 | 9,293 | 1.01 | 1% | +------------------+--------+--------+-------+--------+--------+--------+-------+------+ Co-authored-by: Paul Amonson <[email protected]> --- config/c-compiler.m4 | 32 +++++ configure | 154 ++++++++++++--------- configure.ac | 107 +++++++-------- meson.build | 23 ++++ src/include/pg_config.h.in | 3 + src/include/pg_cpu.h | 23 ++++ src/include/port/pg_crc32c.h | 55 +++----- src/include/port/pg_hw_feat_check.h | 6 + src/port/meson.build | 10 +- src/port/pg_crc32c_avx512.c | 203 ++++++++++++++++++++++++++++ src/port/pg_crc32c_sse42.c | 2 + src/port/pg_crc32c_sse42_choose.c | 51 ------- src/port/pg_crc32c_x86_choose.c | 57 ++++++++ src/port/pg_hw_feat_check.c | 75 +++++++++- 14 files changed, 578 insertions(+), 223 deletions(-) create mode 100644 src/include/pg_cpu.h create mode 100644 src/port/pg_crc32c_avx512.c delete mode 100644 src/port/pg_crc32c_sse42_choose.c create mode 100644 src/port/pg_crc32c_x86_choose.c diff --git a/config/c-compiler.m4 b/config/c-compiler.m4 index e112fd45d4..e08de01739 100644 --- a/config/c-compiler.m4 +++ b/config/c-compiler.m4 @@ -578,6 +578,38 @@ undefine([Ac_cachevar])dnl ])# PGAC_SSE42_CRC32_INTRINSICS +# PGAC_AVX512_CRC32_INTRINSICS +# --------------------------- +# Check if the compiler supports the x86 CRC instructions added in AVX-512, +# using intrinsics with function __attribute__((target("..."))): + +AC_DEFUN([PGAC_AVX512_CRC32_INTRINSICS], +[define([Ac_cachevar], [AS_TR_SH([pgac_cv_avx512_crc32_intrinsics])])dnl +AC_CACHE_CHECK([for _mm512_clmulepi64_epi128 with function attribute], [Ac_cachevar], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h> + #include <stdint.h> + #if defined(__has_attribute) && __has_attribute (target) + __attribute__((target("avx512vl,vpclmulqdq"))) + #endif + static int crc32_avx512_test(void) + { + __m512i x0 = _mm512_set1_epi32(0x1); + __m512i x1 = _mm512_set1_epi32(0x2); + __m512i x2 = _mm512_clmulepi64_epi128(x1, x0, 0x00); // vpclmulqdq + __m128i a1 = _mm_xor_epi64(_mm512_castsi512_si128(x1), _mm512_castsi512_si128(x0)); //avx512vl + int64_t val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); // 64-bit instruction + return (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); + }], + [return crc32_avx512_test();])], + [Ac_cachevar=yes], + [Ac_cachevar=no])]) +if test x"$Ac_cachevar" = x"yes"; then + pgac_avx512_crc32_intrinsics=yes +fi +undefine([Ac_cachevar])dnl +])# PGAC_AVX512_CRC32_INTRINSICS + + # PGAC_ARMV8_CRC32C_INTRINSICS # ---------------------------- # Check if the compiler supports the CRC32C instructions using the __crc32cb, diff --git a/configure b/configure index 518c33b73a..b03b928bfd 100755 --- a/configure +++ b/configure @@ -17159,7 +17159,7 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h fi fi -# Check for Intel SSE 4.2 intrinsics to do CRC calculations. +# Check for Intel SSE 4.2 and AVX-512 intrinsics to do CRC calculations. # { $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5 $as_echo_n "checking for _mm_crc32_u8 and _mm_crc32_u32... " >&6; } @@ -17203,6 +17203,52 @@ if test x"$pgac_cv_sse42_crc32_intrinsics" = x"yes"; then fi +# Check if the _mm512_clmulepi64_epi128 and _mm_xor_epi64 can be used with with +# the __attribute__((target("avx512vl,vpclmulqdq"))). +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_clmulepi64_epi128 with function attribute" >&5 +$as_echo_n "checking for _mm512_clmulepi64_epi128 with function attribute... " >&6; } +if ${pgac_cv_avx512_crc32_intrinsics+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> + #include <stdint.h> + #if defined(__has_attribute) && __has_attribute (target) + __attribute__((target("avx512vl,vpclmulqdq"))) + #endif + static int crc32_avx512_test(void) + { + __m512i x0 = _mm512_set1_epi32(0x1); + __m512i x1 = _mm512_set1_epi32(0x2); + __m512i x2 = _mm512_clmulepi64_epi128(x1, x0, 0x00); // vpclmulqdq + __m128i a1 = _mm_xor_epi64(_mm512_castsi512_si128(x1), _mm512_castsi512_si128(x0)); //avx512vl + int64_t val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); // 64-bit instruction + return (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); + } +int +main () +{ +return crc32_avx512_test(); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_crc32_intrinsics=yes +else + pgac_cv_avx512_crc32_intrinsics=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_crc32_intrinsics" >&5 +$as_echo "$pgac_cv_avx512_crc32_intrinsics" >&6; } +if test x"$pgac_cv_avx512_crc32_intrinsics" = x"yes"; then + pgac_avx512_crc32_intrinsics=yes +fi + + # Are we targeting a processor that supports SSE 4.2? gcc, clang and icc all # define __SSE4_2__ in that case. cat confdefs.h - <<_ACEOF >conftest.$ac_ext @@ -17404,9 +17450,8 @@ fi # If we are targeting a processor that has Intel SSE 4.2 instructions, we can # use the special CRC instructions for calculating CRC-32C. If we're not # targeting such a processor, but we can nevertheless produce code that uses -# the SSE intrinsics, compile both implementations and select which one to use -# at runtime, depending on whether SSE 4.2 is supported by the processor we're -# running on. +# the SSE/AVX-512 intrinsics compile both implementations and select which one +# to use at runtime, depending runtime cpuid information. # # Similarly, if we are targeting an ARM processor that has the CRC # instructions that are part of the ARMv8 CRC Extension, use them. And if @@ -17423,95 +17468,80 @@ fi # # If we are targeting a LoongArch processor, CRC instructions are # always available (at least on 64 bit), so no runtime check is needed. -if test x"$USE_SLICING_BY_8_CRC32C" = x"" && test x"$USE_SSE42_CRC32C" = x"" && test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_ARMV8_CRC32C" = x"" && test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_LOONGARCH_CRC32C" = x""; then - # Use Intel SSE 4.2 if available. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then - USE_SSE42_CRC32C=1 - else - # Intel SSE 4.2, with runtime check? The CPUID instruction is needed for - # the runtime check. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then - USE_SSE42_CRC32C_WITH_RUNTIME_CHECK=1 - else - # Use ARM CRC Extension if available. - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then - USE_ARMV8_CRC32C=1 - else - # ARM CRC Extension, with runtime check? - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then - USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK=1 - else - # LoongArch CRCC instructions. - if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then - USE_LOONGARCH_CRC32C=1 - else - # fall back to slicing-by-8 algorithm, which doesn't require any - # special CPU support. - USE_SLICING_BY_8_CRC32C=1 - fi - fi - fi - fi - fi -fi -# Set PG_CRC32C_OBJS appropriately depending on the selected implementation. { $as_echo "$as_me:${as_lineno-$LINENO}: checking which CRC-32C implementation to use" >&5 $as_echo_n "checking which CRC-32C implementation to use... " >&6; } -if test x"$USE_SSE42_CRC32C" = x"1"; then +if test x"$host_cpu" = x"x86_64"; then + #x86 only: + PG_CRC32C_OBJS="pg_crc32c_sb8.o pg_crc32c_x86_choose.o" + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then $as_echo "#define USE_SSE42_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_sse42.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: SSE 4.2" >&5 -$as_echo "SSE 4.2" >&6; } -else - if test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: CRC32C baseline feature SSE 4.2" >&5 +$as_echo "CRC32C baseline feature SSE 4.2" >&6; } + else + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then $as_echo "#define USE_SSE42_CRC32C_WITH_RUNTIME_CHECK 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_sse42.o pg_crc32c_sb8.o pg_crc32c_sse42_choose.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: SSE 4.2 with runtime check" >&5 -$as_echo "SSE 4.2 with runtime check" >&6; } - else - if test x"$USE_ARMV8_CRC32C" = x"1"; then + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: CRC32C SSE42 with runtime check" >&5 +$as_echo "CRC32C SSE42 with runtime check" >&6; } + fi + fi + if test x"$pgac_avx512_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then + +$as_echo "#define USE_AVX512_CRC32C_WITH_RUNTIME_CHECK 1" >>confdefs.h + + PG_CRC32C_OBJS+=" pg_crc32c_avx512.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: CRC32C AVX-512 with runtime check" >&5 +$as_echo "CRC32C AVX-512 with runtime check" >&6; } + fi +else + # non x86 code: + # Use ARM CRC Extension if available. + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then $as_echo "#define USE_ARMV8_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_armv8.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions" >&5 + PG_CRC32C_OBJS="pg_crc32c_armv8.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions" >&5 $as_echo "ARMv8 CRC instructions" >&6; } - else - if test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then + else + # ARM CRC Extension, with runtime check? + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then $as_echo "#define USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions with runtime check" >&5 + PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions with runtime check" >&5 $as_echo "ARMv8 CRC instructions with runtime check" >&6; } - else - if test x"$USE_LOONGARCH_CRC32C" = x"1"; then + else + # LoongArch CRCC instructions. + if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then $as_echo "#define USE_LOONGARCH_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_loongarch.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: LoongArch CRCC instructions" >&5 + PG_CRC32C_OBJS="pg_crc32c_loongarch.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: LoongArch CRCC instructions" >&5 $as_echo "LoongArch CRCC instructions" >&6; } - else + else + # fall back to slicing-by-8 algorithm, which doesn't require any + # special CPU support. $as_echo "#define USE_SLICING_BY_8_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_sb8.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: slicing-by-8" >&5 + PG_CRC32C_OBJS="pg_crc32c_sb8.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: slicing-by-8" >&5 $as_echo "slicing-by-8" >&6; } - fi fi fi fi fi - # Select semaphore implementation type. if test "$PORTNAME" != "win32"; then if test x"$PREFERRED_SEMAPHORES" = x"NAMED_POSIX" ; then diff --git a/configure.ac b/configure.ac index 247ae97fa4..96a9c2db1f 100644 --- a/configure.ac +++ b/configure.ac @@ -2021,10 +2021,14 @@ if test x"$host_cpu" = x"x86_64"; then fi fi -# Check for Intel SSE 4.2 intrinsics to do CRC calculations. +# Check for Intel SSE 4.2 and AVX-512 intrinsics to do CRC calculations. # PGAC_SSE42_CRC32_INTRINSICS() +# Check if the _mm512_clmulepi64_epi128 and _mm_xor_epi64 can be used with with +# the __attribute__((target("avx512vl,vpclmulqdq"))). +PGAC_AVX512_CRC32_INTRINSICS([]) + # Are we targeting a processor that supports SSE 4.2? gcc, clang and icc all # define __SSE4_2__ in that case. AC_COMPILE_IFELSE([AC_LANG_PROGRAM([], [ @@ -2060,9 +2064,8 @@ AC_SUBST(CFLAGS_CRC) # If we are targeting a processor that has Intel SSE 4.2 instructions, we can # use the special CRC instructions for calculating CRC-32C. If we're not # targeting such a processor, but we can nevertheless produce code that uses -# the SSE intrinsics, compile both implementations and select which one to use -# at runtime, depending on whether SSE 4.2 is supported by the processor we're -# running on. +# the SSE/AVX-512 intrinsics compile both implementations and select which one +# to use at runtime, depending runtime cpuid information. # # Similarly, if we are targeting an ARM processor that has the CRC # instructions that are part of the ARMv8 CRC Extension, use them. And if @@ -2079,76 +2082,58 @@ AC_SUBST(CFLAGS_CRC) # # If we are targeting a LoongArch processor, CRC instructions are # always available (at least on 64 bit), so no runtime check is needed. -if test x"$USE_SLICING_BY_8_CRC32C" = x"" && test x"$USE_SSE42_CRC32C" = x"" && test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_ARMV8_CRC32C" = x"" && test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_LOONGARCH_CRC32C" = x""; then - # Use Intel SSE 4.2 if available. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then - USE_SSE42_CRC32C=1 - else - # Intel SSE 4.2, with runtime check? The CPUID instruction is needed for - # the runtime check. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then - USE_SSE42_CRC32C_WITH_RUNTIME_CHECK=1 + +AC_MSG_CHECKING([which CRC-32C implementation to use]) +if test x"$host_cpu" = x"x86_64"; then + #x86 only: + PG_CRC32C_OBJS="pg_crc32c_sb8.o pg_crc32c_x86_choose.o" + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then + AC_DEFINE(USE_SSE42_CRC32C, 1, [Define to 1 use Intel SSE 4.2 CRC instructions.]) + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + AC_MSG_RESULT(CRC32C baseline feature SSE 4.2) else - # Use ARM CRC Extension if available. - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then - USE_ARMV8_CRC32C=1 - else - # ARM CRC Extension, with runtime check? - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then - USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK=1 - else - # LoongArch CRCC instructions. - if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then - USE_LOONGARCH_CRC32C=1 - else - # fall back to slicing-by-8 algorithm, which doesn't require any - # special CPU support. - USE_SLICING_BY_8_CRC32C=1 - fi + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then + AC_DEFINE(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use Intel SSE 4.2 CRC instructions with a runtime check.]) + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + AC_MSG_RESULT(CRC32C SSE42 with runtime check) fi - fi fi - fi -fi - -# Set PG_CRC32C_OBJS appropriately depending on the selected implementation. -AC_MSG_CHECKING([which CRC-32C implementation to use]) -if test x"$USE_SSE42_CRC32C" = x"1"; then - AC_DEFINE(USE_SSE42_CRC32C, 1, [Define to 1 use Intel SSE 4.2 CRC instructions.]) - PG_CRC32C_OBJS="pg_crc32c_sse42.o" - AC_MSG_RESULT(SSE 4.2) + if test x"$pgac_avx512_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then + AC_DEFINE(USE_AVX512_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use Intel AVX 512 CRC instructions with a runtime check.]) + PG_CRC32C_OBJS+=" pg_crc32c_avx512.o" + AC_MSG_RESULT(CRC32C AVX-512 with runtime check) + fi else - if test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then - AC_DEFINE(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use Intel SSE 4.2 CRC instructions with a runtime check.]) - PG_CRC32C_OBJS="pg_crc32c_sse42.o pg_crc32c_sb8.o pg_crc32c_sse42_choose.o" - AC_MSG_RESULT(SSE 4.2 with runtime check) + # non x86 code: + # Use ARM CRC Extension if available. + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then + AC_DEFINE(USE_ARMV8_CRC32C, 1, [Define to 1 to use ARMv8 CRC Extension.]) + PG_CRC32C_OBJS="pg_crc32c_armv8.o" + AC_MSG_RESULT(ARMv8 CRC instructions) else - if test x"$USE_ARMV8_CRC32C" = x"1"; then - AC_DEFINE(USE_ARMV8_CRC32C, 1, [Define to 1 to use ARMv8 CRC Extension.]) - PG_CRC32C_OBJS="pg_crc32c_armv8.o" - AC_MSG_RESULT(ARMv8 CRC instructions) + # ARM CRC Extension, with runtime check? + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then + AC_DEFINE(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARMv8 CRC Extension with a runtime check.]) + PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" + AC_MSG_RESULT(ARMv8 CRC instructions with runtime check) else - if test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then - AC_DEFINE(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARMv8 CRC Extension with a runtime check.]) - PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" - AC_MSG_RESULT(ARMv8 CRC instructions with runtime check) + # LoongArch CRCC instructions. + if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then + AC_DEFINE(USE_LOONGARCH_CRC32C, 1, [Define to 1 to use LoongArch CRCC instructions.]) + PG_CRC32C_OBJS="pg_crc32c_loongarch.o" + AC_MSG_RESULT(LoongArch CRCC instructions) else - if test x"$USE_LOONGARCH_CRC32C" = x"1"; then - AC_DEFINE(USE_LOONGARCH_CRC32C, 1, [Define to 1 to use LoongArch CRCC instructions.]) - PG_CRC32C_OBJS="pg_crc32c_loongarch.o" - AC_MSG_RESULT(LoongArch CRCC instructions) - else - AC_DEFINE(USE_SLICING_BY_8_CRC32C, 1, [Define to 1 to use software CRC-32C implementation (slicing-by-8).]) - PG_CRC32C_OBJS="pg_crc32c_sb8.o" - AC_MSG_RESULT(slicing-by-8) - fi + # fall back to slicing-by-8 algorithm, which doesn't require any + # special CPU support. + AC_DEFINE(USE_SLICING_BY_8_CRC32C, 1, [Define to 1 to use software CRC-32C implementation (slicing-by-8).]) + PG_CRC32C_OBJS="pg_crc32c_sb8.o" + AC_MSG_RESULT(slicing-by-8) fi fi fi fi AC_SUBST(PG_CRC32C_OBJS) - # Select semaphore implementation type. if test "$PORTNAME" != "win32"; then if test x"$PREFERRED_SEMAPHORES" = x"NAMED_POSIX" ; then diff --git a/meson.build b/meson.build index e5ce437a5c..5833661d71 100644 --- a/meson.build +++ b/meson.build @@ -2222,6 +2222,23 @@ if host_cpu == 'x86' or host_cpu == 'x86_64' have_optimized_crc = true else + avx512_crc_prog = ''' +#include <immintrin.h> +#include <stdint.h> +#if defined(__has_attribute) && __has_attribute (target) +__attribute__((target("avx512vl,vpclmulqdq"))) +#endif +int main(void) +{ + __m512i x0 = _mm512_set1_epi32(0x1); + __m512i x1 = _mm512_set1_epi32(0x2); + __m512i x2 = _mm512_clmulepi64_epi128(x1, x0, 0x00); // vpclmulqdq + __m128i a1 = _mm_xor_epi64(_mm512_castsi512_si128(x1), _mm512_castsi512_si128(x0)); //avx512vl + int64_t val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); // 64-bit instruction + return (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); +} +''' + prog = ''' #include <nmmintrin.h> @@ -2252,6 +2269,12 @@ int main(void) cdata.set('USE_SSE42_CRC32C_WITH_RUNTIME_CHECK', 1) have_optimized_crc = true endif + if cc.links(avx512_crc_prog, + name: 'AVX512 CRC32C with function attributes', + args: test_c_args) + cdata.set('USE_AVX512_CRC32C_WITH_RUNTIME_CHECK', 1) + have_optimized_crc = true + endif endif diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 07b2f798ab..db40e6476d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -697,6 +697,9 @@ /* Define to 1 to use Intel SSE 4.2 CRC instructions with a runtime check. */ #undef USE_SSE42_CRC32C_WITH_RUNTIME_CHECK +/* Define to 1 to use Intel AVX-512 CRC instructions with a runtime check. */ +#undef USE_AVX512_CRC32C_WITH_RUNTIME_CHECK + /* Define to build with systemd support. (--with-systemd) */ #undef USE_SYSTEMD diff --git a/src/include/pg_cpu.h b/src/include/pg_cpu.h new file mode 100644 index 0000000000..223994cb0d --- /dev/null +++ b/src/include/pg_cpu.h @@ -0,0 +1,23 @@ +/* + * pg_cpu.h + * Useful macros to determine CPU types + */ + +#ifndef PG_CPU_H_ +#define PG_CPU_H_ +#if defined( __i386__ ) || defined(i386) || defined(_M_IX86) + /* + * __i386__ is defined by gcc and Intel compiler on Linux, + * _M_IX86 by VS compiler, + * i386 by Sun compilers on opensolaris at least + */ + #define PG_CPU_X86 +#elif defined(__x86_64__) || defined(__amd64__) || defined(__x86_64) || defined(_M_AMD64) + /* + * both __x86_64__ and __amd64__ are defined by gcc + * __x86_64 defined by sun compiler on opensolaris at least + * _M_AMD64 defined by MS compiler + */ + #define PG_CPU_x86_64 +#endif +#endif // PG_CPU_H_ diff --git a/src/include/port/pg_crc32c.h b/src/include/port/pg_crc32c.h index 63c8e3a00b..690273506b 100644 --- a/src/include/port/pg_crc32c.h +++ b/src/include/port/pg_crc32c.h @@ -34,58 +34,43 @@ #define PG_CRC32C_H #include "port/pg_bswap.h" +#include "pg_cpu.h" typedef uint32 pg_crc32c; /* The INIT and EQ macros are the same for all implementations. */ #define INIT_CRC32C(crc) ((crc) = 0xFFFFFFFF) #define EQ_CRC32C(c1, c2) ((c1) == (c2)) - -#if defined(USE_SSE42_CRC32C) -/* Use Intel SSE4.2 instructions. */ -#define COMP_CRC32C(crc, data, len) \ - ((crc) = pg_comp_crc32c_sse42((crc), (data), (len))) #define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) +/* x86 */ +#if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) +extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len); +extern pg_crc32c pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t len); +extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); +#define COMP_CRC32C(crc, data, len) \ + ((crc) = pg_comp_crc32c((crc), (data), (len))) +/* ARMV8 */ #elif defined(USE_ARMV8_CRC32C) -/* Use ARMv8 CRC Extension instructions. */ - +extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t len); #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c_armv8((crc), (data), (len))) -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) +/* ARMV8 with runtime check */ +#elif defined(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK) +extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t len); +extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); +#define COMP_CRC32C(crc, data, len) \ + ((crc) = pg_comp_crc32c((crc), (data), (len))) +/* LoongArch */ #elif defined(USE_LOONGARCH_CRC32C) -/* Use LoongArch CRCC instructions. */ - +extern pg_crc32c pg_comp_crc32c_loongarch(pg_crc32c crc, const void *data, size_t len); #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c_loongarch((crc), (data), (len))) -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) - -extern pg_crc32c pg_comp_crc32c_loongarch(pg_crc32c crc, const void *data, size_t len); - -#elif defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) || defined(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK) - -/* - * Use Intel SSE 4.2 or ARMv8 instructions, but perform a runtime check first - * to check that they are available. - */ -#define COMP_CRC32C(crc, data, len) \ - ((crc) = pg_comp_crc32c((crc), (data), (len))) -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) - -extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); -extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); - -#ifdef USE_SSE42_CRC32C_WITH_RUNTIME_CHECK -extern pg_crc32c pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len); -#endif -#ifdef USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK -extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t len); -#endif #else /* @@ -98,13 +83,11 @@ extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t le #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c_sb8((crc), (data), (len))) #ifdef WORDS_BIGENDIAN +#undef FIN_CRC32C #define FIN_CRC32C(crc) ((crc) = pg_bswap32(crc) ^ 0xFFFFFFFF) -#else -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) #endif extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); - #endif #endif /* PG_CRC32C_H */ diff --git a/src/include/port/pg_hw_feat_check.h b/src/include/port/pg_hw_feat_check.h index 58be900b54..3a73014987 100644 --- a/src/include/port/pg_hw_feat_check.h +++ b/src/include/port/pg_hw_feat_check.h @@ -30,4 +30,10 @@ extern PGDLLIMPORT bool pg_popcount_available(void); * available. */ extern PGDLLIMPORT bool pg_popcount_avx512_available(void); + +/* + * Test to see if all hardware features required by the AVX-512 SIMD + * algorithm are available. + */ +extern PGDLLIMPORT bool pg_crc32c_avx512_available(void); #endif /* PG_HW_FEAT_CHECK_H */ diff --git a/src/port/meson.build b/src/port/meson.build index ec28590473..0ba4a56194 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,8 +8,10 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_avx512.c', - 'pg_crc32c_sse42_choose.c', + 'pg_crc32c_x86_choose.c', + 'pg_crc32c_avx512.c', 'pg_crc32c_sse42.c', + 'pg_crc32c_sb8.c', 'pg_hw_feat_check.c', 'pg_strong_random.c', 'pgcheckdir.c', @@ -83,12 +85,6 @@ endif # Replacement functionality to be built if corresponding configure symbol # is true replace_funcs_pos = [ - # x86/x64 - ['pg_crc32c_sse42', 'USE_SSE42_CRC32C'], - ['pg_crc32c_sse42', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], - ['pg_crc32c_sse42_choose', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], - ['pg_crc32c_sb8', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], - # arm / aarch64 ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C'], ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK', 'crc'], diff --git a/src/port/pg_crc32c_avx512.c b/src/port/pg_crc32c_avx512.c new file mode 100644 index 0000000000..ba4defcefd --- /dev/null +++ b/src/port/pg_crc32c_avx512.c @@ -0,0 +1,203 @@ +/*------------------------------------------------------------------------- + * + * pg_crc32c_avx512.c + * Compute CRC-32C checksum using Intel AVX-512 instructions. + * + * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/port/pg_crc32c_avx512.c + * + *------------------------------------------------------------------------- + */ + +#include "c.h" + +#if defined(USE_AVX512_CRC32C_WITH_RUNTIME_CHECK) + +#include <immintrin.h> + +#include "port/pg_crc32c.h" + + +/******************************************************************* + * pg_crc32c_avx512(): compute the crc32c of the buffer, where the + * buffer length must be at least 256, and a multiple of 64. Based + * on: + * + * "Fast CRC Computation for Generic Polynomials Using PCLMULQDQ + * Instruction" + * V. Gopal, E. Ozturk, et al., 2009 + * + * For This Function: + * Copyright 2015 The Chromium Authors + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google LLC nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + + +pg_attribute_no_sanitize_alignment() +pg_attribute_target("avx512vl,vpclmulqdq") +inline pg_crc32c +pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t length) +{ + static const uint64 k1k2[8] = { + 0xdcb17aa4, 0xb9e02b86, 0xdcb17aa4, 0xb9e02b86, 0xdcb17aa4, + 0xb9e02b86, 0xdcb17aa4, 0xb9e02b86}; + static const uint64 k3k4[8] = { + 0x740eef02, 0x9e4addf8, 0x740eef02, 0x9e4addf8, 0x740eef02, + 0x9e4addf8, 0x740eef02, 0x9e4addf8}; + static const uint64 k9k10[8] = { + 0x6992cea2, 0x0d3b6092, 0x6992cea2, 0x0d3b6092, 0x6992cea2, + 0x0d3b6092, 0x6992cea2, 0x0d3b6092}; + static const uint64 k1k4[8] = { + 0x1c291d04, 0xddc0152b, 0x3da6d0cb, 0xba4fc28e, 0xf20c0dfe, + 0x493c7d27, 0x00000000, 0x00000000}; + + const uint8 *input = (const uint8 *)data; + if (length >= 256) + { + uint64 val; + __m512i x0, x1, x2, x3, x4, x5, x6, x7, x8, y5, y6, y7, y8; + __m128i a1, a2; + + /* + * AVX-512 Optimized crc32c algorithm with mimimum of 256 bytes aligned + * to 32 bytes. + * >>> BEGIN + */ + + /* + * There's at least one block of 256. + */ + x1 = _mm512_loadu_si512((__m512i *)(input + 0x00)); + x2 = _mm512_loadu_si512((__m512i *)(input + 0x40)); + x3 = _mm512_loadu_si512((__m512i *)(input + 0x80)); + x4 = _mm512_loadu_si512((__m512i *)(input + 0xC0)); + + x1 = _mm512_xor_si512(x1, _mm512_castsi128_si512(_mm_cvtsi32_si128(crc))); + + x0 = _mm512_load_si512((__m512i *)k1k2); + + input += 256; + length -= 256; + + /* + * Parallel fold blocks of 256, if any. + */ + while (length >= 256) + { + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x6 = _mm512_clmulepi64_epi128(x2, x0, 0x00); + x7 = _mm512_clmulepi64_epi128(x3, x0, 0x00); + x8 = _mm512_clmulepi64_epi128(x4, x0, 0x00); + + x1 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x2 = _mm512_clmulepi64_epi128(x2, x0, 0x11); + x3 = _mm512_clmulepi64_epi128(x3, x0, 0x11); + x4 = _mm512_clmulepi64_epi128(x4, x0, 0x11); + + y5 = _mm512_loadu_si512((__m512i *)(input + 0x00)); + y6 = _mm512_loadu_si512((__m512i *)(input + 0x40)); + y7 = _mm512_loadu_si512((__m512i *)(input + 0x80)); + y8 = _mm512_loadu_si512((__m512i *)(input + 0xC0)); + + x1 = _mm512_ternarylogic_epi64(x1, x5, y5, 0x96); + x2 = _mm512_ternarylogic_epi64(x2, x6, y6, 0x96); + x3 = _mm512_ternarylogic_epi64(x3, x7, y7, 0x96); + x4 = _mm512_ternarylogic_epi64(x4, x8, y8, 0x96); + + input += 256; + length -= 256; + } + + /* + * Fold 256 bytes into 64 bytes. + */ + x0 = _mm512_load_si512((__m512i *)k9k10); + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x6 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x3 = _mm512_ternarylogic_epi64(x3, x5, x6, 0x96); + + x7 = _mm512_clmulepi64_epi128(x2, x0, 0x00); + x8 = _mm512_clmulepi64_epi128(x2, x0, 0x11); + x4 = _mm512_ternarylogic_epi64(x4, x7, x8, 0x96); + + x0 = _mm512_load_si512((__m512i *)k3k4); + y5 = _mm512_clmulepi64_epi128(x3, x0, 0x00); + y6 = _mm512_clmulepi64_epi128(x3, x0, 0x11); + x1 = _mm512_ternarylogic_epi64(x4, y5, y6, 0x96); + + /* + * Single fold blocks of 64, if any. + */ + while (length >= 64) + { + x2 = _mm512_loadu_si512((__m512i *)input); + + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x1 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x1 = _mm512_ternarylogic_epi64(x1, x2, x5, 0x96); + + input += 64; + length -= 64; + } + + /* + * Fold 512-bits to 128-bits. + */ + x0 = _mm512_loadu_si512((__m512i *)k1k4); + + a2 = _mm512_extracti32x4_epi32(x1, 3); + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x1 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x1 = _mm512_ternarylogic_epi64(x1, x5, _mm512_castsi128_si512(a2), 0x96); + + x0 = _mm512_shuffle_i64x2(x1, x1, 0x4E); + x0 = _mm512_xor_epi64(x1, x0); + a1 = _mm512_extracti32x4_epi32(x0, 1); + a1 = _mm_xor_epi64(a1, _mm512_castsi512_si128(x0)); + + /* + * Fold 128-bits to 32-bits. + */ + val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); + crc = (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); + /* + * AVX-512 Optimized crc32c algorithm with mimimum of 256 bytes aligned + * to 32 bytes. + * <<< END + ******************************************************************/ + } + + /* + * Finish any remaining bytes with legacy AVX algorithm. + */ + return pg_comp_crc32c_sse42(crc, input, length); +} +#endif // AVX512_CRC32 diff --git a/src/port/pg_crc32c_sse42.c b/src/port/pg_crc32c_sse42.c index dcc4904a82..90d155e804 100644 --- a/src/port/pg_crc32c_sse42.c +++ b/src/port/pg_crc32c_sse42.c @@ -14,6 +14,7 @@ */ #include "c.h" +#if defined(USE_SSE42_CRC32C) || defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) #include <nmmintrin.h> #include "port/pg_crc32c.h" @@ -68,3 +69,4 @@ pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len) return crc; } +#endif diff --git a/src/port/pg_crc32c_sse42_choose.c b/src/port/pg_crc32c_sse42_choose.c deleted file mode 100644 index c659917af0..0000000000 --- a/src/port/pg_crc32c_sse42_choose.c +++ /dev/null @@ -1,51 +0,0 @@ -/*------------------------------------------------------------------------- - * - * pg_crc32c_sse42_choose.c - * Choose between Intel SSE 4.2 and software CRC-32C implementation. - * - * On first call, checks if the CPU we're running on supports Intel SSE - * 4.2. If it does, use the special SSE instructions for CRC-32C - * computation. Otherwise, fall back to the pure software implementation - * (slicing-by-8). - * - * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group - * Portions Copyright (c) 1994, Regents of the University of California - * - * - * IDENTIFICATION - * src/port/pg_crc32c_sse42_choose.c - * - *------------------------------------------------------------------------- - */ - -#include "c.h" - -#if defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) -#ifdef HAVE__GET_CPUID -#include <cpuid.h> -#endif - -#ifdef HAVE__CPUID -#include <intrin.h> -#endif - -#include "port/pg_crc32c.h" -#include "port/pg_hw_feat_check.h" - -/* - * This gets called on the first call. It replaces the function pointer - * so that subsequent calls are routed directly to the chosen implementation. - */ -static pg_crc32c -pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len) -{ - if (pg_crc32c_sse42_available()) - pg_comp_crc32c = pg_comp_crc32c_sse42; - else - pg_comp_crc32c = pg_comp_crc32c_sb8; - - return pg_comp_crc32c(crc, data, len); -} - -pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; -#endif diff --git a/src/port/pg_crc32c_x86_choose.c b/src/port/pg_crc32c_x86_choose.c new file mode 100644 index 0000000000..3ce8be11a6 --- /dev/null +++ b/src/port/pg_crc32c_x86_choose.c @@ -0,0 +1,57 @@ +/*------------------------------------------------------------------------- + * + * pg_crc32c_x86_choose.c + * Choose between Intel AVX-512, SSE 4.2 and software CRC-32C implementation. + * + * On first call, checks if the CPU we're running on supports Intel AVX-512. If + * it does, use the special SSE instructions for CRC-32C computation. + * Otherwise, fall back to the pure software implementation (slicing-by-8). + * + * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/port/pg_crc32c_x86_choose.c + * + *------------------------------------------------------------------------- + */ + +#include "c.h" +#include "pg_cpu.h" + +#if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) + +#include "port/pg_crc32c.h" +#include "port/pg_hw_feat_check.h" + +/* + * This gets called on the first call. It replaces the function pointer + * so that subsequent calls are routed directly to the chosen implementation. + * (1) set pg_comp_crc32c pointer and (2) return the computed crc value + */ +static pg_crc32c +pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len) +{ +#ifdef USE_AVX512_CRC32C_WITH_RUNTIME_CHECK + if (pg_crc32c_avx512_available()) { + pg_comp_crc32c = pg_comp_crc32c_avx512; + return pg_comp_crc32c(crc, data, len); + } +#endif +#ifdef USE_SSE42_CRC32C + pg_comp_crc32c = pg_comp_crc32c_sse42; + return pg_comp_crc32c(crc, data, len); +#elif USE_SSE42_CRC32C_WITH_RUNTIME_CHECK + if (pg_crc32c_sse42_available()) { + pg_comp_crc32c = pg_comp_crc32c_sse42; + return pg_comp_crc32c(crc, data, len); + } +#endif + pg_comp_crc32c = pg_comp_crc32c_sb8; + return pg_comp_crc32c(crc, data, len); +} + +pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; + +#endif // x86/x86_64 diff --git a/src/port/pg_hw_feat_check.c b/src/port/pg_hw_feat_check.c index 260aa60502..b2872fa708 100644 --- a/src/port/pg_hw_feat_check.c +++ b/src/port/pg_hw_feat_check.c @@ -11,6 +11,9 @@ *------------------------------------------------------------------------- */ #include "c.h" +#include "pg_cpu.h" + +#if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) #if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) #include <cpuid.h> @@ -135,9 +138,60 @@ bool PGDLLIMPORT pg_popcount_available(void) return is_bit_set_in_exx(exx, ECX, 23); } +/* + * Check for CPU supprt for CPUIDEX: avx512-f + */ +inline static bool +avx512f_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + return is_bit_set_in_exx(exx, EBX, 16); /* avx512-f */ +} + +/* + * Check for CPU supprt for CPUIDEX: vpclmulqdq + */ +inline static bool +vpclmulqdq_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + return is_bit_set_in_exx(exx, ECX, 10); /* vpclmulqdq */ +} + +/* + * Check for CPU supprt for CPUIDEX: vpclmulqdq + */ +inline static bool +avx512vl_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + return is_bit_set_in_exx(exx, EBX, 31); /* avx512-vl */ +} + +/* + * Check for CPU supprt for CPUID: sse4.2 + */ +inline static bool +sse42_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + return is_bit_set_in_exx(exx, ECX, 20); /* sse4.2 */ +} + +/****************************************************************************/ +/* Public API */ +/****************************************************************************/ /* - * Returns true if the CPU supports the instructions required for the AVX-512 - * pg_popcount() implementation. + * Returns true if the CPU supports the instructions required for the + * AVX-512 pg_popcount() implementation. * * PA: The call to 'osxsave_available' MUST preceed the call to * 'zmm_regs_available' function per NB above. @@ -154,10 +208,19 @@ bool PGDLLIMPORT pg_popcount_avx512_available(void) */ bool PGDLLIMPORT pg_crc32c_sse42_available(void) { - exx_t exx[4] = {0, 0, 0, 0}; - - pg_getcpuid(1, exx); + return sse42_available(); +} - return is_bit_set_in_exx(exx, ECX, 20); +/* + * Returns true if the CPU supports the instructions required for the AVX-512 + * pg_crc32c implementation. + */ +bool PGDLLIMPORT +pg_crc32c_avx512_available(void) +{ + return sse42_available() && osxsave_available() && + avx512f_available() && vpclmulqdq_available() && + avx512vl_available() && zmm_regs_available(); } +#endif // #if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) -- 2.34.1 From 6e8f557c857772b0c22607866d1b8930a67df05e Mon Sep 17 00:00:00 2001 From: Matthew Sterrett <[email protected]> Date: Wed, 18 Dec 2024 14:11:33 -0800 Subject: [PATCH v10 4/4] Mark pg_comp_crc32c as PGDLLIMPORT for Windows build --- src/include/port/pg_crc32c.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/include/port/pg_crc32c.h b/src/include/port/pg_crc32c.h index 690273506b..534d07dd5d 100644 --- a/src/include/port/pg_crc32c.h +++ b/src/include/port/pg_crc32c.h @@ -48,7 +48,7 @@ typedef uint32 pg_crc32c; extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t len); -extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); +extern PGDLLIMPORT pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c((crc), (data), (len))) -- 2.34.1 Attachments: [text/plain] v10-0001-Add-a-Postgres-SQL-function-for-crc32c-benchmark.patch (4.9K, ../../[email protected]/2-v10-0001-Add-a-Postgres-SQL-function-for-crc32c-benchmark.patch) download | inline diff: From 74d085d44d41af8ffb01f7bf2377ac487c7d4cc1 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Mon, 6 May 2024 08:34:17 -0700 Subject: [PATCH v10 1/4] Add a Postgres SQL function for crc32c benchmarking. Add a drive_crc32c() function to use for benchmarking crc32c computation. The function takes 2 arguments: (1) count: num of times CRC32C is computed in a loop. (2) num: #bytes in the buffer to calculate crc over. Signed-off-by: Paul Amonson <[email protected]> Signed-off-by: Raghuveer Devulapalli <[email protected]> --- src/test/modules/meson.build | 1 + src/test/modules/test_crc32c/Makefile | 20 ++++++++ src/test/modules/test_crc32c/meson.build | 22 +++++++++ .../modules/test_crc32c/test_crc32c--1.0.sql | 1 + src/test/modules/test_crc32c/test_crc32c.c | 47 +++++++++++++++++++ .../modules/test_crc32c/test_crc32c.control | 4 ++ 6 files changed, 95 insertions(+) create mode 100644 src/test/modules/test_crc32c/Makefile create mode 100644 src/test/modules/test_crc32c/meson.build create mode 100644 src/test/modules/test_crc32c/test_crc32c--1.0.sql create mode 100644 src/test/modules/test_crc32c/test_crc32c.c create mode 100644 src/test/modules/test_crc32c/test_crc32c.control diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index c829b61953..68d8904dd0 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -15,6 +15,7 @@ subdir('ssl_passphrase_callback') subdir('test_bloomfilter') subdir('test_copy_callbacks') subdir('test_custom_rmgrs') +subdir('test_crc32c') subdir('test_ddl_deparse') subdir('test_dsa') subdir('test_dsm_registry') diff --git a/src/test/modules/test_crc32c/Makefile b/src/test/modules/test_crc32c/Makefile new file mode 100644 index 0000000000..5b747c6184 --- /dev/null +++ b/src/test/modules/test_crc32c/Makefile @@ -0,0 +1,20 @@ +MODULE_big = test_crc32c +OBJS = test_crc32c.o +PGFILEDESC = "test" +EXTENSION = test_crc32c +DATA = test_crc32c--1.0.sql + +first: all + +# test_crc32c.o: CFLAGS+=-g + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_crc32c +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_crc32c/meson.build b/src/test/modules/test_crc32c/meson.build new file mode 100644 index 0000000000..7021a6d6cf --- /dev/null +++ b/src/test/modules/test_crc32c/meson.build @@ -0,0 +1,22 @@ +# Copyright (c) 2022-2024, PostgreSQL Global Development Group + +test_crc32c_sources = files( + 'test_crc32c.c', +) + +if host_system == 'windows' + test_crc32c_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_crc32c', + '--FILEDESC', 'test_crc32c - test code for crc32c library',]) +endif + +test_crc32c = shared_module('test_crc32c', + test_crc32c_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_crc32c + +test_install_data += files( + 'test_crc32c.control', + 'test_crc32c--1.0.sql', +) diff --git a/src/test/modules/test_crc32c/test_crc32c--1.0.sql b/src/test/modules/test_crc32c/test_crc32c--1.0.sql new file mode 100644 index 0000000000..32f8f0fb2e --- /dev/null +++ b/src/test/modules/test_crc32c/test_crc32c--1.0.sql @@ -0,0 +1 @@ +CREATE FUNCTION drive_crc32c (count int, num int) RETURNS bigint AS 'test_crc32c.so' LANGUAGE C; diff --git a/src/test/modules/test_crc32c/test_crc32c.c b/src/test/modules/test_crc32c/test_crc32c.c new file mode 100644 index 0000000000..b350caf5ce --- /dev/null +++ b/src/test/modules/test_crc32c/test_crc32c.c @@ -0,0 +1,47 @@ +/* select drive_crc32c(1000000, 1024); */ + +#include "postgres.h" +#include "fmgr.h" +#include "port/pg_crc32c.h" +#include "common/pg_prng.h" + +PG_MODULE_MAGIC; + +/* + * drive_crc32c(count: int, num: int) returns bigint + * + * count is the nuimber of loops to perform + * + * num is the number byte in the buffer to calculate + * crc32c over. + */ +PG_FUNCTION_INFO_V1(drive_crc32c); +Datum +drive_crc32c(PG_FUNCTION_ARGS) +{ + int64 count = PG_GETARG_INT64(0); + int64 num = PG_GETARG_INT64(1); + char* data = malloc((size_t)num); + pg_crc32c crc; + pg_prng_state state; + uint64 seed = 42; + pg_prng_seed(&state, seed); + /* set random data */ + for (uint64 i = 0; i < num; i++) + { + data[i] = pg_prng_uint32(&state) % 255; + } + + INIT_CRC32C(crc); + + while(count--) + { + INIT_CRC32C(crc); + COMP_CRC32C(crc, data, num); + FIN_CRC32C(crc); + } + + free((void *)data); + + PG_RETURN_INT64((int64_t)crc); +} diff --git a/src/test/modules/test_crc32c/test_crc32c.control b/src/test/modules/test_crc32c/test_crc32c.control new file mode 100644 index 0000000000..878a077ee1 --- /dev/null +++ b/src/test/modules/test_crc32c/test_crc32c.control @@ -0,0 +1,4 @@ +comment = 'test' +default_version = '1.0' +module_pathname = '$libdir/test_crc32c' +relocatable = true -- 2.34.1 [text/plain] v10-0002-Refactor-consolidate-x86-ISA-and-OS-runtime-chec.patch (11.7K, ../../[email protected]/3-v10-0002-Refactor-consolidate-x86-ISA-and-OS-runtime-chec.patch) download | inline diff: From 2542c6830d98e146d79844fb84fe3fb1b2945c25 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 23 Jul 2024 11:23:23 -0700 Subject: [PATCH v10 2/4] Refactor: consolidate x86 ISA and OS runtime checks Move all x86 ISA and OS runtime checks into a single file for improved modularity and easier future maintenance. Signed-off-by: Paul Amonson <[email protected]> Signed-off-by: Raghuveer Devulapalli <[email protected]> --- src/include/port/pg_bitutils.h | 1 - src/include/port/pg_hw_feat_check.h | 33 ++++++ src/port/Makefile | 1 + src/port/meson.build | 3 + src/port/pg_bitutils.c | 22 +--- src/port/pg_crc32c_sse42_choose.c | 21 +--- src/port/pg_hw_feat_check.c | 163 ++++++++++++++++++++++++++++ src/port/pg_popcount_avx512.c | 78 ------------- 8 files changed, 205 insertions(+), 117 deletions(-) create mode 100644 src/include/port/pg_hw_feat_check.h create mode 100644 src/port/pg_hw_feat_check.c diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index a3cad46afe..461c7c13cf 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -312,7 +312,6 @@ extern PGDLLIMPORT uint64 (*pg_popcount_masked_optimized) (const char *buf, int * files. */ #ifdef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK -extern bool pg_popcount_avx512_available(void); extern uint64 pg_popcount_avx512(const char *buf, int bytes); extern uint64 pg_popcount_masked_avx512(const char *buf, int bytes, bits8 mask); #endif diff --git a/src/include/port/pg_hw_feat_check.h b/src/include/port/pg_hw_feat_check.h new file mode 100644 index 0000000000..58be900b54 --- /dev/null +++ b/src/include/port/pg_hw_feat_check.h @@ -0,0 +1,33 @@ +/*------------------------------------------------------------------------- + * + * pg_hw_feat_check.h + * Miscellaneous functions for cheing for hardware features at runtime. + * + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * src/include/port/pg_hw_feat_check.h + * + *------------------------------------------------------------------------- + */ +#ifndef PG_HW_FEAT_CHECK_H +#define PG_HW_FEAT_CHECK_H + +/* + * Test to see if all hardware features required by SSE 4.2 crc32c (64 bit) + * are available. + */ +extern PGDLLIMPORT bool pg_crc32c_sse42_available(void); + +/* + * Test to see if all hardware features required by SSE 4.1 POPCNT (64 bit) + * are available. + */ +extern PGDLLIMPORT bool pg_popcount_available(void); + +/* + * Test to see if all hardware features required by AVX-512 POPCNT are + * available. + */ +extern PGDLLIMPORT bool pg_popcount_avx512_available(void); +#endif /* PG_HW_FEAT_CHECK_H */ diff --git a/src/port/Makefile b/src/port/Makefile index 4c22431951..6088b56b71 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -45,6 +45,7 @@ OBJS = \ path.o \ pg_bitutils.o \ pg_popcount_avx512.o \ + pg_hw_feat_check.o \ pg_strong_random.o \ pgcheckdir.o \ pgmkdirp.o \ diff --git a/src/port/meson.build b/src/port/meson.build index c5bceed9cd..ec28590473 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,6 +8,9 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_avx512.c', + 'pg_crc32c_sse42_choose.c', + 'pg_crc32c_sse42.c', + 'pg_hw_feat_check.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index c8399981ee..c11b13dca2 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -20,7 +20,7 @@ #endif #include "port/pg_bitutils.h" - +#include "port/pg_hw_feat_check.h" /* * Array giving the position of the left-most set bit for each possible @@ -109,7 +109,6 @@ static uint64 pg_popcount_slow(const char *buf, int bytes); static uint64 pg_popcount_masked_slow(const char *buf, int bytes, bits8 mask); #ifdef TRY_POPCNT_FAST -static bool pg_popcount_available(void); static int pg_popcount32_choose(uint32 word); static int pg_popcount64_choose(uint64 word); static uint64 pg_popcount_choose(const char *buf, int bytes); @@ -127,25 +126,6 @@ uint64 (*pg_popcount_masked_optimized) (const char *buf, int bytes, bits8 mask) #ifdef TRY_POPCNT_FAST -/* - * Return true if CPUID indicates that the POPCNT instruction is available. - */ -static bool -pg_popcount_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 23)) != 0; /* POPCNT */ -} - /* * These functions get called on the first call to pg_popcount32 etc. * They detect whether we can use the asm implementations, and replace diff --git a/src/port/pg_crc32c_sse42_choose.c b/src/port/pg_crc32c_sse42_choose.c index 56d600f3a9..c659917af0 100644 --- a/src/port/pg_crc32c_sse42_choose.c +++ b/src/port/pg_crc32c_sse42_choose.c @@ -20,6 +20,7 @@ #include "c.h" +#if defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) #ifdef HAVE__GET_CPUID #include <cpuid.h> #endif @@ -29,22 +30,7 @@ #endif #include "port/pg_crc32c.h" - -static bool -pg_crc32c_sse42_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 20)) != 0; /* SSE 4.2 */ -} +#include "port/pg_hw_feat_check.h" /* * This gets called on the first call. It replaces the function pointer @@ -61,4 +47,5 @@ pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len) return pg_comp_crc32c(crc, data, len); } -pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; +pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; +#endif diff --git a/src/port/pg_hw_feat_check.c b/src/port/pg_hw_feat_check.c new file mode 100644 index 0000000000..260aa60502 --- /dev/null +++ b/src/port/pg_hw_feat_check.c @@ -0,0 +1,163 @@ +/*------------------------------------------------------------------------- + * + * pg_hw_feat_check.c + * Test for hardware features at runtime on x86_64 platforms. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_hw_feat_check.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" + +#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) +#include <cpuid.h> +#endif + +#include <immintrin.h> + +#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX) +#include <intrin.h> +#endif + +#include "port/pg_hw_feat_check.h" + +/* Define names for EXX registers to avoid hard to see bugs in code below. */ +typedef unsigned int exx_t; +typedef enum +{ + EAX = 0, + EBX = 1, + ECX = 2, + EDX = 3 +} reg_name; + +/* + * Helper function. + * Test for a bit being set in a exx_t register. + */ +inline static bool is_bit_set_in_exx(exx_t* regs, reg_name ex, int bit) +{ + return ((regs[ex] & (1 << bit)) != 0); +} + +/* + * x86_64 Platform CPUID check for Linux and Visual Studio platforms. + */ +inline static void +pg_getcpuid(unsigned int leaf, exx_t *exx) +{ +#if defined(HAVE__GET_CPUID) + __get_cpuid(leaf, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUID) + __cpuid(exx, 1); +#else +#error cpuid instruction not available +#endif +} + +/* + * x86_64 Platform CPUIDEX check for Linux and Visual Studio platforms. + */ +inline static void +pg_getcpuidex(unsigned int leaf, unsigned int subleaf, exx_t *exx) +{ +#if defined(HAVE__GET_CPUID_COUNT) + __get_cpuid_count(leaf, subleaf, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUIDEX) + __cpuidex(exx, 7, 0); +#else +#error cpuid instruction not available +#endif +} + +/* + * Check for CPU support for CPUID: osxsave + */ +inline static bool +osxsave_available(void) +{ +#if defined(HAVE_XSAVE_INTRINSICS) + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + + return is_bit_set_in_exx(exx, ECX, 27); /* osxsave */ +#else + return false; +#endif +} + +/* + * Does XGETBV say the ZMM registers are enabled? + * + * NB: Caller is responsible for verifying that osxsave_available() returns true + * before calling this. + */ +#ifdef HAVE_XSAVE_INTRINSICS +pg_attribute_target("xsave") +#endif +inline static bool +zmm_regs_available(void) +{ +#if defined(HAVE_XSAVE_INTRINSICS) + return (_xgetbv(0) & 0xe6) == 0xe6; +#else + return false; +#endif +} + +/* + * Does CPUID say there's support for AVX-512 popcount and byte-and-word + * instructions? + */ +inline static bool +avx512_popcnt_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + + return is_bit_set_in_exx(exx, ECX, 14) && is_bit_set_in_exx(exx, EBX, 30); +} + +/* + * Return true if CPUID indicates that the POPCNT instruction is available. + */ +bool PGDLLIMPORT pg_popcount_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + + return is_bit_set_in_exx(exx, ECX, 23); + } + + /* + * Returns true if the CPU supports the instructions required for the AVX-512 + * pg_popcount() implementation. + * + * PA: The call to 'osxsave_available' MUST preceed the call to + * 'zmm_regs_available' function per NB above. + */ +bool PGDLLIMPORT pg_popcount_avx512_available(void) +{ + return osxsave_available() && + zmm_regs_available() && + avx512_popcnt_available(); +} + +/* + * Does CPUID say there's support for SSE 4.2? + */ +bool PGDLLIMPORT pg_crc32c_sse42_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + + return is_bit_set_in_exx(exx, ECX, 20); +} + diff --git a/src/port/pg_popcount_avx512.c b/src/port/pg_popcount_avx512.c index c8a4f2b19f..1123a1a634 100644 --- a/src/port/pg_popcount_avx512.c +++ b/src/port/pg_popcount_avx512.c @@ -14,16 +14,7 @@ #ifdef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK -#if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) -#include <cpuid.h> -#endif - #include <immintrin.h> - -#if defined(HAVE__CPUID) || defined(HAVE__CPUIDEX) -#include <intrin.h> -#endif - #include "port/pg_bitutils.h" /* @@ -33,75 +24,6 @@ */ #ifdef TRY_POPCNT_FAST -/* - * Does CPUID say there's support for XSAVE instructions? - */ -static inline bool -xsave_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - return (exx[2] & (1 << 27)) != 0; /* osxsave */ -} - -/* - * Does XGETBV say the ZMM registers are enabled? - * - * NB: Caller is responsible for verifying that xsave_available() returns true - * before calling this. - */ -#ifdef HAVE_XSAVE_INTRINSICS -pg_attribute_target("xsave") -#endif -static inline bool -zmm_regs_available(void) -{ -#ifdef HAVE_XSAVE_INTRINSICS - return (_xgetbv(0) & 0xe6) == 0xe6; -#else - return false; -#endif -} - -/* - * Does CPUID say there's support for AVX-512 popcount and byte-and-word - * instructions? - */ -static inline bool -avx512_popcnt_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID_COUNT) - __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUIDEX) - __cpuidex(exx, 7, 0); -#else -#error cpuid instruction not available -#endif - return (exx[2] & (1 << 14)) != 0 && /* avx512-vpopcntdq */ - (exx[1] & (1 << 30)) != 0; /* avx512-bw */ -} - -/* - * Returns true if the CPU supports the instructions required for the AVX-512 - * pg_popcount() implementation. - */ -bool -pg_popcount_avx512_available(void) -{ - return xsave_available() && - zmm_regs_available() && - avx512_popcnt_available(); -} - /* * pg_popcount_avx512 * Returns the number of 1-bits in buf -- 2.34.1 [text/plain] v10-0003-Add-AVX-512-CRC32C-algorithm-with-a-runtime-chec.patch (41.9K, ../../[email protected]/4-v10-0003-Add-AVX-512-CRC32C-algorithm-with-a-runtime-chec.patch) download | inline diff: From f08e15c0834616c636d1cb949ed140926265847e Mon Sep 17 00:00:00 2001 From: Raghuveer Devulapalli <[email protected]> Date: Thu, 21 Nov 2024 12:42:09 -0800 Subject: [PATCH v10 3/4] Add AVX-512 CRC32C algorithm with a runtime check Adds pg_crc32c_avx512(): compute the crc32c of the buffer, where the buffer length must be at least 256, and a multiple of 64. Based on: "Fast CRC Computation for Generic Polynomials Using PCLMULQDQ Instruction" V. Gopal, E. Ozturk, et al., 2009" Benchmark numbers to compare against the SSE4.2 CRC32C algorithm was generated by using the drive_crc32c() function added in src/test/modules/test_crc32c/test_crc32c.c. +------------------+----------------+----------------+------------------+-------+------+ | Rate in bytes/us | SDP (SPR) | m6i | m7i | | | +------------------+----------------+----------------+------------------+ Multi-| | | higher is better | SSE42 | AVX512 | SSE42 | AVX512 | SSE42 | AVX512 | plier | % | +==================+=================+=======+========+========+========+=======+======+ | AVG Rate 64-8192 | 10,095 | 82,101 | 8,591 | 38,652 | 11,867 | 83,194 | 6.68 | 568% | +------------------+--------+--------+-------+--------+--------+--------+-------+------+ | AVG Rate 64-255 | 9,034 | 9,136 | 7,619 | 7,437 | 9,030 | 9,293 | 1.01 | 1% | +------------------+--------+--------+-------+--------+--------+--------+-------+------+ Co-authored-by: Paul Amonson <[email protected]> --- config/c-compiler.m4 | 32 +++++ configure | 154 ++++++++++++--------- configure.ac | 107 +++++++-------- meson.build | 23 ++++ src/include/pg_config.h.in | 3 + src/include/pg_cpu.h | 23 ++++ src/include/port/pg_crc32c.h | 55 +++----- src/include/port/pg_hw_feat_check.h | 6 + src/port/meson.build | 10 +- src/port/pg_crc32c_avx512.c | 203 ++++++++++++++++++++++++++++ src/port/pg_crc32c_sse42.c | 2 + src/port/pg_crc32c_sse42_choose.c | 51 ------- src/port/pg_crc32c_x86_choose.c | 57 ++++++++ src/port/pg_hw_feat_check.c | 75 +++++++++- 14 files changed, 578 insertions(+), 223 deletions(-) create mode 100644 src/include/pg_cpu.h create mode 100644 src/port/pg_crc32c_avx512.c delete mode 100644 src/port/pg_crc32c_sse42_choose.c create mode 100644 src/port/pg_crc32c_x86_choose.c diff --git a/config/c-compiler.m4 b/config/c-compiler.m4 index e112fd45d4..e08de01739 100644 --- a/config/c-compiler.m4 +++ b/config/c-compiler.m4 @@ -578,6 +578,38 @@ undefine([Ac_cachevar])dnl ])# PGAC_SSE42_CRC32_INTRINSICS +# PGAC_AVX512_CRC32_INTRINSICS +# --------------------------- +# Check if the compiler supports the x86 CRC instructions added in AVX-512, +# using intrinsics with function __attribute__((target("..."))): + +AC_DEFUN([PGAC_AVX512_CRC32_INTRINSICS], +[define([Ac_cachevar], [AS_TR_SH([pgac_cv_avx512_crc32_intrinsics])])dnl +AC_CACHE_CHECK([for _mm512_clmulepi64_epi128 with function attribute], [Ac_cachevar], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h> + #include <stdint.h> + #if defined(__has_attribute) && __has_attribute (target) + __attribute__((target("avx512vl,vpclmulqdq"))) + #endif + static int crc32_avx512_test(void) + { + __m512i x0 = _mm512_set1_epi32(0x1); + __m512i x1 = _mm512_set1_epi32(0x2); + __m512i x2 = _mm512_clmulepi64_epi128(x1, x0, 0x00); // vpclmulqdq + __m128i a1 = _mm_xor_epi64(_mm512_castsi512_si128(x1), _mm512_castsi512_si128(x0)); //avx512vl + int64_t val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); // 64-bit instruction + return (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); + }], + [return crc32_avx512_test();])], + [Ac_cachevar=yes], + [Ac_cachevar=no])]) +if test x"$Ac_cachevar" = x"yes"; then + pgac_avx512_crc32_intrinsics=yes +fi +undefine([Ac_cachevar])dnl +])# PGAC_AVX512_CRC32_INTRINSICS + + # PGAC_ARMV8_CRC32C_INTRINSICS # ---------------------------- # Check if the compiler supports the CRC32C instructions using the __crc32cb, diff --git a/configure b/configure index 518c33b73a..b03b928bfd 100755 --- a/configure +++ b/configure @@ -17159,7 +17159,7 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h fi fi -# Check for Intel SSE 4.2 intrinsics to do CRC calculations. +# Check for Intel SSE 4.2 and AVX-512 intrinsics to do CRC calculations. # { $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5 $as_echo_n "checking for _mm_crc32_u8 and _mm_crc32_u32... " >&6; } @@ -17203,6 +17203,52 @@ if test x"$pgac_cv_sse42_crc32_intrinsics" = x"yes"; then fi +# Check if the _mm512_clmulepi64_epi128 and _mm_xor_epi64 can be used with with +# the __attribute__((target("avx512vl,vpclmulqdq"))). +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_clmulepi64_epi128 with function attribute" >&5 +$as_echo_n "checking for _mm512_clmulepi64_epi128 with function attribute... " >&6; } +if ${pgac_cv_avx512_crc32_intrinsics+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> + #include <stdint.h> + #if defined(__has_attribute) && __has_attribute (target) + __attribute__((target("avx512vl,vpclmulqdq"))) + #endif + static int crc32_avx512_test(void) + { + __m512i x0 = _mm512_set1_epi32(0x1); + __m512i x1 = _mm512_set1_epi32(0x2); + __m512i x2 = _mm512_clmulepi64_epi128(x1, x0, 0x00); // vpclmulqdq + __m128i a1 = _mm_xor_epi64(_mm512_castsi512_si128(x1), _mm512_castsi512_si128(x0)); //avx512vl + int64_t val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); // 64-bit instruction + return (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); + } +int +main () +{ +return crc32_avx512_test(); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_crc32_intrinsics=yes +else + pgac_cv_avx512_crc32_intrinsics=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_crc32_intrinsics" >&5 +$as_echo "$pgac_cv_avx512_crc32_intrinsics" >&6; } +if test x"$pgac_cv_avx512_crc32_intrinsics" = x"yes"; then + pgac_avx512_crc32_intrinsics=yes +fi + + # Are we targeting a processor that supports SSE 4.2? gcc, clang and icc all # define __SSE4_2__ in that case. cat confdefs.h - <<_ACEOF >conftest.$ac_ext @@ -17404,9 +17450,8 @@ fi # If we are targeting a processor that has Intel SSE 4.2 instructions, we can # use the special CRC instructions for calculating CRC-32C. If we're not # targeting such a processor, but we can nevertheless produce code that uses -# the SSE intrinsics, compile both implementations and select which one to use -# at runtime, depending on whether SSE 4.2 is supported by the processor we're -# running on. +# the SSE/AVX-512 intrinsics compile both implementations and select which one +# to use at runtime, depending runtime cpuid information. # # Similarly, if we are targeting an ARM processor that has the CRC # instructions that are part of the ARMv8 CRC Extension, use them. And if @@ -17423,95 +17468,80 @@ fi # # If we are targeting a LoongArch processor, CRC instructions are # always available (at least on 64 bit), so no runtime check is needed. -if test x"$USE_SLICING_BY_8_CRC32C" = x"" && test x"$USE_SSE42_CRC32C" = x"" && test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_ARMV8_CRC32C" = x"" && test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_LOONGARCH_CRC32C" = x""; then - # Use Intel SSE 4.2 if available. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then - USE_SSE42_CRC32C=1 - else - # Intel SSE 4.2, with runtime check? The CPUID instruction is needed for - # the runtime check. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then - USE_SSE42_CRC32C_WITH_RUNTIME_CHECK=1 - else - # Use ARM CRC Extension if available. - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then - USE_ARMV8_CRC32C=1 - else - # ARM CRC Extension, with runtime check? - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then - USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK=1 - else - # LoongArch CRCC instructions. - if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then - USE_LOONGARCH_CRC32C=1 - else - # fall back to slicing-by-8 algorithm, which doesn't require any - # special CPU support. - USE_SLICING_BY_8_CRC32C=1 - fi - fi - fi - fi - fi -fi -# Set PG_CRC32C_OBJS appropriately depending on the selected implementation. { $as_echo "$as_me:${as_lineno-$LINENO}: checking which CRC-32C implementation to use" >&5 $as_echo_n "checking which CRC-32C implementation to use... " >&6; } -if test x"$USE_SSE42_CRC32C" = x"1"; then +if test x"$host_cpu" = x"x86_64"; then + #x86 only: + PG_CRC32C_OBJS="pg_crc32c_sb8.o pg_crc32c_x86_choose.o" + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then $as_echo "#define USE_SSE42_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_sse42.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: SSE 4.2" >&5 -$as_echo "SSE 4.2" >&6; } -else - if test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: CRC32C baseline feature SSE 4.2" >&5 +$as_echo "CRC32C baseline feature SSE 4.2" >&6; } + else + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then $as_echo "#define USE_SSE42_CRC32C_WITH_RUNTIME_CHECK 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_sse42.o pg_crc32c_sb8.o pg_crc32c_sse42_choose.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: SSE 4.2 with runtime check" >&5 -$as_echo "SSE 4.2 with runtime check" >&6; } - else - if test x"$USE_ARMV8_CRC32C" = x"1"; then + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: CRC32C SSE42 with runtime check" >&5 +$as_echo "CRC32C SSE42 with runtime check" >&6; } + fi + fi + if test x"$pgac_avx512_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then + +$as_echo "#define USE_AVX512_CRC32C_WITH_RUNTIME_CHECK 1" >>confdefs.h + + PG_CRC32C_OBJS+=" pg_crc32c_avx512.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: CRC32C AVX-512 with runtime check" >&5 +$as_echo "CRC32C AVX-512 with runtime check" >&6; } + fi +else + # non x86 code: + # Use ARM CRC Extension if available. + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then $as_echo "#define USE_ARMV8_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_armv8.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions" >&5 + PG_CRC32C_OBJS="pg_crc32c_armv8.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions" >&5 $as_echo "ARMv8 CRC instructions" >&6; } - else - if test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then + else + # ARM CRC Extension, with runtime check? + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then $as_echo "#define USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions with runtime check" >&5 + PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: ARMv8 CRC instructions with runtime check" >&5 $as_echo "ARMv8 CRC instructions with runtime check" >&6; } - else - if test x"$USE_LOONGARCH_CRC32C" = x"1"; then + else + # LoongArch CRCC instructions. + if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then $as_echo "#define USE_LOONGARCH_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_loongarch.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: LoongArch CRCC instructions" >&5 + PG_CRC32C_OBJS="pg_crc32c_loongarch.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: LoongArch CRCC instructions" >&5 $as_echo "LoongArch CRCC instructions" >&6; } - else + else + # fall back to slicing-by-8 algorithm, which doesn't require any + # special CPU support. $as_echo "#define USE_SLICING_BY_8_CRC32C 1" >>confdefs.h - PG_CRC32C_OBJS="pg_crc32c_sb8.o" - { $as_echo "$as_me:${as_lineno-$LINENO}: result: slicing-by-8" >&5 + PG_CRC32C_OBJS="pg_crc32c_sb8.o" + { $as_echo "$as_me:${as_lineno-$LINENO}: result: slicing-by-8" >&5 $as_echo "slicing-by-8" >&6; } - fi fi fi fi fi - # Select semaphore implementation type. if test "$PORTNAME" != "win32"; then if test x"$PREFERRED_SEMAPHORES" = x"NAMED_POSIX" ; then diff --git a/configure.ac b/configure.ac index 247ae97fa4..96a9c2db1f 100644 --- a/configure.ac +++ b/configure.ac @@ -2021,10 +2021,14 @@ if test x"$host_cpu" = x"x86_64"; then fi fi -# Check for Intel SSE 4.2 intrinsics to do CRC calculations. +# Check for Intel SSE 4.2 and AVX-512 intrinsics to do CRC calculations. # PGAC_SSE42_CRC32_INTRINSICS() +# Check if the _mm512_clmulepi64_epi128 and _mm_xor_epi64 can be used with with +# the __attribute__((target("avx512vl,vpclmulqdq"))). +PGAC_AVX512_CRC32_INTRINSICS([]) + # Are we targeting a processor that supports SSE 4.2? gcc, clang and icc all # define __SSE4_2__ in that case. AC_COMPILE_IFELSE([AC_LANG_PROGRAM([], [ @@ -2060,9 +2064,8 @@ AC_SUBST(CFLAGS_CRC) # If we are targeting a processor that has Intel SSE 4.2 instructions, we can # use the special CRC instructions for calculating CRC-32C. If we're not # targeting such a processor, but we can nevertheless produce code that uses -# the SSE intrinsics, compile both implementations and select which one to use -# at runtime, depending on whether SSE 4.2 is supported by the processor we're -# running on. +# the SSE/AVX-512 intrinsics compile both implementations and select which one +# to use at runtime, depending runtime cpuid information. # # Similarly, if we are targeting an ARM processor that has the CRC # instructions that are part of the ARMv8 CRC Extension, use them. And if @@ -2079,76 +2082,58 @@ AC_SUBST(CFLAGS_CRC) # # If we are targeting a LoongArch processor, CRC instructions are # always available (at least on 64 bit), so no runtime check is needed. -if test x"$USE_SLICING_BY_8_CRC32C" = x"" && test x"$USE_SSE42_CRC32C" = x"" && test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_ARMV8_CRC32C" = x"" && test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"" && test x"$USE_LOONGARCH_CRC32C" = x""; then - # Use Intel SSE 4.2 if available. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then - USE_SSE42_CRC32C=1 - else - # Intel SSE 4.2, with runtime check? The CPUID instruction is needed for - # the runtime check. - if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then - USE_SSE42_CRC32C_WITH_RUNTIME_CHECK=1 + +AC_MSG_CHECKING([which CRC-32C implementation to use]) +if test x"$host_cpu" = x"x86_64"; then + #x86 only: + PG_CRC32C_OBJS="pg_crc32c_sb8.o pg_crc32c_x86_choose.o" + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && test x"$SSE4_2_TARGETED" = x"1" ; then + AC_DEFINE(USE_SSE42_CRC32C, 1, [Define to 1 use Intel SSE 4.2 CRC instructions.]) + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + AC_MSG_RESULT(CRC32C baseline feature SSE 4.2) else - # Use ARM CRC Extension if available. - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then - USE_ARMV8_CRC32C=1 - else - # ARM CRC Extension, with runtime check? - if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then - USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK=1 - else - # LoongArch CRCC instructions. - if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then - USE_LOONGARCH_CRC32C=1 - else - # fall back to slicing-by-8 algorithm, which doesn't require any - # special CPU support. - USE_SLICING_BY_8_CRC32C=1 - fi + if test x"$pgac_sse42_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then + AC_DEFINE(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use Intel SSE 4.2 CRC instructions with a runtime check.]) + PG_CRC32C_OBJS+=" pg_crc32c_sse42.o" + AC_MSG_RESULT(CRC32C SSE42 with runtime check) fi - fi fi - fi -fi - -# Set PG_CRC32C_OBJS appropriately depending on the selected implementation. -AC_MSG_CHECKING([which CRC-32C implementation to use]) -if test x"$USE_SSE42_CRC32C" = x"1"; then - AC_DEFINE(USE_SSE42_CRC32C, 1, [Define to 1 use Intel SSE 4.2 CRC instructions.]) - PG_CRC32C_OBJS="pg_crc32c_sse42.o" - AC_MSG_RESULT(SSE 4.2) + if test x"$pgac_avx512_crc32_intrinsics" = x"yes" && (test x"$pgac_cv__get_cpuid" = x"yes" || test x"$pgac_cv__cpuid" = x"yes"); then + AC_DEFINE(USE_AVX512_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use Intel AVX 512 CRC instructions with a runtime check.]) + PG_CRC32C_OBJS+=" pg_crc32c_avx512.o" + AC_MSG_RESULT(CRC32C AVX-512 with runtime check) + fi else - if test x"$USE_SSE42_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then - AC_DEFINE(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use Intel SSE 4.2 CRC instructions with a runtime check.]) - PG_CRC32C_OBJS="pg_crc32c_sse42.o pg_crc32c_sb8.o pg_crc32c_sse42_choose.o" - AC_MSG_RESULT(SSE 4.2 with runtime check) + # non x86 code: + # Use ARM CRC Extension if available. + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes" && test x"$CFLAGS_CRC" = x""; then + AC_DEFINE(USE_ARMV8_CRC32C, 1, [Define to 1 to use ARMv8 CRC Extension.]) + PG_CRC32C_OBJS="pg_crc32c_armv8.o" + AC_MSG_RESULT(ARMv8 CRC instructions) else - if test x"$USE_ARMV8_CRC32C" = x"1"; then - AC_DEFINE(USE_ARMV8_CRC32C, 1, [Define to 1 to use ARMv8 CRC Extension.]) - PG_CRC32C_OBJS="pg_crc32c_armv8.o" - AC_MSG_RESULT(ARMv8 CRC instructions) + # ARM CRC Extension, with runtime check? + if test x"$pgac_armv8_crc32c_intrinsics" = x"yes"; then + AC_DEFINE(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARMv8 CRC Extension with a runtime check.]) + PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" + AC_MSG_RESULT(ARMv8 CRC instructions with runtime check) else - if test x"$USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK" = x"1"; then - AC_DEFINE(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARMv8 CRC Extension with a runtime check.]) - PG_CRC32C_OBJS="pg_crc32c_armv8.o pg_crc32c_sb8.o pg_crc32c_armv8_choose.o" - AC_MSG_RESULT(ARMv8 CRC instructions with runtime check) + # LoongArch CRCC instructions. + if test x"$pgac_loongarch_crc32c_intrinsics" = x"yes"; then + AC_DEFINE(USE_LOONGARCH_CRC32C, 1, [Define to 1 to use LoongArch CRCC instructions.]) + PG_CRC32C_OBJS="pg_crc32c_loongarch.o" + AC_MSG_RESULT(LoongArch CRCC instructions) else - if test x"$USE_LOONGARCH_CRC32C" = x"1"; then - AC_DEFINE(USE_LOONGARCH_CRC32C, 1, [Define to 1 to use LoongArch CRCC instructions.]) - PG_CRC32C_OBJS="pg_crc32c_loongarch.o" - AC_MSG_RESULT(LoongArch CRCC instructions) - else - AC_DEFINE(USE_SLICING_BY_8_CRC32C, 1, [Define to 1 to use software CRC-32C implementation (slicing-by-8).]) - PG_CRC32C_OBJS="pg_crc32c_sb8.o" - AC_MSG_RESULT(slicing-by-8) - fi + # fall back to slicing-by-8 algorithm, which doesn't require any + # special CPU support. + AC_DEFINE(USE_SLICING_BY_8_CRC32C, 1, [Define to 1 to use software CRC-32C implementation (slicing-by-8).]) + PG_CRC32C_OBJS="pg_crc32c_sb8.o" + AC_MSG_RESULT(slicing-by-8) fi fi fi fi AC_SUBST(PG_CRC32C_OBJS) - # Select semaphore implementation type. if test "$PORTNAME" != "win32"; then if test x"$PREFERRED_SEMAPHORES" = x"NAMED_POSIX" ; then diff --git a/meson.build b/meson.build index e5ce437a5c..5833661d71 100644 --- a/meson.build +++ b/meson.build @@ -2222,6 +2222,23 @@ if host_cpu == 'x86' or host_cpu == 'x86_64' have_optimized_crc = true else + avx512_crc_prog = ''' +#include <immintrin.h> +#include <stdint.h> +#if defined(__has_attribute) && __has_attribute (target) +__attribute__((target("avx512vl,vpclmulqdq"))) +#endif +int main(void) +{ + __m512i x0 = _mm512_set1_epi32(0x1); + __m512i x1 = _mm512_set1_epi32(0x2); + __m512i x2 = _mm512_clmulepi64_epi128(x1, x0, 0x00); // vpclmulqdq + __m128i a1 = _mm_xor_epi64(_mm512_castsi512_si128(x1), _mm512_castsi512_si128(x0)); //avx512vl + int64_t val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); // 64-bit instruction + return (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); +} +''' + prog = ''' #include <nmmintrin.h> @@ -2252,6 +2269,12 @@ int main(void) cdata.set('USE_SSE42_CRC32C_WITH_RUNTIME_CHECK', 1) have_optimized_crc = true endif + if cc.links(avx512_crc_prog, + name: 'AVX512 CRC32C with function attributes', + args: test_c_args) + cdata.set('USE_AVX512_CRC32C_WITH_RUNTIME_CHECK', 1) + have_optimized_crc = true + endif endif diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 07b2f798ab..db40e6476d 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -697,6 +697,9 @@ /* Define to 1 to use Intel SSE 4.2 CRC instructions with a runtime check. */ #undef USE_SSE42_CRC32C_WITH_RUNTIME_CHECK +/* Define to 1 to use Intel AVX-512 CRC instructions with a runtime check. */ +#undef USE_AVX512_CRC32C_WITH_RUNTIME_CHECK + /* Define to build with systemd support. (--with-systemd) */ #undef USE_SYSTEMD diff --git a/src/include/pg_cpu.h b/src/include/pg_cpu.h new file mode 100644 index 0000000000..223994cb0d --- /dev/null +++ b/src/include/pg_cpu.h @@ -0,0 +1,23 @@ +/* + * pg_cpu.h + * Useful macros to determine CPU types + */ + +#ifndef PG_CPU_H_ +#define PG_CPU_H_ +#if defined( __i386__ ) || defined(i386) || defined(_M_IX86) + /* + * __i386__ is defined by gcc and Intel compiler on Linux, + * _M_IX86 by VS compiler, + * i386 by Sun compilers on opensolaris at least + */ + #define PG_CPU_X86 +#elif defined(__x86_64__) || defined(__amd64__) || defined(__x86_64) || defined(_M_AMD64) + /* + * both __x86_64__ and __amd64__ are defined by gcc + * __x86_64 defined by sun compiler on opensolaris at least + * _M_AMD64 defined by MS compiler + */ + #define PG_CPU_x86_64 +#endif +#endif // PG_CPU_H_ diff --git a/src/include/port/pg_crc32c.h b/src/include/port/pg_crc32c.h index 63c8e3a00b..690273506b 100644 --- a/src/include/port/pg_crc32c.h +++ b/src/include/port/pg_crc32c.h @@ -34,58 +34,43 @@ #define PG_CRC32C_H #include "port/pg_bswap.h" +#include "pg_cpu.h" typedef uint32 pg_crc32c; /* The INIT and EQ macros are the same for all implementations. */ #define INIT_CRC32C(crc) ((crc) = 0xFFFFFFFF) #define EQ_CRC32C(c1, c2) ((c1) == (c2)) - -#if defined(USE_SSE42_CRC32C) -/* Use Intel SSE4.2 instructions. */ -#define COMP_CRC32C(crc, data, len) \ - ((crc) = pg_comp_crc32c_sse42((crc), (data), (len))) #define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) +/* x86 */ +#if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) +extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len); +extern pg_crc32c pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t len); +extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); +#define COMP_CRC32C(crc, data, len) \ + ((crc) = pg_comp_crc32c((crc), (data), (len))) +/* ARMV8 */ #elif defined(USE_ARMV8_CRC32C) -/* Use ARMv8 CRC Extension instructions. */ - +extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t len); #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c_armv8((crc), (data), (len))) -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) +/* ARMV8 with runtime check */ +#elif defined(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK) +extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t len); +extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); +#define COMP_CRC32C(crc, data, len) \ + ((crc) = pg_comp_crc32c((crc), (data), (len))) +/* LoongArch */ #elif defined(USE_LOONGARCH_CRC32C) -/* Use LoongArch CRCC instructions. */ - +extern pg_crc32c pg_comp_crc32c_loongarch(pg_crc32c crc, const void *data, size_t len); #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c_loongarch((crc), (data), (len))) -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) - -extern pg_crc32c pg_comp_crc32c_loongarch(pg_crc32c crc, const void *data, size_t len); - -#elif defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) || defined(USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK) - -/* - * Use Intel SSE 4.2 or ARMv8 instructions, but perform a runtime check first - * to check that they are available. - */ -#define COMP_CRC32C(crc, data, len) \ - ((crc) = pg_comp_crc32c((crc), (data), (len))) -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) - -extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); -extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); - -#ifdef USE_SSE42_CRC32C_WITH_RUNTIME_CHECK -extern pg_crc32c pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len); -#endif -#ifdef USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK -extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t len); -#endif #else /* @@ -98,13 +83,11 @@ extern pg_crc32c pg_comp_crc32c_armv8(pg_crc32c crc, const void *data, size_t le #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c_sb8((crc), (data), (len))) #ifdef WORDS_BIGENDIAN +#undef FIN_CRC32C #define FIN_CRC32C(crc) ((crc) = pg_bswap32(crc) ^ 0xFFFFFFFF) -#else -#define FIN_CRC32C(crc) ((crc) ^= 0xFFFFFFFF) #endif extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); - #endif #endif /* PG_CRC32C_H */ diff --git a/src/include/port/pg_hw_feat_check.h b/src/include/port/pg_hw_feat_check.h index 58be900b54..3a73014987 100644 --- a/src/include/port/pg_hw_feat_check.h +++ b/src/include/port/pg_hw_feat_check.h @@ -30,4 +30,10 @@ extern PGDLLIMPORT bool pg_popcount_available(void); * available. */ extern PGDLLIMPORT bool pg_popcount_avx512_available(void); + +/* + * Test to see if all hardware features required by the AVX-512 SIMD + * algorithm are available. + */ +extern PGDLLIMPORT bool pg_crc32c_avx512_available(void); #endif /* PG_HW_FEAT_CHECK_H */ diff --git a/src/port/meson.build b/src/port/meson.build index ec28590473..0ba4a56194 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,8 +8,10 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_avx512.c', - 'pg_crc32c_sse42_choose.c', + 'pg_crc32c_x86_choose.c', + 'pg_crc32c_avx512.c', 'pg_crc32c_sse42.c', + 'pg_crc32c_sb8.c', 'pg_hw_feat_check.c', 'pg_strong_random.c', 'pgcheckdir.c', @@ -83,12 +85,6 @@ endif # Replacement functionality to be built if corresponding configure symbol # is true replace_funcs_pos = [ - # x86/x64 - ['pg_crc32c_sse42', 'USE_SSE42_CRC32C'], - ['pg_crc32c_sse42', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], - ['pg_crc32c_sse42_choose', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], - ['pg_crc32c_sb8', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], - # arm / aarch64 ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C'], ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C_WITH_RUNTIME_CHECK', 'crc'], diff --git a/src/port/pg_crc32c_avx512.c b/src/port/pg_crc32c_avx512.c new file mode 100644 index 0000000000..ba4defcefd --- /dev/null +++ b/src/port/pg_crc32c_avx512.c @@ -0,0 +1,203 @@ +/*------------------------------------------------------------------------- + * + * pg_crc32c_avx512.c + * Compute CRC-32C checksum using Intel AVX-512 instructions. + * + * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/port/pg_crc32c_avx512.c + * + *------------------------------------------------------------------------- + */ + +#include "c.h" + +#if defined(USE_AVX512_CRC32C_WITH_RUNTIME_CHECK) + +#include <immintrin.h> + +#include "port/pg_crc32c.h" + + +/******************************************************************* + * pg_crc32c_avx512(): compute the crc32c of the buffer, where the + * buffer length must be at least 256, and a multiple of 64. Based + * on: + * + * "Fast CRC Computation for Generic Polynomials Using PCLMULQDQ + * Instruction" + * V. Gopal, E. Ozturk, et al., 2009 + * + * For This Function: + * Copyright 2015 The Chromium Authors + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following disclaimer + * in the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Google LLC nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + + +pg_attribute_no_sanitize_alignment() +pg_attribute_target("avx512vl,vpclmulqdq") +inline pg_crc32c +pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t length) +{ + static const uint64 k1k2[8] = { + 0xdcb17aa4, 0xb9e02b86, 0xdcb17aa4, 0xb9e02b86, 0xdcb17aa4, + 0xb9e02b86, 0xdcb17aa4, 0xb9e02b86}; + static const uint64 k3k4[8] = { + 0x740eef02, 0x9e4addf8, 0x740eef02, 0x9e4addf8, 0x740eef02, + 0x9e4addf8, 0x740eef02, 0x9e4addf8}; + static const uint64 k9k10[8] = { + 0x6992cea2, 0x0d3b6092, 0x6992cea2, 0x0d3b6092, 0x6992cea2, + 0x0d3b6092, 0x6992cea2, 0x0d3b6092}; + static const uint64 k1k4[8] = { + 0x1c291d04, 0xddc0152b, 0x3da6d0cb, 0xba4fc28e, 0xf20c0dfe, + 0x493c7d27, 0x00000000, 0x00000000}; + + const uint8 *input = (const uint8 *)data; + if (length >= 256) + { + uint64 val; + __m512i x0, x1, x2, x3, x4, x5, x6, x7, x8, y5, y6, y7, y8; + __m128i a1, a2; + + /* + * AVX-512 Optimized crc32c algorithm with mimimum of 256 bytes aligned + * to 32 bytes. + * >>> BEGIN + */ + + /* + * There's at least one block of 256. + */ + x1 = _mm512_loadu_si512((__m512i *)(input + 0x00)); + x2 = _mm512_loadu_si512((__m512i *)(input + 0x40)); + x3 = _mm512_loadu_si512((__m512i *)(input + 0x80)); + x4 = _mm512_loadu_si512((__m512i *)(input + 0xC0)); + + x1 = _mm512_xor_si512(x1, _mm512_castsi128_si512(_mm_cvtsi32_si128(crc))); + + x0 = _mm512_load_si512((__m512i *)k1k2); + + input += 256; + length -= 256; + + /* + * Parallel fold blocks of 256, if any. + */ + while (length >= 256) + { + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x6 = _mm512_clmulepi64_epi128(x2, x0, 0x00); + x7 = _mm512_clmulepi64_epi128(x3, x0, 0x00); + x8 = _mm512_clmulepi64_epi128(x4, x0, 0x00); + + x1 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x2 = _mm512_clmulepi64_epi128(x2, x0, 0x11); + x3 = _mm512_clmulepi64_epi128(x3, x0, 0x11); + x4 = _mm512_clmulepi64_epi128(x4, x0, 0x11); + + y5 = _mm512_loadu_si512((__m512i *)(input + 0x00)); + y6 = _mm512_loadu_si512((__m512i *)(input + 0x40)); + y7 = _mm512_loadu_si512((__m512i *)(input + 0x80)); + y8 = _mm512_loadu_si512((__m512i *)(input + 0xC0)); + + x1 = _mm512_ternarylogic_epi64(x1, x5, y5, 0x96); + x2 = _mm512_ternarylogic_epi64(x2, x6, y6, 0x96); + x3 = _mm512_ternarylogic_epi64(x3, x7, y7, 0x96); + x4 = _mm512_ternarylogic_epi64(x4, x8, y8, 0x96); + + input += 256; + length -= 256; + } + + /* + * Fold 256 bytes into 64 bytes. + */ + x0 = _mm512_load_si512((__m512i *)k9k10); + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x6 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x3 = _mm512_ternarylogic_epi64(x3, x5, x6, 0x96); + + x7 = _mm512_clmulepi64_epi128(x2, x0, 0x00); + x8 = _mm512_clmulepi64_epi128(x2, x0, 0x11); + x4 = _mm512_ternarylogic_epi64(x4, x7, x8, 0x96); + + x0 = _mm512_load_si512((__m512i *)k3k4); + y5 = _mm512_clmulepi64_epi128(x3, x0, 0x00); + y6 = _mm512_clmulepi64_epi128(x3, x0, 0x11); + x1 = _mm512_ternarylogic_epi64(x4, y5, y6, 0x96); + + /* + * Single fold blocks of 64, if any. + */ + while (length >= 64) + { + x2 = _mm512_loadu_si512((__m512i *)input); + + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x1 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x1 = _mm512_ternarylogic_epi64(x1, x2, x5, 0x96); + + input += 64; + length -= 64; + } + + /* + * Fold 512-bits to 128-bits. + */ + x0 = _mm512_loadu_si512((__m512i *)k1k4); + + a2 = _mm512_extracti32x4_epi32(x1, 3); + x5 = _mm512_clmulepi64_epi128(x1, x0, 0x00); + x1 = _mm512_clmulepi64_epi128(x1, x0, 0x11); + x1 = _mm512_ternarylogic_epi64(x1, x5, _mm512_castsi128_si512(a2), 0x96); + + x0 = _mm512_shuffle_i64x2(x1, x1, 0x4E); + x0 = _mm512_xor_epi64(x1, x0); + a1 = _mm512_extracti32x4_epi32(x0, 1); + a1 = _mm_xor_epi64(a1, _mm512_castsi512_si128(x0)); + + /* + * Fold 128-bits to 32-bits. + */ + val = _mm_crc32_u64(0, _mm_extract_epi64(a1, 0)); + crc = (uint32_t)_mm_crc32_u64(val, _mm_extract_epi64(a1, 1)); + /* + * AVX-512 Optimized crc32c algorithm with mimimum of 256 bytes aligned + * to 32 bytes. + * <<< END + ******************************************************************/ + } + + /* + * Finish any remaining bytes with legacy AVX algorithm. + */ + return pg_comp_crc32c_sse42(crc, input, length); +} +#endif // AVX512_CRC32 diff --git a/src/port/pg_crc32c_sse42.c b/src/port/pg_crc32c_sse42.c index dcc4904a82..90d155e804 100644 --- a/src/port/pg_crc32c_sse42.c +++ b/src/port/pg_crc32c_sse42.c @@ -14,6 +14,7 @@ */ #include "c.h" +#if defined(USE_SSE42_CRC32C) || defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) #include <nmmintrin.h> #include "port/pg_crc32c.h" @@ -68,3 +69,4 @@ pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len) return crc; } +#endif diff --git a/src/port/pg_crc32c_sse42_choose.c b/src/port/pg_crc32c_sse42_choose.c deleted file mode 100644 index c659917af0..0000000000 --- a/src/port/pg_crc32c_sse42_choose.c +++ /dev/null @@ -1,51 +0,0 @@ -/*------------------------------------------------------------------------- - * - * pg_crc32c_sse42_choose.c - * Choose between Intel SSE 4.2 and software CRC-32C implementation. - * - * On first call, checks if the CPU we're running on supports Intel SSE - * 4.2. If it does, use the special SSE instructions for CRC-32C - * computation. Otherwise, fall back to the pure software implementation - * (slicing-by-8). - * - * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group - * Portions Copyright (c) 1994, Regents of the University of California - * - * - * IDENTIFICATION - * src/port/pg_crc32c_sse42_choose.c - * - *------------------------------------------------------------------------- - */ - -#include "c.h" - -#if defined(USE_SSE42_CRC32C_WITH_RUNTIME_CHECK) -#ifdef HAVE__GET_CPUID -#include <cpuid.h> -#endif - -#ifdef HAVE__CPUID -#include <intrin.h> -#endif - -#include "port/pg_crc32c.h" -#include "port/pg_hw_feat_check.h" - -/* - * This gets called on the first call. It replaces the function pointer - * so that subsequent calls are routed directly to the chosen implementation. - */ -static pg_crc32c -pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len) -{ - if (pg_crc32c_sse42_available()) - pg_comp_crc32c = pg_comp_crc32c_sse42; - else - pg_comp_crc32c = pg_comp_crc32c_sb8; - - return pg_comp_crc32c(crc, data, len); -} - -pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; -#endif diff --git a/src/port/pg_crc32c_x86_choose.c b/src/port/pg_crc32c_x86_choose.c new file mode 100644 index 0000000000..3ce8be11a6 --- /dev/null +++ b/src/port/pg_crc32c_x86_choose.c @@ -0,0 +1,57 @@ +/*------------------------------------------------------------------------- + * + * pg_crc32c_x86_choose.c + * Choose between Intel AVX-512, SSE 4.2 and software CRC-32C implementation. + * + * On first call, checks if the CPU we're running on supports Intel AVX-512. If + * it does, use the special SSE instructions for CRC-32C computation. + * Otherwise, fall back to the pure software implementation (slicing-by-8). + * + * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/port/pg_crc32c_x86_choose.c + * + *------------------------------------------------------------------------- + */ + +#include "c.h" +#include "pg_cpu.h" + +#if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) + +#include "port/pg_crc32c.h" +#include "port/pg_hw_feat_check.h" + +/* + * This gets called on the first call. It replaces the function pointer + * so that subsequent calls are routed directly to the chosen implementation. + * (1) set pg_comp_crc32c pointer and (2) return the computed crc value + */ +static pg_crc32c +pg_comp_crc32c_choose(pg_crc32c crc, const void *data, size_t len) +{ +#ifdef USE_AVX512_CRC32C_WITH_RUNTIME_CHECK + if (pg_crc32c_avx512_available()) { + pg_comp_crc32c = pg_comp_crc32c_avx512; + return pg_comp_crc32c(crc, data, len); + } +#endif +#ifdef USE_SSE42_CRC32C + pg_comp_crc32c = pg_comp_crc32c_sse42; + return pg_comp_crc32c(crc, data, len); +#elif USE_SSE42_CRC32C_WITH_RUNTIME_CHECK + if (pg_crc32c_sse42_available()) { + pg_comp_crc32c = pg_comp_crc32c_sse42; + return pg_comp_crc32c(crc, data, len); + } +#endif + pg_comp_crc32c = pg_comp_crc32c_sb8; + return pg_comp_crc32c(crc, data, len); +} + +pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len) = pg_comp_crc32c_choose; + +#endif // x86/x86_64 diff --git a/src/port/pg_hw_feat_check.c b/src/port/pg_hw_feat_check.c index 260aa60502..b2872fa708 100644 --- a/src/port/pg_hw_feat_check.c +++ b/src/port/pg_hw_feat_check.c @@ -11,6 +11,9 @@ *------------------------------------------------------------------------- */ #include "c.h" +#include "pg_cpu.h" + +#if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) #if defined(HAVE__GET_CPUID) || defined(HAVE__GET_CPUID_COUNT) #include <cpuid.h> @@ -135,9 +138,60 @@ bool PGDLLIMPORT pg_popcount_available(void) return is_bit_set_in_exx(exx, ECX, 23); } +/* + * Check for CPU supprt for CPUIDEX: avx512-f + */ +inline static bool +avx512f_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + return is_bit_set_in_exx(exx, EBX, 16); /* avx512-f */ +} + +/* + * Check for CPU supprt for CPUIDEX: vpclmulqdq + */ +inline static bool +vpclmulqdq_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + return is_bit_set_in_exx(exx, ECX, 10); /* vpclmulqdq */ +} + +/* + * Check for CPU supprt for CPUIDEX: vpclmulqdq + */ +inline static bool +avx512vl_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuidex(7, 0, exx); + return is_bit_set_in_exx(exx, EBX, 31); /* avx512-vl */ +} + +/* + * Check for CPU supprt for CPUID: sse4.2 + */ +inline static bool +sse42_available(void) +{ + exx_t exx[4] = {0, 0, 0, 0}; + + pg_getcpuid(1, exx); + return is_bit_set_in_exx(exx, ECX, 20); /* sse4.2 */ +} + +/****************************************************************************/ +/* Public API */ +/****************************************************************************/ /* - * Returns true if the CPU supports the instructions required for the AVX-512 - * pg_popcount() implementation. + * Returns true if the CPU supports the instructions required for the + * AVX-512 pg_popcount() implementation. * * PA: The call to 'osxsave_available' MUST preceed the call to * 'zmm_regs_available' function per NB above. @@ -154,10 +208,19 @@ bool PGDLLIMPORT pg_popcount_avx512_available(void) */ bool PGDLLIMPORT pg_crc32c_sse42_available(void) { - exx_t exx[4] = {0, 0, 0, 0}; - - pg_getcpuid(1, exx); + return sse42_available(); +} - return is_bit_set_in_exx(exx, ECX, 20); +/* + * Returns true if the CPU supports the instructions required for the AVX-512 + * pg_crc32c implementation. + */ +bool PGDLLIMPORT +pg_crc32c_avx512_available(void) +{ + return sse42_available() && osxsave_available() && + avx512f_available() && vpclmulqdq_available() && + avx512vl_available() && zmm_regs_available(); } +#endif // #if defined(PG_CPU_X86) || defined(PG_CPU_x86_64) -- 2.34.1 [text/plain] v10-0004-Mark-pg_comp_crc32c-as-PGDLLIMPORT-for-Windows-b.patch (1.1K, ../../[email protected]/5-v10-0004-Mark-pg_comp_crc32c-as-PGDLLIMPORT-for-Windows-b.patch) download | inline diff: From 6e8f557c857772b0c22607866d1b8930a67df05e Mon Sep 17 00:00:00 2001 From: Matthew Sterrett <[email protected]> Date: Wed, 18 Dec 2024 14:11:33 -0800 Subject: [PATCH v10 4/4] Mark pg_comp_crc32c as PGDLLIMPORT for Windows build --- src/include/port/pg_crc32c.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/include/port/pg_crc32c.h b/src/include/port/pg_crc32c.h index 690273506b..534d07dd5d 100644 --- a/src/include/port/pg_crc32c.h +++ b/src/include/port/pg_crc32c.h @@ -48,7 +48,7 @@ typedef uint32 pg_crc32c; extern pg_crc32c pg_comp_crc32c_sb8(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_sse42(pg_crc32c crc, const void *data, size_t len); extern pg_crc32c pg_comp_crc32c_avx512(pg_crc32c crc, const void *data, size_t len); -extern pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); +extern PGDLLIMPORT pg_crc32c (*pg_comp_crc32c) (pg_crc32c crc, const void *data, size_t len); #define COMP_CRC32C(crc, data, len) \ ((crc) = pg_comp_crc32c((crc), (data), (len))) -- 2.34.1 ^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: Proposal for Updating CRC32C with AVX-512 Algorithm. @ 2025-01-21 17:45 Devulapalli, Raghuveer <[email protected]> parent: Sterrett, Matthew <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Devulapalli, Raghuveer @ 2025-01-21 17:45 UTC (permalink / raw) To: Sterrett, Matthew <[email protected]>; Nathan Bossart <[email protected]>; +Cc: Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; Shankaran, Akash <[email protected]> > Hello! I'm Matthew Sterrett and I'm a coworker of Raghuveer; he asked me to > look into the Windows build failures related to pg_comp_crc32c. > > It seems that the only thing that was required to fix that is to mark > pg_comp_crc32c as PGDLLIMPORT, so I added a patch that does just that. > I'm new to working with mailing lists, so please tell me if I messed anything up! Thanks Matthew for fixing the windows CI failure. Looks like the CI all pass https://cirrus-ci.com/build/5105570367143936 with v10. Is there any additional feedback for this patch? Raghuveer ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Proposal for Updating CRC32C with AVX-512 Algorithm. @ 2025-01-22 09:48 John Naylor <[email protected]> parent: Devulapalli, Raghuveer <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: John Naylor @ 2025-01-22 09:48 UTC (permalink / raw) To: Devulapalli, Raghuveer <[email protected]>; +Cc: Sterrett, Matthew <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; Shankaran, Akash <[email protected]> On Wed, Jan 22, 2025 at 12:46 AM Devulapalli, Raghuveer <[email protected]> wrote: > > Is there any additional feedback for this patch? Hi Raghuveer, I raised one question and one concern upthread. I will repeat them here for convenience. #1 - The choice of AVX-512. There is no such thing as a "CRC instruction operating on 8 bytes", and the proposed algorithm is a multistep process using carryless multiplication and requiring at least 256 bytes of input. The Chromium sources cited as the source for this patch also contain an implementation using 128-bit instructions, and which only requires at least 64 bytes of input. Is there a reason that not tested or proposed as well? That would be much easier to read/maintain, work on more systems, and might give a speed boost on smaller inputs. These are useful properties to have. https://github.com/chromium/chromium/blob/main/third_party/zlib/crc32_simd.c#L215 #2 - The legal status of the algorithm from following Intel white paper, which is missing from its original location, archived here: https://web.archive.org/web/20220802143127/https://www.intel.com/content/dam/www/public/us/en/docume... This algorithm is the most portable and can in fact be coded with plain C, no additional instructions. The only disadvantage is that with pure C it's only useful on input with hundreds of bytes. But that limitation is not that different from the AVX-512 proposal in this regard. My question on this paper is about this passage: "The basic concepts in this paper are derived from and explained in detail in the patents and pending applications [4][5][6]." ... [4] Determining a Message Residue, Gopal et al. United States Patent 7,886,214 [5] Determining a Message Residue Gueron et al. United States Patent Application 20090019342 [6] Determining a Message Residue Gopal et al. United States Patent Application 20090158132 Looking at Linux kernel sources, it seems a patch using this technique was contributed by Intel over a decade ago: https://github.com/torvalds/linux/blob/master/arch/x86/crypto/crc32c-pcl-intel-asm_64.S ...so I'm unclear if these patents are applicable to software implementations. They also seem to be expired, but I am not a lawyer. Could you look into this please? Even if we do end up with AVX-512, this would be a good fallback. -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: Proposal for Updating CRC32C with AVX-512 Algorithm. @ 2025-01-24 20:34 Devulapalli, Raghuveer <[email protected]> parent: John Naylor <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Devulapalli, Raghuveer @ 2025-01-24 20:34 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Sterrett, Matthew <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; Shankaran, Akash <[email protected]> Hi John, Thanks for your summary and here are responses: > #1 - The choice of AVX-512. There is no such thing as a "CRC instruction operating > on 8 bytes", and the proposed algorithm is a multistep process using carryless > multiplication and requiring at least 256 bytes of input. The Chromium sources > cited as the source for this patch also contain an implementation using 128-bit > instructions, and which only requires at least 64 bytes of input. Is there a reason > that not tested or proposed as well? That would be much easier to read/maintain, > work on more systems, and might give a speed boost on smaller inputs. These are > useful properties to have. > > https://github.com/chromium/chromium/blob/main/third_party/zlib/crc32_simd > .c#L215 Agreed. postgres already has the SSE42 version pg_comp_crc32c_sse42, but I didn’t realize it uses the crc32 instruction which processes only 8 bytes at a time. This can certainly be upgraded to process 64bytes at a time and should be faster. Since most of the AVX-512 stuff is almost ready, I propose to do this in a follow up patch immediately. Let me know if you disagree. The AVX512 version processes 256 bytes at a time and will most certainly be faster than the improved SSE42 version, which is why the chromium library has both AVX512 and SSE42. > > #2 - The legal status of the algorithm from following Intel white paper, which is > missing from its original location, archived here: > > https://web.archive.org/web/20220802143127/https://www.intel.com/content/ > dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32- > instruction-paper.pdf > > https://github.com/torvalds/linux/blob/master/arch/x86/crypto/crc32c-pcl-intel- > asm_64.S > > ...so I'm unclear if these patents are applicable to software implementations. > They also seem to be expired, but I am not a lawyer. > Could you look into this please? Even if we do end up with AVX-512, this would be > a good fallback. Given that SSE42 is pretty much available in all x86 processors at this point, do we need a fallback C version specially after we improve the SSE42 version. Raghuveer ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Proposal for Updating CRC32C with AVX-512 Algorithm. @ 2025-01-29 06:46 John Naylor <[email protected]> parent: Devulapalli, Raghuveer <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: John Naylor @ 2025-01-29 06:46 UTC (permalink / raw) To: Devulapalli, Raghuveer <[email protected]>; +Cc: Sterrett, Matthew <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; Shankaran, Akash <[email protected]> On Sat, Jan 25, 2025 at 3:35 AM Devulapalli, Raghuveer <[email protected]> wrote: > > #1 - The choice of AVX-512. There is no such thing as a "CRC instruction operating > > on 8 bytes", and the proposed algorithm is a multistep process using carryless > > multiplication and requiring at least 256 bytes of input. The Chromium sources > > cited as the source for this patch also contain an implementation using 128-bit > > instructions, and which only requires at least 64 bytes of input. Is there a reason > > that not tested or proposed as well? That would be much easier to read/maintain, > > work on more systems, and might give a speed boost on smaller inputs. These are > > useful properties to have. > > > > https://github.com/chromium/chromium/blob/main/third_party/zlib/crc32_simd > > .c#L215 > > Agreed. postgres already has the SSE42 version pg_comp_crc32c_sse42, but I didn’t > realize it uses the crc32 instruction which processes only 8 bytes at a time. This can > certainly be upgraded to process 64bytes at a time and should be faster. Since most > of the AVX-512 stuff is almost ready, I propose to do this in a follow up patch immediately. It doesn't make sense to me that more limited/difficult hardware support (and more complex coding for that) and a larger input threshold should be a prerequisite for something that doesn't have these disadvantages. > Let me know if you disagree. The AVX512 version processes 256 bytes at a time and will > most certainly be faster than the improved SSE42 version, which is why the chromium > library has both AVX512 and SSE42. It looks like chromium simply vendored the zlib library. Input destined for compression is always going to be "large". That's not true in general for our use case, and we mentioned that fact seven months ago, when Andres said upthread [1]: "This is extremely workload dependent, it's not hard to find workloads with lots of very small record and very few big ones...". Given that feedback, it would have made a lot of sense to mention the 64-byte alternative back then, especially since it's the exact same pclmull algorithm based on the same paper, and is found in the same zlib .c file, but for some reason that was not done. More broadly, the best strategy is to start with the customer and work backward to the technology. It's more risky to pick the technology upfront and try to find ways to use it. My goal here is to help you make the right tradeoffs. Here's my view: 1. If we can have a relatively low input size threshold for improvement, it's possibly worth a bit of additional complexity in configure and run-time checks. There is a complicating factor in testing that though: the latency of carryless multiplication instructions varies drastically on different microarchitectures. 2. If we can improve large inputs in a simple fashion, with no additional hardware support, that's worth doing in any case. 3. Complex hardware support (6 CPUIDs!) that only works on large inputs (a minority of workloads) looks to be the worst of both worlds and it's not the tradeoff we should make. Further, we verified upthread that Intel's current and near-future product line includes server chips (some with over 100 cores, so not exactly low-end) that don't support AVX-512 at all. I have no idea how common they will be, but they will certainly be found in cloud datacenters somewhere. Shouldn't we have an answer for them as well? > > #2 - The legal status of the algorithm from following Intel white paper, which is > > missing from its original location, archived here: > > > > https://web.archive.org/web/20220802143127/https://www.intel.com/content/ > > dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32- > > instruction-paper.pdf > > > > https://github.com/torvalds/linux/blob/master/arch/x86/crypto/crc32c-pcl-intel- > > asm_64.S > > > > ...so I'm unclear if these patents are applicable to software implementations. > > They also seem to be expired, but I am not a lawyer. > > Could you look into this please? Even if we do end up with AVX-512, this would be > > a good fallback. > > Given that SSE42 is pretty much available in all x86 processors at this point, do we need a > fallback C version specially after we improve the SSE42 version. I know you had extended time off work, but I've already shared my findings and explained my reasoning [2]. The title of the paper is "Fast CRC Computation for iSCSI Polynomial Using CRC32 Instruction", so unsurprisingly it does improve the SSE42 version. With a few dozen lines of code, I can get ~3x speedup on page-sized inputs. At the very least we want to use this technique on Arm [3], and the only blocker now is the question regarding the patents. I'm interested to hear the response on this. [1] https://www.postgresql.org/message-id/20240612201135.kk77tiqcux77lgev%40awork3.anarazel.de [2] https://www.postgresql.org/message-id/CANWCAZbr4sO1bPoS%2BE%3DiRWnrBZp7zUKZEJk39KYt_Pu9%2BX1-SQ%40ma... [3] https://commitfest.postgresql.org/51/4620/ -- John Naylor Amazon Web Services ^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: Proposal for Updating CRC32C with AVX-512 Algorithm. @ 2025-02-05 20:58 Devulapalli, Raghuveer <[email protected]> parent: John Naylor <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Devulapalli, Raghuveer @ 2025-02-05 20:58 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: Sterrett, Matthew <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Bruce Momjian <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; Shankaran, Akash <[email protected]> Hi John, > Further, we verified upthread that Intel's current and near-future product line > includes server chips (some with over 100 cores, so not exactly low-end) that > don't support AVX-512 at all. I have no idea how common they will be, but they > will certainly be found in cloud datacenters somewhere. Shouldn't we have an > answer for them as well? Just submitted a patch to improve the SSE4.2 version using the source you referenced. See https://www.postgresql.org/message-id/PH8PR11MB82869FF741DFA4E9A029FF13FBF72%40PH8PR11MB8286.namprd1... > I know you had extended time off work, but I've already shared my findings and > explained my reasoning [2]. The title of the paper is "Fast CRC Computation for > iSCSI Polynomial Using CRC32 Instruction", so unsurprisingly it does improve the > SSE42 version. With a few dozen lines of code, I can get ~3x speedup on page- > sized inputs. At the very least we want to use this technique on Arm [3], and the > only blocker now is the question regarding the patents. I'm interested to hear the > response on this. Still figuring this out. Will respond as soon as I can. Thanks, Raghuveer ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-04-01 17:35 Fujii Masao <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Fujii Masao @ 2025-04-01 17:35 UTC (permalink / raw) To: Yugo NAGATA <[email protected]>; Laurenz Albe <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On 2025/01/23 19:22, Yugo NAGATA wrote: > On Wed, 22 Jan 2025 13:30:17 +0100 > Laurenz Albe <[email protected]> wrote: > >> On Fri, 2024-09-13 at 16:18 +0900, Yugo Nagata wrote: >>> I've attached a updated patch. The test is rewritten using has_largeobject_privilege() >>> function instead of calling loread & lowrite, which makes the test a bit simpler. >>> Thare are no other changes. >> >> When I tried to apply this patch, I found that it doesn't apply any >> more since commit f391d9dc93 renamed tab-complete.c to tab-complete.in.c. >> >> Attached is a rebased patch. > > Thank you for updating the patch! > >> I agree that large objects are a feature that should fade out (alas, >> the JDBC driver still uses it for BLOBs). But this patch is not big >> or complicated and is unlikely to create a big maintenance burden. >> >> So I am somewhat for committing it. It works as advertised. >> If you are fine with my rebased patch, I can mark it as "ready for >> committer". If it actually gets committed depends on whether there >> is a committer who thinks it worth the effort or not. > > I confirmed the patch and I am fine with it. I've started reviewing this patch since it's marked as "ready for committer". I know of several systems that use large objects, and I believe this feature would be beneficial for them. Overall, I like the idea. The latest patch looks good to me. I just have one minor comment: > only the privileges for schemas, tables (including views and foreign > tables), sequences, functions, and types (including domains) can be > altered. In alter_default_privileges.sgml, this part should also mention large objects? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-04-03 15:21 Fujii Masao <[email protected]> parent: Fujii Masao <[email protected]> 0 siblings, 2 replies; 31+ messages in thread From: Fujii Masao @ 2025-04-03 15:21 UTC (permalink / raw) To: Yugo NAGATA <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On 2025/04/03 23:04, Yugo NAGATA wrote: > On Wed, 2 Apr 2025 02:35:35 +0900 > Fujii Masao <[email protected]> wrote: > >> >> >> On 2025/01/23 19:22, Yugo NAGATA wrote: >>> On Wed, 22 Jan 2025 13:30:17 +0100 >>> Laurenz Albe <[email protected]> wrote: >>> >>>> On Fri, 2024-09-13 at 16:18 +0900, Yugo Nagata wrote: >>>>> I've attached a updated patch. The test is rewritten using has_largeobject_privilege() >>>>> function instead of calling loread & lowrite, which makes the test a bit simpler. >>>>> Thare are no other changes. >>>> >>>> When I tried to apply this patch, I found that it doesn't apply any >>>> more since commit f391d9dc93 renamed tab-complete.c to tab-complete.in.c. >>>> >>>> Attached is a rebased patch. >>> >>> Thank you for updating the patch! >>> >>>> I agree that large objects are a feature that should fade out (alas, >>>> the JDBC driver still uses it for BLOBs). But this patch is not big >>>> or complicated and is unlikely to create a big maintenance burden. >>>> >>>> So I am somewhat for committing it. It works as advertised. >>>> If you are fine with my rebased patch, I can mark it as "ready for >>>> committer". If it actually gets committed depends on whether there >>>> is a committer who thinks it worth the effort or not. >>> >>> I confirmed the patch and I am fine with it. >> >> I've started reviewing this patch since it's marked as "ready for committer". > > Thank you for your reviewing this patch! > >> I know of several systems that use large objects, and I believe >> this feature would be beneficial for them. Overall, I like the idea. >> >> >> The latest patch looks good to me. I just have one minor comment: >> >>> only the privileges for schemas, tables (including views and foreign >>> tables), sequences, functions, and types (including domains) can be >>> altered. >> >> In alter_default_privileges.sgml, this part should also mention large objects? > > Agreed. I attached a updated patch. Thanks for updating the patch! If there are no objections, I'll proceed with committing it using the following commit log. ---------------- Extend ALTER DEFAULT PRIVILEGES to define default privileges for large objects. Previously, ALTER DEFAULT PRIVILEGES did not support large objects. This meant that to grant privileges to users other than the owner, permissions had to be manually assigned each time a large object was created, which was inconvenient. This commit extends ALTER DEFAULT PRIVILEGES to allow defining default access privileges for large objects. With this change, specified privileges will automatically apply to newly created large objects, making privilege management more efficient. As a side effect, this commit introduces the new keyword OBJECTS since it's used in the syntax of ALTER DEFAULT PRIVILEGES. Original patch by Haruka Takatsuka, with some fixes and tests by Yugo Nagata, and rebased by Laurenz Albe. Author: Takatsuka Haruka <[email protected]> Co-authored-by: Yugo Nagata <[email protected]> Co-authored-by: Laurenz Albe <[email protected]> Reviewed-by: Masao Fujii <[email protected]> Discussion: https://postgr.es/m/[email protected] ---------------- Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-04-04 14:47 Nathan Bossart <[email protected]> parent: Fujii Masao <[email protected]> 1 sibling, 0 replies; 31+ messages in thread From: Nathan Bossart @ 2025-04-04 14:47 UTC (permalink / raw) To: Fujii Masao <[email protected]>; +Cc: Yugo NAGATA <[email protected]>; Laurenz Albe <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Fri, Apr 04, 2025 at 07:18:11PM +0900, Fujii Masao wrote: > I've pushed the patch. Thanks! Just a heads up, I fixed a pgindent issue in this commit (see commits e1a8b1ad58 and 742317a80f). I'd ordinarily just report it, but since we're nearing feature freeze, I just fixed it because my workflow (and presumably others') involves running pgindent on the entire tree periodically. -- nathan ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-06-11 02:49 Yugo Nagata <[email protected]> parent: Fujii Masao <[email protected]> 1 sibling, 1 reply; 31+ messages in thread From: Yugo Nagata @ 2025-06-11 02:49 UTC (permalink / raw) To: Yugo NAGATA <[email protected]>; +Cc: Fujii Masao <[email protected]>; Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Tue, 8 Apr 2025 12:28:57 +0900 Yugo NAGATA <[email protected]> wrote: > On Fri, 4 Apr 2025 19:18:11 +0900 > Fujii Masao <[email protected]> wrote: > > > > > > > On 2025/04/04 0:21, Fujii Masao wrote: > > > Thanks for updating the patch! > > > > > > If there are no objections, I'll proceed with committing it using the following commit log. > > > > I've pushed the patch. Thanks! > > Thank you! > > > > While testing the feature, I noticed that psql doesn't complete > > "ALTER DEFAULT PRIVILEGES GRANT/REVOKE ... ON LARGE OBJECTS" or > > "GRANT/REVOKE ... ON LARGE OBJECT ..." with TO/FROM. The attached > > patch adds tab-completion support for both cases. > > This patch looks good to me. This works as expected. > > While looking into this patch, I found that the tab completion suggests > TO/FROM even after "LARGE OBJECT", but it is not correct because > there should be largeobject id at that place. This is same for the > "FOREIGN SERVER", server names should be suggested ratar than TO/FROM > in this case. > > The additional patch 0002 fixed to prevents to suggest TO or FROM right > after LARGE OBJECT or FOREIGN SERVER. Also, it allows to suggest list of > foreign server names after FOREIGN SERVER. > While looking at the thread [1], I've remembered this thread. The patches in this thread are partially v18-related, but include enhancement or fixes for existing feature, so should they be postponed to v19, or should be separated properly to v18 part and other? [1] https://www.postgresql.org/message-id/70372bdd-4399-4d5b-ab4f-6d4487a4911a%40oss.nttdata.com Best regards, Yugo Nagata -- Yugo Nagata <[email protected]> ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-06-11 04:33 Fujii Masao <[email protected]> parent: Yugo Nagata <[email protected]> 0 siblings, 2 replies; 31+ messages in thread From: Fujii Masao @ 2025-06-11 04:33 UTC (permalink / raw) To: Yugo Nagata <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On 2025/06/11 11:49, Yugo Nagata wrote: > While looking at the thread [1], I've remembered this thread. > The patches in this thread are partially v18-related, but include > enhancement or fixes for existing feature, so should they be postponed > to v19, or should be separated properly to v18 part and other? > > [1] https://www.postgresql.org/message-id/70372bdd-4399-4d5b-ab4f-6d4487a4911a%40oss.nttdata.com I see these patches more as enhancements to psql tab-completion, rather than fixes for clear oversights in the original commit. For example, if tab-completion for ALTER DEFAULT PRIVILEGES had completely missed LARGE OBJECTS, that would be an obvious oversight. But these patches go beyond that kind of issue. That said, if others think it's appropriate to include them in v18 for consistency or completeness, I'm fine with that. Regarding the 0002 patch: - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) - COMPLETE_WITH("TO"); - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) - COMPLETE_WITH("FROM"); + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) + { + if (TailMatches("FOREIGN", "SERVER")) + COMPLETE_WITH_QUERY(Query_for_list_of_servers); + else if (!TailMatches("LARGE", "OBJECT")) + { + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) + COMPLETE_WITH("TO"); + else + COMPLETE_WITH("FROM"); + } + } Wouldn't this change break the case where "GRANT ... ON TABLE ... <TAB>" is supposed to complete with "TO"? Regards, -- Fujii Masao NTT DATA Japan Corporation ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-06-11 04:57 Yugo Nagata <[email protected]> parent: Fujii Masao <[email protected]> 1 sibling, 1 reply; 31+ messages in thread From: Yugo Nagata @ 2025-06-11 04:57 UTC (permalink / raw) To: Fujii Masao <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Wed, 11 Jun 2025 13:33:07 +0900 Fujii Masao <[email protected]> wrote: > > > On 2025/06/11 11:49, Yugo Nagata wrote: > > While looking at the thread [1], I've remembered this thread. > > The patches in this thread are partially v18-related, but include > > enhancement or fixes for existing feature, so should they be postponed > > to v19, or should be separated properly to v18 part and other? > > > > [1] https://www.postgresql.org/message-id/70372bdd-4399-4d5b-ab4f-6d4487a4911a%40oss.nttdata.com > > I see these patches more as enhancements to psql tab-completion, > rather than fixes for clear oversights in the original commit. > > For example, if tab-completion for ALTER DEFAULT PRIVILEGES had > completely missed LARGE OBJECTS, that would be an obvious oversight. > But these patches go beyond that kind of issue. > > That said, if others think it's appropriate to include them in v18 > for consistency or completeness, I'm fine with that. > > Regarding the 0002 patch: > > - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) > - COMPLETE_WITH("TO"); > - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > - COMPLETE_WITH("FROM"); > + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > + { > + if (TailMatches("FOREIGN", "SERVER")) > + COMPLETE_WITH_QUERY(Query_for_list_of_servers); > + else if (!TailMatches("LARGE", "OBJECT")) > + { > + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) > + COMPLETE_WITH("TO"); > + else > + COMPLETE_WITH("FROM"); > + } > + } > > Wouldn't this change break the case where "GRANT ... ON TABLE ... <TAB>" > is supposed to complete with "TO"? Sorry, I made a stupid mistake. > + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) This should be "GRANT|REVOKE". I've attached update patches. (There is no change on 0001.) Best regards, Yugo Nagata -- Yugo Nagata <[email protected]> ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-06-11 05:03 Yugo Nagata <[email protected]> parent: Fujii Masao <[email protected]> 1 sibling, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2025-06-11 05:03 UTC (permalink / raw) To: Fujii Masao <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Wed, 11 Jun 2025 13:33:07 +0900 Fujii Masao <[email protected]> wrote: > > > On 2025/06/11 11:49, Yugo Nagata wrote: > > While looking at the thread [1], I've remembered this thread. > > The patches in this thread are partially v18-related, but include > > enhancement or fixes for existing feature, so should they be postponed > > to v19, or should be separated properly to v18 part and other? > > > > [1] https://www.postgresql.org/message-id/70372bdd-4399-4d5b-ab4f-6d4487a4911a%40oss.nttdata.com > > I see these patches more as enhancements to psql tab-completion, > rather than fixes for clear oversights in the original commit. > > For example, if tab-completion for ALTER DEFAULT PRIVILEGES had > completely missed LARGE OBJECTS, that would be an obvious oversight. > But these patches go beyond that kind of issue. Thank you for your clarification. I agreed. Best regards, Yugo Nagata -- Yugo Nagata <[email protected]> ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-07-09 11:42 Fujii Masao <[email protected]> parent: Yugo Nagata <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Fujii Masao @ 2025-07-09 11:42 UTC (permalink / raw) To: Yugo Nagata <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On 2025/06/11 13:57, Yugo Nagata wrote: > On Wed, 11 Jun 2025 13:33:07 +0900 > Fujii Masao <[email protected]> wrote: > >> >> >> On 2025/06/11 11:49, Yugo Nagata wrote: >>> While looking at the thread [1], I've remembered this thread. >>> The patches in this thread are partially v18-related, but include >>> enhancement or fixes for existing feature, so should they be postponed >>> to v19, or should be separated properly to v18 part and other? >>> >>> [1] https://www.postgresql.org/message-id/70372bdd-4399-4d5b-ab4f-6d4487a4911a%40oss.nttdata.com >> >> I see these patches more as enhancements to psql tab-completion, >> rather than fixes for clear oversights in the original commit. >> >> For example, if tab-completion for ALTER DEFAULT PRIVILEGES had >> completely missed LARGE OBJECTS, that would be an obvious oversight. >> But these patches go beyond that kind of issue. >> >> That said, if others think it's appropriate to include them in v18 >> for consistency or completeness, I'm fine with that. >> >> Regarding the 0002 patch: >> >> - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) >> - COMPLETE_WITH("TO"); >> - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) >> - COMPLETE_WITH("FROM"); >> + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) >> + { >> + if (TailMatches("FOREIGN", "SERVER")) >> + COMPLETE_WITH_QUERY(Query_for_list_of_servers); >> + else if (!TailMatches("LARGE", "OBJECT")) >> + { >> + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) >> + COMPLETE_WITH("TO"); >> + else >> + COMPLETE_WITH("FROM"); >> + } >> + } >> >> Wouldn't this change break the case where "GRANT ... ON TABLE ... <TAB>" >> is supposed to complete with "TO"? > > Sorry, I made a stupid mistake. > >> + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > > This should be "GRANT|REVOKE". > > I've attached update patches. (There is no change on 0001.) Thanks for updating the patch! At first I've pushed the 0001 patch. As for the 0002 patch: + if (TailMatches("FOREIGN", "SERVER")) + COMPLETE_WITH_QUERY(Query_for_list_of_servers); This part seems not needed, since we already have the following tab-completion code: /* FOREIGN SERVER */ else if (TailMatches("FOREIGN", "SERVER")) COMPLETE_WITH_QUERY(Query_for_list_of_servers); Thought? Regards, -- Fujii Masao NTT DATA Japan Corporation ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-07-10 01:30 Yugo Nagata <[email protected]> parent: Fujii Masao <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Yugo Nagata @ 2025-07-10 01:30 UTC (permalink / raw) To: Fujii Masao <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Wed, 9 Jul 2025 20:42:42 +0900 Fujii Masao <[email protected]> wrote: > > > On 2025/06/11 13:57, Yugo Nagata wrote: > > On Wed, 11 Jun 2025 13:33:07 +0900 > > Fujii Masao <[email protected]> wrote: > > > >> > >> > >> On 2025/06/11 11:49, Yugo Nagata wrote: > >>> While looking at the thread [1], I've remembered this thread. > >>> The patches in this thread are partially v18-related, but include > >>> enhancement or fixes for existing feature, so should they be postponed > >>> to v19, or should be separated properly to v18 part and other? > >>> > >>> [1] https://www.postgresql.org/message-id/70372bdd-4399-4d5b-ab4f-6d4487a4911a%40oss.nttdata.com > >> > >> I see these patches more as enhancements to psql tab-completion, > >> rather than fixes for clear oversights in the original commit. > >> > >> For example, if tab-completion for ALTER DEFAULT PRIVILEGES had > >> completely missed LARGE OBJECTS, that would be an obvious oversight. > >> But these patches go beyond that kind of issue. > >> > >> That said, if others think it's appropriate to include them in v18 > >> for consistency or completeness, I'm fine with that. > >> > >> Regarding the 0002 patch: > >> > >> - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) > >> - COMPLETE_WITH("TO"); > >> - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > >> - COMPLETE_WITH("FROM"); > >> + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > >> + { > >> + if (TailMatches("FOREIGN", "SERVER")) > >> + COMPLETE_WITH_QUERY(Query_for_list_of_servers); > >> + else if (!TailMatches("LARGE", "OBJECT")) > >> + { > >> + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) > >> + COMPLETE_WITH("TO"); > >> + else > >> + COMPLETE_WITH("FROM"); > >> + } > >> + } > >> > >> Wouldn't this change break the case where "GRANT ... ON TABLE ... <TAB>" > >> is supposed to complete with "TO"? > > > > Sorry, I made a stupid mistake. > > > >> + else if (Matches("GRANT/REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > > > > This should be "GRANT|REVOKE". > > > > I've attached update patches. (There is no change on 0001.) > > Thanks for updating the patch! At first I've pushed the 0001 patch. > > As for the 0002 patch: > > + if (TailMatches("FOREIGN", "SERVER")) > + COMPLETE_WITH_QUERY(Query_for_list_of_servers); > > This part seems not needed, since we already have the following tab-completion code: > > /* FOREIGN SERVER */ > else if (TailMatches("FOREIGN", "SERVER")) > COMPLETE_WITH_QUERY(Query_for_list_of_servers); > > Thought? You're right. I must have overlooked something. I think I saw "TO" being suggested after "FOREIGN SERVER" when no foreign servers were defined. The attached patch still prevents "TO/FROM" from being suggested after "FOREIGN SERVER" in such cases. But perhaps this corner case doesn't really need to be handled? Regards, Yugo Nagata -- Yugo Nagata <[email protected]> ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-07-10 04:23 Fujii Masao <[email protected]> parent: Yugo Nagata <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Fujii Masao @ 2025-07-10 04:23 UTC (permalink / raw) To: Yugo Nagata <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On 2025/07/10 10:30, Yugo Nagata wrote: > You're right. I must have overlooked something. I think I saw "TO" being > suggested after "FOREIGN SERVER" when no foreign servers were defined. > > The attached patch still prevents "TO/FROM" from being suggested after > "FOREIGN SERVER" in such cases. Thanks for updating the patch! Based on your patch, I'm thinking of simplifying the code like this: - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) - COMPLETE_WITH("TO"); - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) - COMPLETE_WITH("FROM"); + else if (Matches("GRANT|REVOKE", MatchAnyN, "ON", MatchAny, MatchAny) && + !TailMatches("FOREIGN", "SERVER") && !TailMatches("LARGE", "OBJECT")) + { + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) + COMPLETE_WITH("TO"); + else + COMPLETE_WITH("FROM"); + } > But perhaps this corner case doesn't really > need to be handled? Probably I failed to get your point here. Could you clarify what you meant? Regards, -- Fujii Masao NTT DATA Japan Corporation ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-07-10 05:11 Yugo Nagata <[email protected]> parent: Fujii Masao <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Yugo Nagata @ 2025-07-10 05:11 UTC (permalink / raw) To: Fujii Masao <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Thu, 10 Jul 2025 13:23:47 +0900 Fujii Masao <[email protected]> wrote: > > > On 2025/07/10 10:30, Yugo Nagata wrote: > > You're right. I must have overlooked something. I think I saw "TO" being > > suggested after "FOREIGN SERVER" when no foreign servers were defined. > > > > The attached patch still prevents "TO/FROM" from being suggested after > > "FOREIGN SERVER" in such cases. > > Based on your patch, I'm thinking of simplifying the code like this: > > - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) > - COMPLETE_WITH("TO"); > - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) > - COMPLETE_WITH("FROM"); > + else if (Matches("GRANT|REVOKE", MatchAnyN, "ON", MatchAny, MatchAny) && > + !TailMatches("FOREIGN", "SERVER") && !TailMatches("LARGE", "OBJECT")) > + { > + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) > + COMPLETE_WITH("TO"); > + else > + COMPLETE_WITH("FROM"); > + } Thank you for your review! I agree with your suggestion and have updated the patch accordingly. > > > But perhaps this corner case doesn't really > > need to be handled? > > Probably I failed to get your point here. Could you clarify what you meant? I'm sorry for not explaining it clearly. Currently, TO or FROM could be suggested immediately after FOREIGN SERVER, but only when no foreign servers are defined. When foreign servers do exist, their names are correctly suggested instead, as expected. The patch fixed the behavior so that TO or FROM are not suggested after FOREIGN SERVER, even when no foreign servers are defined. However, I've started to wonder if it's worth fixing such a corner case. What do you think? Regards, Yugo Nagata -- Yugo Nagata <[email protected]> ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-07-15 10:07 Fujii Masao <[email protected]> parent: Yugo Nagata <[email protected]> 0 siblings, 1 reply; 31+ messages in thread From: Fujii Masao @ 2025-07-15 10:07 UTC (permalink / raw) To: Yugo Nagata <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On 2025/07/10 14:11, Yugo Nagata wrote: > On Thu, 10 Jul 2025 13:23:47 +0900 > Fujii Masao <[email protected]> wrote: > >> >> >> On 2025/07/10 10:30, Yugo Nagata wrote: >>> You're right. I must have overlooked something. I think I saw "TO" being >>> suggested after "FOREIGN SERVER" when no foreign servers were defined. >>> >>> The attached patch still prevents "TO/FROM" from being suggested after >>> "FOREIGN SERVER" in such cases. > >> >> Based on your patch, I'm thinking of simplifying the code like this: >> >> - else if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) >> - COMPLETE_WITH("TO"); >> - else if (Matches("REVOKE", MatchAnyN, "ON", MatchAny, MatchAny)) >> - COMPLETE_WITH("FROM"); >> + else if (Matches("GRANT|REVOKE", MatchAnyN, "ON", MatchAny, MatchAny) && >> + !TailMatches("FOREIGN", "SERVER") && !TailMatches("LARGE", "OBJECT")) >> + { >> + if (Matches("GRANT", MatchAnyN, "ON", MatchAny, MatchAny)) >> + COMPLETE_WITH("TO"); >> + else >> + COMPLETE_WITH("FROM"); >> + } > > Thank you for your review! > I agree with your suggestion and have updated the patch accordingly. I've pushed the patch. Thanks! >>> But perhaps this corner case doesn't really >>> need to be handled? >> >> Probably I failed to get your point here. Could you clarify what you meant? > > I'm sorry for not explaining it clearly. > > Currently, TO or FROM could be suggested immediately after FOREIGN SERVER, but > only when no foreign servers are defined. When foreign servers do exist, > their names are correctly suggested instead, as expected. > > The patch fixed the behavior so that TO or FROM are not suggested after FOREIGN SERVER, > even when no foreign servers are defined. However, I've started to wonder if it's worth > fixing such a corner case. What do you think? I think it's worth doing. This issue can lead to unexpected behavior and is something users might run into. If the fix were overly complex for a minor issue, it might not be justified. But that's not the case here. Regards, -- Fujii Masao NTT DATA Japan Corporation ^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Extend ALTER DEFAULT PRIVILEGES for large objects @ 2025-07-16 07:22 Yugo Nagata <[email protected]> parent: Fujii Masao <[email protected]> 0 siblings, 0 replies; 31+ messages in thread From: Yugo Nagata @ 2025-07-16 07:22 UTC (permalink / raw) To: Fujii Masao <[email protected]>; +Cc: Laurenz Albe <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; pgsql-hackers On Tue, 15 Jul 2025 19:07:16 +0900 Fujii Masao <[email protected]> wrote: > I've pushed the patch. Thanks! Thank you! > > >>> But perhaps this corner case doesn't really > >>> need to be handled? > >> > >> Probably I failed to get your point here. Could you clarify what you meant? > > > > I'm sorry for not explaining it clearly. > > > > Currently, TO or FROM could be suggested immediately after FOREIGN SERVER, but > > only when no foreign servers are defined. When foreign servers do exist, > > their names are correctly suggested instead, as expected. > > > > The patch fixed the behavior so that TO or FROM are not suggested after FOREIGN SERVER, > > even when no foreign servers are defined. However, I've started to wonder if it's worth > > fixing such a corner case. What do you think? > > I think it's worth doing. This issue can lead to unexpected behavior > and is something users might run into. If the fix were overly complex > for a minor issue, it might not be justified. But that's not the case here. Thank you for your explanation. It makes sense to me. Regards, Yugo Nagata -- Yugo Nagata <[email protected]> ^ permalink raw reply [nested|flat] 31+ messages in thread
end of thread, other threads:[~2025-07-16 07:22 UTC | newest] Thread overview: 31+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-10-26 21:14 [PATCH v5 11/14] hio: Use ExtendBufferedRelBy() Andres Freund <[email protected]> 2022-10-26 21:14 [PATCH v5 12/15] hio: Use ExtendBufferedRelBy() Andres Freund <[email protected]> 2022-10-26 21:14 [PATCH v4 12/15] hio: Use ExtendBufferedRelBy() Andres Freund <[email protected]> 2023-03-29 01:39 [PATCH v6 14/17] hio: Use ExtendBufferedRelBy() Andres Freund <[email protected]> 2023-03-29 01:39 [PATCH v7 12/14] hio: Use ExtendBufferedRelBy() Andres Freund <[email protected]> 2024-03-08 08:43 [PATCH v3] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-03-08 08:43 [PATCH v5] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-03-08 08:43 [PATCH v2] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-03-08 08:43 [PATCH] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-03-08 08:43 [PATCH v3] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-03-08 08:43 [PATCH v2] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-03-08 08:43 [PATCH] Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2024-12-19 00:00 Re: Proposal for Updating CRC32C with AVX-512 Algorithm. Sterrett, Matthew <[email protected]> 2025-01-21 17:45 ` RE: Proposal for Updating CRC32C with AVX-512 Algorithm. Devulapalli, Raghuveer <[email protected]> 2025-01-22 09:48 ` Re: Proposal for Updating CRC32C with AVX-512 Algorithm. John Naylor <[email protected]> 2025-01-24 20:34 ` RE: Proposal for Updating CRC32C with AVX-512 Algorithm. Devulapalli, Raghuveer <[email protected]> 2025-01-29 06:46 ` Re: Proposal for Updating CRC32C with AVX-512 Algorithm. John Naylor <[email protected]> 2025-02-05 20:58 ` RE: Proposal for Updating CRC32C with AVX-512 Algorithm. Devulapalli, Raghuveer <[email protected]> 2025-04-01 17:35 Re: Extend ALTER DEFAULT PRIVILEGES for large objects Fujii Masao <[email protected]> 2025-04-03 15:21 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Fujii Masao <[email protected]> 2025-04-04 14:47 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Nathan Bossart <[email protected]> 2025-06-11 02:49 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2025-06-11 04:33 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Fujii Masao <[email protected]> 2025-06-11 04:57 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2025-07-09 11:42 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Fujii Masao <[email protected]> 2025-07-10 01:30 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2025-07-10 04:23 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Fujii Masao <[email protected]> 2025-07-10 05:11 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2025-07-15 10:07 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Fujii Masao <[email protected]> 2025-07-16 07:22 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]> 2025-06-11 05:03 ` Re: Extend ALTER DEFAULT PRIVILEGES for large objects Yugo Nagata <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox