public inbox for [email protected]help / color / mirror / Atom feed
[PATCH] cfe-12-gist_over_cfe-11-persistent squash commit 3+ messages / 3 participants [nested] [flat]
* [PATCH] cfe-12-gist_over_cfe-11-persistent squash commit @ 2021-03-12 01:02 Bruce Momjian <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Bruce Momjian @ 2021-03-12 01:02 UTC (permalink / raw) --- src/backend/access/gist/README | 10 ++++++++++ src/backend/access/gist/gist.c | 21 +++++++++++++++------ src/backend/access/gist/gistbuild.c | 9 ++++++--- src/backend/access/gist/gistvacuum.c | 18 ++++++++++++------ 4 files changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/gist/README b/src/backend/access/gist/README index 25cab0047b..6b1f84634f 100644 --- a/src/backend/access/gist/README +++ b/src/backend/access/gist/README @@ -461,6 +461,16 @@ value. The page is not recycled, until that XID is no longer visible to anyone. That's much more conservative than necessary, but let's keep it simple. +GiST and Encryption +------------------- + +GiST uses LSNs and NSNs for concurrency. This means that unlogged and +temporary relations also need LSNs. GiST has a mechanism to assign LSNs +to such relations, but sometimes uses fixed LSNs and or calls +gistGetFakeLSN(), which can cause duplicate LSNs to be used. Therefore, +when encryption is enabled and fake LSNs are needed, the GiST code calls +LSNForEncryption(). See src/backend/crypto/README for more details. + Authors: Teodor Sigaev <[email protected]> diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 0683f42c25..5ca5f17db3 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -501,13 +501,16 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, * we don't need to be able to detect concurrent splits yet.) */ if (is_build) - recptr = GistBuildLSN; + recptr = !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(rel)); else { if (RelationNeedsWAL(rel)) recptr = gistXLogSplit(is_leaf, dist, oldrlink, oldnsn, leftchildbuf, markfollowright); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else recptr = gistGetFakeLSN(rel); } @@ -568,7 +571,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, MarkBufferDirty(leftchildbuf); if (is_build) - recptr = GistBuildLSN; + recptr = !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(rel)); else { if (RelationNeedsWAL(rel)) @@ -586,6 +590,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, deloffs, ndeloffs, itup, ntup, leftchildbuf); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else recptr = gistGetFakeLSN(rel); } @@ -1665,6 +1671,8 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel) if (ndeletable > 0) { + XLogRecPtr recptr; + TransactionId latestRemovedXid = InvalidTransactionId; if (XLogStandbyInfoActive() && RelationNeedsWAL(rel)) @@ -1690,16 +1698,17 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel) /* XLOG stuff */ if (RelationNeedsWAL(rel)) { - XLogRecPtr recptr; - recptr = gistXLogDelete(buffer, deletable, ndeletable, latestRemovedXid); - PageSetLSN(page, recptr); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - PageSetLSN(page, gistGetFakeLSN(rel)); + recptr = gistGetFakeLSN(rel); + + PageSetLSN(page, recptr); END_CRIT_SECTION(); } diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index 1054f6f1f2..a8b82483e8 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -299,7 +299,8 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) GISTInitBuffer(buffer, F_LEAF); MarkBufferDirty(buffer); - PageSetLSN(page, GistBuildLSN); + PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(index))); UnlockReleaseBuffer(buffer); @@ -451,7 +452,8 @@ gist_indexsortbuild(GISTBuildState *state) /* Write out the root */ RelationOpenSmgr(state->indexrel); - PageSetLSN(pagestate->page, GistBuildLSN); + PageSetLSN(pagestate->page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(state->indexrel))); PageSetChecksumInplace(pagestate->page, GIST_ROOT_BLKNO); smgrwrite(state->indexrel->rd_smgr, MAIN_FORKNUM, GIST_ROOT_BLKNO, pagestate->page, true); @@ -573,7 +575,8 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state) if (blkno != state->pages_written) elog(ERROR, "unexpected block number to flush GiST sorting build"); - PageSetLSN(page, GistBuildLSN); + PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(state->indexrel))); PageSetChecksumInplace(page, blkno); smgrextend(state->indexrel->rd_smgr, MAIN_FORKNUM, blkno, page, true); diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c index 0663193531..f783edd627 100644 --- a/src/backend/access/gist/gistvacuum.c +++ b/src/backend/access/gist/gistvacuum.c @@ -174,6 +174,8 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, vstate.callback_state = callback_state; if (RelationNeedsWAL(rel)) vstate.startNSN = GetInsertRecPtr(); + else if (FileEncryptionEnabled) + vstate.startNSN = LSNForEncryption(RelationIsPermanent(rel)); else vstate.startNSN = gistGetFakeLSN(rel); @@ -353,6 +355,8 @@ restart: */ if (ntodelete > 0) { + XLogRecPtr recptr; + START_CRIT_SECTION(); MarkBufferDirty(buffer); @@ -361,16 +365,15 @@ restart: GistMarkTuplesDeleted(page); if (RelationNeedsWAL(rel)) - { - XLogRecPtr recptr; - recptr = gistXLogUpdate(buffer, todelete, ntodelete, NULL, 0, InvalidBuffer); - PageSetLSN(page, recptr); - } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - PageSetLSN(page, gistGetFakeLSN(rel)); + recptr = gistGetFakeLSN(rel); + + PageSetLSN(page, recptr); END_CRIT_SECTION(); @@ -657,8 +660,11 @@ gistdeletepage(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (RelationNeedsWAL(info->index)) recptr = gistXLogPageDelete(leafBuffer, txid, parentBuffer, downlink); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(info->index)); else recptr = gistGetFakeLSN(info->index); + PageSetLSN(parentPage, recptr); PageSetLSN(leafPage, recptr); -- 2.20.1 --LQksG6bCIzRHxTLp Content-Type: application/gzip Content-Disposition: attachment; filename="cfe-13-rel_over_cfe-12-gist.diff.gz" Content-Transfer-Encoding: base64 H4sICMS9SmAAA2NmZS0xMy1yZWxfb3Zlcl9jZmUtMTItZ2lzdC5kaWZmAOQ8aXfbRpKfpV/R 9mw8pHiBN+msneigZI51ragk3mTz8BpAg0QEAgwOU0qc+e1T1d0AcZKUk+xk3/LZPIDq6q6j 6+qCzj13SQbUGHdNpvUVY6z0espgyGjbHPU6w06HKkaHDoZDpa0MyZXrkBlbkfaQKMpr/o90 FKV9eA5oXpMTL9QZAC1/sqhD/lPDn18vxc9m6L89PKMBe03uF2GdtNvkinowutOGt9dKF3E1 lL6iHM5C7SemB6/JD7fH96fvfiS6yRrtbsNjtup+ZJ7Kf3cac8sPiP9zSP0F0d3l0goODxuN xiH8cALP0lqa7bpLeLccn3lBUyep1yf43yW1Q+J7ekuj+gNzjBbVdeb7LUTN3zaDALzfJjV4 Nfgk20ZpoWUbfCiMandgVCH8AlbO31Z0zsRMuKh2yaIWjK5aHlt7VsDwO4xA8GEZekcLPMbk xwZ9Gc0bcN+V3ELwHqkVU+uvOL34EfEX4Qcl6AOPOj5dth5tdy65iuCdLLjmuoEPwKvNtwR4 jjc6DShgbPmB68VcjMGHucXr3tMqcFtX9IGZls0y+pBHL8C10GSODsgrDltXJXh72OP6UPxK 41m5frCkfsC8xNeYxzkmSGpwXhOGwMdyHoN3UKGydGVG2K5ObfguhTLK8SGCRya0dHf1ZFhe rCOd4ZYZ+AiPWY4VpOVYAr+Sy0opudxK+VkCWAtn0dxjfmrLphhlObodGiy585qL3Pbux4uK 4NPyXKTkSTojkhBdNCTBWU7CIjlFG4c0igdEXF1kSQD4To8gH32iL6gzZ0addMcdIjaS5Tp+ pVatk96QGMxm4kKjenhoWKZJGmD4AkJbpVZOK711aDkGeyRGd6gzZg773WZT15Qepfp40CVt RRn0emhCt2A/BP5sm+Hrr0mjPRzWQYPwo60QuKLZ3CKy5Sp4qtwxmyJJhC+mekgOyBH5cHlz oZ6dHM8m6und5Ph+QlxPXLw/uZzdnkZXPaa7ntEk4ESYx0zXY3WyZsRhzBCIgoXlE/aROWS9 wDdqqzb8tN8sQWGX1G5ysNZh7eAWZDRxuEJMnZVNdVZZsoCijOtkej29V89v7t5ff3NVJ9GS p/4t85bUYU5QEauvAyJ8gWBPLm9urtSryf3x7fHFRD25fH99U/0SpsOJZiw4XTD9wQ+X+clK RoIeHviw77nBF/M13nqGiteyKyxCUYfxuLIK6JhHjqpkM2PghYyvDUynClsAr1ZepadovMV3 1XNcgzXxLTNnWht3OFFtB4DUzL5O+2Z7aGigmX0Dvipjpd9Pa+YOREI/dwChlvaVbh0CmRr/ 7KGa4k0um8DlDIk1FSKPOplZvzBigov0VwBSJxfT2f3sHpXyCAf6AQQ3nOWWSSqWr3KVr3IZ gNauAo+8IS/OYdNLpQPEE4dqNjPIV+QCMJzggMvZNXnNB8ELfpy73ga+UqSIsLgqasvBAbN9 RnD2COwatoX/3fGlgOGqGi8F1/zh0p3PVrYV4HptRs1InYVGGwBSJ65teLC/H/g3x3fqxGZm oC9gsWAS0yOW1HswXdt2QWfniyCzrELiM8t6DslE4D5sHBz8im8HW2jH2zuJbxw8m/rUkDz5 eHsX/emlPYcBEfYC6i5YcA4xDmDjwBz2t7wG5MAIchXsKqkIIOTClwS+8ze4AO+Ntw57DIRq S+OGCPgdYV/EHICOb7Rht46utT9U6p3B//mN1pA832PDkT0VUwAd3Jimz4LrcKkxj4A9tF24 AMQoGzWTF39o/8i31rYhtfwIIvDgapIDp/631LaMCmg4gDrhMtovcl0xDuVHQB1DfSnvJqZt y2u/HTZKdtw3KwPEVxFhanL7RJPUYzLqxArCFfzG9yRkcgcKE7MXSTVOUW0bQbU8PUSQ85fZ x7ViM57ha21fttbKuEqkvfjTrHdts492kkzShiZtY0hkCyCQ0ewHx63uH5pEmfr2iEFCySCF jrRRpzfqslGzqRldA4Jnpo33D1IibLsjlQgSrWiv36tDbo0f7U5kRFW+IszXOWgFTSW3bjM0 kuSI20q0Q5EcblbMmUFYV+F3Gm/5eGn5syyWIILbzzGo4An3UoTsGoRWtI4IZufEDz2GeHxC 4ctHClPUCXUMVEYWI67zu44bYNUHEDpBU4T3xz5mJJUXuDnumH4bgE2YOh+5VUBCLwoJrVbJ q1eokkBDpYTmT5+4zm5BQl68SfFFEFaQcOTYfHU8vd6ed2SZlshAUPrq3c3N/e7kIzdx0dhN 8pGZNJGGpBecwRJlILnZZP4hooMh6nV/1C3Xa9W0Q3+heowaTypHVqrphabiWdHA79PeP0p9 n6O/sdLuobVZtS3R1TJl/R0aCmsT1vnLhIyK9LKehEMtg2ATrOPeSshH10k6093pErLF2ELL nAGSDqE3puPRYDgcDZtNRhn8MpUe3ekQsshK/UEWkNdYlE4f9w3/HOK+URFCpZB96BASA4LA zwTWJ3DrQcaKpuWBf8ELdRJaTtDtEIf/9KsyKI45V+A9pF/+PVrCNYPYVK5il7HKgib0Ypsu xIPq5BfmuVgRhTiJ1olJIfbYTzHyZfdiKeXgpHpAsNDW6MjstpvNXlfrmR1Gx93d6pHHV64h eVhUkq7QEfwYo4rACBVvqxIU9IN/ChsaBwsp8R+Uxg6ej6UjNcobi43GBlaGpmkBRfEnGIct piQxU3JEAjeKeIdpya+kcHyx0dmsoFTVctjqXAaD0RhlMBh3ZDHUo2shBFEzLZBBnbyD+/fh ymYEgnQhEnQL+Lm/NNAVffjwgZy5UYmUBG7kewglWJ8mrkmCBePGkmBJ9Svpfvbc2nH+sFOA kGWlBbhFgtsNQanYPktuO/d/9hitcANmgOTO14ewxYfGwOzCzu+MKe0Pxmx3ppBFVrrts4Dc MQw6vPg+iPRNC/4/FN/v7yaTuP69Z9k9O+ZZBffU4H93qT13drtDXQSU1NKO3jOG5oB1+s3m wND0tjYwO7uL7jlse6iphOR2sT+st7tgGPGzzwMYLYAdzZnPmXRy/93GMB6tpWlEscrYLhnP yMSfHPwaVQNerKUN0IK1r6L35/Eul1ThLfKG8Mi6CugxilIqJ5en72ffiwINGFPDdf4eEJ9B sC6VimClFIAbiEMYUWE+i6EBeBP6F49Ne591MuYttWBR2pKiiqdJKnITMNVqG6hYRwuZsIGL Ffc37k5+R6iXoiK5azdx/n7pAAC2jnZuhoLGhEK1zMPJDaHodNBp9/oKazZHBh12B9Ro6zs3 RAG+0i1RACuM94gH9fAxwi0B93cZb+HALdCsR8BoOXPe/gJsFzYYnTtYbY/5K9cx+G1Io2FD CWPOs1OByGPA8ydmbDXSn22gZ7e8NrD/8aiYqXzYvoa6EEPOXufKEm1RlmgPx/tJAjb8pfXA 1pbP+MZGvnsu5PU8ZxM7e7bCRHvqWAHSnKbwcnJ8LgO3P4Ht+xeF6iVDnsnufA1oC6tHA87q sfL5rHZCMKQBRsz+s9lOPkW/rr+5vJz9eVJA9M+UQmbIM6WwGb1dCrssaqZ3q9CipWGkJe2O dYUOtL4GoUWnOxgOTYP2d6e+GVylVjQDJ8rkokrOayKH5G+yKYe8xA5B1xGNK5CL0ICqYWDZ fnPxMgUmGoQesN8qfYc9Mj0MXK8FFhv4tgRJI0QtNzZqLkqNXlq+Tg2IgzPXV3MVqNBZ7jL6 TLzIa6S9tsKLpL12WxR7TkDHZ9gbh2F65aNr8QNQHi+dCgqxCliRTjOGfo8JI6/c4HawqG39 wk54NpwodspBGEMFoNOWA2G8aWCdsNEWEVapvhQ372m7YaJSSZ8qem/U64O+sLHJxr3RqG2W 60sxrry+FMMhazvc4cJ7Tl9kYyEISHdtscmzurIBCZ5WWRFKhcPmgYZOHeo9PUNdbEtb/dxa /exbc2D/nrqEmYPfWmJrY+joXLNLJZXvm9S2349SWo0O6ZgaVGk2h2Z7pHQUbbxFQnk8eenk YfhO7nXBH9TwgxevIkt76q6eZgKyMruae7GDAJRg+ZJXDD+obyo6eHxQ4UDgNTBviEvFUVmw Ko/IefqA5nnqf8s8y7SYMeEBOTOSIWnyUPp2+q2KGfN3x3fX0+sL8Cl45W5ye3N3r2L3gjiA 3o05WlviZNZj9op5PjgxUBRIU96Qu8nl7eRuBjZ+cn06UeH71fH15Po+OUrSts/C0DdARr+C 7Kwyubu7uZNZwEGFeZ4OWoVXT2/OJurZ8f2xenpzd/fN7f3krBrBEYBb+vPKS0ucUIh0xnII r9CQL0IMU71ILF/4L0VtrDdoc2s96AjP/9kiFowNfaauqS3oSaXbAFWQY2fVQAogrgxvOwbJ igm8/DPE9PyTEITclvtk+5q1bXejvdyB1KajdMYG5DljsL2DwWi4JfHPYinYyRkIHt51eXTX FY4rcFeiYcew8Ly/2Wzxf4cksmL/UUmCVDn6CF9zbrsatZEbNyf/mMH4/wH+SwPqwg9ywB23 ixBphIAmRhevllvo5vJhJ1vj/u9CcqO7h6BvvK2XLEG/Ij7KnDJ+NZuKPlT0drfN2pzJLYN9 bGEAW87QeAZkqFJXgJ91bD8Hl1VrHTX+qNdhjRzhfxLPF13AHcl72iCCrOJTGwpYFtGlPfuv S3LB5ULOsNjnrjA0IheeG66i8RzH9AxUf3o+PT2+n95c4yXYNduIlQP/UPJa2ECTiLOiTnN0 zckbKR9bSzlm8FMeJNKWY7q5Yal+9JcFd3iouiMOSNzLNZ8X3jUNuZCW4PR3jIAt3BT2L2fX wrpBnoTVMnEQvYoSloYGeh+4mGFZOtYGwJhSsIMPjGPDs/E19RElP0GQBQRKHBftHKZgjOoL kXJJHv/NgA3rMHLyzTmYPhU8zmz6/eTgoD3AZWJsa+kkdDC8AaQ8KQES1U11TLU+/pAe/CPv fbqdn1qrBfNOg0dyBKErGGf8+oZgrvNl/v4ZS9+PJ8eYGatz8jQoPXVlW5Ux6WQJxJN27NlU y1djropeLZzmsLYl2haxew0b03gLADnl6vCePZGjB/YkGr542FDegOWxIPQcAQpjgFoM9C9Y AFgq768u7tT3k/9Wp2fg9i9lC1mSdRjicqapevCoCvFWbi/U0+ntu8mdejyZqaf3dymyK2nZ QUIJ8zbewlsaTFy1mVONk01JTbyAqugEc+cy9MC42sGOCMuRPEsWTXlx6zF4uSEjlvD/EhlR fLChQ6xgDzoMVkjHb2LrEilcvm/nFh6j8D27toKFqCdFQQ/KGHeZUC45jBc3EnpbrpmFlXOu gRl2iJ68PJP4smokERyJ3ktkCk4L4Ziv8j4ehFRdT/WtpWVTT2qzE2DACVaMSaWVzS4bpXwh NqyUMpo1LFHGJRbRUUQW9CO21mDAiZ02wqyBVWqsMoChH1LbfuIDBKooUK2ghGDpVY6hSUjl BswWe9SZkBO251xYs3tuDuUsPoP4CiS9blYFsnfuGjyfF5+GyTW5DkyNOGxbrsNMnQBEJ2l8 ksB1BTJY+BPhMaAPUqfiCCFpyjmWLIVglw2Xg/l0KWlEuAJuAAH8RPin0BfIwfrjMpp8lBh6 bPsQEK8XGMxshkfa5JM5DMNohzc4Ie1PJLCWCZdj+ZIa1+BZTr1gNRt0sHSQQ2phIW9o5Qhh EoGMM0vjJ4+4aDxsFMzEqvVm8VxePOziooc3ITcgB9wUMhu2sEvQe4KsIEu5RqZazDYACBnJ AqYHAhcnxsfHBPzcwSeeEq35CvmMmCjhcOrjdpG9X6CQaykOcSAkHvjCUcsE11tSzxMqvuGO v3BD24DpeANcVuuf3+9YrZJPn8iLLV4r3nE8K8F4YNMkAJEABY4wHYwa8dGsQaaD5Uac5WZF fw7ZGW+xoWKRUiZYj+QF+OlZHYQdb0CxuWw6JxCFiGY50EmxlemGTM78SKlA8OhMohM0gUPq toxsFnSF6zRBR+UGwa0km+2EJAUu1wG7YLhrsHEgPkhT5yHEaXhQsvZceLfkUJD3kgfAaDDS yonrEMh8d8lKlFxwT0hfLjOItAmAonht+m1doBLKCqZmCXToaI7Q38faUmZhyRt0Q5HEZ0JM +ESF7PYDWV0dfzi+nF5cV1B4rlnJy27TBYuVc4lL3BZoGm83c4N8pAJE8o0GV0pHo7x98or8 kw87vzy+UE+m91fHs/d8hUqEYT9triaCpEKlBnTkRQnDqlGan+oYzXWAFm8WOXNpGJksFpVH iaJ44djWA7OfKi82cYxEtvGLkHRti2cqIoLM++vAq6NxsZxVGJCow0e8UDduzIQrx67UxP1o LPjqOcQhqbGvhA/n9yGJCm1+/lgEmQvuxXo+pqHSAX9yFRgM1IlSrZZGWVFfE99mX4QvN7WT RIAh1osqVkR2MhY7Y58Vi8lhf5FYbFugJcPmdKD1ZymyDHw38/4bFdkNAxz8l1Vkyas/SpFP XX5cpQe4yOi4NKHRuO5ESgw/fldOvC0hLtZr0OocUwWtV2wJRrmSZzlR6pnKQhy2CKJs3yGV EdGeAnB5XDJLttRXT5VVnbyqcEreQSzFPLF/wCOtDNXGpzulT0SIje8RPfErUntTcj+eXdS7 RZWFVHolK5D7SOJKMDM/UfJmOjS7ot4DL/NAzBInJxjNTr/FOFrGbBAy2jzYiEAgTA0h9fRk KIV/UMWmsCgRfICBs1F8SwqIQscCnw3BEnaRTn0Rr0BQpbEAEGBsFrKvNgFJqfvlqn60qtXw wcTHkZKmAyIxyHkerWW4hIl/Di0MyuUiucKCfWiAfcBIrKP0RlEcb2K4aPEOF4EIe73hl4fZ GbDdj2Ol38rru2V/l0PbByo65DaHw/ZwoHSVZnPUa/f6I6U/3tIuVIYtX/0tgxQPACii/18p O+rGSjQXQvHBpLUqvi6qm887q2xZtqx45s4waRgsCm/w9+iEu93tiiaQbncoDgpuY4KvQBcr 4Mogep/rdWk24PvHH34Uj7YtQYlv5/dsucISmR8dYG+Kbn/IsfeBbrs+S97b1jlR9hdctH2g pGKZg27HHGpdjTWbdDDsGN3xaLhFscqw5RWrDJI/NMAbn6QYkmLLVLALDsLlZ8nd6Hh3f80q abnI6J5m4eE6f+xhyzk5kjbu4mapjbvt6EiZGkIPVKH8mdNG3lnD9S19tlcn5+L4TxMmSpwE iieFRSc6uALe9ckN2Jx6GnolPGJGgxQ9EV52ECy7UIElJ+LRFU06AeFlnzW04Dj5M8+TDzar IJuL+x4v/yoG4dr54RhOenKlfj+5u1FvrlUeA2FVAvtQVYMugQxD9LBWZZ+Gwh98rcGXYb3D /17HOT4VKORXER9nzNfJEf5hhvQpMVDsVGVfIzmIuAPuCIa9M7BSzi9U5KPP0puVFtx/FS2+ PEUFB8b0hWPpvMAIJMXMQydmG5tTFYo9bGAoUCTYcxDwwsLffYnGfaBP4PXlkQoymqxl3Qfc ecDAL4c+rz4knuKIEXJvJz0xEnjv8mYgIDERGuIpXoF2hWbjbUDnzQJd4aEZb9+FbD1JXGJY pBfiwcHc3Imz7cz8Eq4MFT5+Hj0h/gysWaqSOEUDMW93FQ/MR1UZ2aANUjeYjxEIRD0zLos1 EwU3XifywRZAdMKVxxWhB0cWPVHkYohE/lXdtfa2jVzRz/avYL1wV4olWdRbSXYLO5YToc4D fmxQIABBimSsri2qopSs0ea/976GM3xKjrNAu8jaMjkcznvO3HvO1XIV4UoZxGIx8gI2GOGZ 4hY3Mk+gyzwxed5v4jVnRaqcxNoGXb1czb+wNIiWTywjPOBHSanvcbuWwd38QS2V7IQr2DGc eeSs5/gamsvTd1fXl8719O3EuZpcO69uLi9hQNQgEQyU1VpYnd0ec5m7vZEQeGi6qinJ8zWt UBTtGXJOYDLhXtVaroIv82iDsRjgcoTrNPlN8FWz318kD2TvQPq/GvkoRZSSRIlF9Staky0k VGLXbFNEFZFBKM6aHI5pKSmbSSX6KFZBmo+anaBeWkAPKXlxpgv39gzKaEoEtTNyMSLJVWII nU4xHnpet98fBW631bLbXRgPw9Gogl1Snt9W/GKkJTDZH5MEin7BhQu8zQPuBIUdRbt8wYae OeUam983/MFOV2l91kk0ob8RgIR+EFoXp2eT05vXcCOEKbxYh7V4DSdOeNfBxal1cnHx/pVV O9w0Dn34V7cO/U+Lg0YiY8vRlVrQfe+ytCVVJtohbQJu+IuF9D+82jB5zqFv6VFtAEOEw/0g +pUId1uYgJgsWgaLmjlAV0y9evtwyl059RXV6n94P/3uiY8al6oNtGTWP23OQzedoNsJJ32r Re6hrZM9EwayeLalE6lggu7IG868YOa2WmHP7o6GrudXULxLMiuf45mENNQ7NNKZNWgg/peb BYBav3X7q3kxfoiPmU39K460wgONqHB+BJ9bEbetPE8niQjJx5FhD5ep4YBXKbkpDP1wFd3D X+q4u47oDzGuzTarWCm11WP4ADYVejDld6Lg/qZCpVEgnytnenU5eV0L4cQerx2E4xLwCfNy 8OFcbhJmqTyBwZuk152vguAMivXHHdLucGTSytyhKtud76+zgkjRIiCu3T6Zgdl22dTl0zly 6VSWVJn9o90SystlcXBwcXA4AwvPM5gUp/gzXv6w6mxxh2Ec+iJu6XYA/RzZ3R5TXHctocpq QaYszDoKQweuRGLdTy5wbBR1uckCGYqhBkiGmkpW5xq6j9kjTfAQjW1BvI7JPtlUjK1X7z/8 wzm9OSezKhpRn1kig0wcClQHWtfI+2vV8g1k/U2esp5bOotEwkdIV5WRjsVUi6RIgL6na3F+ esECSkbeZtht7qN4jeyINay090h74FbuyY7f1uNqp1b+CVaaeSjFsm7iQLSfvCdQDBcAich2 cNFI6d6x6VrvdwS48fMv8mQt1YA0b7IpUu1XTzcKyvoJf1J4WNFHcTjV0eiRlUvMEbwoOczw dr668zVD9drHk+m1M/kNtiOHin0+vZjA4f3kjDWvPPig4Io6H/qN5KyTr2d1+lytC8uFqle+ SRG9OMOXVruQpE6tM+Y4KONBo9N+ZPPsGcx1BhcIB2iaEI3308Fh/OnguXV4fwDrm+RTr+fK hy5rztCDh39/kYIzH0kKH/BUNDFLcsLzcNSt1jgX4D7J3qzNAnalz4EvmSi0hcHbYjbqoy0f j434OGk3ZzAwyT/J3E1cIROegWRDb0xxZhTEwdoUrHNHKmRe4vzRKmm16nFEOnFM6RZJxNNH e7s42Hn7MJLvIDamZ+DUTWufdZx65TcOGwfdu4jQ71A24ComwsfL6fVEdzbaoOu8dtb8eG2O ba41xVKSjztCriSOdgX2UWlUtKH2bNgf+cPRuNUaj/tdz+sEVRqp4ry24K0kHR3lycdw1BVX AxxkoblvZICqA0U8XZx5FLETt3323/KE8/kybOYL9x66Cy0g0bKuLaXv/450qp+JB4MEJzR6 BCtc4GkldGfrDYx3nD8tFQWFHKd0vuqg5xRu4Qn+kHhwKC9BXvnSXd82UCCCHyTKol4cChIc VSbIACr8pUCO1NBhqLO155duKiJ6cTekE0nfj91Zvz/wei56BNx+uzMYt+1ge99nMivv/ExC VkKzELrMH3C73iwdP1i7eTGlpMB4k8V3dmHD7wyu5bSUuU0iz+P74N4PvM1nBb1HKBA9Gg0Y lJExHca76eImdIIf8YP8GWs6FSNRHKlcJoGhiBf3myXG+UoHOk0L4kXBqrs9g8zBfUH2gHKW iT6PVr+ZcK12j+8t1XJvustJiQxtAOgT0S9eEpTKjkmOYWC3jcZ9QmPwKUc61wlhmCEq+4Vm YzIZ5ZAu9Ju8VIvap4SAo+kVrLGy9pRbmmM5U9AEer9QOJer6Auz9W6pVdAzTTEWZhgwGu3E aIiQfGAVuIsieDZ2kVi8iDafb4klSuRRqCezdgUjLqELW9aFi77vTYxNFIWCcDvDRheatde1 G72kYQusBFUNy4tuGQEiMYMzl141OSaoZYTjiS6NaCZ8j4et6Wp46ngto0YJX0UIJJA/vs3U eOCuMhU7OfYYkbLdxGLeYIqxWPDY2gvwxW/OF4mTjl1APHjfBV+FPom8WbxGQZyvo1NVXRjQ Uq26ochIy+3TtIePzB+eKR4pwUPaRiyS1SBwQ3YHjB/kaODZh1Q28caL8ZS0EIIy2smYzwqD h7bWUM4ZP5O/IUCy8solxwQMywV6M9jabbQiZ+WuVu5DA0czRQc0zkAwUv61IZsbHoBI36Wo RKYXgijQkhfaCQOAASQZcklt89XlACVcS2hr7Sghiu0CieRUIYokZXBdoSd0NwtjDZvZ6Pu3 wX20enjFTgC2d15Hy9TVhgEQDUKO5NHIBkcw0KQpsdDpt60nahSoJxQri3lWBRj3qfOlaHmX KfP0wcyNlm2J7a1AVTYrnVmk/w8qbbI6d610KQrMfsONV3VXMUGCWce327POsNVqdzvtcWiP exW+lGwuecCXTUFybDKr9jpVsRBKAh3AdI+P3fghB9mSm0s4dLmr72J/xMt55tY/5+tj+H87 r6g7tolXBL8NXhHWuYJV1BDn51+m8Q00/0pTkZIAY0Xcoh3IRRXsot3oRcXfe+SV31OHRq89 6A/7/ngEA6gX2nbodQcFkQiL89DDp/g+DR6MTHLUs9XgEXMiEvzffYB1PiYAatvYDrAHNlHO RfiGhAG87cKW4BrqHa0ozSdlBUHxAwzFVQnOKcgPvrxmWy/RlMR0djfUQdd02rPJxeR6cpak til1Sv3iBcGCv5gJQAU8TRFaKJBBn3lFZn5vTq6c1yeXpyevJyrLHmVJUhGJWBQt9BtQouID NEuxbIhiwNIT9eKHgDjMFpF8SecB8DpmLIk2ItpRZ7DfewGLvBZaqcSNR4qnlFo3p8aAtbX9 h32O0BiBMRQDpVI19OFZjEJmSlNcR9yQCIOay80KVhdmnDJfE26L4kqgDOWn+w1LmJeiMA4i roYvWp0N8hMzwqzTh/K5kv3OL6/q7q4a/tlsNJgF3qhTruEve4Oh4e+M/mwJ/21y4UPEXy2W 1/Lbj9LyV32f2p8l2f9pHi7Qpy6k7zc5gfmbtAzf4AdIIKf8nqNCPAlN/m6DC7wpNZaPMQ87 YkSTZzg1ZbZACQpQUUwkQ1GSTke6LG3brDNowjCQACAJPW/TkL9IJ3+8KliRVfJoKpP140Uu lVlDY5IXRtY66AzVqYWu6/Lv5fOq7yu6dn/sDXvjcBi2WuHYHw+D4azdK98K8/nkp3g+DfNQ yBDSz2yJUDsSyr3BrbDQJAYJ4+ygZXI2UjfdHKE6eT02b8m9+TrImsbUrSgMEx72oMcBKPts vEG4hxNPJCTaboBrM5z+M1cI5ahn0resZ/rvF+n5k9ItiYNTNBb6+Ya19B3ZXer6+SL5iyWB W61mPvu64A/ce5b4xRK4EyJ1DKY3757a6JOoUmGDQSCIL/S1THWeTEhfENCIAlbBekpt50fW vykmDf1Xq+ExnrwH6B39cDJ13v82uSQfg/UcUIn1HzMxTB9kNKuk0yvnzeTkAyXEKjTN9tPW tVTUIMiuuVdifTPSNXCNqsyu8rBDK1zlW9I+m6rMGroFAHlspxZLX1Ljd7uEP7sDCXt9RdYN 9krBgnWnxgQstDX1+VipbNiaWa/j7pa+ouLhHsgjCADZZeda96gTwyDtUZh57IDxvrlofo+N +cV+U7KgtfWJllS9iu+YW8FxPEVo3GHBLzE5p26bJVRNlvpeMyzSie9PYQUrKiZet3B5Uy2I jWmi51RmkfFHo/DdlDX+eB2sUVSS3ueE3trrdxp9DDQ5atg2fbdCtmlxs6EY/u/VvDcLnS3T Is2rpzoBFKVv6qJawR/KcS6v4iNrxgqctbsW7bi5cflYK7IeSboMu9t9k85/pCw2g0IebT/T 760curu9+VFGrKe82VIklRQ8YgCxEz7SX0NcjFmS+0k4+O64O/DavttqDQZjwEZ+f2Rvx0c6 n3J8pNOwT4OwBv408RE64M+ml4yPzHZ/NFtMr5/J8zswQzK9/USqWGEPJlUkClHT2u+0Ou2W vb//X+BdTqWQfwAA --LQksG6bCIzRHxTLp-- ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Using Expanded Objects other than Arrays from plpgsql @ 2024-10-20 17:13 Tom Lane <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Tom Lane @ 2024-10-20 17:13 UTC (permalink / raw) To: Michel Pelletier <[email protected]>; +Cc: pgsql-general <[email protected]> Michel Pelletier <[email protected]> writes: > I found this thread from the original path implementation from Tom Lane in > 2015: > https://www.postgresql.org/message-id/E1Ysvgz-0000s0-DP%40gemulon.postgresql.org >> In this initial implementation, a few heuristics have been hard-wired >> into plpgsql to improve performance for arrays that are stored in >> plpgsql variables. We would like to generalize those hacks so that >> other datatypes can obtain similar improvements, but figuring out some >> appropriate APIs is left as a task for future work. Yeah, we thought that it wouldn't be appropriate to try to design general APIs till we had more kinds of expandable objects. Maybe now is a good time to push forward on that. > My first thought was to add a flag to CREATE TYPE like "EXPANDED = true" or > some other better name that indicates that the object can be safely taken > ownership of in its expanded state and not copied. Isn't that inherent in the notion of R/W vs R/O expanded-object pointers? > And then there is just removing the existing restriction on arrays only. > Is any other expanded object out there really interested in being > flattened/expanded over and over again? I'm not sure. It seems certain that if the object is already expanded (either R/W or R/O), the paths for that in plpgsql_exec_function could be taken regardless of its specific type. The thing that is debatable is "if the object is in a flat state, should we forcibly expand it here?". That could be a win if the function later does object accesses that would benefit --- but it might never do so. We chose to always expand arrays, and we've gotten little pushback on that, but the tradeoffs might be different for other sorts of expanded objects. The other problem is that plpgsql only knows how to do such expansion for arrays, and it's not obvious how to extend that part. But it seems like we could get an easy win by adjusting plpgsql_exec_function along the lines of l. 549: - if (!var->isnull && var->datatype->typisarray) + if (!var->isnull) l. 564: - else + else if (var->datatype->typisarray) How far does that improve matters for you? The comment above line 549 cross-references exec_assign_value, but it looks like that's already set up to be similarly type-agnostic about values that are already expanded. regards, tom lane ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Using Expanded Objects other than Arrays from plpgsql @ 2024-10-20 18:25 Michel Pelletier <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Michel Pelletier @ 2024-10-20 18:25 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: pgsql-general <[email protected]> On Sun, Oct 20, 2024 at 10:13 AM Tom Lane <[email protected]> wrote: > Michel Pelletier <[email protected]> writes: > > I found this thread from the original path implementation from Tom Lane > in > > >> appropriate APIs is left as a task for future work. > > Yeah, we thought that it wouldn't be appropriate to try to design > general APIs till we had more kinds of expandable objects. Maybe > now is a good time to push forward on that. > Great, I'm happy to be involved! > My first thought was to add a flag to CREATE TYPE like "EXPANDED = true" > or > > some other better name that indicates that the object can be safely taken > > ownership of in its expanded state and not copied. > > Isn't that inherent in the notion of R/W vs R/O expanded-object > pointers? > I'm not sure I'm qualified enough to agree with you but I see your point. > > And then there is just removing the existing restriction on arrays only. > > Is any other expanded object out there really interested in being > > flattened/expanded over and over again? > > I'm not sure. It seems certain that if the object is already expanded > (either R/W or R/O), the paths for that in plpgsql_exec_function could > be taken regardless of its specific type. Agreed. > The thing that is debatable > is "if the object is in a flat state, should we forcibly expand it > here?". That could be a win if the function later does object > accesses that would benefit --- but it might never do so. We chose > to always expand arrays, and we've gotten little pushback on that, > but the tradeoffs might be different for other sorts of expanded > objects. > The OneSparse objects will always expand themselves on first use via a DatumGetExpandedArray like macro wrapper, there is no run-time benefit to their flat representation, so I'm fine with not forcing their expansion ahead of time, but once I expand it I'd like it to stay expanded until the function returns (as much as possible) the serialization costs for very large sparse matrices is heavy. The other problem is that plpgsql only knows how to do such expansion > for arrays, and it's not obvious how to extend that part. > Perhaps a third member function for ExpandedObjectMethods that formalizes the expansion interface like found in DatumGetExpandedArray? I closely follow that same pattern in my code. > > But it seems like we could get an easy win by adjusting > plpgsql_exec_function along the lines of > ... > > How far does that improve matters for you? > I'll give it a try in my local benchmarking code and get back to you, thank you for the fast reply and the insight! -Michel ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2024-10-20 18:25 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-03-12 01:02 [PATCH] cfe-12-gist_over_cfe-11-persistent squash commit Bruce Momjian <[email protected]> 2024-10-20 17:13 Re: Using Expanded Objects other than Arrays from plpgsql Tom Lane <[email protected]> 2024-10-20 18:25 ` Re: Using Expanded Objects other than Arrays from plpgsql Michel Pelletier <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox