Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lODmT-0003nL-Uz for pgsql-announce@arkaria.postgresql.org; Mon, 22 Mar 2021 06:11:14 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lODmS-0000nI-TW for pgsql-announce@arkaria.postgresql.org; Mon, 22 Mar 2021 06:11:12 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lODmR-0000nB-N3 for pgsql-announce@lists.postgresql.org; Mon, 22 Mar 2021 06:11:12 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lODmK-0004lJ-Ar for pgsql-announce@lists.postgresql.org; Mon, 22 Mar 2021 06:11:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:From:To:Subject: MIME-Version:Content-Type:Sender:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:In-Reply-To:References; bh=tvARRPVmGJR+MMTD9fOqp5ixwzOEVoEs/kar+wbWqZ4=; b=MptKnsAkxFi9gXVWX24spoyLWL 39BuCUrVkm7mp1DA5PLQDUiZPMwigzP9AwxEf05FwSoX9jUDEgHyXrepnCJYqSTJIFO2NsrtMAhF+ YCxKkcZtR3Py/5o5H6WyRo5thahaVGoOEvY7uBH9QwBoBuT8mkhNBKeiwEjTyffwtToCldCjx3kNj YuJnm73RrwNdigMcki350mx3oZp3/zIw49+hosKfuurZzZdEA14HEkzKSEF6FFdO38IFuNN/Rrd/I 1nMLgSQPEkLdY3Z/dmDfho4wFcuiSnVgeL3TTvfN1ctOQUqL1Hmm2PMtyJMGkPAc21Gr4kzug6sKT iG+3UWjA==; Received: from wrigleys.postgresql.org ([2a02:c0:301:0:ffff::34]) by mahout.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lODmJ-0001RC-Hj for pgsql-announce@lists.postgresql.org; Mon, 22 Mar 2021 06:11:04 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lODmI-0006I7-Cl for pgsql-announce@lists.postgresql.org; Mon, 22 Mar 2021 06:11:02 +0000 Content-Type: multipart/mixed; boundary="===============1844203319814237320==" MIME-Version: 1.0 Subject: PostgreSQL Weekly News - March 21, 2021 To: PostgreSQL Announce From: PWN via PostgreSQL Announce Reply-To: david@fetter.org Date: Mon, 22 Mar 2021 06:10:46 +0000 Message-ID: <161639344616.652.6203233043817949403@wrigleys.postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-pglister-tags: pwn X-pglister-tagsig: 77475b1ed382fb8605150491f6dc3d04e15b3e9feba175ac61cc5be1ee78f7b0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --===============1844203319814237320== Content-Type: multipart/alternative; boundary="===============3798017092296226947==" MIME-Version: 1.0 --===============3798017092296226947== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable # PostgreSQL Weekly News - March 21, 2021 Person of the week: [https://postgresql.life/post/anastasia_lubennikova/](h= ttps://postgresql.life/post/anastasia_lubennikova/) # PostgreSQL Product News pspg 4.4.0 a pager designed for PostgreSQL, released. [https://github.com/okbob/pspg/releases/tag/4.4.0](https://github.com/okbob= /pspg/releases/tag/4.4.0) pg_activity 2.1.2, a top-like application for PostgreSQL server activity monitoring, released. [https://github.com/dalibo/pg_activity/releases/tag/v2.1.2](https://github.= com/dalibo/pg_activity/releases/tag/v2.1.2) # PostgreSQL Jobs for March [https://archives.postgresql.org/pgsql-jobs/2021-03/](https://archives.post= gresql.org/pgsql-jobs/2021-03/) # PostgreSQL in the News Planet PostgreSQL: [https://planet.postgresql.org/](https://planet.postgres= ql.org/) PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.o= rg. # Applied Patches Peter Geoghegan pushed: - Notice that heap page has dead items during VACUUM. Consistently set a fl= ag variable that tracks whether the current heap page has a dead item during= lazy vacuum's heap scan. We missed the common case where there is an preexist= ing (or even a new) LP_DEAD heap line pointer. Also make it clear that the variable might be affected by an existing line pointer, say from an earli= er opportunistic pruning operation. This distinction is important because i= t's the main reason why we can't just use the nearby tups_vacuumed variable instead. No backpatch. In theory failing to set the page level flag var= iable had no consequences. Currently it is only used to defensively check if a= page marked all visible has dead items, which should never happen anyway (if it does then the table must be corrupt). Author: Masahiko Sawada Diagnosed-By: Masahiko Sawada Discussion: [https://postgr.es/m/CAD21AoAtZb4+HJT_8RoOXvu4HM-Zd4HKS3YSMCH6+-W=3DbDyh-= w@mail.gmail.com](https://postgr.es/m/CAD21AoAtZb4+HJT_8RoOXvu4HM-Zd4HKS3YS= MCH6+-W=3DbDyh-w@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/0ea71c93a06ddc38e0b72e48f1d512e= 5383a9c1b](https://git.postgresql.org/pg/commitdiff/0ea71c93a06ddc38e0b72e4= 8f1d512e5383a9c1b) - amcheck: Reduce debug message verbosity. Empty sibling pages can occasion= ally be much more common than any other event that we report on at elevel DEBU= G1. Increase the elevel for relevant cases to DEBUG2 to avoid overwhelming the user with relatively insignificant details. [https://git.postgresql.org/pg/commitdiff/65445469d6de3670f3e027236613e09= 3e119ec55](https://git.postgresql.org/pg/commitdiff/65445469d6de3670f3e0272= 36613e093e119ec55) - Fix comment about promising tuples. Oversight in commit d168b666823, which added bottom-up index deletion. [https://git.postgresql.org/pg/commitdiff/fbe4cb3bd49f9e524f53ef77c775c1b= ad4d0312a](https://git.postgresql.org/pg/commitdiff/fbe4cb3bd49f9e524f53ef7= 7c775c1bad4d0312a) Fujii Masao pushed: - Make archiver process an auxiliary process. This commit changes WAL archi= ver process so that it's treated as an auxiliary process and can use shared memory. This is an infrastructure patch required for upcoming shared-memo= ry based stats collector patch series. These patch series basically need any processes including archiver that can report the statistics to access to shared memory. Since this patch itself is useful to simplify the code and= when users monitor the status of archiver, it's committed separately in advanc= e. This commit simplifies the code for WAL archiving. For example, previously backends need to signal to archiver via postmaster when they notify archi= ver that there are some WAL files to archive. On the other hand, this commit removes that signal to postmaster and enables backends to notify archier directly using shared latch. Also, as the side of this change, the information about archiver process becomes viewable at pg_stat_activity v= iew. Author: Kyotaro Horiguchi Reviewed-by: Andres Freund, =C3=81lvaro Herrera= , Julien Rouhaud, Tomas Vondra, Arthur Zakirov, Fujii Masao Discussion: [https://postgr.es/m/20180629.173418.190173462.horiguchi.kyotaro@lab.ntt.= co.jp](https://postgr.es/m/20180629.173418.190173462.horiguchi.kyotaro@lab.= ntt.co.jp) [https://git.postgresql.org/pg/commitdiff/d75288fb27b8fe0a926aaab7d75816f= 091ecdc27](https://git.postgresql.org/pg/commitdiff/d75288fb27b8fe0a926aaab= 7d75816f091ecdc27) - Fix comments in postmaster.c. Commit 86c23a6eb2 changed the option to spe= cify that postgres will stop all other server processes by sending the signal SIGSTOP, from -s to -T. But previously there were comments incorrectly explaining that SIGSTOP behavior is set by -s option. This commit fixes t= hem. Author: Kyotaro Horiguchi Reviewed-by: Fujii Masao Discussion: [https://postgr.es/m/20210316.165141.1400441966284654043.horikyota.ntt@gm= ail.com](https://postgr.es/m/20210316.165141.1400441966284654043.horikyota.= ntt@gmail.com) [https://git.postgresql.org/pg/commitdiff/fd31214075cc740e43edc71ca1c385c= 8c53047b7](https://git.postgresql.org/pg/commitdiff/fd31214075cc740e43edc71= ca1c385c8c53047b7) Thomas Munro pushed: - Drop SERIALIZABLE workaround from parallel query tests. SERIALIZABLE no l= onger inhibits parallelism, so we can drop some outdated workarounds and commen= ts from regression tests. The change came in release 12, commit bb16aba5, b= ut it's not really worth back-patching. Also fix a typo. Reviewed-by: Bhar= ath Rupireddy Discussion: [https://postgr.es/m/CA%2BhUKGJUaHeK%3DHLATxF1JOKDjKJVrBKA-zmbPAebOM0Se2F= QRg%40mail.gmail.com](https://postgr.es/m/CA%2BhUKGJUaHeK%3DHLATxF1JOKDjKJV= rBKA-zmbPAebOM0Se2FQRg%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/eeb60e45d82d5840b713a8741ae5522= 38d57e8b9](https://git.postgresql.org/pg/commitdiff/eeb60e45d82d5840b713a87= 41ae552238d57e8b9) - Enable parallelism in REFRESH MATERIALIZED VIEW. Pass CURSOR_OPT_PARALLEL= _OK to pg_plan_query() so that parallel plans are considered when running the underlying SELECT query. This wasn't done in commit e9baa5e9, which did = this for CREATE MATERIALIZED VIEW, because it wasn't yet known to be safe. Si= nce REFRESH always inserts into a freshly created table before later merging = or swapping the data into place with separate operations, we can enable such plans here too. Author: Bharath Rupireddy Reviewed-by: Hou, Zhijie Reviewed-by: Luc Vlaming Reviewed-by: Thomas Munro Discussion: [https://postgr.es/m/CALj2ACXg-4hNKJC6nFnepRHYT4t5jJVstYvri%2BtKQHy7ydcr8= A%40mail.gmail.com](https://postgr.es/m/CALj2ACXg-4hNKJC6nFnepRHYT4t5jJVstY= vri%2BtKQHy7ydcr8A%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/9e7ccd9ef64d05e87ceb1985d459bef= 9031205c0](https://git.postgresql.org/pg/commitdiff/9e7ccd9ef64d05e87ceb198= 5d459bef9031205c0) - Fix transaction.sql tests in higher isolation levels. It seems like a use= ful sanity check to be able to run "installcheck" against a cluster running w= ith default_transaction_level set to serializable or repeatable read. Only o= ne thing currently fails in those configurations, so let's fix that. No back-patch for now, because it fails in many other places in some of the stable branches. We'd have to go back and fix those too if we included t= his configuration in automated testing. Reviewed-by: Tom Lane Discussion: [https://postgr.es/m/CA%2BhUKGJUaHeK%3DHLATxF1JOKDjKJVrBKA-zmbPAebOM0Se2F= QRg%40mail.gmail.com](https://postgr.es/m/CA%2BhUKGJUaHeK%3DHLATxF1JOKDjKJV= rBKA-zmbPAebOM0Se2FQRg%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/37929599499fe5760a9dbab48a10a89= 8879a0d44](https://git.postgresql.org/pg/commitdiff/37929599499fe5760a9dbab= 48a10a898879a0d44) - Fix race in Parallel Hash Join batch cleanup. With very unlucky timing and parallel_leader_participation off, PHJ could attempt to access per-batch = state just as it was being freed. There was code intended to prevent that by checking for a cleared pointer, but it was buggy. Fix, by introducing an extra barrier phase. The new phase PHJ_BUILD_RUNNING means that it's saf= e to access the per-batch state to find a batch to help with, and PHJ_BUILD_DO= NE means that it is too late. The last to detach will free the array of per-= batch state as before, but now it will also atomically advance the phase at the= same time, so that late attachers can avoid the hazard, without the data race. This mirrors the way per-batch hash tables are freed (see phases PHJ_BATCH_PROBING and PHJ_BATCH_DONE). Revealed by a one-off build farm failure, where BarrierAttach() failed a sanity check assertion, because t= he memory had been clobbered by dsa_free(). Back-patch to 11, where the code arrived. Reported-by: Michael Paquier Discussion: [https://postgr.es/m/20200929061142.GA29096%40paquier.xyz](https://postgr= .es/m/20200929061142.GA29096%40paquier.xyz) [https://git.postgresql.org/pg/commitdiff/3b8981b6e1a2aea0f18384c803e21e9= 391de669a](https://git.postgresql.org/pg/commitdiff/3b8981b6e1a2aea0f18384c= 803e21e9391de669a) - Update the names of Parallel Hash Join phases. Commit 3048898e dropped -I= NG from some wait event names that correspond to barrier phases. Update the phases' names to match. While we're here making cosmetic changes, also r= ename "DONE" to "FREE". That pairs better with "ALLOCATE", and describes the activity that actually happens in that phase (as we do for the other phas= es) rather than describing a state. The distinction is clearer after bugfix commit 3b8981b6 split the phase into two. As for the growth barriers, re= name their "ALLOCATE" phase to "REALLOCATE", which is probably a better descri= ption of what happens then. Also improve the comments about the phases a bit. Discussion: [https://postgr.es/m/CA%2BhUKG%2BMDpwF2Eo2LAvzd%3DpOh81wUTsrwU1uAwR-v6OGB= B6%2B7g%40mail.gmail.com](https://postgr.es/m/CA%2BhUKG%2BMDpwF2Eo2LAvzd%3D= pOh81wUTsrwU1uAwR-v6OGBB6%2B7g%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/378802e3713c6c0fce31d2390c134cd= 5d7c30157](https://git.postgresql.org/pg/commitdiff/378802e3713c6c0fce31d23= 90c134cd5d7c30157) - Revert "Fix race in Parallel Hash Join batch cleanup.". This reverts comm= it 378802e3713c6c0fce31d2390c134cd5d7c30157. This reverts commit 3b8981b6e1a2aea0f18384c803e21e9391de669a. Discussion: [https://postgr.es/m/CA%2BhUKGJmcqAE3MZeDCLLXa62cWM0AJbKmp2JrJYaJ86bz36LF= A%40mail.gmail.com](https://postgr.es/m/CA%2BhUKGJmcqAE3MZeDCLLXa62cWM0AJbK= mp2JrJYaJ86bz36LFA%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/7f7f25f15edb6eacec58179ef5285e8= 74aa4435b](https://git.postgresql.org/pg/commitdiff/7f7f25f15edb6eacec58179= ef5285e874aa4435b) - Provide recovery_init_sync_method=3Dsyncfs. Since commit 2ce439f3 we have= opened every file in the data directory and called fsync() at the start of crash recovery. This can be very slow if there are many files, leading to field complaints of systems taking minutes or even hours to begin crash recover= y. Provide an alternative method, for Linux only, where we call syncfs() on = every possibly different filesystem under the data directory. This is equivale= nt, but avoids faulting in potentially many inodes from potentially slow stor= age. The new mode comes with some caveats, described in the documentation, so = the default value for the new setting is "fsync", preserving the older behavi= or. Reported-by: Michael Brown Reviewed-by: Fuj= ii Masao Reviewed-by: Paul Guo Reviewed-by: Bruce Momjian Reviewed-by: Justin Pryzby Reviewed-by: David Steele Discussion: [https://postgr.es/m/11bc2bb7-ecb5-3ad0-b39f-df632734cd81%40discourse.org= ](https://postgr.es/m/11bc2bb7-ecb5-3ad0-b39f-df632734cd81%40discourse.org) Discussion: [https://postgr.es/m/CAEET0ZHGnbXmi8yF3ywsDZvb3m9CbdsGZgfTXscQ6agcbzcZAw%= 40mail.gmail.com](https://postgr.es/m/CAEET0ZHGnbXmi8yF3ywsDZvb3m9CbdsGZgfT= XscQ6agcbzcZAw%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/61752afb26404dfc99a535c7a53f7f0= 4dc110263](https://git.postgresql.org/pg/commitdiff/61752afb26404dfc99a535c= 7a53f7f04dc110263) Tom Lane pushed: - Work around issues in MinGW-64's setjmp/longjmp support. It's hard to avo= id the conclusion that there is something wrong with setjmp/longjmp on MinGW= -64, as we have seen failures come and go after entirely-unrelated-looking cha= nges in our own code. Other projects such as Ruby have given up and started u= sing gcc's setjmp/longjmp builtins on that platform; this patch just follows t= hat lead. Note that this is a pretty fundamental ABI break for functions containining either setjmp or longjmp, so we can't really consider a back-patch. Per reports from Regina Obe and Heath Lord, as well as recent failures on buildfarm member walleye, and less-recent failures on fairywr= en. Juan Jos=C3=A9 Santamar=C3=ADa Flecha Discussion: [https://postgr.es/m/000401d716a0$1ed0fc70$5c72f550$@pcorp.us](https://po= stgr.es/m/000401d716a0$1ed0fc70$5c72f550$@pcorp.us) Discussion: [https://postgr.es/m/CA+BEBhvHhM-Bn628pf-LsjqRh3Ang7qCSBG0Ga+7KwhGqrNUPw@= mail.gmail.com](https://postgr.es/m/CA+BEBhvHhM-Bn628pf-LsjqRh3Ang7qCSBG0Ga= +7KwhGqrNUPw@mail.gmail.com) Discussion: [https://postgr.es/m/f1caef93-9640-022e-9211-bbe8755a56b0@2ndQuadrant.com= ](https://postgr.es/m/f1caef93-9640-022e-9211-bbe8755a56b0@2ndQuadrant.com) [https://git.postgresql.org/pg/commitdiff/146cb3889c3ccb3fce198fe7464a129= 6a9e107c3](https://git.postgresql.org/pg/commitdiff/146cb3889c3ccb3fce198fe= 7464a1296a9e107c3) - Improve logging of bad parameter values in BIND messages. Since commit ba79cb5dc, values of bind parameters have been logged during errors in extended query mode. However, we only did that after we'd collected and converted all the parameter values, thus failing to offer any useful localization of invalid-parameter problems. Add a separate callback that= 's used during parameter collection, and have it print the parameter number, along with the input string if text input format is used. Justin Pryzby = and Tom Lane Discussion: [https://postgr.es/m/20210104170939.GH9712@telsasoft.com](https://postgr.= es/m/20210104170939.GH9712@telsasoft.com) Discussion: [https://postgr.es/m/CANfkH5k-6nNt-4cSv1vPB80nq2BZCzhFVR5O4VznYbsX0wZmow@= mail.gmail.com](https://postgr.es/m/CANfkH5k-6nNt-4cSv1vPB80nq2BZCzhFVR5O4V= znYbsX0wZmow@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/1ea396362be1615e926ea69d666c770= 081a0d3ef](https://git.postgresql.org/pg/commitdiff/1ea396362be1615e926ea69= d666c770081a0d3ef) - Avoid corner-case memory leak in SSL parameter processing. After reading = the root cert list from the ssl_ca_file, immediately install it as client CA = list of the new SSL context. That gives the SSL context ownership of the list= , so that SSL_CTX_free will free it. This avoids a permanent memory leak if we= fail further down in be_tls_init(), which could happen if bogus CRL data is offered. The leak could only amount to something if the CRL parameters g= et broken after server start (else we'd just quit) and then the server is SIGHUP'd many times without fixing the CRL data. That's rather unlikely perhaps, but it seems worth fixing, if only because the code is clearer t= his way. While we're here, add some comments about the memory management asp= ects of this logic. Noted by Jelte Fennema and independently by Andres Freund. Back-patch to v10; before commit de41869b6 it doesn't matter, since we'd = not re-execute this code during SIGHUP. Discussion: [https://postgr.es/m/16160-18367e56e9a28264@postgresql.org](https://postg= r.es/m/16160-18367e56e9a28264@postgresql.org) [https://git.postgresql.org/pg/commitdiff/4b12ab18c9d0735d760bf7667b15870= 7b06e5df8](https://git.postgresql.org/pg/commitdiff/4b12ab18c9d0735d760bf76= 67b158707b06e5df8) - Doc: improve discussion of variable substitution in PL/pgSQL. This was a = bit disjointed, partly because of a not-well-considered decision to document = SQL commands that don't return result rows as though they had nothing in comm= on with commands that do. Rearrange so that we have one discussion of varia= ble substitution that clearly applies to all types of SQL commands, and then handle the question of processing command output separately. Clarify that EXPLAIN, CREATE TABLE AS SELECT, and similar commands that incorporate an optimizable statement will act like optimizable statements for the purpos= es of variable substitution. Do a bunch of minor wordsmithing in the same area. David Johnston and Tom Lane, reviewed by Pavel Stehule and David Steele Discussion: [`https://postgr.es/m/CAKFQuwYvMKucM5fnZvHSo-ah4S=3D_n9gmKeu6EAo=3D_fTroh= unqQ@mail.gmail.com`](https://postgr.es/m/CAKFQuwYvMKucM5fnZvHSo-ah4S=3D_n9= gmKeu6EAo=3D_fTrohunqQ@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/c783e656d41816b0328cb4bff27f11b= 70200770e](https://git.postgresql.org/pg/commitdiff/c783e656d41816b0328cb4b= ff27f11b70200770e) - Add end-to-end testing of pg_basebackup's tar-format output. The existing= test script does run pg_basebackup with the -Ft option, but it makes no real attempt to verify the sanity of the results. We wouldn't know if the outp= ut is incompatible with standard "tar" programs, nor if the server fails to sta= rt from the restored output. Notably, this means that xlog.c's read_tablespace_map() is not being meaningfully tested, since that code is used only in the tar-format case. (We do have reasonable coverage of restoring from plain-format output, though it's over in src/test/recovery= not here.) Hence, attempt to untar the output and start a server from it, ra= ther just hoping it's OK. This test assumes that the local "tar" has the "-C directory" switch. Although that's not promised by POSIX, my research suggests that all non-extinct tar implementations have it. Should the buildfarm's opinion differ, we can complicate the test a bit to avoid requiring that. Possibly this should be back-patched, but I'm unsure abo= ut whether it could work on Windows before d66b23b03. [https://git.postgresql.org/pg/commitdiff/081876d75ea15c3bd2ee5ba64a794fd= 8ea46d794](https://git.postgresql.org/pg/commitdiff/081876d75ea15c3bd2ee5ba= 64a794fd8ea46d794) - Prevent buffer overrun in read_tablespace_map(). Robert Foggia of Trustwa= ve reported that read_tablespace_map() fails to prevent an overrun of its on-stack input buffer. Since the tablespace map file is presumed trustwor= thy, this does not seem like an interesting security vulnerability, but still = we should fix it just in the name of robustness. While here, document that pg_basebackup's --tablespace-mapping option doesn't work with tar-format output, because it doesn't. To make it work, we'd have to modify the tablespace_map file within the tarball sent by the server, which might be possible but I'm not volunteering. (Less-painful solutions would require changing the basebackup protocol so that the source server could adjust t= he map. That's not very appetizing either.) [https://git.postgresql.org/pg/commitdiff/a50e4fd028a1ece2b1a04df2c9ae658= 1783e9eef](https://git.postgresql.org/pg/commitdiff/a50e4fd028a1ece2b1a04df= 2c9ae6581783e9eef) - Code review for server's handling of "tablespace map" files. While lookin= g at Robert Foggia's report, I noticed a passel of other issues in the same ar= ea: * The scheme for backslash-quoting newlines in pathnames is just wrong; it will misbehave if the last ordinary character in a pathname is a backslas= h. I'm not sure why we're bothering to allow newlines in tablespace paths, b= ut if we're going to do it we should do it without introducing other problems. Hence, backslashes themselves have to be backslashed too. * The author h= adn't read the sscanf man page very carefully, because this code would drop any leading whitespace from the path. (I doubt that a tablespace path with leading whitespace could happen in practice; but if we're bothering to al= low newlines in the path, it sure seems like leading whitespace is little less implausible.) Using sscanf for the task of finding the first space is overkill anyway. * While I'm not 100% sure what the rationale for escapi= ng both \r and \n is, if the idea is to allow Windows newlines in the file t= hen this code failed, because it'd throw an error if it saw \r followed by \n= . * There's no cross-check for an incomplete final line in the map file, which would be a likely apparent symptom of the improper-escaping bug. On the generation end, aside from the escaping issue we have: * If needtblspcma= pfile is true then do_pg_start_backup will pass back escaped strings in tablespaceinfo->path values, which no caller wants or is prepared to deal with. I'm not sure if there's a live bug from that, but it looks like th= ere might be (given the dubious assumption that anyone actually has newlines = in their tablespace paths). * It's not being very paranoid about the possib= ility of random stuff in the pg_tblspc directory. IMO we should ignore anything without an OID-like name. The escaping rule change doesn't seem back-patchable: it'll require doubling of backslashes in the tablespace_m= ap file, which is basically a basebackup format change. The odds of that ca= using trouble are considerably more than the odds of the existing bug causing trouble. The rest of this seems somewhat unlikely to cause problems too, = so no back-patch. [https://git.postgresql.org/pg/commitdiff/8620a7f6dbdf978e57cdb9f42802a04= 18656d863](https://git.postgresql.org/pg/commitdiff/8620a7f6dbdf978e57cdb9f= 42802a0418656d863) - Doc: remove duplicated step in RLS example. Seems to have been a copy-and-paste mistake in 093129c9d. Per report from max1@inbox.ru. Discussion: [https://postgr.es/m/161591740692.24273.4202054598867879464@wrigleys.post= gresql.org](https://postgr.es/m/161591740692.24273.4202054598867879464@wrig= leys.postgresql.org) [https://git.postgresql.org/pg/commitdiff/70945649d734d16be22c3d1d90dd8c3= d3c1e9d89](https://git.postgresql.org/pg/commitdiff/70945649d734d16be22c3d1= d90dd8c3d3c1e9d89) - Fix misuse of foreach_delete_current(). Our coding convention requires th= is macro's result to be assigned back to the original List variable. In this usage, since the List could not become empty, there was no actual bug ---= but some compilers warned about it. Oversight in be45be9c3. Discussion: [https://postgr.es/m/35077b31-2d62-1e31-0e2e-ddb52d590b73@enterprisedb.co= m](https://postgr.es/m/35077b31-2d62-1e31-0e2e-ddb52d590b73@enterprisedb.co= m) [https://git.postgresql.org/pg/commitdiff/1d581ce7129d7a33cd4ad27f8f246ab= fa1fd2db9](https://git.postgresql.org/pg/commitdiff/1d581ce7129d7a33cd4ad27= f8f246abfa1fd2db9) - Don't leak rd_statlist when a relcache entry is dropped. Although these l= ists are usually NIL, and even when not empty are unlikely to be large, consta= nt relcache update traffic could eventually result in visible bloat of CacheMemoryContext. Found via valgrind testing. Back-patch to v10 where = this field was added. Discussion: [https://postgr.es/m/3816764.1616104288@sss.pgh.pa.us](https://postgr.es/= m/3816764.1616104288@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/28644fac10731e30e70b622986a6fbb= eb5a5b2f9](https://git.postgresql.org/pg/commitdiff/28644fac10731e30e70b622= 986a6fbbeb5a5b2f9) - Don't leak malloc'd error string in libpqrcv_check_conninfo(). We leaked = the error report from PQconninfoParse, when there was one. It seems unlikely= that real usage patterns would repeat the failure often enough to create serio= us bloat, but let's back-patch anyway to keep the code similar in all branch= es. Found via valgrind testing. Back-patch to v10 where this code was added. Discussion: [https://postgr.es/m/3816764.1616104288@sss.pgh.pa.us](https://postgr.es/= m/3816764.1616104288@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/9bacdf9f536a3720976ae258238cb46= c691ce9b2](https://git.postgresql.org/pg/commitdiff/9bacdf9f536a3720976ae25= 8238cb46c691ce9b2) - Don't leak compiled regex(es) when an ispell cache entry is dropped. The = text search cache mechanisms assume that we can clean up an invalidated dictio= nary cache entry simply by resetting the associated long-lived memory context. However, that does not work for ispell affixes that make use of regular expressions, because the regex library deals in plain old malloc. Hence,= we leaked compiled regex(es) any time we dropped such a cache entry. That c= ould quickly add up, since even a fairly trivial regex can use up tens of kB, = and a large one can eat megabytes. Add a memory context callback to ensure tha= t a regex gets freed when its owning cache entry is cleared. Found via valgr= ind testing. This problem is ancient, so back-patch to all supported branches. Discussion: [https://postgr.es/m/3816764.1616104288@sss.pgh.pa.us](https://postgr.es/= m/3816764.1616104288@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/d303849b059c3c315e5a8d4239016f8= 328f3296c](https://git.postgresql.org/pg/commitdiff/d303849b059c3c315e5a8d4= 239016f8328f3296c) - Don't run RelationInitTableAccessMethod in a long-lived context. Some code paths in this function perform syscache lookups, which can lead to table accesses and possibly leakage of cruft into the caller's context. If said context is CacheMemoryContext, we eventually will have visible bloat. But fixing this is no harder than moving one memory context switch step. (The other callers don't have a problem.) Andres Freund and I independently f= ound this via valgrind testing. Back-patch to v12 where this code was added. Discussion: [https://postgr.es/m/20210317023101.anvejcfotwka6gaa@alap3.anarazel.de](h= ttps://postgr.es/m/20210317023101.anvejcfotwka6gaa@alap3.anarazel.de) Discussion: [https://postgr.es/m/3816764.1616104288@sss.pgh.pa.us](https://postgr.es/= m/3816764.1616104288@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/415ffdc2205e209b6a73fb42a3fdd6e= 57e16c7b2](https://git.postgresql.org/pg/commitdiff/415ffdc2205e209b6a73fb4= 2a3fdd6e57e16c7b2) - Don't leak malloc'd strings when a GUC setting is rejected. Because guc.c prefers to keep all its string values in malloc'd not palloc'd storage, i= t has to be more careful than usual to avoid leaks. Error exits out of string = GUC hook checks failed to clear the proposed value string, and error exits ou= t of ProcessGUCArray() failed to clear the malloc'd results of ParseLongOption= (). Found via valgrind testing. This problem is ancient, so back-patch to all supported branches. Discussion: [https://postgr.es/m/3816764.1616104288@sss.pgh.pa.us](https://postgr.es/= m/3816764.1616104288@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/377b7a83007d277d32ef19f7c7590c8= 668d504cb](https://git.postgresql.org/pg/commitdiff/377b7a83007d277d32ef19f= 7c7590c8668d504cb) - Blindly try to fix test script's tar invocation for MSYS. Buildfarm member fairywren doesn't like the test case I added in commit 081876d75. I'm guessing the reason is that I shouldn't be using a perl2host-ified path i= n the tar command line. [https://git.postgresql.org/pg/commitdiff/27ab1981e7c9b8fcbcb143c5f6f7064= 41a52bbc8](https://git.postgresql.org/pg/commitdiff/27ab1981e7c9b8fcbcb143c= 5f6f706441a52bbc8) - Avoid leaking memory in RestoreGUCState(), and improve comments. RestoreGUCState applied InitializeOneGUCOption to already-live GUC entrie= s, causing any malloc'd subsidiary data to be forgotten. We do want the effe= ct of resetting the GUC to its compiled-in default, and InitializeOneGUCOption = seems like the best way to do that, so add code to free any existing subsidiary= data beforehand. The interaction between can_skip_gucvar, SerializeGUCState, = and RestoreGUCState is way more subtle than their opaque comments would sugge= st to an unwary reader. Rewrite and enlarge the comments to try to make it cle= arer what's happening. Remove a long-obsolete assertion in read_nondefault_variables: the behavior of set_config_option hasn't depen= ded on IsInitProcessingMode since f5d9698a8 installed a better way of control= ling it. Although this is fixing a clear memory leak, the leak is quite unlik= ely to involve any large amount of data, and it can only happen once in the lifetime of a worker process. So it seems unnecessary to take any risk of back-patching. Discussion: [https://postgr.es/m/4105247.1616174862@sss.pgh.pa.us](https://postgr.es/= m/4105247.1616174862@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/f0c2a5bba6c566fad781802537eb17f= 2977702bc](https://git.postgresql.org/pg/commitdiff/f0c2a5bba6c566fad781802= 537eb17f2977702bc) - Fix memory leak when rejecting bogus DH parameters. While back-patching e0e569e1d, I noted that there were some other places where we ought to be applying DH_free(); namely, where we load some DH parameters from a file = and then reject them as not being sufficiently secure. While it seems really unlikely that anybody would hit these code paths in production, let alone= do so repeatedly, let's fix it for consistency. Back-patch to v10 where this code was introduced. Discussion: [https://postgr.es/m/16160-18367e56e9a28264@postgresql.org](https://postg= r.es/m/16160-18367e56e9a28264@postgresql.org) [https://git.postgresql.org/pg/commitdiff/e835e89a0fd267871e7fbddc39ad00e= e3d0cb55c](https://git.postgresql.org/pg/commitdiff/e835e89a0fd267871e7fbdd= c39ad00ee3d0cb55c) - Fix up pg_dump's handling of per-attribute compression options. The appro= ach used in commit bbe0a81db would've been disastrous for portability of dump= s. Instead handle non-default compression options in separate ALTER TABLE commands. This reduces chatter for the common case where most columns are compressed the same way, and it makes it possible to restore the dump to a server that lacks any knowledge of per-attribute compression options (so = long as you're willing to ignore syntax errors from the ALTER TABLE commands). There's a whole lot left to do to mop up after bbe0a81db, but I'm fast-tracking this part because we need to see if it's enough to make the buildfarm's cross-version-upgrade tests happy. Justin Pryzby and Tom Lane Discussion: [https://postgr.es/m/20210119190720.GL8560@telsasoft.com](https://postgr.= es/m/20210119190720.GL8560@telsasoft.com) [https://git.postgresql.org/pg/commitdiff/aa25d1089ac00bbc3f97d2efe8f54c3= d4beed5d1](https://git.postgresql.org/pg/commitdiff/aa25d1089ac00bbc3f97d2e= fe8f54c3d4beed5d1) - Suppress various new compiler warnings. Compilers that don't understand t= hat elog(ERROR) doesn't return issued warnings here. In the cases in libpq_pipeline.c, we were not exactly helping things by failing to mark pg_fatal() as noreturn. Per buildfarm. [https://git.postgresql.org/pg/commitdiff/9fb9691a88ae8df9bc30e0f0f72de7c= 96e73e125](https://git.postgresql.org/pg/commitdiff/9fb9691a88ae8df9bc30e0f= 0f72de7c96e73e125) =C3=81lvaro Herrera pushed: - Implement pipeline mode in libpq. Pipeline mode in libpq lets an applicat= ion avoid the Sync messages in the FE/BE protocol that are implicit in the old libpq API after each query. The application can then insert Sync at its leisure with a new libpq function PQpipelineSync. This can lead to substantial reductions in query latency. Co-authored-by: Craig Ringer Co-authored-by: Matthieu Garrigues Co-authored-by: =C3=81lvaro Herrera Reviewed-by: Andres Freund Reviewed-by: Aya Iwata Reviewed-by: Daniel V= =C3=A9rit=C3=A9 Reviewed-by: David G. Johnston Reviewed-by: Justin Pryzby Reviewed-by: Kirk Jamison Reviewed-by: Michael Pa= quier Reviewed-by: Nikhil Sontakke Reviewed-by: Vaishnavi Prabakaran Reviewed-by: Zhihong Yu Discussion: [https://postgr.es/m/CAMsr+YFUjJytRyV4J-16bEoiZyH=3D4nj+sQ7JP9ajwz=3DB4dM= MZw@mail.gmail.com](https://postgr.es/m/CAMsr+YFUjJytRyV4J-16bEoiZyH=3D4nj+= sQ7JP9ajwz=3DB4dMMZw@mail.gmail.com) Discussion: [https://postgr.es/m/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MArvLhs_pg75gg@= mail.gmail.com](https://postgr.es/m/CAJkzx4T5E-2cQe3dtv2R78dYFvz+in8PY7A8MA= rvLhs_pg75gg@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/acb7e4eb6b1c614c68a62fb3a6a5bba= 1af0a2659](https://git.postgresql.org/pg/commitdiff/acb7e4eb6b1c614c68a62fb= 3a6a5bba1af0a2659) - Add libpq pipeline mode support to pgbench. New metacommands \startpipeli= ne and \endpipeline allow the user to run queries in libpq pipeline mode. Author: Daniel V=C3=A9rit=C3=A9 Reviewed-by: = =C3=81lvaro Herrera Discussion: [https://postgr.es/m/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.or= g](https://postgr.es/m/b4e34135-2bd9-4b8a-94ca-27d760da26d7@manitou-mail.or= g) [https://git.postgresql.org/pg/commitdiff/9aa491abbf07ca8385a429385be8d68= 517384fdf](https://git.postgresql.org/pg/commitdiff/9aa491abbf07ca8385a4293= 85be8d68517384fdf) - (Blind) fix Perl splitting of strings at newlines. I forgot that Windows represents newlines as \r\n, so splitting a string at /\s/ creates additi= onal empty strings. Let's rewrite that as /\s+/ to see if that avoids those. (There's precedent for using that pattern on Windows in other scripts.) Previously: 91bdf499b37b, 8ed428dc977f, 650b96707672. Per buildfarm, via= Tom Lane. Discussion: [https://postgr.es/m/3144460.1615860259@sss.pgh.pa.us](https://postgr.es/= m/3144460.1615860259@sss.pgh.pa.us) [https://git.postgresql.org/pg/commitdiff/015061690c6526ff9f9f7af2940e1c1= 541654b68](https://git.postgresql.org/pg/commitdiff/015061690c6526ff9f9f7af= 2940e1c1541654b68) - Fix new memory leaks in libpq. My oversight in commit 9aa491abbf07. Per coverity. [https://git.postgresql.org/pg/commitdiff/2b526ed2e1cbaa54e5ad0c12d129448= 2f2757b17](https://git.postgresql.org/pg/commitdiff/2b526ed2e1cbaa54e5ad0c1= 2d1294482f2757b17) Micha=C3=ABl Paquier pushed: - Add some basic tests for progress reporting of COPY. This tests some basic features for progress reporting of COPY, relying on an INSERT trigger that gets fired when doing COPY FROM with a file or stdin, checking for sizes, number of tuples processed, and number of tuples excluded by a WHERE clau= se. Author: Josef =C5=A0im=C3=A1nek, Matthias van de Meent Reviewed-by: Micha= el Paquier, Justin Pryzby, Bharath Rupireddy, Tomas Vondra Discussion: [https://postgr.es/m/CAEze2WiOcgdH4aQA8NtZq-4dgvnJzp8PohdeKchPkhMY-jWZXA@= mail.gmail.com](https://postgr.es/m/CAEze2WiOcgdH4aQA8NtZq-4dgvnJzp8PohdeKc= hPkhMY-jWZXA@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/15639d5e8f6f278219681fec8a5668a= 92fb7e874](https://git.postgresql.org/pg/commitdiff/15639d5e8f6f278219681fe= c8a5668a92fb7e874) - Fix comment in indexing.c. 578b229, that removed support for WITH OIDS, h= as changed CatalogTupleInsert() to not return an Oid, but one comment was st= ill mentioning that. Author: Vik Fearing Discussion: [https://postgr.es/m/fef01975-ed10-3601-7b9e-80ecef72d00b@postgresfriends= .org](https://postgr.es/m/fef01975-ed10-3601-7b9e-80ecef72d00b@postgresfrie= nds.org) [https://git.postgresql.org/pg/commitdiff/9fd2952cf4920d563e9cea51634c5b3= 64d57f71a](https://git.postgresql.org/pg/commitdiff/9fd2952cf4920d563e9cea5= 1634c5b364d57f71a) - Improve tab completion of IMPORT FOREIGN SCHEMA with \h in psql. Only "IM= PORT" was showing as result of the completion, while IMPORT FOREIGN SCHEMA is t= he only command using this keyword in first position. This changes the completion to show the full command name instead of just "IMPORT". Reviewed-by: Georgios Kokolatos, Julien Rouhaud Discussion: [https://postgr.es/m/YFL6JneBiuMWYyoh@paquier.xyz](https://postgr.es/m/YF= L6JneBiuMWYyoh@paquier.xyz) [https://git.postgresql.org/pg/commitdiff/5b2266e33fc74142d23685bdf54f64a= d598fbdea](https://git.postgresql.org/pg/commitdiff/5b2266e33fc74142d23685b= df54f64ad598fbdea) Stephen Frost pushed: - Use pre-fetching for ANALYZE. When we have posix_fadvise() available, we = can improve the performance of an ANALYZE by quite a bit by using it to infor= m the kernel of the blocks that we're going to be asking for. Similar to bitmap index scans, the number of buffers pre-fetched is based off of the maintenance_io_concurrency setting (for the particular tablespace or, if = not set, globally, via get_tablespace_maintenance_io_concurrency()). Reviewe= d-By: Heikki Linnakangas, Tomas Vondra Discussion: [https://www.postgresql.org/message-id/VI1PR0701MB69603A433348EDCF783C6EC= BF6EF0%40VI1PR0701MB6960.eurprd07.prod.outlook.com](https://www.postgresql.= org/message-id/VI1PR0701MB69603A433348EDCF783C6ECBF6EF0%40VI1PR0701MB6960.e= urprd07.prod.outlook.com) [https://git.postgresql.org/pg/commitdiff/c6fc50cb40285141fad401321ae21be= cbaea1c59](https://git.postgresql.org/pg/commitdiff/c6fc50cb40285141fad4013= 21ae21becbaea1c59) - Improve logging of auto-vacuum and auto-analyze. When logging auto-vacuum= and auto-analyze activity, include the I/O timing if track_io_timing is enabl= ed. Also, for auto-analyze, add the read rate and the dirty rate, similar to = how that information has historically been logged for auto-vacuum. Stephen F= rost and Jakub Wartak Reviewed-By: Heikki Linnakangas, Tomas Vondra Discussio= n: [https://www.postgresql.org/message-id/VI1PR0701MB69603A433348EDCF783C6EC= BF6EF0%40VI1PR0701MB6960.eurprd07.prod.outlook.com](https://www.postgresql.= org/message-id/VI1PR0701MB69603A433348EDCF783C6ECBF6EF0%40VI1PR0701MB6960.e= urprd07.prod.outlook.com) [https://git.postgresql.org/pg/commitdiff/94d13d474dc61800e8a17cc1959c558= 15b050ecd](https://git.postgresql.org/pg/commitdiff/94d13d474dc61800e8a17cc= 1959c55815b050ecd) Robert Haas pushed: - Fix a confusing amcheck corruption message. Don't complain about the last TOAST chunk number being different from what we expected if there are no = TOAST chunks at all. In such a case, saying that the final chunk number is 0 is= not really accurate, and the fact the value is missing from the TOAST table is reported separately anyway. Mark Dilger Discussion: [http://postgr.es/m/AA5506CE-7D2A-42E4-A51D-358635E3722D@enterprisedb.com= ](http://postgr.es/m/AA5506CE-7D2A-42E4-A51D-358635E3722D@enterprisedb.com) [https://git.postgresql.org/pg/commitdiff/4078ce65a0f7197180a9be2c6460ea4= bf909bd98](https://git.postgresql.org/pg/commitdiff/4078ce65a0f7197180a9be2= c6460ea4bf909bd98) - Allow configurable LZ4 TOAST compression. There is now a per-column COMPRESSION option which can be set to pglz (the default, and the only op= tion in up until now) or lz4. Or, if you like, you can set the new default_toast_compression GUC to lz4, and then that will be the default f= or new table columns for which no value is specified. We don't have lz4 supp= ort in the PostgreSQL code, so to use lz4 compression, PostgreSQL must be bui= lt --with-lz4. In general, TOAST compression means compression of individual column values, not the whole tuple, and those values can either be compre= ssed inline within the tuple or compressed and then stored externally in the T= OAST table, so those properties also apply to this feature. Prior to this com= mit, a TOAST pointer has two unused bits as part of the va_extsize field, and a compessed datum has two unused bits as part of the va_rawsize field. These bits are unused because the length of a varlena is limited to 1GB; we now= use them to indicate the compression type that was used. This means we only h= ave bit space for 2 more built-in compresison types, but we could work around= that problem, if necessary, by introducing a new vartag_external value for any further types we end up wanting to add. Hopefully, it won't be too import= ant to offer a wide selection of algorithms here, since each one we add not o= nly takes more coding but also adds a build dependency for every packager. Nevertheless, it seems worth doing at least this much, because LZ4 gets b= etter compression than PGLZ with less CPU usage. It's possible for LZ4-compres= sed datums to leak into composite type values stored on disk, just as it is f= or PGLZ. It's also possible for LZ4-compressed attributes to be copied into a different table via SQL commands such as CREATE TABLE AS or INSERT .. SEL= ECT. It would be expensive to force such values to be decompressed, so Postgre= SQL has never done so. For the same reasons, we also don't force recompressio= n of already-compressed values even if the target table prefers a different compression method than was used for the source data. These architectural decisions are perhaps arguable but revisiting them is well beyond the sco= pe of what seemed possible to do as part of this project. However, it's relati= vely cheap to recompress as part of VACUUM FULL or CLUSTER, so this commit adj= usts those commands to do so, if the configured compression method of the table happens not to match what was used for some column value stored therein. Dilip Kumar. The original patches on which this work was based were writt= en by Ildus Kurbangaliev, and those were patches were based on even earlier wor= k by Nikita Glukhov, but the design has since changed very substantially, since allow a potentially large number of compression methods that could be add= ed and dropped on a running system proved too problematic given some of the architectural issues mentioned above; the choice of which specific compre= ssion method to add first is now different; and a lot of the code has been heav= ily refactored. More recently, Justin Przyby helped quite a bit with testing= and reviewing and this version also includes some code contributions from him. Other design input and review from Tomas Vondra, =C3=81lvaro Herrera, And= res Freund, Oleg Bartunov, Alexander Korotkov, and me. Discussion: [http://postgr.es/m/20170907194236.4cefce96%40wp.localdomain](http://post= gr.es/m/20170907194236.4cefce96%40wp.localdomain) Discussion: [http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FMdQrHKe= bw%40mail.gmail.com](http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSN= OTeqdaNRR9FMdQrHKebw%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/bbe0a81db69bd10bd166907c3701492= a29aca294](https://git.postgresql.org/pg/commitdiff/bbe0a81db69bd10bd166907= c3701492a29aca294) - Fix use-after-ReleaseSysCache problem in ATExecAlterColumnType. Introduce= d by commit bbe0a81db69bd10bd166907c3701492a29aca294. Per buildfarm member pr= ion. [https://git.postgresql.org/pg/commitdiff/d00fbdc431192c3e429b3e91c43d364= e5c7ba680](https://git.postgresql.org/pg/commitdiff/d00fbdc431192c3e429b3e9= 1c43d364e5c7ba680) Amit Kapila pushed: - Doc: Add a description of substream in pg_subscription. Commit 464824323e added a new column substream in pg_subscription but forgot to update the = docs. Reported-by: Peter Smith Author: Amit Kapila Reviewed-by: Peter Smith Discussion: [https://postgr.es/m/CAHut+PuPGGASnh2Dy37VYODKULVQo-5oE=3DShc6gwtRizDt=3D= =3DcA@mail.gmail.com](https://postgr.es/m/CAHut+PuPGGASnh2Dy37VYODKULVQo-5o= E=3DShc6gwtRizDt=3D=3DcA@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/7efeb214ad832fa96ea950d0906b1d2= b96316d15](https://git.postgresql.org/pg/commitdiff/7efeb214ad832fa96ea950d= 0906b1d2b96316d15) - Fix race condition in drop subscription's handling of tablesync slots. Co= mmit ce0fdbfe97 made tablesync slots permanent and allow Drop Subscription to = drop such slots. However, it is possible that before tablesync worker could ge= t the acknowledgment of slot creation, drop subscription stops it and that can = lead to a dangling slot on the publisher. Prevent cancel/die interrupts while creating a slot in the tablesync worker. Reported-by: Thomas Munro as per buildfarm Author: Amit Kapila Reviewed-by: Vignesh C, Takamichi Osumi Discussion: [https://postgr.es/m/CA+hUKGJG9dWpw1cOQ2nzWU8PHjm=3DPTraB+KgE5648K9nTfwvx= g@mail.gmail.com](https://postgr.es/m/CA+hUKGJG9dWpw1cOQ2nzWU8PHjm=3DPTraB+= KgE5648K9nTfwvxg@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/6b67d72b604cb913e39324b81b61ab1= 94d94cba0](https://git.postgresql.org/pg/commitdiff/6b67d72b604cb913e39324b= 81b61ab194d94cba0) - Add a new GUC and a reloption to enable inserts in parallel-mode. Commit 05c8482f7f added the implementation of parallel SELECT for "INSERT INTO .= .. SELECT ..." which may incur non-negligible overhead in the additional parallel-safety checks that it performs, even when, in the end, those che= cks determine that parallelism can't be used. This is normally only ever a pr= oblem in the case of when the target table has a large number of partitions. A= new GUC option "enable_parallel_insert" is added, to allow insert in parallel-mode. The default is on. In addition to the GUC option, the use= r may want a mechanism to allow inserts in parallel-mode with finer granularity= at table level. The new table option "parallel_insert_enabled" allows this. = The default is true. Author: "Hou, Zhijie" Reviewed-by: Greg Nancarrow, Amit Langote, Takayuki Tsunakawa, Amit Kapila Discussion: [https://postgr.es/m/CAA4eK1K-cW7svLC2D7DHoGHxdAdg3P37BLgebqBOC2ZLc9a6QQ%= 40mail.gmail.com](https://postgr.es/m/CAA4eK1K-cW7svLC2D7DHoGHxdAdg3P37BLge= bqBOC2ZLc9a6QQ%40mail.gmail.com) Discussion: [https://postgr.es/m/CAJcOf-cXnB5cnMKqWEp2E2z7Mvcd04iLVmV=3DqpFJrR3AcrTS3= g@mail.gmail.com](https://postgr.es/m/CAJcOf-cXnB5cnMKqWEp2E2z7Mvcd04iLVmV= =3DqpFJrR3AcrTS3g@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/c8f78b616167bf8e24bc5dc69112c37= 755ed3058](https://git.postgresql.org/pg/commitdiff/c8f78b616167bf8e24bc5dc= 69112c37755ed3058) - Doc: Update description for parallel insert reloption. Commit c8f78b6161 = added a new reloption to enable inserts in parallel-mode but forgot to update a= t one of the places about the same in docs. In passing, fix a typo in the same commit. Reported-by: Justin Pryzby Author: Justin Pryzby Reviewed-by: "H= ou, Zhijie", Amit Kapila Discussion: [https://postgr.es/m/20210318025228.GE11765@telsasoft.com](https://postgr= .es/m/20210318025228.GE11765@telsasoft.com) [https://git.postgresql.org/pg/commitdiff/ed62d3737c1b823f796d974060b1d02= 95a3dd831](https://git.postgresql.org/pg/commitdiff/ed62d3737c1b823f796d974= 060b1d0295a3dd831) Peter Eisentraut pushed: - Small error message improvement. [https://git.postgresql.org/pg/commitdiff/e1ae40f381d0582981b1e63856bd4b0= 60cfe2d53](https://git.postgresql.org/pg/commitdiff/e1ae40f381d0582981b1e63= 856bd4b060cfe2d53) - Move lwlock-release probe back where it belongs. The documentation specifically states that lwlock-release fires before any released waiters= have been awakened. It worked that way until ab5194e6f617a9a9e7aadb3dd1cee948a42d0755, where is seems to have been misplaced accidentally. Move it back where it belongs. Author: Craig Ri= nger Discussion: [https://www.postgresql.org/message-id/CAGRY4nwxKUS_RvXFW-ugrZBYxPFFM5kjw= KT5O+0+Stuga5b4+Q@mail.gmail.com](https://www.postgresql.org/message-id/CAG= RY4nwxKUS_RvXFW-ugrZBYxPFFM5kjwKT5O+0+Stuga5b4+Q@mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/96ae658e6238c5e69819fb1557c2c14= a555506d8](https://git.postgresql.org/pg/commitdiff/96ae658e6238c5e69819fb1= 557c2c14a555506d8) Andres Freund pushed: - Fix memory lifetime issues of replication slot stats. When accessing replication slot stats, introduced in 98681675002d, pgstat_read_statsfile= s() reads the data into newly allocated memory. Unfortunately the current mem= ory context at that point is the callers, leading to leaks and use-after-free dangers. The fix is trivial, explicitly use pgStatLocalContext. There's = some potential for further improvements, but that's outside of the scope of th= is bugfix. No backpatch necessary, feature is only in HEAD. Author: Andres Freund Discussion: [https://postgr.es/m/20210317230447.c7uc4g3vbs4wi32i@alap3.anarazel.de](h= ttps://postgr.es/m/20210317230447.c7uc4g3vbs4wi32i@alap3.anarazel.de) [https://git.postgresql.org/pg/commitdiff/5f79580ad69f6e696365bdc63bc265f= 45bd77211](https://git.postgresql.org/pg/commitdiff/5f79580ad69f6e696365bdc= 63bc265f45bd77211) Magnus Hagander pushed: - Fix function name in error hint. pg_read_file() is the function that's in core, pg_file_read() is in adminpack. But when using pg_file_read() in adminpack it calls the *C* level function pg_read_file() in core, which probably threw the original author off. But the error hint should be abou= t the SQL function. Reported-By: Sergei Kornilov Backpatch-through: 11 Discuss= ion: [https://postgr.es/m/373021616060475@mail.yandex.ru](https://postgr.es/m/= 373021616060475@mail.yandex.ru) [https://git.postgresql.org/pg/commitdiff/da18d829c28197efb04805a43f129f6= 2650e50c8](https://git.postgresql.org/pg/commitdiff/da18d829c28197efb04805a= 43f129f62650e50c8) Tom=C3=A1=C5=A1 Vondra pushed: - Remove temporary files after backend crash. After a crash of a backend us= ing temporary files, the files used to be left behind, on the basis that it m= ight be useful for debugging. But we don't have any reports of anyone actually doing that, and it means the disk usage may grow over time due to repeated backend failures (possibly even hitting ENOSPC). So this behavior is a bit unfortunate, and fixing it required either manual cleanup (deleting files, which is error-prone) or restart of the instance (i.e. service disruption= ). This implements automatic cleanup of temporary files, controled by a new = GUC remove_temp_files_after_crash. By default the files are removed, but it c= an be disabled to restore the old behavior if needed. Author: Euler Taveira Reviewed-by: Tomas Vondra, Michael Paquier, Anastasia Lubennikova, Thomas Munro Discussion: [https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%2B6_6xTJ-Yn5WSvoHLY= 1Ww%40mail.gmail.com](https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%= 2B6_6xTJ-Yn5WSvoHLY1Ww%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/cd91de0d17952b5763466cfa663e983= 18f26d357](https://git.postgresql.org/pg/commitdiff/cd91de0d17952b5763466cf= a663e98318f26d357) - Implement GROUP BY DISTINCT. With grouping sets, it's possible that some = of the grouping sets are duplicate. This is especially common with CUBE and ROLLUP clauses. For example GROUP BY CUBE (a,b), CUBE (b,c) is equivalent= to GROUP BY GROUPING SETS ( (a, b, c), (a, b, c), (a, b, c), = (a, b), (a, b), (a, b), (a), (a), (a), (c, a), (c= , a), (c, a), (c), (b, c), (b), () ) Some of the grouping se= ts are calculated multiple times, which is mostly unnecessary. This commit implements a new GROUP BY DISTINCT feature, as defined in the SQL standar= d, which eliminates the duplicate sets. Author: Vik Fearing Reviewed-by: Er= ik Rijkers, Georgios Kokolatos, Tomas Vondra Discussion: [https://postgr.es/m/bf3805a8-d7d1-ae61-fece-761b7ff41ecc@postgresfriends= .org](https://postgr.es/m/bf3805a8-d7d1-ae61-fece-761b7ff41ecc@postgresfrie= nds.org) [https://git.postgresql.org/pg/commitdiff/be45be9c33a85e72cdaeb9967e9f6d2= d00199e09](https://git.postgresql.org/pg/commitdiff/be45be9c33a85e72cdaeb99= 67e9f6d2d00199e09) - Fix TAP test for remove_temp_files_after_crash. The test included in cd91de0d17 had two simple flaws. Firstly, the number of rows was low and= on some platforms (e.g. 32-bit) the sort did not require on-disk sort, so on those machines it was not testing the automatic removal. The test was ho= wever failing, because without any temporary files the base/pgsql_tmp directory= was not even created. Fixed by increasing the rowcount to 5000, which should= be high engough on any platform. Secondly, the test used a simple sleep to = wait for the temporary file to be created. This is obviously problematic, bec= ause on slow machines (or with valgrind, CLOBBER_CACHE_ALWAYS etc.) it may tak= e a while to create the temporary file. But we also want the tests run reaso= nably fast. Fixed by instead relying on a UNIQUE constraint, blocking the query= that created the temporary file. Author: Euler Taveira Reviewed-by: Tomas Von= dra Discussion: [https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%2B6_6xTJ-Yn5WSvoHLY= 1Ww%40mail.gmail.com](https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%= 2B6_6xTJ-Yn5WSvoHLY1Ww%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/a16b2b960f0eec2fe367e86017b3c24= ed688ba2b](https://git.postgresql.org/pg/commitdiff/a16b2b960f0eec2fe367e86= 017b3c24ed688ba2b) - Fix race condition in remove_temp_files_after_crash TAP test. The TAP tes= t was written so that it was not waiting for the correct SQL command, but for o= utput from the preceding one. This resulted in race conditions, allowing the commands to run in a different order, not block as expected and so on. T= his fixes it by inverting the order of commands where possible, so that obser= ving the output guarantees the data was inserted properly, and waiting for a l= ock to appear in pg_locks. Discussion: [https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%2B6_6xTJ-Yn5WSvoHLY= 1Ww%40mail.gmail.com](https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%= 2B6_6xTJ-Yn5WSvoHLY1Ww%40mail.gmail.com) [https://git.postgresql.org/pg/commitdiff/e589c4890b05044a04207c2797e7c8a= f6693ea5f](https://git.postgresql.org/pg/commitdiff/e589c4890b05044a04207c2= 797e7c8af6693ea5f) - Use lfirst_int in cmp_list_len_contents_asc. The function added in be45be= 9c33 is comparing integer lists (IntList) by length and contents, but there we= re two bugs. Firstly, it used intVal() to extract the value, but that's for Value nodes, not for extracting int values from IntList. Secondly, it ca= lled it directly on the ListCell, without doing lfirst(). So just do lfirst_i= nt() instead. Interestingly enough, this did not cause any crashes on the buildfarm, but valgrind rightfully complained about it. Discussion: [https://postgr.es/m/bf3805a8-d7d1-ae61-fece-761b7ff41ecc@postgresfriends= .org](https://postgr.es/m/bf3805a8-d7d1-ae61-fece-761b7ff41ecc@postgresfrie= nds.org) [https://git.postgresql.org/pg/commitdiff/b822ae13ea93c18326d58d47829bbc6= 6d36fae5c](https://git.postgresql.org/pg/commitdiff/b822ae13ea93c18326d58d4= 7829bbc66d36fae5c) - Use valid compression method in brin_form_tuple. When compressing the BRIN summary, we can't simply use the compression method from the indexed attribute. The summary may use a different data type, e.g. fixed-length attribute may have varlena summary, leading to compression failures. For= the built-in BRIN opclasses this happens to work, because the summary uses the same data type as the attribute. When the data types match, we can inher= it use the compression method specified for the attribute (it's copied into = the index descriptor). Otherwise we don't have much choice and have to use the default one. Author: Tomas Vondra Reviewed-by: Justin Pryzby Discussion: [https://postgr.es/m/e0367f27-392c-321a-7411-a58e1a7e4817%40enterprisedb.= com](https://postgr.es/m/e0367f27-392c-321a-7411-a58e1a7e4817%40enterprised= b.com) [https://git.postgresql.org/pg/commitdiff/882b2cdc08c4100e273f24742e2118b= e98708a07](https://git.postgresql.org/pg/commitdiff/882b2cdc08c4100e273f247= 42e2118be98708a07) Andrew Dunstan pushed: - Unbreak recovery test on Windows. On Windows we need to send explicit quit messages to psql or the TAP tests can hang. [https://git.postgresql.org/pg/commitdiff/677271a3a125e294b33b891669f594a= 2c8cb36ce](https://git.postgresql.org/pg/commitdiff/677271a3a125e294b33b891= 669f594a2c8cb36ce) # Pending Patches Erica Zhang sent in another revision of a patch to add some tests for pg_stat_statements compatibility verification under contrib. J=C3=BCrgen Purtz sent in another revision of a patch to change the JOIN tu= torial to focus more on explicit joins. Noah Misch sent in a patch to merge similar algorithms into roles_is_member_of(). David Rowley sent in another revision of a patch to cache PathTarget and RestrictInfo's volatility, allow estimate_num_groups() to pass back further details about the estimation, allow the users of simplehash.h to perform di= rect deletions, add a Result Cache executor node, and remove code duplication in nodeResultCache.c. Vigneshwaran C sent in a patch to fail prepared transaction if transaction = has locked system tables/user catalog tables. Laurenz Albe sent in another revision of a patch to improve \e, \ef and \ev= if the editor is quit without saving by erasing the query buffer rather than executing the previous statement. Nathan Bossart sent in another revision of a patch to avoid creating archive status ".ready" files too early, and keep track of notified-ready-for-archi= ve position through crashes. Mike Insoo sent in a patch to check whether, when decoding a catalog modify= ing commit record, it=E2=80=99s part of the RUNNING_XACT xid=E2=80=99s processe= d @ the restart_lsn. If so, then add its xid & subxacts in the committed txns list in the logical decoding snapshot. Justin Pryzby sent in three more revisions of a patch to make it possible t= o use different compression methods for FPI. Jim Mlodgenski sent in another revision of a patch to add a parser hook and= a test that uses it. Julien Rouhaud sent in another revision of a patch to add a new OUTDATED filtering facility for REINDEX command, with a corresponding option for the reindexdb utility. When used, REINDEX will only process indexes that have = an outdated dependency. For now, only dependency on collations are supported = but we'll likely support other kind of dependency in the future. Georgios Kokolatos and Micha=C3=ABl Paquier traded patches to make dbsize a= bit more consistent. Amit Kapila, Justin Pryzby, and Greg Nancarrow traded patches to make it possible to execute INSERT .... SELECT in parallel. Bruce Momjian sent in two more revisions of a patch to implement key manage= ment. Bharath Rupireddy sent in four more revisions of a patch to add a version of pg_terminate_backend() with wait and timeout. Masahiro Ikeda sent in another revision of a patch to make the WAL receiver report WAL statistics. Iwata Aya sent in four more revisions of a patch to implement a libpq debug= log. Amul Sul sent in two more revisions of a patch to add the infrastructure AL= TER SYSTEM READ {ONLY | WRITE} will use. Jacob Champion and Micha=C3=ABl Paquier traded patches to log authenticated= identity from all auth backends. Masahiro Ikeda sent in four revisions of a patch to make the stats collector shutdown without writing the statsfiles if the immediate shutdown is reques= ted. Mark Rofail and Andreas Karlsson traded patches to implement foreign key ar= rays. Ajin Cherian, Peter Smith, and Takamichi Osumi traded patches to implement logical decoding of two-phase transactions. =C3=81lvaro Herrera sent in three more revisions of a patch to implement AL= TER TABLE .. DETACH PARTITION CONCURRENTLY. Amit Langote sent in another revision of a patch to allow batching of inser= ts during cross-partition updates. IIvan Panchenko sent in another revision of a patch to add a login trigger. Peter Eisentraut sent in another revision of a patch to make it possible to= get dynamic result sets from procedures. Declaring a cursor WITH RETURN in a procedure makes the cursor's data be returned as a result of the CALL invocation. The procedure needs to be declared with the DYNAMIC RESULT SETS attribute. Amit Kapila sent in a doc patch to add a description of substream in pg_subscription. Amit Kapila sent in a patch to prevent cancel/die interrupts while creating tablesync slot. Bharath Rupireddy sent in another revision of a patch to refactor the code = that implements REFRESH MATERIALIZED VIEW for comprehensibility. Ranier Vilela sent in a patch to cover a corner case in renaming files on Windows. Hou Zhijie sent in another revision of a patch to extend the safety check to support parallel insert into fk relations, and skip the CommandCounterIncre= ment. Fujii Masao sent in two more revisions of a patch to improve pgbench's \sle= ep command. Andres Freund and Kyotaro HORIGUCHI traded patches to make the stats collec= tor use shared memory instead of files for temporary storage in the stats colle= ctor. Masahiko Sawada sent in another revision of a patch to fix transactions involving multiple postgres foreign servers. Bharath Rupireddy sent in another revision of a patch to mprove PID XXXX is= not a PostgreSQL server process message. Instead of the message "is not a serv= er process" just say "is not a backend process". This would make things clear = that the given PID is not a backend process but it can be some other process's p= id or some other postmaster child process's pid or an invalid pid. Zeng Wenjing sent in another revision of a patch to implement global tempor= ary tables. Tom=C3=A1=C5=A1 Vondra sent in another revision of a patch to implement ind= ex skip scans. Atsushi Torikoshi sent in another revision of a patch to add memorycontext = elog print. John Naylor sent in two more revisions of a patch to fix bug in heap space management that was overly cautious about fillfactor. Previously, if there= were leftover ununused line pointers in a page, a table with a low fill factor w= ould not be available for new tuples that were large enough to exceed fillfactor, even though the page contained no tuples at all. This lead to unnecessary relation extension. Fix by allowing some slack space equal to 1/8 the maxi= mum space that could be taken up by line pointers. This is somewhat arbitrary, = but should allow many cases to succeed where they didn't before. Per report fr= om Floris van Nee. Pavel St=C4=9Bhule sent in another revision of a patch to add routine label= s to PL/pgsql. Micha=C3=ABl Paquier sent in another revision of a patch to fix an infelici= ty between PITR and 2PC. Bertrand Drouvot sent in another revision of a patch to implement minimal logical decoding on standbys. Amit Langote sent in a patch to tweak handling of serialization failures du= ring cascaded RI update/delete. Peter Eisentraut sent in another revision of a patch to ibpq: Set Server Na= me Indication (SNI) for SSL connections By default, have libpq set the TLS extension "Server Name Indication" (SNI). This allows an SNI-aware SSL pro= xy to route connections. (This requires a proxy that is aware of the PostgreSQL protocol, not just any SSL proxy.) In the future, this could also allow the server to use different SSL certificates for different host specifications. (That would require new server functionality. This would be the client-side functionality for that.) Since SNI makes the host name appear in cleartext = in the network traffic, this might be undesirable in some cases. Therefore, a= lso add a libpq connection option "sslsni" to turn it off. Amit Langote sent in another revision of a patch to mark fully grouped rela= tions partitioned if input relation is, and allow setting parallel_workers on partitioned tables. Ibrar Ahmed sent in another revision of a patch to make it possible for INS= ERT SELECT to use BulkInsertState and multi_insert. John Naylor sent in another revision of a patch to add a 'noError' argument= to encoding conversion functions, and use same to do COPY FROM encoding conversion/verification in larger chunks. Thomas Munro sent in another revision of a patch to use prefetching for WAL= s. Peter Geoghegan sent in a patch to add a way to ignore pgindent runs and si= milar when using `git blame`. Peter Eisentraut sent in another revision of a patch to add a result_format_auto_binary_types setting. Fujii Masao sent in another revision of a patch to change the type of WalReceiverWaitStart wait event from Client to IPC. Marina Polyakova sent in a patch to reduce the time required to recover dat= abase from archive by allowing concurrent loading of WAL files. Peter Eisentraut sent in another revision of a patch to implement SQL-stand= ard function bodies. Andrei Zubkov sent in a patch to correct deallocation events accounting in pg_stat_statements. James Coleman sent in another revision of a patch to Improve standby connec= tion denied error message. Currently when a standby is finished starting up but hot_standby is configured to off, the error message when a client connects = is "the database system is starting up", which is needless confusing (and not really all that accurate either). Instead send a helpful error message so = that the user immediately knows that their server is configured to deny these connections. John Naylor sent in another revision of a patch to add a 'noError' argument= to encoding conversion functions, then use same to replace pg_utf8_verifystr()= with two faster implementations: one using SSE-4.1 instructions on Intel, the ot= her using a bespoke fallback function, rather than one that relies on pg_utf8_verifychar() and pg_utf8_isvalid(). This one is loosely based on the fallback that is part of the simdjson library. Peter Eisentraut sent in another revision of a patch to change the return t= ype of EXTRACT to numeric. James Coleman sent in another revision of a patch to hash lookup const arra= ys in OR'd ScalarArrayOps. Amit Khandekar sent in another revision of a patch to speed up xor'ing of t= wo gist index signatures for tsvectors by using 64-bit chunks instead of char values, and avoid function pointer dereferencing for the pg_popcount32/64() call. Etsuro Fujita sent in another revision of a patch to implement asynchronous append on postgres_fdw nodes. Konstantin Knizhnik sent in another revision of a patch to add two statisti= cs GUCs to auto_explain: add_statistics_threshold, and add_statistics_suggest_= only. Gilles Darold sent in another revision of a patch to add hooks at the XactCommand level. Tatsuo Ishii sent in three more revisions of a patch to make it possible to= use COPY FREEZE in pgbench. Julien Rouhaud sent in another revision of a patch to add queryId to the pg_catalog.pg_stat_activity view. Amit Langote sent in another revision of a patch to export get_partition_for_tuple() to make it visible to things other than execPartition.c can use it, and use it elsewhere to avoid using SPI for som= e RI checks. Peter Smith sent in a patch to fixed a bug where the code was de-referencin= g the hash entry returned by HASH_REMOVE in stream_cleanup_files. Thomas Munro sent in another revision of a patch to support PSQL_WATCH_PAGE= R for psql's \watch command, and try sigwait() to make it work. Micha=C3=ABl Paquier sent in a patch to clean up the Kerberos auth test. Thomas Munro sent in another revision of a patch to provide recovery_init_sync_method=3Dwal. Gilles Darold and Erik Rijkers traded patches to add some new functions for counting and otherwise characterizing regexp matches. David Fetter sent in another revision of a patch to implement popcount (now called bit_count). Justin Pryzby sent in another revision of a patch to add a way to use diffe= rent compression methods for FPI. Matthias van de Meent sent in a patch to ensure that when [+] is an option = for a psql metacommand, \? output mentions that it is, and not otherwise. Jan Wieck and Tom Lane traded patches to ensure that pg_upgrade preserves datdba. Konstantin Knizhnik sent in another revision of a patch to implement a buil= t-in connection pooler. --===============3798017092296226947== Content-Type: text/html; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable PostgreSQL Weekly News - March 21, 2021
 

PostgreSQL Weekly News - March 21, 2021

PostgreSQL Weekly News - March 21, 2021

Person of the week: https://postgresql.life/post/anastasia_lubennikova/<= /p>

PostgreSQL Product News

pspg 4.4.0 a pager designed for PostgreSQL,= released. https://github.com/okbob/pspg/releas= es/tag/4.4.0

pg_activity 2.1.2, a top-like application f= or PostgreSQL server activity monitoring, released. https://github.com/dalibo/p= g_activity/releases/tag/v2.1.2

PostgreSQL Jobs for March

h= ttps://archives.postgresql.org/pgsql-jobs/2021-03/

PostgreSQL in the News

Planet PostgreSQL: htt= ps://planet.postgresql.org/

PostgreSQL Weekly News is brought to you th= is week by David Fetter

Submit news and announcements by Sunday at = 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Peter Geoghegan pushed:

Fujii Masao pushed:

  • Make archiver process an auxiliary process.= This commit changes WAL archiver process so that it's treated as an auxiliary process and can use shared memory. This is an infrastructure patch required for upcoming shared-memo= ry based stats collector patch series. These patch series basically need any processes including archiver that can report the statistics to access to shared memory. Since this patch itself is useful to simplify the code and= when users monitor the status of archiver, it's committed separately in advanc= e. This commit simplifies the code for WAL archiving. For example, previously backends need to signal to archiver via postmaster when they notify archi= ver that there are some WAL files to archive. On the other hand, this commit removes that signal to postmaster and enables backends to notify archier directly using shared latch. Also, as the side of this change, the information about archiver process becomes viewable at pg_stat_activity v= iew. Author: Kyotaro Horiguchi Reviewed-by: Andres Freund, =C3=81lvaro Herrera= , Julien Rouhaud, Tomas Vondra, Arthur Zakirov, Fujii Masao Discussion: https= ://postgr.es/m/20180629.173418.190173462.horiguchi.kyotaro@lab.ntt.co.jp htt= ps://git.postgresql.org/pg/commitdiff/d75288fb27b8fe0a926aaab7d75816f091ecd= c27

  • Fix comments in postmaster.c. Commit 86c23a= 6eb2 changed the option to specify that postgres will stop all other server processes by sending the signal SIGSTOP, from -s to -T. But previously there were comments incorrectly explaining that SIGSTOP behavior is set by -s option. This commit fixes t= hem. Author: Kyotaro Horiguchi Reviewed-by: Fujii Masao Discussion: htt= ps://postgr.es/m/20210316.165141.1400441966284654043.horikyota.ntt@gmail.co= m htt= ps://git.postgresql.org/pg/commitdiff/fd31214075cc740e43edc71ca1c385c8c5304= 7b7

Thomas Munro pushed:

Tom Lane pushed:

  • Work around issues in MinGW-64's setjmp/lon= gjmp support. It's hard to avoid the conclusion that there is something wrong with setjmp/longjmp on MinGW= -64, as we have seen failures come and go after entirely-unrelated-looking cha= nges in our own code. Other projects such as Ruby have given up and started u= sing gcc's setjmp/longjmp builtins on that platform; this patch just follows t= hat lead. Note that this is a pretty fundamental ABI break for functions containining either setjmp or longjmp, so we can't really consider a back-patch. Per reports from Regina Obe and Heath Lord, as well as recent failures on buildfarm member walleye, and less-recent failures on fairywr= en. Juan Jos=C3=A9 Santamar=C3=ADa Flecha Discussion: https://postgr.es/m/00= 0401d716a0$1ed0fc70$5c72f550$@pcorp.us Discussion: https://postgr.es/m/CA+BEBhvHhM-Bn628pf-LsjqRh3Ang7qCSBG0Ga+7KwhGqrNUPw= @mail.gmail.com Discussion: https://po= stgr.es/m/f1caef93-9640-022e-9211-bbe8755a56b0@2ndQuadrant.com htt= ps://git.postgresql.org/pg/commitdiff/146cb3889c3ccb3fce198fe7464a1296a9e10= 7c3

  • Improve logging of bad parameter values in = BIND messages. Since commit ba79cb5dc, values of bind parameters have been logged during errors in extended query mode. However, we only did that after we'd collected and converted all the parameter values, thus failing to offer any useful localization of invalid-parameter problems. Add a separate callback that= 's used during parameter collection, and have it print the parameter number, along with the input string if text input format is used. Justin Pryzby = and Tom Lane Discussion: https://postgr.es/m/2021010= 4170939.GH9712@telsasoft.com Discussion: https://postgr.es/m/CANfkH5k-6nNt-4cSv1vPB80nq2BZCzhFVR5O4VznYbsX0wZmow= @mail.gmail.com htt= ps://git.postgresql.org/pg/commitdiff/1ea396362be1615e926ea69d666c770081a0d= 3ef

  • Avoid corner-case memory leak in SSL parame= ter processing. After reading the root cert list from the ssl_ca_file, immediately install it as client CA = list of the new SSL context. That gives the SSL context ownership of the list= , so that SSL_CTX_free will free it. This avoids a permanent memory leak if we= fail further down in be_tls_init(), which could happen if bogus CRL data is offered. The leak could only amount to something if the CRL parameters g= et broken after server start (else we'd just quit) and then the server is SIGHUP'd many times without fixing the CRL data. That's rather unlikely perhaps, but it seems worth fixing, if only because the code is clearer t= his way. While we're here, add some comments about the memory management asp= ects of this logic. Noted by Jelte Fennema and independently by Andres Freund. Back-patch to v10; before commit de41869b6 it doesn't matter, since we'd = not re-execute this code during SIGHUP. Discussion: https://postgr.es/m/16160= -18367e56e9a28264@postgresql.org htt= ps://git.postgresql.org/pg/commitdiff/4b12ab18c9d0735d760bf7667b158707b06e5= df8

  • Doc: improve discussion of variable substit= ution in PL/pgSQL. This was a bit disjointed, partly because of a not-well-considered decision to document = SQL commands that don't return result rows as though they had nothing in comm= on with commands that do. Rearrange so that we have one discussion of varia= ble substitution that clearly applies to all types of SQL commands, and then handle the question of processing command output separately. Clarify that EXPLAIN, CREATE TABLE AS SELECT, and similar commands that incorporate an optimizable statement will act like optimizable statements for the purpos= es of variable substitution. Do a bunch of minor wordsmithing in the same area. David Johnston and Tom Lane, reviewed by Pavel Stehule and David Steele Discussion: https://postgr.es/m/CAKFQuwYvMKucM5fnZvHSo-ah4S=3D_n9gmKeu6EA= o=3D_fTrohunqQ@mail.gmail.com htt= ps://git.postgresql.org/pg/commitdiff/c783e656d41816b0328cb4bff27f11b702007= 70e

  • Add end-to-end testing of pg_basebackup's t= ar-format output. The existing test script does run pg_basebackup with the -Ft option, but it makes no real attempt to verify the sanity of the results. We wouldn't know if the outp= ut is incompatible with standard "tar" programs, nor if the server fails to sta= rt from the restored output. Notably, this means that xlog.c's read_tablespace_map() is not being meaningfully tested, since that code is used only in the tar-format case. (We do have reasonable coverage of restoring from plain-format output, though it's over in src/test/recovery= not here.) Hence, attempt to untar the output and start a server from it, ra= ther just hoping it's OK. This test assumes that the local "tar" has the "-C directory" switch. Although that's not promised by POSIX, my research suggests that all non-extinct tar implementations have it. Should the buildfarm's opinion differ, we can complicate the test a bit to avoid requiring that. Possibly this should be back-patched, but I'm unsure abo= ut whether it could work on Windows before d66b23b03. h= ttps://git.postgresql.org/pg/commitdiff/081876d75ea15c3bd2ee5ba64a794fd8ea4= 6d794

  • Prevent buffer overrun in read_tablespace_m= ap(). Robert Foggia of Trustwave reported that read_tablespace_map() fails to prevent an overrun of its on-stack input buffer. Since the tablespace map file is presumed trustwor= thy, this does not seem like an interesting security vulnerability, but still = we should fix it just in the name of robustness. While here, document that pg_basebackup's --tablespace-mapping option doesn't work with tar-format output, because it doesn't. To make it work, we'd have to modify the tablespace_map file within the tarball sent by the server, which might be possible but I'm not volunteering. (Less-painful solutions would require changing the basebackup protocol so that the source server could adjust t= he map. That's not very appetizing either.) h= ttps://git.postgresql.org/pg/commitdiff/a50e4fd028a1ece2b1a04df2c9ae6581783= e9eef

  • Code review for server's handling of "table= space map" files. While looking at Robert Foggia's report, I noticed a passel of other issues in the same ar= ea:

  • The scheme for backslash-quoting newlines i= n pathnames is just wrong; it will misbehave if the last ordinary character in a pathname is a backslas= h. I'm not sure why we're bothering to allow newlines in tablespace paths, b= ut if we're going to do it we should do it without introducing other problems. Hence, backslashes themselves have to be backslashed too. * The author h= adn't read the sscanf man page very carefully, because this code would drop any leading whitespace from the path. (I doubt that a tablespace path with leading whitespace could happen in practice; but if we're bothering to al= low newlines in the path, it sure seems like leading whitespace is little less implausible.) Using sscanf for the task of finding the first space is overkill anyway. * While I'm not 100% sure what the rationale for escapi= ng both \r and \n is, if the idea is to allow Windows newlines in the file t= hen this code failed, because it'd throw an error if it saw \r followed by \n= . * There's no cross-check for an incomplete final line in the map file, which would be a likely apparent symptom of the improper-escaping bug. On the generation end, aside from the escaping issue we have: * If needtblspcma= pfile is true then do_pg_start_backup will pass back escaped strings in tablespaceinfo->path values, which no caller wants or is prepared to d= eal with. I'm not sure if there's a live bug from that, but it looks like th= ere might be (given the dubious assumption that anyone actually has newlines = in their tablespace paths). * It's not being very paranoid about the possib= ility of random stuff in the pg_tblspc directory. IMO we should ignore anything without an OID-like name. The escaping rule change doesn't seem back-patchable: it'll require doubling of backslashes in the tablespace_m= ap file, which is basically a basebackup format change. The odds of that ca= using trouble are considerably more than the odds of the existing bug causing trouble. The rest of this seems somewhat unlikely to cause problems too, = so no back-patch. h= ttps://git.postgresql.org/pg/commitdiff/8620a7f6dbdf978e57cdb9f42802a041865= 6d863

  • Doc: remove duplicated step in RLS example.= Seems to have been a copy-and-paste mistake in 093129c9d. Per report from max1@inbox.ru. Discussion: = https://postgr.es/m/161591740692.24273.4202054598867879464@wrigleys.postgre= sql.org htt= ps://git.postgresql.org/pg/commitdiff/70945649d734d16be22c3d1d90dd8c3d3c1e9= d89

  • Fix misuse of foreach_delete_current(). Our= coding convention requires this macro's result to be assigned back to the original List variable. In this usage, since the List could not become empty, there was no actual bug ---= but some compilers warned about it. Oversight in be45be9c3. Discussion: https://p= ostgr.es/m/35077b31-2d62-1e31-0e2e-ddb52d590b73@enterprisedb.com htt= ps://git.postgresql.org/pg/commitdiff/1d581ce7129d7a33cd4ad27f8f246abfa1fd2= db9

  • Don't leak rd_statlist when a relcache entr= y is dropped. Although these lists are usually NIL, and even when not empty are unlikely to be large, consta= nt relcache update traffic could eventually result in visible bloat of CacheMemoryContext. Found via valgrind testing. Back-patch to v10 where = this field was added. Discussion: https://postgr.es/m/3816764.16= 16104288@sss.pgh.pa.us htt= ps://git.postgresql.org/pg/commitdiff/28644fac10731e30e70b622986a6fbbeb5a5b= 2f9

  • Don't leak malloc'd error string in libpqrc= v_check_conninfo(). We leaked the error report from PQconninfoParse, when there was one. It seems unlikely= that real usage patterns would repeat the failure often enough to create serio= us bloat, but let's back-patch anyway to keep the code similar in all branch= es. Found via valgrind testing. Back-patch to v10 where this code was added. Discussion: https://postgr.es/m/3816764.16= 16104288@sss.pgh.pa.us htt= ps://git.postgresql.org/pg/commitdiff/9bacdf9f536a3720976ae258238cb46c691ce= 9b2

  • Don't leak compiled regex(es) when an ispel= l cache entry is dropped. The text search cache mechanisms assume that we can clean up an invalidated dictio= nary cache entry simply by resetting the associated long-lived memory context. However, that does not work for ispell affixes that make use of regular expressions, because the regex library deals in plain old malloc. Hence,= we leaked compiled regex(es) any time we dropped such a cache entry. That c= ould quickly add up, since even a fairly trivial regex can use up tens of kB, = and a large one can eat megabytes. Add a memory context callback to ensure tha= t a regex gets freed when its owning cache entry is cleared. Found via valgr= ind testing. This problem is ancient, so back-patch to all supported branches. Discussion: https://postgr.es/m/3816764.16= 16104288@sss.pgh.pa.us htt= ps://git.postgresql.org/pg/commitdiff/d303849b059c3c315e5a8d4239016f8328f32= 96c

  • Don't run RelationInitTableAccessMethod in = a long-lived context. Some code paths in this function perform syscache lookups, which can lead to table accesses and possibly leakage of cruft into the caller's context. If said context is CacheMemoryContext, we eventually will have visible bloat. But fixing this is no harder than moving one memory context switch step. (The other callers don't have a problem.) Andres Freund and I independently f= ound this via valgrind testing. Back-patch to v12 where this code was added. Discussion: https://postg= r.es/m/20210317023101.anvejcfotwka6gaa@alap3.anarazel.de Discussion: https://postgr.es/m/3816764.16= 16104288@sss.pgh.pa.us htt= ps://git.postgresql.org/pg/commitdiff/415ffdc2205e209b6a73fb42a3fdd6e57e16c= 7b2

  • Don't leak malloc'd strings when a GUC sett= ing is rejected. Because guc.c prefers to keep all its string values in malloc'd not palloc'd storage, i= t has to be more careful than usual to avoid leaks. Error exits out of string = GUC hook checks failed to clear the proposed value string, and error exits ou= t of ProcessGUCArray() failed to clear the malloc'd results of ParseLongOption= (). Found via valgrind testing. This problem is ancient, so back-patch to all supported branches. Discussion: https://postgr.es/m/3816764.16= 16104288@sss.pgh.pa.us htt= ps://git.postgresql.org/pg/commitdiff/377b7a83007d277d32ef19f7c7590c8668d50= 4cb

  • Blindly try to fix test script's tar invoca= tion for MSYS. Buildfarm member fairywren doesn't like the test case I added in commit 081876d75. I'm guessing the reason is that I shouldn't be using a perl2host-ified path i= n the tar command line. h= ttps://git.postgresql.org/pg/commitdiff/27ab1981e7c9b8fcbcb143c5f6f706441a5= 2bbc8

  • Avoid leaking memory in RestoreGUCState(), = and improve comments. RestoreGUCState applied InitializeOneGUCOption to already-live GUC entrie= s, causing any malloc'd subsidiary data to be forgotten. We do want the effe= ct of resetting the GUC to its compiled-in default, and InitializeOneGUCOption = seems like the best way to do that, so add code to free any existing subsidiary= data beforehand. The interaction between can_skip_gucvar, SerializeGUCState, = and RestoreGUCState is way more subtle than their opaque comments would sugge= st to an unwary reader. Rewrite and enlarge the comments to try to make it cle= arer what's happening. Remove a long-obsolete assertion in read_nondefault_variables: the behavior of set_config_option hasn't depen= ded on IsInitProcessingMode since f5d9698a8 installed a better way of control= ling it. Although this is fixing a clear memory leak, the leak is quite unlik= ely to involve any large amount of data, and it can only happen once in the lifetime of a worker process. So it seems unnecessary to take any risk of back-patching. Discussion: https://postgr.es/m/4105247.16= 16174862@sss.pgh.pa.us htt= ps://git.postgresql.org/pg/commitdiff/f0c2a5bba6c566fad781802537eb17f297770= 2bc

  • Fix memory leak when rejecting bogus DH par= ameters. While back-patching e0e569e1d, I noted that there were some other places where we ought to be applying DH_free(); namely, where we load some DH parameters from a file = and then reject them as not being sufficiently secure. While it seems really unlikely that anybody would hit these code paths in production, let alone= do so repeatedly, let's fix it for consistency. Back-patch to v10 where this code was introduced. Discussion: https://postgr.es/m/16160= -18367e56e9a28264@postgresql.org htt= ps://git.postgresql.org/pg/commitdiff/e835e89a0fd267871e7fbddc39ad00ee3d0cb= 55c

  • Fix up pg_dump's handling of per-attribute = compression options. The approach used in commit bbe0a81db would've been disastrous for portability of dump= s. Instead handle non-default compression options in separate ALTER TABLE commands. This reduces chatter for the common case where most columns are compressed the same way, and it makes it possible to restore the dump to a server that lacks any knowledge of per-attribute compression options (so = long as you're willing to ignore syntax errors from the ALTER TABLE commands). There's a whole lot left to do to mop up after bbe0a81db, but I'm fast-tracking this part because we need to see if it's enough to make the buildfarm's cross-version-upgrade tests happy. Justin Pryzby and Tom Lane Discussion: https://postgr.es/m/2021011= 9190720.GL8560@telsasoft.com htt= ps://git.postgresql.org/pg/commitdiff/aa25d1089ac00bbc3f97d2efe8f54c3d4beed= 5d1

  • Suppress various new compiler warnings. Com= pilers that don't understand that elog(ERROR) doesn't return issued warnings here. In the cases in libpq_pipeline.c, we were not exactly helping things by failing to mark pg_fatal() as noreturn. Per buildfarm. h= ttps://git.postgresql.org/pg/commitdiff/9fb9691a88ae8df9bc30e0f0f72de7c96e7= 3e125

=C3=81lvaro Herrera pushed:

Micha=C3=ABl Paquier pushed:

Stephen Frost pushed:

Robert Haas pushed:

  • Fix a confusing amcheck corruption message.= Don't complain about the last TOAST chunk number being different from what we expected if there are no = TOAST chunks at all. In such a case, saying that the final chunk number is 0 is= not really accurate, and the fact the value is missing from the TOAST table is reported separately anyway. Mark Dilger Discussion: http://pos= tgr.es/m/AA5506CE-7D2A-42E4-A51D-358635E3722D@enterprisedb.com htt= ps://git.postgresql.org/pg/commitdiff/4078ce65a0f7197180a9be2c6460ea4bf909b= d98

  • Allow configurable LZ4 TOAST compression. T= here is now a per-column COMPRESSION option which can be set to pglz (the default, and the only op= tion in up until now) or lz4. Or, if you like, you can set the new default_toast_compression GUC to lz4, and then that will be the default f= or new table columns for which no value is specified. We don't have lz4 supp= ort in the PostgreSQL code, so to use lz4 compression, PostgreSQL must be bui= lt --with-lz4. In general, TOAST compression means compression of individual column values, not the whole tuple, and those values can either be compre= ssed inline within the tuple or compressed and then stored externally in the T= OAST table, so those properties also apply to this feature. Prior to this com= mit, a TOAST pointer has two unused bits as part of the va_extsize field, and a compessed datum has two unused bits as part of the va_rawsize field. These bits are unused because the length of a varlena is limited to 1GB; we now= use them to indicate the compression type that was used. This means we only h= ave bit space for 2 more built-in compresison types, but we could work around= that problem, if necessary, by introducing a new vartag_external value for any further types we end up wanting to add. Hopefully, it won't be too import= ant to offer a wide selection of algorithms here, since each one we add not o= nly takes more coding but also adds a build dependency for every packager. Nevertheless, it seems worth doing at least this much, because LZ4 gets b= etter compression than PGLZ with less CPU usage. It's possible for LZ4-compres= sed datums to leak into composite type values stored on disk, just as it is f= or PGLZ. It's also possible for LZ4-compressed attributes to be copied into a different table via SQL commands such as CREATE TABLE AS or INSERT .. SEL= ECT. It would be expensive to force such values to be decompressed, so Postgre= SQL has never done so. For the same reasons, we also don't force recompressio= n of already-compressed values even if the target table prefers a different compression method than was used for the source data. These architectural decisions are perhaps arguable but revisiting them is well beyond the sco= pe of what seemed possible to do as part of this project. However, it's relati= vely cheap to recompress as part of VACUUM FULL or CLUSTER, so this commit adj= usts those commands to do so, if the configured compression method of the table happens not to match what was used for some column value stored therein. Dilip Kumar. The original patches on which this work was based were writt= en by Ildus Kurbangaliev, and those were patches were based on even earlier wor= k by Nikita Glukhov, but the design has since changed very substantially, since allow a potentially large number of compression methods that could be add= ed and dropped on a running system proved too problematic given some of the architectural issues mentioned above; the choice of which specific compre= ssion method to add first is now different; and a lot of the code has been heav= ily refactored. More recently, Justin Przyby helped quite a bit with testing= and reviewing and this version also includes some code contributions from him. Other design input and review from Tomas Vondra, =C3=81lvaro Herrera, And= res Freund, Oleg Bartunov, Alexander Korotkov, and me. Discussion: http://postgr.es/m/2017= 0907194236.4cefce96%40wp.localdomain Discussion: http://postgr.es/m/CAFiTN-uUpX3ck%3DK0mLEk-G_kUQY%3DSNOTeqdaNRR9FM= dQrHKebw%40mail.gmail.com htt= ps://git.postgresql.org/pg/commitdiff/bbe0a81db69bd10bd166907c3701492a29aca= 294

  • Fix use-after-ReleaseSysCache problem in AT= ExecAlterColumnType. Introduced by commit bbe0a81db69bd10bd166907c3701492a29aca294. Per buildfarm member pr= ion. h= ttps://git.postgresql.org/pg/commitdiff/d00fbdc431192c3e429b3e91c43d364e5c7= ba680

Amit Kapila pushed:

Peter Eisentraut pushed:

Andres Freund pushed:

Magnus Hagander pushed:

Tom=C3=A1=C5=A1 Vondra pushed:

  • Remove temporary files after backend crash.= After a crash of a backend using temporary files, the files used to be left behind, on the basis that it m= ight be useful for debugging. But we don't have any reports of anyone actually doing that, and it means the disk usage may grow over time due to repeated backend failures (possibly even hitting ENOSPC). So this behavior is a bit unfortunate, and fixing it required either manual cleanup (deleting files, which is error-prone) or restart of the instance (i.e. service disruption= ). This implements automatic cleanup of temporary files, controled by a new = GUC remove_temp_files_after_crash. By default the files are removed, but it c= an be disabled to restore the old behavior if needed. Author: Euler Taveira Reviewed-by: Tomas Vondra, Michael Paquier, Anastasia Lubennikova, Thomas Munro Discussion: https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%2B6_6xTJ-Yn5= WSvoHLY1Ww%40mail.gmail.com htt= ps://git.postgresql.org/pg/commitdiff/cd91de0d17952b5763466cfa663e98318f26d= 357

  • Implement GROUP BY DISTINCT. With grouping = sets, it's possible that some of the grouping sets are duplicate. This is especially common with CUBE and ROLLUP clauses. For example GROUP BY CUBE (a,b), CUBE (b,c) is equivalent= to GROUP BY GROUPING SETS ( (a, b, c), (a, b, c), (a, b, c), = (a, b), (a, b), (a, b), (a), (a), (a), (c, a), (c= , a), (c, a), (c), (b, c), (b), () ) Some of the grouping se= ts are calculated multiple times, which is mostly unnecessary. This commit implements a new GROUP BY DISTINCT feature, as defined in the SQL standar= d, which eliminates the duplicate sets. Author: Vik Fearing Reviewed-by: Er= ik Rijkers, Georgios Kokolatos, Tomas Vondra Discussion: https:= //postgr.es/m/bf3805a8-d7d1-ae61-fece-761b7ff41ecc@postgresfriends.org htt= ps://git.postgresql.org/pg/commitdiff/be45be9c33a85e72cdaeb9967e9f6d2d00199= e09

  • Fix TAP test for remove_temp_files_after_cr= ash. The test included in cd91de0d17 had two simple flaws. Firstly, the number of rows was low and= on some platforms (e.g. 32-bit) the sort did not require on-disk sort, so on those machines it was not testing the automatic removal. The test was ho= wever failing, because without any temporary files the base/pgsql_tmp directory= was not even created. Fixed by increasing the rowcount to 5000, which should= be high engough on any platform. Secondly, the test used a simple sleep to = wait for the temporary file to be created. This is obviously problematic, bec= ause on slow machines (or with valgrind, CLOBBER_CACHE_ALWAYS etc.) it may tak= e a while to create the temporary file. But we also want the tests run reaso= nably fast. Fixed by instead relying on a UNIQUE constraint, blocking the query= that created the temporary file. Author: Euler Taveira Reviewed-by: Tomas Von= dra Discussion: https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%2B6_6xTJ-Yn5= WSvoHLY1Ww%40mail.gmail.com htt= ps://git.postgresql.org/pg/commitdiff/a16b2b960f0eec2fe367e86017b3c24ed688b= a2b

  • Fix race condition in remove_temp_files_aft= er_crash TAP test. The TAP test was written so that it was not waiting for the correct SQL command, but for o= utput from the preceding one. This resulted in race conditions, allowing the commands to run in a different order, not block as expected and so on. T= his fixes it by inverting the order of commands where possible, so that obser= ving the output guarantees the data was inserted properly, and waiting for a l= ock to appear in pg_locks. Discussion: https://postgr.es/m/CAH503wDKdYzyq7U-QJqGn%3DGm6XmoK%2B6_6xTJ-Yn5= WSvoHLY1Ww%40mail.gmail.com htt= ps://git.postgresql.org/pg/commitdiff/e589c4890b05044a04207c2797e7c8af6693e= a5f

  • Use lfirst_int in cmp_list_len_contents_asc= . The function added in be45be9c33 is comparing integer lists (IntList) by length and contents, but there we= re two bugs. Firstly, it used intVal() to extract the value, but that's for Value nodes, not for extracting int values from IntList. Secondly, it ca= lled it directly on the ListCell, without doing lfirst(). So just do lfirst_i= nt() instead. Interestingly enough, this did not cause any crashes on the buildfarm, but valgrind rightfully complained about it. Discussion: https:= //postgr.es/m/bf3805a8-d7d1-ae61-fece-761b7ff41ecc@postgresfriends.org htt= ps://git.postgresql.org/pg/commitdiff/b822ae13ea93c18326d58d47829bbc66d36fa= e5c

  • Use valid compression method in brin_form_t= uple. When compressing the BRIN summary, we can't simply use the compression method from the indexed attribute. The summary may use a different data type, e.g. fixed-length attribute may have varlena summary, leading to compression failures. For= the built-in BRIN opclasses this happens to work, because the summary uses the same data type as the attribute. When the data types match, we can inher= it use the compression method specified for the attribute (it's copied into = the index descriptor). Otherwise we don't have much choice and have to use the default one. Author: Tomas Vondra Reviewed-by: Justin Pryzby pry= ;zby@telsasoft= ;.com Discussion: https:/= /postgr.es/m/e0367f27-392c-321a-7411-a58e1a7e4817%40enterprisedb.com htt= ps://git.postgresql.org/pg/commitdiff/882b2cdc08c4100e273f24742e2118be98708= a07

Andrew Dunstan pushed:

Pending Patches

Erica Zhang sent in another revision of a p= atch to add some tests for pg_stat_statements compatibility verification under contrib.

J=C3=BCrgen Purtz sent in another revision = of a patch to change the JOIN tutorial to focus more on explicit joins.

Noah Misch sent in a patch to merge similar= algorithms into roles_is_member_of().

David Rowley sent in another revision of a = patch to cache PathTarget and RestrictInfo's volatility, allow estimate_num_groups() to pass back further details about the estimation, allow the users of simplehash.h to perform di= rect deletions, add a Result Cache executor node, and remove code duplication in nodeResultCache.c.

Vigneshwaran C sent in a patch to fail prep= ared transaction if transaction has locked system tables/user catalog tables.

Laurenz Albe sent in another revision of a = patch to improve \e, \ef and \ev if the editor is quit without saving by erasing the query buffer rather than executing the previous statement.

Nathan Bossart sent in another revision of = a patch to avoid creating archive status ".ready" files too early, and keep track of notified-ready-for-archi= ve position through crashes.

Mike Insoo sent in a patch to check whether= , when decoding a catalog modifying commit record, it=E2=80=99s part of the RUNNING_XACT xid=E2=80=99s processe= d @ the restart_lsn. If so, then add its xid & subxacts in the committed txns list in the lo= gical decoding snapshot.

Justin Pryzby sent in three more revisions = of a patch to make it possible to use different compression methods for FPI.

Jim Mlodgenski sent in another revision of = a patch to add a parser hook and a test that uses it.

Julien Rouhaud sent in another revision of = a patch to add a new OUTDATED filtering facility for REINDEX command, with a corresponding option for the reindexdb utility. When used, REINDEX will only process indexes that have = an outdated dependency. For now, only dependency on collations are supported = but we'll likely support other kind of dependency in the future.

Georgios Kokolatos and Micha=C3=ABl Paquier= traded patches to make dbsize a bit more consistent.

Amit Kapila, Justin Pryzby, and Greg Nancar= row traded patches to make it possible to execute INSERT .... SELECT in parallel.

Bruce Momjian sent in two more revisions of= a patch to implement key management.

Bharath Rupireddy sent in four more revisio= ns of a patch to add a version of pg_terminate_backend() with wait and timeout.

Masahiro Ikeda sent in another revision of = a patch to make the WAL receiver report WAL statistics.

Iwata Aya sent in four more revisions of a = patch to implement a libpq debug log.

Amul Sul sent in two more revisions of a pa= tch to add the infrastructure ALTER SYSTEM READ {ONLY | WRITE} will use.

Jacob Champion and Micha=C3=ABl Paquier tra= ded patches to log authenticated identity from all auth backends.

Masahiro Ikeda sent in four revisions of a = patch to make the stats collector shutdown without writing the statsfiles if the immediate shutdown is reques= ted.

Mark Rofail and Andreas Karlsson traded pat= ches to implement foreign key arrays.

Ajin Cherian, Peter Smith, and Takamichi Os= umi traded patches to implement logical decoding of two-phase transactions.

=C3=81lvaro Herrera sent in three more revi= sions of a patch to implement ALTER TABLE .. DETACH PARTITION CONCURRENTLY.

Amit Langote sent in another revision of a = patch to allow batching of inserts during cross-partition updates.

IIvan Panchenko sent in another revision of= a patch to add a login trigger.

Peter Eisentraut sent in another revision o= f a patch to make it possible to get dynamic result sets from procedures. Declaring a cursor WITH RETURN in a procedure makes the cursor's data be returned as a result of the CALL invocation. The procedure needs to be declared with the DYNAMIC RESULT SETS attribute.

Amit Kapila sent in a doc patch to add a de= scription of substream in pg_subscription.

Amit Kapila sent in a patch to prevent canc= el/die interrupts while creating tablesync slot.

Bharath Rupireddy sent in another revision = of a patch to refactor the code that implements REFRESH MATERIALIZED VIEW for comprehensibility.

Ranier Vilela sent in a patch to cover a co= rner case in renaming files on Windows.

Hou Zhijie sent in another revision of a pa= tch to extend the safety check to support parallel insert into fk relations, and skip the CommandCounterIncre= ment.

Fujii Masao sent in two more revisions of a= patch to improve pgbench's \sleep command.

Andres Freund and Kyotaro HORIGUCHI traded = patches to make the stats collector use shared memory instead of files for temporary storage in the stats colle= ctor.

Masahiko Sawada sent in another revision of= a patch to fix transactions involving multiple postgres foreign servers.

Bharath Rupireddy sent in another revision = of a patch to mprove PID XXXX is not a PostgreSQL server process message. Instead of the message "is not a serv= er process" just say "is not a backend process". This would make things clear = that the given PID is not a backend process but it can be some other process's p= id or some other postmaster child process's pid or an invalid pid.

Zeng Wenjing sent in another revision of a = patch to implement global temporary tables.

Tom=C3=A1=C5=A1 Vondra sent in another revi= sion of a patch to implement index skip scans.

Atsushi Torikoshi sent in another revision = of a patch to add memorycontext elog print.

John Naylor sent in two more revisions of a= patch to fix bug in heap space management that was overly cautious about fillfactor. Previously, if there= were leftover ununused line pointers in a page, a table with a low fill factor w= ould not be available for new tuples that were large enough to exceed fillfactor, even though the page contained no tuples at all. This lead to unnecessary relation extension. Fix by allowing some slack space equal to 1/8 the maxi= mum space that could be taken up by line pointers. This is somewhat arbitrary, = but should allow many cases to succeed where they didn't before. Per report fr= om Floris van Nee.

Pavel St=C4=9Bhule sent in another revision= of a patch to add routine labels to PL/pgsql.

Micha=C3=ABl Paquier sent in another revisi= on of a patch to fix an infelicity between PITR and 2PC.

Bertrand Drouvot sent in another revision o= f a patch to implement minimal logical decoding on standbys.

Amit Langote sent in a patch to tweak handl= ing of serialization failures during cascaded RI update/delete.

Peter Eisentraut sent in another revision o= f a patch to ibpq: Set Server Name Indication (SNI) for SSL connections By default, have libpq set the TLS extension "Server Name Indication" (SNI). This allows an SNI-aware SSL pro= xy to route connections. (This requires a proxy that is aware of the PostgreSQL protocol, not just any SSL proxy.) In the future, this could also allow the server to use different SSL certificates for different host specifications. (That would require new server functionality. This would be the client-side functionality for that.) Since SNI makes the host name appear in cleartext = in the network traffic, this might be undesirable in some cases. Therefore, a= lso add a libpq connection option "sslsni" to turn it off.

Amit Langote sent in another revision of a = patch to mark fully grouped relations partitioned if input relation is, and allow setting parallel_workers on partitioned tables.

Ibrar Ahmed sent in another revision of a p= atch to make it possible for INSERT SELECT to use BulkInsertState and multi_insert.

John Naylor sent in another revision of a p= atch to add a 'noError' argument to encoding conversion functions, and use same to do COPY FROM encoding conversion/verification in larger chunks.

Thomas Munro sent in another revision of a = patch to use prefetching for WALs.

Peter Geoghegan sent in a patch to add a wa= y to ignore pgindent runs and similar when using git blame.

Peter Eisentraut sent in another revision o= f a patch to add a result_format_auto_binary_types setting.

Fujii Masao sent in another revision of a p= atch to change the type of WalReceiverWaitStart wait event from Client to IPC.

Marina Polyakova sent in a patch to reduce = the time required to recover database from archive by allowing concurrent loading of WAL files.

Peter Eisentraut sent in another revision o= f a patch to implement SQL-standard function bodies.

Andrei Zubkov sent in a patch to correct de= allocation events accounting in pg_stat_statements.

James Coleman sent in another revision of a= patch to Improve standby connection denied error message. Currently when a standby is finished starting up but hot_standby is configured to off, the error message when a client connects = is "the database system is starting up", which is needless confusing (and not really all that accurate either). Instead send a helpful error message so = that the user immediately knows that their server is configured to deny these connections.

John Naylor sent in another revision of a p= atch to add a 'noError' argument to encoding conversion functions, then use same to replace pg_utf8_verifystr()= with two faster implementations: one using SSE-4.1 instructions on Intel, the ot= her using a bespoke fallback function, rather than one that relies on pg_utf8_verifychar() and pg_utf8_isvalid(). This one is loosely based on the fallback that is part of the simdjson library.

Peter Eisentraut sent in another revision o= f a patch to change the return type of EXTRACT to numeric.

James Coleman sent in another revision of a= patch to hash lookup const arrays in OR'd ScalarArrayOps.

Amit Khandekar sent in another revision of = a patch to speed up xor'ing of two gist index signatures for tsvectors by using 64-bit chunks instead of char values, and avoid function pointer dereferencing for the pg_popcount32/64() call.

Etsuro Fujita sent in another revision of a= patch to implement asynchronous append on postgres_fdw nodes.

Konstantin Knizhnik sent in another revisio= n of a patch to add two statistics GUCs to auto_explain: add_statistics_threshold, and add_statistics_suggest_= only.

Gilles Darold sent in another revision of a= patch to add hooks at the XactCommand level.

Tatsuo Ishii sent in three more revisions o= f a patch to make it possible to use COPY FREEZE in pgbench.

Julien Rouhaud sent in another revision of = a patch to add queryId to the pg_catalog.pg_stat_activity view.

Amit Langote sent in another revision of a = patch to export get_partition_for_tuple() to make it visible to things other than execPartition.c can use it, and use it elsewhere to avoid using SPI for som= e RI checks.

Peter Smith sent in a patch to fixed a bug = where the code was de-referencing the hash entry returned by HASH_REMOVE in stream_cleanup_files.

Thomas Munro sent in another revision of a = patch to support PSQL_WATCH_PAGER for psql's \watch command, and try sigwait() to make it work.

Micha=C3=ABl Paquier sent in a patch to cle= an up the Kerberos auth test.

Thomas Munro sent in another revision of a = patch to provide recovery_init_sync_method=3Dwal.

Gilles Darold and Erik Rijkers traded patch= es to add some new functions for counting and otherwise characterizing regexp matches.

David Fetter sent in another revision of a = patch to implement popcount (now called bit_count).

Justin Pryzby sent in another revision of a= patch to add a way to use different compression methods for FPI.

Matthias van de Meent sent in a patch to en= sure that when [+] is an option for a psql metacommand, \? output mentions that it is, and not otherwise.

Jan Wieck and Tom Lane traded patches to en= sure that pg_upgrade preserves datdba.

Konstantin Knizhnik sent in another revisio= n of a patch to implement a built-in connection pooler.
This email was sent to you from PWN. It was delivered on their behalf by the PostgreSQL project. Any questions about the content of the message shou= ld be sent to PWN.

You were sent this email as a subscriber of the pgsql-announce mai= linglist, for the content tag PWN. To unsubscribe from further emails, or change which emails you want to receive, please click th= e personal unsubscribe link that you can find in the headers of this email, or visit https://lists.postgresql.org/unsubscribe/.
 
--===============3798017092296226947==-- --===============1844203319814237320==--