[PATCH 1/2] Refactor checks for deleted GiST pages.

public inbox for [email protected]  
help / color / mirror / Atom feed

[PATCH 1/2] Refactor checks for deleted GiST pages.
94+ messages / 14 participants
[nested] [flat]

* [PATCH 1/2] Refactor checks for deleted GiST pages.
@ 2019-04-04 15:06  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Heikki Linnakangas @ 2019-04-04 15:06 UTC (permalink / raw)

The explicit check in gistScanPage() isn't currently really necessary, as
a deleted page is always empty, so the loop would fall through without
doing anything, anyway. But it's a marginal optimization, and it gives a
nice place to attach a comment to explain how it works.
---
 src/backend/access/gist/gist.c    | 40 ++++++++++++-------------------
 src/backend/access/gist/gistget.c | 14 +++++++++++
 2 files changed, 29 insertions(+), 25 deletions(-)

diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 2db790c840..028b06b264 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -693,14 +693,15 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 			continue;
 		}
 
-		if (stack->blkno != GIST_ROOT_BLKNO &&
-			stack->parent->lsn < GistPageGetNSN(stack->page))
+		if ((stack->blkno != GIST_ROOT_BLKNO &&
+			 stack->parent->lsn < GistPageGetNSN(stack->page)) ||
+			GistPageIsDeleted(stack->page))
 		{
 			/*
-			 * Concurrent split detected. There's no guarantee that the
-			 * downlink for this page is consistent with the tuple we're
-			 * inserting anymore, so go back to parent and rechoose the best
-			 * child.
+			 * Concurrent split or page deletion detected. There's no
+			 * guarantee that the downlink for this page is consistent with
+			 * the tuple we're inserting anymore, so go back to parent and
+			 * rechoose the best child.
 			 */
 			UnlockReleaseBuffer(stack->buffer);
 			xlocked = false;
@@ -719,9 +720,6 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 			GISTInsertStack *item;
 			OffsetNumber downlinkoffnum;
 
-			/* currently, internal pages are never deleted */
-			Assert(!GistPageIsDeleted(stack->page));
-
 			downlinkoffnum = gistchoose(state.r, stack->page, itup, giststate);
 			iid = PageGetItemId(stack->page, downlinkoffnum);
 			idxtuple = (IndexTuple) PageGetItem(stack->page, iid);
@@ -842,12 +840,13 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 					 * leaf/inner is enough to recognize split for root
 					 */
 				}
-				else if (GistFollowRight(stack->page) ||
-						 stack->parent->lsn < GistPageGetNSN(stack->page))
+				else if ((GistFollowRight(stack->page) ||
+						  stack->parent->lsn < GistPageGetNSN(stack->page)) &&
+						 GistPageIsDeleted(stack->page))
 				{
 					/*
-					 * The page was split while we momentarily unlocked the
-					 * page. Go back to parent.
+					 * The page was split or deleted while we momentarily
+					 * unlocked the page. Go back to parent.
 					 */
 					UnlockReleaseBuffer(stack->buffer);
 					xlocked = false;
@@ -856,18 +855,6 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 				}
 			}
 
-			/*
-			 * The page might have been deleted after we scanned the parent
-			 * and saw the downlink.
-			 */
-			if (GistPageIsDeleted(stack->page))
-			{
-				UnlockReleaseBuffer(stack->buffer);
-				xlocked = false;
-				state.stack = stack = stack->parent;
-				continue;
-			}
-
 			/* now state.stack->(page, buffer and blkno) points to leaf page */
 
 			gistinserttuple(&state, stack, giststate, itup,
@@ -931,6 +918,9 @@ gistFindPath(Relation r, BlockNumber child, OffsetNumber *downlinkoffnum)
 			break;
 		}
 
+		/* currently, internal pages are never deleted */
+		Assert(!GistPageIsDeleted(page));
+
 		top->lsn = BufferGetLSNAtomic(buffer);
 
 		/*
diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c
index 8108fbb7d8..77ae2fb339 100644
--- a/src/backend/access/gist/gistget.c
+++ b/src/backend/access/gist/gistget.c
@@ -377,6 +377,20 @@ gistScanPage(IndexScanDesc scan, GISTSearchItem *pageItem, double *myDistances,
 		MemoryContextSwitchTo(oldcxt);
 	}
 
+	/*
+	 * Check if the page was deleted after we saw the downlink. There's
+	 * nothing of interest on a deleted page. Note that we must do this
+	 * after checking the NSN for concurrent splits! It's possible that
+	 * the page originally contained some tuples that are visible to use,
+	 * but was split so that all the visible tuples were moved to another
+	 * page, and then this page was deleted.
+	 */
+	if (GistPageIsDeleted(page))
+	{
+		UnlockReleaseBuffer(buffer);
+		return;
+	}
+
 	so->nPageData = so->curPageData = 0;
 	scan->xs_hitup = NULL;		/* might point into pageDataCxt */
 	if (so->pageDataCxt)
-- 
2.20.1


--------------78EA05D1AADD10C0CCF3EEC8
Content-Type: text/x-patch;
 name="0002-Use-full-64-bit-XID-for-checking-if-a-deleted-GiST-p.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0002-Use-full-64-bit-XID-for-checking-if-a-deleted-GiST-p.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* [PATCH 1/2] Refactor checks for deleted GiST pages.
@ 2019-04-04 15:06  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Heikki Linnakangas @ 2019-04-04 15:06 UTC (permalink / raw)

The explicit check in gistScanPage() isn't currently really necessary, as
a deleted page is always empty, so the loop would fall through without
doing anything, anyway. But it's a marginal optimization, and it gives a
nice place to attach a comment to explain how it works.
---
 src/backend/access/gist/gist.c    | 40 ++++++++++++-------------------
 src/backend/access/gist/gistget.c | 14 +++++++++++
 2 files changed, 29 insertions(+), 25 deletions(-)

diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 470b121e7da..79030e77153 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -709,14 +709,15 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 			continue;
 		}
 
-		if (stack->blkno != GIST_ROOT_BLKNO &&
-			stack->parent->lsn < GistPageGetNSN(stack->page))
+		if ((stack->blkno != GIST_ROOT_BLKNO &&
+			 stack->parent->lsn < GistPageGetNSN(stack->page)) ||
+			GistPageIsDeleted(stack->page))
 		{
 			/*
-			 * Concurrent split detected. There's no guarantee that the
-			 * downlink for this page is consistent with the tuple we're
-			 * inserting anymore, so go back to parent and rechoose the best
-			 * child.
+			 * Concurrent split or page deletion detected. There's no
+			 * guarantee that the downlink for this page is consistent with
+			 * the tuple we're inserting anymore, so go back to parent and
+			 * rechoose the best child.
 			 */
 			UnlockReleaseBuffer(stack->buffer);
 			xlocked = false;
@@ -735,9 +736,6 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 			GISTInsertStack *item;
 			OffsetNumber downlinkoffnum;
 
-			/* currently, internal pages are never deleted */
-			Assert(!GistPageIsDeleted(stack->page));
-
 			downlinkoffnum = gistchoose(state.r, stack->page, itup, giststate);
 			iid = PageGetItemId(stack->page, downlinkoffnum);
 			idxtuple = (IndexTuple) PageGetItem(stack->page, iid);
@@ -858,12 +856,13 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 					 * leaf/inner is enough to recognize split for root
 					 */
 				}
-				else if (GistFollowRight(stack->page) ||
-						 stack->parent->lsn < GistPageGetNSN(stack->page))
+				else if ((GistFollowRight(stack->page) ||
+						  stack->parent->lsn < GistPageGetNSN(stack->page)) &&
+						 GistPageIsDeleted(stack->page))
 				{
 					/*
-					 * The page was split while we momentarily unlocked the
-					 * page. Go back to parent.
+					 * The page was split or deleted while we momentarily
+					 * unlocked the page. Go back to parent.
 					 */
 					UnlockReleaseBuffer(stack->buffer);
 					xlocked = false;
@@ -872,18 +871,6 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 				}
 			}
 
-			/*
-			 * The page might have been deleted after we scanned the parent
-			 * and saw the downlink.
-			 */
-			if (GistPageIsDeleted(stack->page))
-			{
-				UnlockReleaseBuffer(stack->buffer);
-				xlocked = false;
-				state.stack = stack = stack->parent;
-				continue;
-			}
-
 			/* now state.stack->(page, buffer and blkno) points to leaf page */
 
 			gistinserttuple(&state, stack, giststate, itup,
@@ -947,6 +934,9 @@ gistFindPath(Relation r, BlockNumber child, OffsetNumber *downlinkoffnum)
 			break;
 		}
 
+		/* currently, internal pages are never deleted */
+		Assert(!GistPageIsDeleted(page));
+
 		top->lsn = BufferGetLSNAtomic(buffer);
 
 		/*
diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c
index 8108fbb7d8e..77ae2fb3395 100644
--- a/src/backend/access/gist/gistget.c
+++ b/src/backend/access/gist/gistget.c
@@ -377,6 +377,20 @@ gistScanPage(IndexScanDesc scan, GISTSearchItem *pageItem, double *myDistances,
 		MemoryContextSwitchTo(oldcxt);
 	}
 
+	/*
+	 * Check if the page was deleted after we saw the downlink. There's
+	 * nothing of interest on a deleted page. Note that we must do this
+	 * after checking the NSN for concurrent splits! It's possible that
+	 * the page originally contained some tuples that are visible to use,
+	 * but was split so that all the visible tuples were moved to another
+	 * page, and then this page was deleted.
+	 */
+	if (GistPageIsDeleted(page))
+	{
+		UnlockReleaseBuffer(buffer);
+		return;
+	}
+
 	so->nPageData = so->curPageData = 0;
 	scan->xs_hitup = NULL;		/* might point into pageDataCxt */
 	if (so->pageDataCxt)
-- 
2.20.1


--------------A4A2DD62EBB140914640E8DD
Content-Type: text/x-patch;
 name="0002-Use-full-64-bit-XID-for-checking-if-a-deleted-GiST-p.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0002-Use-full-64-bit-XID-for-checking-if-a-deleted-GiST-p.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* [PATCH 1/2] Refactor checks for deleted GiST pages.
@ 2019-07-22 12:57  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Heikki Linnakangas @ 2019-07-22 12:57 UTC (permalink / raw)

The explicit check in gistScanPage() isn't currently really necessary, as
a deleted page is always empty, so the loop would fall through without
doing anything, anyway. But it's a marginal optimization, and it gives a
nice place to attach a comment to explain how it works.
---
 src/backend/access/gist/gist.c    | 40 ++++++++++++-------------------
 src/backend/access/gist/gistget.c | 14 +++++++++++
 2 files changed, 29 insertions(+), 25 deletions(-)

diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 169bf6fcfed..e9ca4b82527 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -709,14 +709,15 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 			continue;
 		}
 
-		if (stack->blkno != GIST_ROOT_BLKNO &&
-			stack->parent->lsn < GistPageGetNSN(stack->page))
+		if ((stack->blkno != GIST_ROOT_BLKNO &&
+			 stack->parent->lsn < GistPageGetNSN(stack->page)) ||
+			GistPageIsDeleted(stack->page))
 		{
 			/*
-			 * Concurrent split detected. There's no guarantee that the
-			 * downlink for this page is consistent with the tuple we're
-			 * inserting anymore, so go back to parent and rechoose the best
-			 * child.
+			 * Concurrent split or page deletion detected. There's no
+			 * guarantee that the downlink for this page is consistent with
+			 * the tuple we're inserting anymore, so go back to parent and
+			 * rechoose the best child.
 			 */
 			UnlockReleaseBuffer(stack->buffer);
 			xlocked = false;
@@ -735,9 +736,6 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 			GISTInsertStack *item;
 			OffsetNumber downlinkoffnum;
 
-			/* currently, internal pages are never deleted */
-			Assert(!GistPageIsDeleted(stack->page));
-
 			downlinkoffnum = gistchoose(state.r, stack->page, itup, giststate);
 			iid = PageGetItemId(stack->page, downlinkoffnum);
 			idxtuple = (IndexTuple) PageGetItem(stack->page, iid);
@@ -858,12 +856,13 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 					 * leaf/inner is enough to recognize split for root
 					 */
 				}
-				else if (GistFollowRight(stack->page) ||
-						 stack->parent->lsn < GistPageGetNSN(stack->page))
+				else if ((GistFollowRight(stack->page) ||
+						  stack->parent->lsn < GistPageGetNSN(stack->page)) &&
+						 GistPageIsDeleted(stack->page))
 				{
 					/*
-					 * The page was split while we momentarily unlocked the
-					 * page. Go back to parent.
+					 * The page was split or deleted while we momentarily
+					 * unlocked the page. Go back to parent.
 					 */
 					UnlockReleaseBuffer(stack->buffer);
 					xlocked = false;
@@ -872,18 +871,6 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace,
 				}
 			}
 
-			/*
-			 * The page might have been deleted after we scanned the parent
-			 * and saw the downlink.
-			 */
-			if (GistPageIsDeleted(stack->page))
-			{
-				UnlockReleaseBuffer(stack->buffer);
-				xlocked = false;
-				state.stack = stack = stack->parent;
-				continue;
-			}
-
 			/* now state.stack->(page, buffer and blkno) points to leaf page */
 
 			gistinserttuple(&state, stack, giststate, itup,
@@ -947,6 +934,9 @@ gistFindPath(Relation r, BlockNumber child, OffsetNumber *downlinkoffnum)
 			break;
 		}
 
+		/* currently, internal pages are never deleted */
+		Assert(!GistPageIsDeleted(page));
+
 		top->lsn = BufferGetLSNAtomic(buffer);
 
 		/*
diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c
index 46d08e06350..c5fe2ea3998 100644
--- a/src/backend/access/gist/gistget.c
+++ b/src/backend/access/gist/gistget.c
@@ -377,6 +377,20 @@ gistScanPage(IndexScanDesc scan, GISTSearchItem *pageItem, double *myDistances,
 		MemoryContextSwitchTo(oldcxt);
 	}
 
+	/*
+	 * Check if the page was deleted after we saw the downlink. There's
+	 * nothing of interest on a deleted page. Note that we must do this
+	 * after checking the NSN for concurrent splits! It's possible that
+	 * the page originally contained some tuples that are visible to use,
+	 * but was split so that all the visible tuples were moved to another
+	 * page, and then this page was deleted.
+	 */
+	if (GistPageIsDeleted(page))
+	{
+		UnlockReleaseBuffer(buffer);
+		return;
+	}
+
 	so->nPageData = so->curPageData = 0;
 	scan->xs_hitup = NULL;		/* might point into pageDataCxt */
 	if (so->pageDataCxt)
-- 
2.20.1


--------------2155CE93C68FA8184E2F2BE8
Content-Type: text/x-patch;
 name="0002-Use-full-64-bit-XID-for-checking-if-a-deleted-GiST-p.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0002-Use-full-64-bit-XID-for-checking-if-a-deleted-GiST-p.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* make MaxBackends available in _PG_init
@ 2020-09-21 03:43  Wang, Shenhao <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Wang, Shenhao @ 2020-09-21 03:43 UTC (permalink / raw)
  To: [email protected] <[email protected]>

Hi Hackers:

I find the value in function _PG_init, the value of MaxBackends is 0.
In source, I find that the postmaster will first load library, and then calculate the value of MaxBackends.

In the old version, the MaxBackends was calculated by:
	 MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
		GetNumShmemAttachedBgworkers();
Because any extension can register workers which will affect the return value of GetNumShmemAttachedBgworkers.
InitializeMaxBackends must be called after shared_preload_libraries. This is also mentioned in comments.

Now, function GetNumShmemAttachedBgworkers was deleted and replaced by guc max_worker_processes, 
so if we changed the calling order like:
	Step1: calling InitializeMaxBackends.
	Step2: calling process_shared_preload_libraries

In this order extension can get the correct value of MaxBackends in _PG_init.

Any thoughts?

Regards




Attachments:

  [application/octet-stream] 0001-Make-MaxBackends-available-in-_PG_init.patch (2.2K, ../../4f20d57b2aeb447b8eb1495319940c5f@G08CNEXMBPEKD06.g08.fujitsu.local/2-0001-Make-MaxBackends-available-in-_PG_init.patch)
  download | inline diff:
From 1bc1f06337bfffc57d4ac25147a7b981ef89bcbf Mon Sep 17 00:00:00 2001
From: Shenhao Wang <[email protected]>
Date: Fri, 18 Sep 2020 18:35:54 +0800
Subject: [PATCH] Make MaxBackends available in _PG_init

---
 src/backend/postmaster/postmaster.c | 17 +++++++----------
 src/backend/utils/init/postinit.c   |  4 +---
 2 files changed, 8 insertions(+), 13 deletions(-)

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 959e3b8873..8da50cb2a1 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -989,13 +989,16 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher. It's probably a good idea to call it
+	 * before any modules had chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
+	/*
+	 * Calculate MaxBackends.
+	 */
+	InitializeMaxBackends();
+
 	/*
 	 * process any libraries that should be preloaded at postmaster start
 	 */
@@ -1012,12 +1015,6 @@ PostmasterMain(int argc, char *argv[])
 	}
 #endif
 
-	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
-	 */
-	InitializeMaxBackends();
-
 	/*
 	 * Set up shared memory and semaphores.
 	 */
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index d4ab4c7e23..06d813f76b 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -513,9 +513,7 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
-- 
2.21.0



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2020-09-21 11:52  Bharath Rupireddy <[email protected]>
  parent: Wang, Shenhao <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Bharath Rupireddy @ 2020-09-21 11:52 UTC (permalink / raw)
  To: Wang, Shenhao <[email protected]>; +Cc: [email protected] <[email protected]>

On Mon, Sep 21, 2020 at 9:14 AM Wang, Shenhao
<[email protected]> wrote:
>
> In source, I find that the postmaster will first load library, and then calculate the value of MaxBackends.
>
> In the old version, the MaxBackends was calculated by:
>          MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
>                 GetNumShmemAttachedBgworkers();
> Because any extension can register workers which will affect the return value of GetNumShmemAttachedBgworkers.
> InitializeMaxBackends must be called after shared_preload_libraries. This is also mentioned in comments.
>
> Now, function GetNumShmemAttachedBgworkers was deleted and replaced by guc max_worker_processes,
> so if we changed the calling order like:
>         Step1: calling InitializeMaxBackends.
>         Step2: calling process_shared_preload_libraries
>

Yes, the GetNumShmemAttachedBgworkers() was removed by commit #
dfbba2c86cc8f09cf3ffca3d305b4ce54a7fb49a. ASAICS, changing the order
of InitializeMaxBackends() and process_shared_preload_libraries() has
no problem, as InitializeMaxBackends() doesn't calculate the
MaxBackends based on bgworker infra code, it does calculate based on
GUCs.

Having said that, I'm not quite sure whether any of the bgworker
registration code, for that matter process_shared_preload_libraries()
code path will somewhere use MaxBackends?

>
> In this order extension can get the correct value of MaxBackends in _PG_init.
>

Is there any specific use case that any of the _PG_init will use MaxBackends?

I think the InitializeMaxBackends() function comments still say as shown below.

 * This must be called after modules have had the chance to register background
 * workers in shared_preload_libraries, and before shared memory size is
 * determined.

What happens with your patch in EXEC_BACKEND cases? (On linux
EXEC_BACKEND (include/pg_config_manual.h) can be enabled for testing
purposes).

With Regards,
Bharath Rupireddy.
EnterpriseDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 18:18  Bossart, Nathan <[email protected]>
  parent: Bharath Rupireddy <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-02 18:18 UTC (permalink / raw)
  To: Bharath Rupireddy <[email protected]>; +Cc: [email protected] <[email protected]>

On 9/21/20, 4:52 AM, "Bharath Rupireddy" <[email protected]> wrote:
> On Mon, Sep 21, 2020 at 9:14 AM Wang, Shenhao
> <[email protected]> wrote:
>>
>> In source, I find that the postmaster will first load library, and then calculate the value of MaxBackends.
>>
>> In the old version, the MaxBackends was calculated by:
>>          MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
>>                 GetNumShmemAttachedBgworkers();
>> Because any extension can register workers which will affect the return value of GetNumShmemAttachedBgworkers.
>> InitializeMaxBackends must be called after shared_preload_libraries. This is also mentioned in comments.
>>
>> Now, function GetNumShmemAttachedBgworkers was deleted and replaced by guc max_worker_processes,
>> so if we changed the calling order like:
>>         Step1: calling InitializeMaxBackends.
>>         Step2: calling process_shared_preload_libraries
>>
>
> Yes, the GetNumShmemAttachedBgworkers() was removed by commit #
> dfbba2c86cc8f09cf3ffca3d305b4ce54a7fb49a. ASAICS, changing the order
> of InitializeMaxBackends() and process_shared_preload_libraries() has
> no problem, as InitializeMaxBackends() doesn't calculate the
> MaxBackends based on bgworker infra code, it does calculate based on
> GUCs.
>
> Having said that, I'm not quite sure whether any of the bgworker
> registration code, for that matter process_shared_preload_libraries()
> code path will somewhere use MaxBackends?
>
>>
>> In this order extension can get the correct value of MaxBackends in _PG_init.
>>
>
> Is there any specific use case that any of the _PG_init will use MaxBackends?

I just encountered the same thing, so I am bumping this thread.  I was
trying to use MaxBackends in a call to RequestAddinShmemSpace() in a
_PG_init() function for a module, but since MaxBackends is not yet
initialized, you essentially need to open-code InitializeMaxBackends()
instead.

I think the comments about needing to register background workers
before initializing MaxBackends have been incorrect since the addition
of max_worker_processes in v9.4 (6bc8ef0b).  Furthermore, I think the
suggested reordering is a good idea because it is not obvious that
MaxBackends will be uninitialized in _PG_init(), and use-cases like
the RequestAddinShmemSpace() one are not guaranteed to fail when
MaxBackends is used incorrectly (presumably due to the 100 KB buffer
added in CreateSharedMemoryAndSemaphores()).

I've attached a new version of the proposed patch with some slight
adjustments and an attempt at a commit message.

Nathan



Attachments:

  [application/octet-stream] v1-0001-Calculate-MaxBackends-earlier-in-PostmasterMain.patch (2.9K, ../../[email protected]/2-v1-0001-Calculate-MaxBackends-earlier-in-PostmasterMain.patch)
  download | inline diff:
From 09f678653820b62d0823ea6c951adb3ff2f470ea Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Mon, 2 Aug 2021 17:42:25 +0000
Subject: [PATCH v1 1/1] Calculate MaxBackends earlier in PostmasterMain().

Presently, InitializeMaxBackends() is called after processing
shared_preload_libraries because it used to tally up the number of
registered background workers requested by the libraries.  Since
6bc8ef0b, InitializeMaxBackends() has simply used the
max_worker_processes GUC instead, so all the comments about needing
to register background workers before initializing MaxBackends are
no longer correct.

In addition to revising the comments, this patch reorders
InitializeMaxBackends() to before shared_preload_libraries is
processed so that modules can make use of MaxBackends in their
_PG_init() functions.
---
 src/backend/postmaster/postmaster.c | 19 +++++++++----------
 src/backend/utils/init/postinit.c   |  4 +---
 2 files changed, 10 insertions(+), 13 deletions(-)

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 00d051d520..5eff4610fd 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -990,10 +990,15 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Calculate MaxBackends.  This is done before processing
+	 * shared_preload_libraries so that such libraries can make use of it in
+	 * _PG_init().
+	 */
+	InitializeMaxBackends();
+
+	/*
+	 * Register the apply launcher.  It's probably a good idea to call it before
+	 * any modules had chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1013,12 +1018,6 @@ PostmasterMain(int argc, char *argv[])
 	}
 #endif
 
-	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
-	 */
-	InitializeMaxBackends();
-
 	/*
 	 * Set up shared memory and semaphores.
 	 */
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 51d1bbef30..f8136dfc6f 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -502,9 +502,7 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
-- 
2.16.6



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 20:06  Robert Haas <[email protected]>
  parent: Bossart, Nathan <[email protected]>
  0 siblings, 2 replies; 94+ messages in thread

From: Robert Haas @ 2021-08-02 20:06 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan <[email protected]> wrote:
> I just encountered the same thing, so I am bumping this thread.  I was
> trying to use MaxBackends in a call to RequestAddinShmemSpace() in a
> _PG_init() function for a module, but since MaxBackends is not yet
> initialized, you essentially need to open-code InitializeMaxBackends()
> instead.
>
> I think the comments about needing to register background workers
> before initializing MaxBackends have been incorrect since the addition
> of max_worker_processes in v9.4 (6bc8ef0b).  Furthermore, I think the
> suggested reordering is a good idea because it is not obvious that
> MaxBackends will be uninitialized in _PG_init(), and use-cases like
> the RequestAddinShmemSpace() one are not guaranteed to fail when
> MaxBackends is used incorrectly (presumably due to the 100 KB buffer
> added in CreateSharedMemoryAndSemaphores()).
>
> I've attached a new version of the proposed patch with some slight
> adjustments and an attempt at a commit message.

I think this is a good idea. Anyone object?

-- 
Robert Haas
EDB: http://www.enterprisedb.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 20:27  Alvaro Herrera <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 0 replies; 94+ messages in thread

From: Alvaro Herrera @ 2021-08-02 20:27 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 2021-Aug-02, Robert Haas wrote:

> On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan <[email protected]> wrote:
> >
> > I think the comments about needing to register background workers
> > before initializing MaxBackends have been incorrect since the addition
> > of max_worker_processes in v9.4 (6bc8ef0b).

> I think this is a good idea. Anyone object?

No objection here.  AFAICS Nathan is correct.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Los dioses no protegen a los insensatos.  Éstos reciben protección de
otros insensatos mejor dotados" (Luis Wu, Mundo Anillo)





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 20:37  Andres Freund <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 2 replies; 94+ messages in thread

From: Andres Freund @ 2021-08-02 20:37 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

Hi,

On 2021-08-02 16:06:15 -0400, Robert Haas wrote:
> On Mon, Aug 2, 2021 at 2:18 PM Bossart, Nathan <[email protected]> wrote:
> > I just encountered the same thing, so I am bumping this thread.  I was
> > trying to use MaxBackends in a call to RequestAddinShmemSpace() in a
> > _PG_init() function for a module, but since MaxBackends is not yet
> > initialized, you essentially need to open-code InitializeMaxBackends()
> > instead.
> >
> > I think the comments about needing to register background workers
> > before initializing MaxBackends have been incorrect since the addition
> > of max_worker_processes in v9.4 (6bc8ef0b).  Furthermore, I think the
> > suggested reordering is a good idea because it is not obvious that
> > MaxBackends will be uninitialized in _PG_init(), and use-cases like
> > the RequestAddinShmemSpace() one are not guaranteed to fail when
> > MaxBackends is used incorrectly (presumably due to the 100 KB buffer
> > added in CreateSharedMemoryAndSemaphores()).
> >
> > I've attached a new version of the proposed patch with some slight
> > adjustments and an attempt at a commit message.
> 
> I think this is a good idea. Anyone object?

I'm not so sure it's a good idea. I've seen several shared_preload_library
using extensions that adjust some GUCs (e.g. max_prepared_transactions)
because they need some more resources internally - that's perhaps not a great
idea, but there's also not an obviously better way.

ISTM this would better be solved with making the hook or config logic for
libraries a bit more elaborate (e.g. having a hook you can register for that's
called once all libraries are loaded).

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 21:57  Bossart, Nathan <[email protected]>
  parent: Andres Freund <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-02 21:57 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; Robert Haas <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 8/2/21, 1:37 PM, "Andres Freund" <[email protected]> wrote:
> On 2021-08-02 16:06:15 -0400, Robert Haas wrote:
>> I think this is a good idea. Anyone object?
>
> I'm not so sure it's a good idea. I've seen several shared_preload_library
> using extensions that adjust some GUCs (e.g. max_prepared_transactions)
> because they need some more resources internally - that's perhaps not a great
> idea, but there's also not an obviously better way.

Interesting.  I hadn't heard of extensions adjusting GUCs in
_PG_init() before.  I think the other way to handle that scenario is
to check the GUCs in _PG_init() and fail startup if necessary.
However, while it's probably good to avoid changing GUCs from what
users specified, failing startup isn't exactly user-friendly, either.
In any case, changing a GUC in _PG_init() seems quite risky.  You're
effectively bypassing all of the usual checks.

> ISTM this would better be solved with making the hook or config logic for
> libraries a bit more elaborate (e.g. having a hook you can register for that's
> called once all libraries are loaded).

My worry is that this requires even more specialized knowledge to get
things right.  If I need to do anything based on the value of a GUC,
I have to register another hook.  In this new hook, we'd likely want
to somehow prevent modules from further adjusting GUCs.  We'd also
need modules to check the GUCs they care about again in case another
module's _PG_init() adjusted it in an incompatible way.  If you detect
a problem in this new hook, you probably need to fail startup.

Perhaps I am making a mountain out of a molehill and the modules that
are adjusting GUCs in _PG_init() are doing so in a generally safe way,
but IMO it ought to ordinarily be discouraged.

Nathan

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 22:11  Andres Freund <[email protected]>
  parent: Bossart, Nathan <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Andres Freund @ 2021-08-02 22:11 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

Hi,

On 2021-08-02 21:57:18 +0000, Bossart, Nathan wrote:
> On 8/2/21, 1:37 PM, "Andres Freund" <[email protected]> wrote:
> > On 2021-08-02 16:06:15 -0400, Robert Haas wrote:
> >> I think this is a good idea. Anyone object?
> >
> > I'm not so sure it's a good idea. I've seen several shared_preload_library
> > using extensions that adjust some GUCs (e.g. max_prepared_transactions)
> > because they need some more resources internally - that's perhaps not a great
> > idea, but there's also not an obviously better way.
> 
> Interesting.  I hadn't heard of extensions adjusting GUCs in
> _PG_init() before.  I think the other way to handle that scenario is
> to check the GUCs in _PG_init() and fail startup if necessary.

The problem is that that makes it hard to configure things for the users own
needs. If e.g. the user's workload needs a certain number of prepared
transactions and the extension internally as well (as e.g. citus does), the
extension can't know if the configured number is "aimed" to be for the
extension, or for the user's own needs. If you instead just increase
max_prepared_transactions in _PG_init(), the story is different.

> However, while it's probably good to avoid changing GUCs from what
> users specified, failing startup isn't exactly user-friendly, either.
> In any case, changing a GUC in _PG_init() seems quite risky.  You're
> effectively bypassing all of the usual checks.

It doesn't need to bypass all - you can override them using GUC mechanisms at
that point.

> > ISTM this would better be solved with making the hook or config logic for
> > libraries a bit more elaborate (e.g. having a hook you can register for that's
> > called once all libraries are loaded).
> 
> My worry is that this requires even more specialized knowledge to get
> things right.  If I need to do anything based on the value of a GUC,
> I have to register another hook.  In this new hook, we'd likely want
> to somehow prevent modules from further adjusting GUCs.  We'd also
> need modules to check the GUCs they care about again in case another
> module's _PG_init() adjusted it in an incompatible way.

I think this is overblown. We already size resources *after*
shared_preload_libraries' _PG_init() runs, because that's the whole point of
shared_preload_libraries. What's proposed in this thread is to *disallow*
increasing resource usage in s_p_l _PG_init(), to make one specific case
simpler - but it'll actually also make things more complicated, because other
resources will still only be sized after all of s_p_l has been processed.

> If you detect a problem in this new hook, you probably need to fail startup.

Same is true for _PG_init().

> Perhaps I am making a mountain out of a molehill and the modules that
> are adjusting GUCs in _PG_init() are doing so in a generally safe way,
> but IMO it ought to ordinarily be discouraged.

It's not something I would recommend doing blithely - but I also don't think
we have a better answer for some cases.

Greetings,

Andres Freund

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 22:35  Bossart, Nathan <[email protected]>
  parent: Andres Freund <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-02 22:35 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 8/2/21, 3:12 PM, "Andres Freund" <[email protected]> wrote:
> I think this is overblown. We already size resources *after*
> shared_preload_libraries' _PG_init() runs, because that's the whole point of
> shared_preload_libraries. What's proposed in this thread is to *disallow*
> increasing resource usage in s_p_l _PG_init(), to make one specific case
> simpler - but it'll actually also make things more complicated, because other
> resources will still only be sized after all of s_p_l has been processed.

True.  Perhaps the comments should reference the possibility that a
library will adjust resource usage to explain why
InitializeMaxBackends() is where it is.

Nathan



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 22:42  Andres Freund <[email protected]>
  parent: Bossart, Nathan <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Andres Freund @ 2021-08-02 22:42 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

Hi,

On 2021-08-02 22:35:13 +0000, Bossart, Nathan wrote:
> On 8/2/21, 3:12 PM, "Andres Freund" <[email protected]> wrote:
> > I think this is overblown. We already size resources *after*
> > shared_preload_libraries' _PG_init() runs, because that's the whole point of
> > shared_preload_libraries. What's proposed in this thread is to *disallow*
> > increasing resource usage in s_p_l _PG_init(), to make one specific case
> > simpler - but it'll actually also make things more complicated, because other
> > resources will still only be sized after all of s_p_l has been processed.
> 
> True.  Perhaps the comments should reference the possibility that a
> library will adjust resource usage to explain why
> InitializeMaxBackends() is where it is.

I've wondered, independent of this thread, about not making MaxBackends
externally visible, and requiring a function call to access it. It should be
easier to find cases of misuse if we errored out when accessed at the wrong
time. And we could use that opportunity to add flags that determine which
types of backends are included (e.g. not including autovac, or additionally
including aux workers or prepared xacts).

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-02 22:58  Bossart, Nathan <[email protected]>
  parent: Andres Freund <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-02 22:58 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 8/2/21, 3:42 PM, "Andres Freund" <[email protected]> wrote:
> I've wondered, independent of this thread, about not making MaxBackends
> externally visible, and requiring a function call to access it. It should be
> easier to find cases of misuse if we errored out when accessed at the wrong
> time. And we could use that opportunity to add flags that determine which
> types of backends are included (e.g. not including autovac, or additionally
> including aux workers or prepared xacts).

I'm not opposed to this.  I can work on putting a patch together if no
opposition materializes.

Nathan



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* RE: make MaxBackends available in _PG_init
@ 2021-08-03 00:19  [email protected] <[email protected]>
  parent: Bossart, Nathan <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: [email protected] @ 2021-08-03 00:19 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

Hi,

Bossart, Nathan <[email protected]> wrote:

> I just encountered the same thing, so I am bumping this thread.
Thank you for bumping this thread.

> > I've wondered, independent of this thread, about not making MaxBackends
> > externally visible, and requiring a function call to access it. It should be
> > easier to find cases of misuse if we errored out when accessed at the wrong
> > time. 

> I'm not opposed to this.  I can work on putting a patch together if no
> opposition materializes.

I think this is a good idea.

Shenhao Wang
Best regards


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-03 15:25  Robert Haas <[email protected]>
  parent: Andres Freund <[email protected]>
  1 sibling, 2 replies; 94+ messages in thread

From: Robert Haas @ 2021-08-03 15:25 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On Mon, Aug 2, 2021 at 4:37 PM Andres Freund <[email protected]> wrote:
> I'm not so sure it's a good idea. I've seen several shared_preload_library
> using extensions that adjust some GUCs (e.g. max_prepared_transactions)
> because they need some more resources internally - that's perhaps not a great
> idea, but there's also not an obviously better way.

Blech.

I'm fine with solving the problem some other way, but I say again, blech.

-- 
Robert Haas
EDB: http://www.enterprisedb.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-03 23:13  Bossart, Nathan <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 0 replies; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-03 23:13 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; Andres Freund <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 8/2/21, 4:02 PM, "Bossart, Nathan" <[email protected]> wrote:
> On 8/2/21, 3:42 PM, "Andres Freund" <[email protected]> wrote:
>> I've wondered, independent of this thread, about not making MaxBackends
>> externally visible, and requiring a function call to access it. It should be
>> easier to find cases of misuse if we errored out when accessed at the wrong
>> time. And we could use that opportunity to add flags that determine which
>> types of backends are included (e.g. not including autovac, or additionally
>> including aux workers or prepared xacts).
>
> I'm not opposed to this.  I can work on putting a patch together if no
> opposition materializes.

Here is a first attempt.

Nathan



Attachments:

  [application/octet-stream] v1-0001-Disallow-external-access-to-MaxBackends.patch (30.3K, ../../[email protected]/2-v1-0001-Disallow-external-access-to-MaxBackends.patch)
  download | inline diff:
From 67bc167c81cef9b582e37f81c59c4ebd0cbb3a8f Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Tue, 3 Aug 2021 23:05:55 +0000
Subject: [PATCH v1 1/1] Disallow external access to MaxBackends.

Presently, MaxBackends is externally visible, but it may still be
uninitialized in places where it would be convenient to use (e.g.,
_PG_init()).  This change makes MaxBackends static to postinit.c to
disallow such direct access.  Instead, MaxBackends should now be
accessed via GetMaxBackends().  GetMaxBackends() also accepts a
couple of flags to easily include commonly added values such as
max_prepared_transactions.

Separately, adjust the comments about needing to register
background workers before initializing MaxBackends.  Since
6bc8ef0b, InitializeMaxBackends() has used max_worker_processes
instead of tallying up the number of registered background workers,
so background worker registration is no longer a prerequisite.  The
ordering of this logic is still useful for allowing libraries to
adjust GUCs, so the comments have been updated to mention that use-
case.
---
 src/backend/access/nbtree/nbtutils.c        |  4 +--
 src/backend/access/transam/multixact.c      |  2 +-
 src/backend/access/transam/twophase.c       |  2 +-
 src/backend/bootstrap/bootstrap.c           |  2 +-
 src/backend/commands/async.c                | 10 +++---
 src/backend/libpq/pqcomm.c                  |  2 +-
 src/backend/postmaster/postmaster.c         | 14 ++++-----
 src/backend/storage/ipc/dsm.c               |  2 +-
 src/backend/storage/ipc/procarray.c         |  2 +-
 src/backend/storage/ipc/procsignal.c        |  2 +-
 src/backend/storage/ipc/sinvaladt.c         |  4 +--
 src/backend/storage/lmgr/deadlock.c         | 31 +++++++++---------
 src/backend/storage/lmgr/lock.c             | 20 ++++++------
 src/backend/storage/lmgr/predicate.c        | 10 +++---
 src/backend/storage/lmgr/proc.c             | 17 +++++-----
 src/backend/utils/activity/backend_status.c | 10 +++---
 src/backend/utils/adt/lockfuncs.c           |  4 +--
 src/backend/utils/init/postinit.c           | 49 +++++++++++++++++++++++------
 src/include/miscadmin.h                     | 12 ++++++-
 19 files changed, 118 insertions(+), 81 deletions(-)

diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index d524310723..578d4531a2 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -2065,7 +2065,7 @@ BTreeShmemSize(void)
 	Size		size;
 
 	size = offsetof(BTVacInfo, vacuums);
-	size = add_size(size, mul_size(MaxBackends, sizeof(BTOneVacInfo)));
+	size = add_size(size, mul_size(GetMaxBackends(0), sizeof(BTOneVacInfo)));
 	return size;
 }
 
@@ -2094,7 +2094,7 @@ BTreeShmemInit(void)
 		btvacinfo->cycle_ctr = (BTCycleId) time(NULL);
 
 		btvacinfo->num_vacuums = 0;
-		btvacinfo->max_vacuums = MaxBackends;
+		btvacinfo->max_vacuums = GetMaxBackends(0);
 	}
 	else
 		Assert(found);
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index e6c70ed0bc..7e9728bf88 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -285,7 +285,7 @@ typedef struct MultiXactStateData
  * Last element of OldestMemberMXactId and OldestVisibleMXactId arrays.
  * Valid elements are (1..MaxOldestSlot); element 0 is never used.
  */
-#define MaxOldestSlot	(MaxBackends + max_prepared_xacts)
+#define MaxOldestSlot	(GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 /* Pointers to the state data in shared memory */
 static MultiXactStateData *MultiXactState;
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 6d3efb49a4..1ba84983f2 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -293,7 +293,7 @@ TwoPhaseShmemInit(void)
 			 * prepared transaction. Currently multixact.c uses that
 			 * technique.
 			 */
-			gxacts[i].dummyBackendId = MaxBackends + 1 + i;
+			gxacts[i].dummyBackendId = GetMaxBackends(0) + 1 + i;
 		}
 	}
 	else
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 67cd5ac6e9..ceb1983544 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -395,7 +395,7 @@ AuxiliaryProcessMain(int argc, char *argv[])
 		 * This will need rethinking if we ever want more than one of a
 		 * particular auxiliary process type.
 		 */
-		ProcSignalInit(MaxBackends + MyAuxProcType + 1);
+		ProcSignalInit(GetMaxBackends(0) + MyAuxProcType + 1);
 
 		/* finish setting up bufmgr.c */
 		InitBufferPoolBackend();
diff --git a/src/backend/commands/async.c b/src/backend/commands/async.c
index 4b16fb5682..29c50595c2 100644
--- a/src/backend/commands/async.c
+++ b/src/backend/commands/async.c
@@ -511,7 +511,7 @@ AsyncShmemSize(void)
 	Size		size;
 
 	/* This had better match AsyncShmemInit */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends(0) + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	size = add_size(size, SimpleLruShmemSize(NUM_NOTIFY_BUFFERS, 0));
@@ -534,7 +534,7 @@ AsyncShmemInit(void)
 	 * The used entries in the backend[] array run from 1 to MaxBackends; the
 	 * zero'th entry is unused but must be allocated.
 	 */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends(0) + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	asyncQueueControl = (AsyncQueueControl *)
@@ -549,7 +549,7 @@ AsyncShmemInit(void)
 		QUEUE_FIRST_LISTENER = InvalidBackendId;
 		asyncQueueControl->lastQueueFillWarn = 0;
 		/* zero'th entry won't be used, but let's initialize it anyway */
-		for (int i = 0; i <= MaxBackends; i++)
+		for (int i = 0; i <= GetMaxBackends(0); i++)
 		{
 			QUEUE_BACKEND_PID(i) = InvalidPid;
 			QUEUE_BACKEND_DBOID(i) = InvalidOid;
@@ -1685,8 +1685,8 @@ SignalBackends(void)
 	 * preallocate the arrays?	But in practice this is only run in trivial
 	 * transactions, so there should surely be space available.
 	 */
-	pids = (int32 *) palloc(MaxBackends * sizeof(int32));
-	ids = (BackendId *) palloc(MaxBackends * sizeof(BackendId));
+	pids = (int32 *) palloc(GetMaxBackends(0) * sizeof(int32));
+	ids = (BackendId *) palloc(GetMaxBackends(0) * sizeof(BackendId));
 	count = 0;
 
 	LWLockAcquire(NotifyQueueLock, LW_EXCLUSIVE);
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 89a5f901aa..4d74f9a3f5 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -556,7 +556,7 @@ StreamServerPort(int family, const char *hostName, unsigned short portNumber,
 		 * intended to provide a clamp on the request on platforms where an
 		 * overly large request provokes a kernel error (are there any?).
 		 */
-		maxconn = MaxBackends * 2;
+		maxconn = GetMaxBackends(0) * 2;
 		if (maxconn > PG_SOMAXCONN)
 			maxconn = PG_SOMAXCONN;
 
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 00d051d520..bf05b15455 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -990,10 +990,8 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher.  It's probably a good idea to call it before
+	 * any modules had a chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1014,8 +1012,8 @@ PostmasterMain(int argc, char *argv[])
 #endif
 
 	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
+	 * Now that loadable modules have had their chance to alter any GUCs,
+	 * calculate MaxBackends.
 	 */
 	InitializeMaxBackends();
 
@@ -6184,7 +6182,7 @@ save_backend_variables(BackendParameters *param, Port *port,
 	param->query_id_enabled = query_id_enabled;
 	param->max_safe_fds = max_safe_fds;
 
-	param->MaxBackends = MaxBackends;
+	param->MaxBackends = GetMaxBackends(0);
 
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
@@ -6418,7 +6416,7 @@ restore_backend_variables(BackendParameters *param, Port *port)
 	query_id_enabled = param->query_id_enabled;
 	max_safe_fds = param->max_safe_fds;
 
-	MaxBackends = param->MaxBackends;
+	SetMaxBackends(param->MaxBackends);
 
 #ifdef WIN32
 	PostmasterHandle = param->PostmasterHandle;
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index b461a5f7e9..bb279bb8d1 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -165,7 +165,7 @@ dsm_postmaster_startup(PGShmemHeader *shim)
 
 	/* Determine size for new control segment. */
 	maxitems = PG_DYNSHMEM_FIXED_SLOTS
-		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * MaxBackends;
+		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * GetMaxBackends(0);
 	elog(DEBUG2, "dynamic shared memory system will support %u segments",
 		 maxitems);
 	segsize = dsm_control_bytes_needed(maxitems);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index c7816fcfb3..9afd0209c8 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -365,7 +365,7 @@ ProcArrayShmemSize(void)
 	Size		size;
 
 	/* Size of the ProcArray structure itself */
-#define PROCARRAY_MAXPROCS	(MaxBackends + max_prepared_xacts)
+#define PROCARRAY_MAXPROCS	(GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 	size = offsetof(ProcArrayStruct, pgprocnos);
 	size = add_size(size, mul_size(sizeof(int), PROCARRAY_MAXPROCS));
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index defb75aa26..86b7c39782 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -85,7 +85,7 @@ typedef struct
  * possible auxiliary process type.  (This scheme assumes there is not
  * more than one of any auxiliary process type at a time.)
  */
-#define NumProcSignalSlots	(MaxBackends + NUM_AUXPROCTYPES)
+#define NumProcSignalSlots	(GetMaxBackends(0) + NUM_AUXPROCTYPES)
 
 /* Check whether the relevant type bit is set in the flags. */
 #define BARRIER_SHOULD_CHECK(flags, type) \
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index 946bd8e3cb..2e85a99f68 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -205,7 +205,7 @@ SInvalShmemSize(void)
 	Size		size;
 
 	size = offsetof(SISeg, procState);
-	size = add_size(size, mul_size(sizeof(ProcState), MaxBackends));
+	size = add_size(size, mul_size(sizeof(ProcState), GetMaxBackends(0)));
 
 	return size;
 }
@@ -231,7 +231,7 @@ CreateSharedInvalidationState(void)
 	shmInvalBuffer->maxMsgNum = 0;
 	shmInvalBuffer->nextThreshold = CLEANUP_MIN;
 	shmInvalBuffer->lastBackend = 0;
-	shmInvalBuffer->maxBackends = MaxBackends;
+	shmInvalBuffer->maxBackends = GetMaxBackends(0);
 	SpinLockInit(&shmInvalBuffer->msgnumLock);
 
 	/* The buffer[] array is initially all unused, so we need not fill it */
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index 67733c0d1a..6ffc481009 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -143,6 +143,7 @@ void
 InitDeadLockChecking(void)
 {
 	MemoryContext oldcxt;
+	int max_backends = GetMaxBackends(0);
 
 	/* Make sure allocations are permanent */
 	oldcxt = MemoryContextSwitchTo(TopMemoryContext);
@@ -151,16 +152,16 @@ InitDeadLockChecking(void)
 	 * FindLockCycle needs at most MaxBackends entries in visitedProcs[] and
 	 * deadlockDetails[].
 	 */
-	visitedProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
-	deadlockDetails = (DEADLOCK_INFO *) palloc(MaxBackends * sizeof(DEADLOCK_INFO));
+	visitedProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
+	deadlockDetails = (DEADLOCK_INFO *) palloc(max_backends * sizeof(DEADLOCK_INFO));
 
 	/*
 	 * TopoSort needs to consider at most MaxBackends wait-queue entries, and
 	 * it needn't run concurrently with FindLockCycle.
 	 */
 	topoProcs = visitedProcs;	/* re-use this space */
-	beforeConstraints = (int *) palloc(MaxBackends * sizeof(int));
-	afterConstraints = (int *) palloc(MaxBackends * sizeof(int));
+	beforeConstraints = (int *) palloc(max_backends * sizeof(int));
+	afterConstraints = (int *) palloc(max_backends * sizeof(int));
 
 	/*
 	 * We need to consider rearranging at most MaxBackends/2 wait queues
@@ -169,8 +170,8 @@ InitDeadLockChecking(void)
 	 * MaxBackends total waiters.
 	 */
 	waitOrders = (WAIT_ORDER *)
-		palloc((MaxBackends / 2) * sizeof(WAIT_ORDER));
-	waitOrderProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
+		palloc((max_backends / 2) * sizeof(WAIT_ORDER));
+	waitOrderProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
 
 	/*
 	 * Allow at most MaxBackends distinct constraints in a configuration. (Is
@@ -180,7 +181,7 @@ InitDeadLockChecking(void)
 	 * limits the maximum recursion depth of DeadLockCheckRecurse. Making it
 	 * really big might potentially allow a stack-overflow problem.
 	 */
-	maxCurConstraints = MaxBackends;
+	maxCurConstraints = max_backends;
 	curConstraints = (EDGE *) palloc(maxCurConstraints * sizeof(EDGE));
 
 	/*
@@ -191,7 +192,7 @@ InitDeadLockChecking(void)
 	 * last MaxBackends entries in possibleConstraints[] are reserved as
 	 * output workspace for FindLockCycle.
 	 */
-	maxPossibleConstraints = MaxBackends * 4;
+	maxPossibleConstraints = max_backends * 4;
 	possibleConstraints =
 		(EDGE *) palloc(maxPossibleConstraints * sizeof(EDGE));
 
@@ -327,7 +328,7 @@ DeadLockCheckRecurse(PGPROC *proc)
 	if (nCurConstraints >= maxCurConstraints)
 		return true;			/* out of room for active constraints? */
 	oldPossibleConstraints = nPossibleConstraints;
-	if (nPossibleConstraints + nEdges + MaxBackends <= maxPossibleConstraints)
+	if (nPossibleConstraints + nEdges + GetMaxBackends(0) <= maxPossibleConstraints)
 	{
 		/* We can save the edge list in possibleConstraints[] */
 		nPossibleConstraints += nEdges;
@@ -388,7 +389,7 @@ TestConfiguration(PGPROC *startProc)
 	/*
 	 * Make sure we have room for FindLockCycle's output.
 	 */
-	if (nPossibleConstraints + MaxBackends > maxPossibleConstraints)
+	if (nPossibleConstraints + GetMaxBackends(0) > maxPossibleConstraints)
 		return -1;
 
 	/*
@@ -486,7 +487,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 				 * record total length of cycle --- outer levels will now fill
 				 * deadlockDetails[]
 				 */
-				Assert(depth <= MaxBackends);
+				Assert(depth <= GetMaxBackends(0));
 				nDeadlockDetails = depth;
 
 				return true;
@@ -500,7 +501,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 		}
 	}
 	/* Mark proc as seen */
-	Assert(nVisitedProcs < MaxBackends);
+	Assert(nVisitedProcs < GetMaxBackends(0));
 	visitedProcs[nVisitedProcs++] = checkProc;
 
 	/*
@@ -698,7 +699,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends(0));
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -771,7 +772,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends(0));
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -834,7 +835,7 @@ ExpandConstraints(EDGE *constraints,
 		waitOrders[nWaitOrders].procs = waitOrderProcs + nWaitOrderProcs;
 		waitOrders[nWaitOrders].nProcs = lock->waitProcs.size;
 		nWaitOrderProcs += lock->waitProcs.size;
-		Assert(nWaitOrderProcs <= MaxBackends);
+		Assert(nWaitOrderProcs <= GetMaxBackends(0));
 
 		/*
 		 * Do the topo sort.  TopoSort need not examine constraints after this
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..581bbc29d5 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -55,7 +55,7 @@
 int			max_locks_per_xact; /* set by guc.c */
 
 #define NLOCKENTS() \
-	mul_size(max_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_locks_per_xact, GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 
 /*
@@ -2938,12 +2938,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 			vxids = (VirtualTransactionId *)
 				MemoryContextAlloc(TopMemoryContext,
 								   sizeof(VirtualTransactionId) *
-								   (MaxBackends + max_prepared_xacts + 1));
+								   (GetMaxBackends(GMB_MAX_PREPARED_XACTS) + 1));
 	}
 	else
 		vxids = (VirtualTransactionId *)
 			palloc0(sizeof(VirtualTransactionId) *
-					(MaxBackends + max_prepared_xacts + 1));
+					(GetMaxBackends(GMB_MAX_PREPARED_XACTS) + 1));
 
 	/* Compute hash code and partition lock, and look up conflicting modes. */
 	hashcode = LockTagHashCode(locktag);
@@ -3100,7 +3100,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 
 	LWLockRelease(partitionLock);
 
-	if (count > MaxBackends + max_prepared_xacts)	/* should never happen */
+	if (count > GetMaxBackends(GMB_MAX_PREPARED_XACTS))	/* should never happen */
 		elog(PANIC, "too many conflicting locks found");
 
 	vxids[count].backendId = InvalidBackendId;
@@ -3651,7 +3651,7 @@ GetLockStatusData(void)
 	data = (LockData *) palloc(sizeof(LockData));
 
 	/* Guess how much space we'll need. */
-	els = MaxBackends;
+	els = GetMaxBackends(0);
 	el = 0;
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * els);
 
@@ -3685,7 +3685,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends(0);
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3717,7 +3717,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends(0);
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3846,7 +3846,7 @@ GetBlockerStatusData(int blocked_pid)
 	 * for the procs[] array; the other two could need enlargement, though.)
 	 */
 	data->nprocs = data->nlocks = data->npids = 0;
-	data->maxprocs = data->maxlocks = data->maxpids = MaxBackends;
+	data->maxprocs = data->maxlocks = data->maxpids = GetMaxBackends(0);
 	data->procs = (BlockedProcData *) palloc(sizeof(BlockedProcData) * data->maxprocs);
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * data->maxlocks);
 	data->waiter_pids = (int *) palloc(sizeof(int) * data->maxpids);
@@ -3949,7 +3949,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 		if (data->nlocks >= data->maxlocks)
 		{
-			data->maxlocks += MaxBackends;
+			data->maxlocks += GetMaxBackends(0);
 			data->locks = (LockInstanceData *)
 				repalloc(data->locks, sizeof(LockInstanceData) * data->maxlocks);
 		}
@@ -3978,7 +3978,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 	if (queue_size > data->maxpids - data->npids)
 	{
-		data->maxpids = Max(data->maxpids + MaxBackends,
+		data->maxpids = Max(data->maxpids + GetMaxBackends(0),
 							data->npids + queue_size);
 		data->waiter_pids = (int *) repalloc(data->waiter_pids,
 											 sizeof(int) * data->maxpids);
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 56267bdc3c..d6199aa82d 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -257,7 +257,7 @@
 	(&MainLWLockArray[PREDICATELOCK_MANAGER_LWLOCK_OFFSET + (i)].lock)
 
 #define NPREDICATELOCKTARGETENTS() \
-	mul_size(max_predicate_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_predicate_locks_per_xact, GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 #define SxactIsOnFinishedList(sxact) (!SHMQueueIsDetached(&((sxact)->finishedLink)))
 
@@ -1222,7 +1222,7 @@ InitPredicateLocks(void)
 	 * Compute size for serializable transaction hashtable. Note these
 	 * calculations must agree with PredicateLockShmemSize!
 	 */
-	max_table_size = (MaxBackends + max_prepared_xacts);
+	max_table_size = (GetMaxBackends(GMB_MAX_PREPARED_XACTS));
 
 	/*
 	 * Allocate a list to hold information on transactions participating in
@@ -1374,7 +1374,7 @@ PredicateLockShmemSize(void)
 	size = add_size(size, size / 10);
 
 	/* transaction list */
-	max_table_size = MaxBackends + max_prepared_xacts;
+	max_table_size = GetMaxBackends(GMB_MAX_PREPARED_XACTS);
 	max_table_size *= 10;
 	size = add_size(size, PredXactListDataSize);
 	size = add_size(size, mul_size((Size) max_table_size,
@@ -1905,7 +1905,7 @@ GetSerializableTransactionSnapshotInt(Snapshot snapshot,
 	{
 		++(PredXact->WritableSxactCount);
 		Assert(PredXact->WritableSxactCount <=
-			   (MaxBackends + max_prepared_xacts));
+			   (GetMaxBackends(GMB_MAX_PREPARED_XACTS)));
 	}
 
 	MySerializableXact = sxact;
@@ -5107,7 +5107,7 @@ predicatelock_twophase_recover(TransactionId xid, uint16 info,
 		{
 			++(PredXact->WritableSxactCount);
 			Assert(PredXact->WritableSxactCount <=
-				   (MaxBackends + max_prepared_xacts));
+				   (GetMaxBackends(GMB_MAX_PREPARED_XACTS)));
 		}
 
 		/*
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index b7d9da0aa9..9600afec16 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -102,8 +102,7 @@ Size
 ProcGlobalShmemSize(void)
 {
 	Size		size = 0;
-	Size		TotalProcs =
-	add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+	Size		TotalProcs = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS | GMB_MAX_PREPARED_XACTS);
 
 	/* ProcGlobal */
 	size = add_size(size, sizeof(PROC_HDR));
@@ -127,7 +126,7 @@ ProcGlobalSemas(void)
 	 * We need a sema per backend (including autovacuum), plus one for each
 	 * auxiliary process.
 	 */
-	return MaxBackends + NUM_AUXILIARY_PROCS;
+	return GetMaxBackends(GMB_NUM_AUXILIARY_PROCS);
 }
 
 /*
@@ -162,7 +161,7 @@ InitProcGlobal(void)
 	int			i,
 				j;
 	bool		found;
-	uint32		TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
+	uint32		TotalProcs = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS | GMB_MAX_PREPARED_XACTS);
 
 	/* Create the ProcGlobal shared structure */
 	ProcGlobal = (PROC_HDR *)
@@ -197,7 +196,7 @@ InitProcGlobal(void)
 	MemSet(procs, 0, TotalProcs * sizeof(PGPROC));
 	ProcGlobal->allProcs = procs;
 	/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
-	ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+	ProcGlobal->allProcCount = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS);
 
 	/*
 	 * Allocate arrays mirroring PGPROC fields in a dense manner. See
@@ -223,7 +222,7 @@ InitProcGlobal(void)
 		 * dummy PGPROCs don't need these though - they're never associated
 		 * with a real process
 		 */
-		if (i < MaxBackends + NUM_AUXILIARY_PROCS)
+		if (i < GetMaxBackends(GMB_NUM_AUXILIARY_PROCS))
 		{
 			procs[i].sem = PGSemaphoreCreate();
 			InitSharedLatch(&(procs[i].procLatch));
@@ -260,7 +259,7 @@ InitProcGlobal(void)
 			ProcGlobal->bgworkerFreeProcs = &procs[i];
 			procs[i].procgloballist = &ProcGlobal->bgworkerFreeProcs;
 		}
-		else if (i < MaxBackends)
+		else if (i < GetMaxBackends(0))
 		{
 			/* PGPROC for walsender, add to walsenderFreeProcs list */
 			procs[i].links.next = (SHM_QUEUE *) ProcGlobal->walsenderFreeProcs;
@@ -288,8 +287,8 @@ InitProcGlobal(void)
 	 * Save pointers to the blocks of PGPROC structures reserved for auxiliary
 	 * processes and prepared transactions.
 	 */
-	AuxiliaryProcs = &procs[MaxBackends];
-	PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+	AuxiliaryProcs = &procs[GetMaxBackends(0)];
+	PreparedXactProcs = &procs[GetMaxBackends(GMB_NUM_AUXILIARY_PROCS)];
 
 	/* Create ProcStructLock spinlock, too */
 	ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 2901f9f5a9..4ff834bbc5 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -35,7 +35,7 @@
  * includes autovacuum workers and background workers as well.
  * ----------
  */
-#define NumBackendStatSlots (MaxBackends + NUM_AUXPROCTYPES)
+#define NumBackendStatSlots (GetMaxBackends(0) + NUM_AUXPROCTYPES)
 
 
 /* ----------
@@ -251,7 +251,7 @@ pgstat_beinit(void)
 	/* Initialize MyBEEntry */
 	if (MyBackendId != InvalidBackendId)
 	{
-		Assert(MyBackendId >= 1 && MyBackendId <= MaxBackends);
+		Assert(MyBackendId >= 1 && MyBackendId <= GetMaxBackends(0));
 		MyBEEntry = &BackendStatusArray[MyBackendId - 1];
 	}
 	else
@@ -267,7 +267,7 @@ pgstat_beinit(void)
 		 * MaxBackends + AuxBackendType + 1 as the index of the slot for an
 		 * auxiliary process.
 		 */
-		MyBEEntry = &BackendStatusArray[MaxBackends + MyAuxProcType];
+		MyBEEntry = &BackendStatusArray[GetMaxBackends(0) + MyAuxProcType];
 	}
 
 	/* Set up a process-exit hook to clean up */
@@ -892,7 +892,7 @@ pgstat_get_backend_current_activity(int pid, bool checkUser)
 	int			i;
 
 	beentry = BackendStatusArray;
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= GetMaxBackends(0); i++)
 	{
 		/*
 		 * Although we expect the target backend's entry to be stable, that
@@ -978,7 +978,7 @@ pgstat_get_crashed_backend_activity(int pid, char *buffer, int buflen)
 	if (beentry == NULL || BackendActivityBuffer == NULL)
 		return NULL;
 
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= GetMaxBackends(0); i++)
 	{
 		if (beentry->st_procpid == pid)
 		{
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 5dc0a5882c..d262965800 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -561,11 +561,11 @@ pg_safe_snapshot_blocking_pids(PG_FUNCTION_ARGS)
 	Datum	   *blocker_datums;
 
 	/* A buffer big enough for any possible blocker list without truncation */
-	blockers = (int *) palloc(MaxBackends * sizeof(int));
+	blockers = (int *) palloc(GetMaxBackends(0) * sizeof(int));
 
 	/* Collect a snapshot of processes waited for by GetSafeSnapshot */
 	num_blockers =
-		GetSafeSnapshotBlockingPids(blocked_pid, blockers, MaxBackends);
+		GetSafeSnapshotBlockingPids(blocked_pid, blockers, GetMaxBackends(0));
 
 	/* Convert int array to Datum array */
 	if (num_blockers > 0)
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 51d1bbef30..bf02bc0283 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -25,6 +25,7 @@
 #include "access/session.h"
 #include "access/sysattr.h"
 #include "access/tableam.h"
+#include "access/twophase.h"
 #include "access/xact.h"
 #include "access/xlog.h"
 #include "catalog/catalog.h"
@@ -63,6 +64,9 @@
 #include "utils/syscache.h"
 #include "utils/timeout.h"
 
+static int MaxBackends = 0;
+static int MaxBackendsInitialized = false;
+
 static HeapTuple GetDatabaseTuple(const char *dbname);
 static HeapTuple GetDatabaseTupleByOid(Oid dboid);
 static void PerformAuthentication(Port *port);
@@ -502,9 +506,8 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called after modules have had the change to alter GUCs in
+ * shared_preload_libraries, and before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
@@ -514,15 +517,41 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 void
 InitializeMaxBackends(void)
 {
-	Assert(MaxBackends == 0);
-
 	/* the extra unit accounts for the autovacuum launcher */
-	MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
-		max_worker_processes + max_wal_senders;
+	SetMaxBackends(MaxConnections + autovacuum_max_workers + 1 +
+		max_worker_processes + max_wal_senders);
+}
+
+int
+GetMaxBackends(uint32 flags)
+{
+	int ret;
+
+	if (!MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends not yet initialized");
+
+	ret = MaxBackends;
+
+	if (flags & GMB_MAX_PREPARED_XACTS)
+		ret += max_prepared_xacts;
 
-	/* internal error because the values were all checked previously */
-	if (MaxBackends > MAX_BACKENDS)
+	if (flags & GMB_NUM_AUXILIARY_PROCS)
+		ret += NUM_AUXILIARY_PROCS;
+
+	return ret;
+}
+
+void
+SetMaxBackends(int max_backends)
+{
+	if (MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends already initialized");
+
+	if (max_backends > MAX_BACKENDS)
 		elog(ERROR, "too many backends configured");
+
+	MaxBackends = max_backends;
+	MaxBackendsInitialized = true;
 }
 
 /*
@@ -603,7 +632,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 
 	SharedInvalBackendInit(false);
 
-	if (MyBackendId > MaxBackends || MyBackendId <= 0)
+	if (MyBackendId > GetMaxBackends(0) || MyBackendId <= 0)
 		elog(FATAL, "bad backend ID: %d", MyBackendId);
 
 	/* Now that we have a BackendId, we can participate in ProcSignal */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 68d840d699..67902d2541 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -172,7 +172,6 @@ extern PGDLLIMPORT char *DataDir;
 extern PGDLLIMPORT int data_directory_mode;
 
 extern PGDLLIMPORT int NBuffers;
-extern PGDLLIMPORT int MaxBackends;
 extern PGDLLIMPORT int MaxConnections;
 extern PGDLLIMPORT int max_worker_processes;
 extern PGDLLIMPORT int max_parallel_workers;
@@ -455,9 +454,20 @@ extern AuxProcType MyAuxProcType;
  *			POSTGRES initialization and cleanup definitions.                 *
  *****************************************************************************/
 
+/*
+ * Option flag bits for GetMaxBackends().
+ */
+typedef enum GMBOption
+{
+	GMB_MAX_PREPARED_XACTS = 1 << 0,	/* include max_prepared_xacts */
+	GMB_NUM_AUXILIARY_PROCS = 1 << 1	/* include NUM_AUXILIARY_PROCS */
+} GMBOption;
+
 /* in utils/init/postinit.c */
 extern void pg_split_opts(char **argv, int *argcp, const char *optstr);
 extern void InitializeMaxBackends(void);
+extern int GetMaxBackends(uint32 flags);
+extern void SetMaxBackends(int max_backends);
 extern void InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 						 Oid useroid, char *out_dbname, bool override_allow_connections);
 extern void BaseInit(void);
-- 
2.16.6



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-07 18:01  Bossart, Nathan <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-07 18:01 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; Andres Freund <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 8/3/21, 4:14 PM, "Bossart, Nathan" <[email protected]> wrote:
> On 8/2/21, 4:02 PM, "Bossart, Nathan" <[email protected]> wrote:
>> On 8/2/21, 3:42 PM, "Andres Freund" <[email protected]> wrote:
>>> I've wondered, independent of this thread, about not making MaxBackends
>>> externally visible, and requiring a function call to access it. It should be
>>> easier to find cases of misuse if we errored out when accessed at the wrong
>>> time. And we could use that opportunity to add flags that determine which
>>> types of backends are included (e.g. not including autovac, or additionally
>>> including aux workers or prepared xacts).
>>
>> I'm not opposed to this.  I can work on putting a patch together if no
>> opposition materializes.
>
> Here is a first attempt.

Here is a rebased version of the patch.

Nathan



Attachments:

  [application/octet-stream] v2-0001-Disallow-external-access-to-MaxBackends.patch (30.3K, ../../[email protected]/2-v2-0001-Disallow-external-access-to-MaxBackends.patch)
  download | inline diff:
From ee6b30cc910bcc53670bed65237518a59e88ae4b Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Sat, 7 Aug 2021 17:58:07 +0000
Subject: [PATCH v2 1/1] Disallow external access to MaxBackends.

Presently, MaxBackends is externally visible, but it may still be
uninitialized in places where it would be convenient to use (e.g.,
_PG_init()).  This change makes MaxBackends static to postinit.c to
disallow such direct access.  Instead, MaxBackends should now be
accessed via GetMaxBackends().  GetMaxBackends() also accepts a
couple of flags to easily include commonly added values such as
max_prepared_transactions.

Separately, adjust the comments about needing to register
background workers before initializing MaxBackends.  Since
6bc8ef0b, InitializeMaxBackends() has used max_worker_processes
instead of tallying up the number of registered background workers,
so background worker registration is no longer a prerequisite.  The
ordering of this logic is still useful for allowing libraries to
adjust GUCs, so the comments have been updated to mention that use-
case.
---
 src/backend/access/nbtree/nbtutils.c        |  4 +--
 src/backend/access/transam/multixact.c      |  2 +-
 src/backend/access/transam/twophase.c       |  2 +-
 src/backend/commands/async.c                | 10 +++---
 src/backend/libpq/pqcomm.c                  |  2 +-
 src/backend/postmaster/auxprocess.c         |  2 +-
 src/backend/postmaster/postmaster.c         | 14 ++++-----
 src/backend/storage/ipc/dsm.c               |  2 +-
 src/backend/storage/ipc/procarray.c         |  2 +-
 src/backend/storage/ipc/procsignal.c        |  2 +-
 src/backend/storage/ipc/sinvaladt.c         |  4 +--
 src/backend/storage/lmgr/deadlock.c         | 31 +++++++++---------
 src/backend/storage/lmgr/lock.c             | 20 ++++++------
 src/backend/storage/lmgr/predicate.c        | 10 +++---
 src/backend/storage/lmgr/proc.c             | 17 +++++-----
 src/backend/utils/activity/backend_status.c | 10 +++---
 src/backend/utils/adt/lockfuncs.c           |  4 +--
 src/backend/utils/init/postinit.c           | 49 +++++++++++++++++++++++------
 src/include/miscadmin.h                     | 12 ++++++-
 19 files changed, 118 insertions(+), 81 deletions(-)

diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index d524310723..578d4531a2 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -2065,7 +2065,7 @@ BTreeShmemSize(void)
 	Size		size;
 
 	size = offsetof(BTVacInfo, vacuums);
-	size = add_size(size, mul_size(MaxBackends, sizeof(BTOneVacInfo)));
+	size = add_size(size, mul_size(GetMaxBackends(0), sizeof(BTOneVacInfo)));
 	return size;
 }
 
@@ -2094,7 +2094,7 @@ BTreeShmemInit(void)
 		btvacinfo->cycle_ctr = (BTCycleId) time(NULL);
 
 		btvacinfo->num_vacuums = 0;
-		btvacinfo->max_vacuums = MaxBackends;
+		btvacinfo->max_vacuums = GetMaxBackends(0);
 	}
 	else
 		Assert(found);
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index e6c70ed0bc..7e9728bf88 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -285,7 +285,7 @@ typedef struct MultiXactStateData
  * Last element of OldestMemberMXactId and OldestVisibleMXactId arrays.
  * Valid elements are (1..MaxOldestSlot); element 0 is never used.
  */
-#define MaxOldestSlot	(MaxBackends + max_prepared_xacts)
+#define MaxOldestSlot	(GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 /* Pointers to the state data in shared memory */
 static MultiXactStateData *MultiXactState;
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 6d3efb49a4..1ba84983f2 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -293,7 +293,7 @@ TwoPhaseShmemInit(void)
 			 * prepared transaction. Currently multixact.c uses that
 			 * technique.
 			 */
-			gxacts[i].dummyBackendId = MaxBackends + 1 + i;
+			gxacts[i].dummyBackendId = GetMaxBackends(0) + 1 + i;
 		}
 	}
 	else
diff --git a/src/backend/commands/async.c b/src/backend/commands/async.c
index 4b16fb5682..29c50595c2 100644
--- a/src/backend/commands/async.c
+++ b/src/backend/commands/async.c
@@ -511,7 +511,7 @@ AsyncShmemSize(void)
 	Size		size;
 
 	/* This had better match AsyncShmemInit */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends(0) + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	size = add_size(size, SimpleLruShmemSize(NUM_NOTIFY_BUFFERS, 0));
@@ -534,7 +534,7 @@ AsyncShmemInit(void)
 	 * The used entries in the backend[] array run from 1 to MaxBackends; the
 	 * zero'th entry is unused but must be allocated.
 	 */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends(0) + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	asyncQueueControl = (AsyncQueueControl *)
@@ -549,7 +549,7 @@ AsyncShmemInit(void)
 		QUEUE_FIRST_LISTENER = InvalidBackendId;
 		asyncQueueControl->lastQueueFillWarn = 0;
 		/* zero'th entry won't be used, but let's initialize it anyway */
-		for (int i = 0; i <= MaxBackends; i++)
+		for (int i = 0; i <= GetMaxBackends(0); i++)
 		{
 			QUEUE_BACKEND_PID(i) = InvalidPid;
 			QUEUE_BACKEND_DBOID(i) = InvalidOid;
@@ -1685,8 +1685,8 @@ SignalBackends(void)
 	 * preallocate the arrays?	But in practice this is only run in trivial
 	 * transactions, so there should surely be space available.
 	 */
-	pids = (int32 *) palloc(MaxBackends * sizeof(int32));
-	ids = (BackendId *) palloc(MaxBackends * sizeof(BackendId));
+	pids = (int32 *) palloc(GetMaxBackends(0) * sizeof(int32));
+	ids = (BackendId *) palloc(GetMaxBackends(0) * sizeof(BackendId));
 	count = 0;
 
 	LWLockAcquire(NotifyQueueLock, LW_EXCLUSIVE);
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 89a5f901aa..4d74f9a3f5 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -556,7 +556,7 @@ StreamServerPort(int family, const char *hostName, unsigned short portNumber,
 		 * intended to provide a clamp on the request on platforms where an
 		 * overly large request provokes a kernel error (are there any?).
 		 */
-		maxconn = MaxBackends * 2;
+		maxconn = GetMaxBackends(0) * 2;
 		if (maxconn > PG_SOMAXCONN)
 			maxconn = PG_SOMAXCONN;
 
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 7452f908b2..208aa5803d 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -116,7 +116,7 @@ AuxiliaryProcessMain(AuxProcType auxtype)
 	 * This will need rethinking if we ever want more than one of a particular
 	 * auxiliary process type.
 	 */
-	ProcSignalInit(MaxBackends + MyAuxProcType + 1);
+	ProcSignalInit(GetMaxBackends(0) + MyAuxProcType + 1);
 
 	/*
 	 * Auxiliary processes don't run transactions, but they may need a
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index fc0bc8d99e..271fbce4b8 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -997,10 +997,8 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher.  It's probably a good idea to call it before
+	 * any modules had a chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1021,8 +1019,8 @@ PostmasterMain(int argc, char *argv[])
 #endif
 
 	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
+	 * Now that loadable modules have had their chance to alter any GUCs,
+	 * calculate MaxBackends.
 	 */
 	InitializeMaxBackends();
 
@@ -6187,7 +6185,7 @@ save_backend_variables(BackendParameters *param, Port *port,
 	param->query_id_enabled = query_id_enabled;
 	param->max_safe_fds = max_safe_fds;
 
-	param->MaxBackends = MaxBackends;
+	param->MaxBackends = GetMaxBackends(0);
 
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
@@ -6421,7 +6419,7 @@ restore_backend_variables(BackendParameters *param, Port *port)
 	query_id_enabled = param->query_id_enabled;
 	max_safe_fds = param->max_safe_fds;
 
-	MaxBackends = param->MaxBackends;
+	SetMaxBackends(param->MaxBackends);
 
 #ifdef WIN32
 	PostmasterHandle = param->PostmasterHandle;
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index b461a5f7e9..bb279bb8d1 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -165,7 +165,7 @@ dsm_postmaster_startup(PGShmemHeader *shim)
 
 	/* Determine size for new control segment. */
 	maxitems = PG_DYNSHMEM_FIXED_SLOTS
-		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * MaxBackends;
+		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * GetMaxBackends(0);
 	elog(DEBUG2, "dynamic shared memory system will support %u segments",
 		 maxitems);
 	segsize = dsm_control_bytes_needed(maxitems);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index c7816fcfb3..9afd0209c8 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -365,7 +365,7 @@ ProcArrayShmemSize(void)
 	Size		size;
 
 	/* Size of the ProcArray structure itself */
-#define PROCARRAY_MAXPROCS	(MaxBackends + max_prepared_xacts)
+#define PROCARRAY_MAXPROCS	(GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 	size = offsetof(ProcArrayStruct, pgprocnos);
 	size = add_size(size, mul_size(sizeof(int), PROCARRAY_MAXPROCS));
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index defb75aa26..86b7c39782 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -85,7 +85,7 @@ typedef struct
  * possible auxiliary process type.  (This scheme assumes there is not
  * more than one of any auxiliary process type at a time.)
  */
-#define NumProcSignalSlots	(MaxBackends + NUM_AUXPROCTYPES)
+#define NumProcSignalSlots	(GetMaxBackends(0) + NUM_AUXPROCTYPES)
 
 /* Check whether the relevant type bit is set in the flags. */
 #define BARRIER_SHOULD_CHECK(flags, type) \
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index 946bd8e3cb..2e85a99f68 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -205,7 +205,7 @@ SInvalShmemSize(void)
 	Size		size;
 
 	size = offsetof(SISeg, procState);
-	size = add_size(size, mul_size(sizeof(ProcState), MaxBackends));
+	size = add_size(size, mul_size(sizeof(ProcState), GetMaxBackends(0)));
 
 	return size;
 }
@@ -231,7 +231,7 @@ CreateSharedInvalidationState(void)
 	shmInvalBuffer->maxMsgNum = 0;
 	shmInvalBuffer->nextThreshold = CLEANUP_MIN;
 	shmInvalBuffer->lastBackend = 0;
-	shmInvalBuffer->maxBackends = MaxBackends;
+	shmInvalBuffer->maxBackends = GetMaxBackends(0);
 	SpinLockInit(&shmInvalBuffer->msgnumLock);
 
 	/* The buffer[] array is initially all unused, so we need not fill it */
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index 67733c0d1a..6ffc481009 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -143,6 +143,7 @@ void
 InitDeadLockChecking(void)
 {
 	MemoryContext oldcxt;
+	int max_backends = GetMaxBackends(0);
 
 	/* Make sure allocations are permanent */
 	oldcxt = MemoryContextSwitchTo(TopMemoryContext);
@@ -151,16 +152,16 @@ InitDeadLockChecking(void)
 	 * FindLockCycle needs at most MaxBackends entries in visitedProcs[] and
 	 * deadlockDetails[].
 	 */
-	visitedProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
-	deadlockDetails = (DEADLOCK_INFO *) palloc(MaxBackends * sizeof(DEADLOCK_INFO));
+	visitedProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
+	deadlockDetails = (DEADLOCK_INFO *) palloc(max_backends * sizeof(DEADLOCK_INFO));
 
 	/*
 	 * TopoSort needs to consider at most MaxBackends wait-queue entries, and
 	 * it needn't run concurrently with FindLockCycle.
 	 */
 	topoProcs = visitedProcs;	/* re-use this space */
-	beforeConstraints = (int *) palloc(MaxBackends * sizeof(int));
-	afterConstraints = (int *) palloc(MaxBackends * sizeof(int));
+	beforeConstraints = (int *) palloc(max_backends * sizeof(int));
+	afterConstraints = (int *) palloc(max_backends * sizeof(int));
 
 	/*
 	 * We need to consider rearranging at most MaxBackends/2 wait queues
@@ -169,8 +170,8 @@ InitDeadLockChecking(void)
 	 * MaxBackends total waiters.
 	 */
 	waitOrders = (WAIT_ORDER *)
-		palloc((MaxBackends / 2) * sizeof(WAIT_ORDER));
-	waitOrderProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
+		palloc((max_backends / 2) * sizeof(WAIT_ORDER));
+	waitOrderProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
 
 	/*
 	 * Allow at most MaxBackends distinct constraints in a configuration. (Is
@@ -180,7 +181,7 @@ InitDeadLockChecking(void)
 	 * limits the maximum recursion depth of DeadLockCheckRecurse. Making it
 	 * really big might potentially allow a stack-overflow problem.
 	 */
-	maxCurConstraints = MaxBackends;
+	maxCurConstraints = max_backends;
 	curConstraints = (EDGE *) palloc(maxCurConstraints * sizeof(EDGE));
 
 	/*
@@ -191,7 +192,7 @@ InitDeadLockChecking(void)
 	 * last MaxBackends entries in possibleConstraints[] are reserved as
 	 * output workspace for FindLockCycle.
 	 */
-	maxPossibleConstraints = MaxBackends * 4;
+	maxPossibleConstraints = max_backends * 4;
 	possibleConstraints =
 		(EDGE *) palloc(maxPossibleConstraints * sizeof(EDGE));
 
@@ -327,7 +328,7 @@ DeadLockCheckRecurse(PGPROC *proc)
 	if (nCurConstraints >= maxCurConstraints)
 		return true;			/* out of room for active constraints? */
 	oldPossibleConstraints = nPossibleConstraints;
-	if (nPossibleConstraints + nEdges + MaxBackends <= maxPossibleConstraints)
+	if (nPossibleConstraints + nEdges + GetMaxBackends(0) <= maxPossibleConstraints)
 	{
 		/* We can save the edge list in possibleConstraints[] */
 		nPossibleConstraints += nEdges;
@@ -388,7 +389,7 @@ TestConfiguration(PGPROC *startProc)
 	/*
 	 * Make sure we have room for FindLockCycle's output.
 	 */
-	if (nPossibleConstraints + MaxBackends > maxPossibleConstraints)
+	if (nPossibleConstraints + GetMaxBackends(0) > maxPossibleConstraints)
 		return -1;
 
 	/*
@@ -486,7 +487,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 				 * record total length of cycle --- outer levels will now fill
 				 * deadlockDetails[]
 				 */
-				Assert(depth <= MaxBackends);
+				Assert(depth <= GetMaxBackends(0));
 				nDeadlockDetails = depth;
 
 				return true;
@@ -500,7 +501,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 		}
 	}
 	/* Mark proc as seen */
-	Assert(nVisitedProcs < MaxBackends);
+	Assert(nVisitedProcs < GetMaxBackends(0));
 	visitedProcs[nVisitedProcs++] = checkProc;
 
 	/*
@@ -698,7 +699,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends(0));
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -771,7 +772,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends(0));
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -834,7 +835,7 @@ ExpandConstraints(EDGE *constraints,
 		waitOrders[nWaitOrders].procs = waitOrderProcs + nWaitOrderProcs;
 		waitOrders[nWaitOrders].nProcs = lock->waitProcs.size;
 		nWaitOrderProcs += lock->waitProcs.size;
-		Assert(nWaitOrderProcs <= MaxBackends);
+		Assert(nWaitOrderProcs <= GetMaxBackends(0));
 
 		/*
 		 * Do the topo sort.  TopoSort need not examine constraints after this
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..581bbc29d5 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -55,7 +55,7 @@
 int			max_locks_per_xact; /* set by guc.c */
 
 #define NLOCKENTS() \
-	mul_size(max_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_locks_per_xact, GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 
 /*
@@ -2938,12 +2938,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 			vxids = (VirtualTransactionId *)
 				MemoryContextAlloc(TopMemoryContext,
 								   sizeof(VirtualTransactionId) *
-								   (MaxBackends + max_prepared_xacts + 1));
+								   (GetMaxBackends(GMB_MAX_PREPARED_XACTS) + 1));
 	}
 	else
 		vxids = (VirtualTransactionId *)
 			palloc0(sizeof(VirtualTransactionId) *
-					(MaxBackends + max_prepared_xacts + 1));
+					(GetMaxBackends(GMB_MAX_PREPARED_XACTS) + 1));
 
 	/* Compute hash code and partition lock, and look up conflicting modes. */
 	hashcode = LockTagHashCode(locktag);
@@ -3100,7 +3100,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 
 	LWLockRelease(partitionLock);
 
-	if (count > MaxBackends + max_prepared_xacts)	/* should never happen */
+	if (count > GetMaxBackends(GMB_MAX_PREPARED_XACTS))	/* should never happen */
 		elog(PANIC, "too many conflicting locks found");
 
 	vxids[count].backendId = InvalidBackendId;
@@ -3651,7 +3651,7 @@ GetLockStatusData(void)
 	data = (LockData *) palloc(sizeof(LockData));
 
 	/* Guess how much space we'll need. */
-	els = MaxBackends;
+	els = GetMaxBackends(0);
 	el = 0;
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * els);
 
@@ -3685,7 +3685,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends(0);
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3717,7 +3717,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends(0);
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3846,7 +3846,7 @@ GetBlockerStatusData(int blocked_pid)
 	 * for the procs[] array; the other two could need enlargement, though.)
 	 */
 	data->nprocs = data->nlocks = data->npids = 0;
-	data->maxprocs = data->maxlocks = data->maxpids = MaxBackends;
+	data->maxprocs = data->maxlocks = data->maxpids = GetMaxBackends(0);
 	data->procs = (BlockedProcData *) palloc(sizeof(BlockedProcData) * data->maxprocs);
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * data->maxlocks);
 	data->waiter_pids = (int *) palloc(sizeof(int) * data->maxpids);
@@ -3949,7 +3949,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 		if (data->nlocks >= data->maxlocks)
 		{
-			data->maxlocks += MaxBackends;
+			data->maxlocks += GetMaxBackends(0);
 			data->locks = (LockInstanceData *)
 				repalloc(data->locks, sizeof(LockInstanceData) * data->maxlocks);
 		}
@@ -3978,7 +3978,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 	if (queue_size > data->maxpids - data->npids)
 	{
-		data->maxpids = Max(data->maxpids + MaxBackends,
+		data->maxpids = Max(data->maxpids + GetMaxBackends(0),
 							data->npids + queue_size);
 		data->waiter_pids = (int *) repalloc(data->waiter_pids,
 											 sizeof(int) * data->maxpids);
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 56267bdc3c..d6199aa82d 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -257,7 +257,7 @@
 	(&MainLWLockArray[PREDICATELOCK_MANAGER_LWLOCK_OFFSET + (i)].lock)
 
 #define NPREDICATELOCKTARGETENTS() \
-	mul_size(max_predicate_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_predicate_locks_per_xact, GetMaxBackends(GMB_MAX_PREPARED_XACTS))
 
 #define SxactIsOnFinishedList(sxact) (!SHMQueueIsDetached(&((sxact)->finishedLink)))
 
@@ -1222,7 +1222,7 @@ InitPredicateLocks(void)
 	 * Compute size for serializable transaction hashtable. Note these
 	 * calculations must agree with PredicateLockShmemSize!
 	 */
-	max_table_size = (MaxBackends + max_prepared_xacts);
+	max_table_size = (GetMaxBackends(GMB_MAX_PREPARED_XACTS));
 
 	/*
 	 * Allocate a list to hold information on transactions participating in
@@ -1374,7 +1374,7 @@ PredicateLockShmemSize(void)
 	size = add_size(size, size / 10);
 
 	/* transaction list */
-	max_table_size = MaxBackends + max_prepared_xacts;
+	max_table_size = GetMaxBackends(GMB_MAX_PREPARED_XACTS);
 	max_table_size *= 10;
 	size = add_size(size, PredXactListDataSize);
 	size = add_size(size, mul_size((Size) max_table_size,
@@ -1905,7 +1905,7 @@ GetSerializableTransactionSnapshotInt(Snapshot snapshot,
 	{
 		++(PredXact->WritableSxactCount);
 		Assert(PredXact->WritableSxactCount <=
-			   (MaxBackends + max_prepared_xacts));
+			   (GetMaxBackends(GMB_MAX_PREPARED_XACTS)));
 	}
 
 	MySerializableXact = sxact;
@@ -5107,7 +5107,7 @@ predicatelock_twophase_recover(TransactionId xid, uint16 info,
 		{
 			++(PredXact->WritableSxactCount);
 			Assert(PredXact->WritableSxactCount <=
-				   (MaxBackends + max_prepared_xacts));
+				   (GetMaxBackends(GMB_MAX_PREPARED_XACTS)));
 		}
 
 		/*
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index b7d9da0aa9..9600afec16 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -102,8 +102,7 @@ Size
 ProcGlobalShmemSize(void)
 {
 	Size		size = 0;
-	Size		TotalProcs =
-	add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+	Size		TotalProcs = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS | GMB_MAX_PREPARED_XACTS);
 
 	/* ProcGlobal */
 	size = add_size(size, sizeof(PROC_HDR));
@@ -127,7 +126,7 @@ ProcGlobalSemas(void)
 	 * We need a sema per backend (including autovacuum), plus one for each
 	 * auxiliary process.
 	 */
-	return MaxBackends + NUM_AUXILIARY_PROCS;
+	return GetMaxBackends(GMB_NUM_AUXILIARY_PROCS);
 }
 
 /*
@@ -162,7 +161,7 @@ InitProcGlobal(void)
 	int			i,
 				j;
 	bool		found;
-	uint32		TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
+	uint32		TotalProcs = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS | GMB_MAX_PREPARED_XACTS);
 
 	/* Create the ProcGlobal shared structure */
 	ProcGlobal = (PROC_HDR *)
@@ -197,7 +196,7 @@ InitProcGlobal(void)
 	MemSet(procs, 0, TotalProcs * sizeof(PGPROC));
 	ProcGlobal->allProcs = procs;
 	/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
-	ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+	ProcGlobal->allProcCount = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS);
 
 	/*
 	 * Allocate arrays mirroring PGPROC fields in a dense manner. See
@@ -223,7 +222,7 @@ InitProcGlobal(void)
 		 * dummy PGPROCs don't need these though - they're never associated
 		 * with a real process
 		 */
-		if (i < MaxBackends + NUM_AUXILIARY_PROCS)
+		if (i < GetMaxBackends(GMB_NUM_AUXILIARY_PROCS))
 		{
 			procs[i].sem = PGSemaphoreCreate();
 			InitSharedLatch(&(procs[i].procLatch));
@@ -260,7 +259,7 @@ InitProcGlobal(void)
 			ProcGlobal->bgworkerFreeProcs = &procs[i];
 			procs[i].procgloballist = &ProcGlobal->bgworkerFreeProcs;
 		}
-		else if (i < MaxBackends)
+		else if (i < GetMaxBackends(0))
 		{
 			/* PGPROC for walsender, add to walsenderFreeProcs list */
 			procs[i].links.next = (SHM_QUEUE *) ProcGlobal->walsenderFreeProcs;
@@ -288,8 +287,8 @@ InitProcGlobal(void)
 	 * Save pointers to the blocks of PGPROC structures reserved for auxiliary
 	 * processes and prepared transactions.
 	 */
-	AuxiliaryProcs = &procs[MaxBackends];
-	PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+	AuxiliaryProcs = &procs[GetMaxBackends(0)];
+	PreparedXactProcs = &procs[GetMaxBackends(GMB_NUM_AUXILIARY_PROCS)];
 
 	/* Create ProcStructLock spinlock, too */
 	ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 2901f9f5a9..4ff834bbc5 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -35,7 +35,7 @@
  * includes autovacuum workers and background workers as well.
  * ----------
  */
-#define NumBackendStatSlots (MaxBackends + NUM_AUXPROCTYPES)
+#define NumBackendStatSlots (GetMaxBackends(0) + NUM_AUXPROCTYPES)
 
 
 /* ----------
@@ -251,7 +251,7 @@ pgstat_beinit(void)
 	/* Initialize MyBEEntry */
 	if (MyBackendId != InvalidBackendId)
 	{
-		Assert(MyBackendId >= 1 && MyBackendId <= MaxBackends);
+		Assert(MyBackendId >= 1 && MyBackendId <= GetMaxBackends(0));
 		MyBEEntry = &BackendStatusArray[MyBackendId - 1];
 	}
 	else
@@ -267,7 +267,7 @@ pgstat_beinit(void)
 		 * MaxBackends + AuxBackendType + 1 as the index of the slot for an
 		 * auxiliary process.
 		 */
-		MyBEEntry = &BackendStatusArray[MaxBackends + MyAuxProcType];
+		MyBEEntry = &BackendStatusArray[GetMaxBackends(0) + MyAuxProcType];
 	}
 
 	/* Set up a process-exit hook to clean up */
@@ -892,7 +892,7 @@ pgstat_get_backend_current_activity(int pid, bool checkUser)
 	int			i;
 
 	beentry = BackendStatusArray;
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= GetMaxBackends(0); i++)
 	{
 		/*
 		 * Although we expect the target backend's entry to be stable, that
@@ -978,7 +978,7 @@ pgstat_get_crashed_backend_activity(int pid, char *buffer, int buflen)
 	if (beentry == NULL || BackendActivityBuffer == NULL)
 		return NULL;
 
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= GetMaxBackends(0); i++)
 	{
 		if (beentry->st_procpid == pid)
 		{
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 5dc0a5882c..d262965800 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -561,11 +561,11 @@ pg_safe_snapshot_blocking_pids(PG_FUNCTION_ARGS)
 	Datum	   *blocker_datums;
 
 	/* A buffer big enough for any possible blocker list without truncation */
-	blockers = (int *) palloc(MaxBackends * sizeof(int));
+	blockers = (int *) palloc(GetMaxBackends(0) * sizeof(int));
 
 	/* Collect a snapshot of processes waited for by GetSafeSnapshot */
 	num_blockers =
-		GetSafeSnapshotBlockingPids(blocked_pid, blockers, MaxBackends);
+		GetSafeSnapshotBlockingPids(blocked_pid, blockers, GetMaxBackends(0));
 
 	/* Convert int array to Datum array */
 	if (num_blockers > 0)
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 87dc060b20..89a03edaab 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -25,6 +25,7 @@
 #include "access/session.h"
 #include "access/sysattr.h"
 #include "access/tableam.h"
+#include "access/twophase.h"
 #include "access/xact.h"
 #include "access/xlog.h"
 #include "catalog/catalog.h"
@@ -63,6 +64,9 @@
 #include "utils/syscache.h"
 #include "utils/timeout.h"
 
+static int MaxBackends = 0;
+static int MaxBackendsInitialized = false;
+
 static HeapTuple GetDatabaseTuple(const char *dbname);
 static HeapTuple GetDatabaseTupleByOid(Oid dboid);
 static void PerformAuthentication(Port *port);
@@ -476,9 +480,8 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called after modules have had the change to alter GUCs in
+ * shared_preload_libraries, and before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
@@ -488,15 +491,41 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 void
 InitializeMaxBackends(void)
 {
-	Assert(MaxBackends == 0);
-
 	/* the extra unit accounts for the autovacuum launcher */
-	MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
-		max_worker_processes + max_wal_senders;
+	SetMaxBackends(MaxConnections + autovacuum_max_workers + 1 +
+		max_worker_processes + max_wal_senders);
+}
+
+int
+GetMaxBackends(uint32 flags)
+{
+	int ret;
+
+	if (!MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends not yet initialized");
+
+	ret = MaxBackends;
+
+	if (flags & GMB_MAX_PREPARED_XACTS)
+		ret += max_prepared_xacts;
 
-	/* internal error because the values were all checked previously */
-	if (MaxBackends > MAX_BACKENDS)
+	if (flags & GMB_NUM_AUXILIARY_PROCS)
+		ret += NUM_AUXILIARY_PROCS;
+
+	return ret;
+}
+
+void
+SetMaxBackends(int max_backends)
+{
+	if (MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends already initialized");
+
+	if (max_backends > MAX_BACKENDS)
 		elog(ERROR, "too many backends configured");
+
+	MaxBackends = max_backends;
+	MaxBackendsInitialized = true;
 }
 
 /*
@@ -585,7 +614,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 
 	SharedInvalBackendInit(false);
 
-	if (MyBackendId > MaxBackends || MyBackendId <= 0)
+	if (MyBackendId > GetMaxBackends(0) || MyBackendId <= 0)
 		elog(FATAL, "bad backend ID: %d", MyBackendId);
 
 	/* Now that we have a BackendId, we can participate in ProcSignal */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 2e2e9a364a..28e5cecae0 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -172,7 +172,6 @@ extern PGDLLIMPORT char *DataDir;
 extern PGDLLIMPORT int data_directory_mode;
 
 extern PGDLLIMPORT int NBuffers;
-extern PGDLLIMPORT int MaxBackends;
 extern PGDLLIMPORT int MaxConnections;
 extern PGDLLIMPORT int max_worker_processes;
 extern PGDLLIMPORT int max_parallel_workers;
@@ -452,9 +451,20 @@ extern AuxProcType MyAuxProcType;
  *			POSTGRES initialization and cleanup definitions.                 *
  *****************************************************************************/
 
+/*
+ * Option flag bits for GetMaxBackends().
+ */
+typedef enum GMBOption
+{
+	GMB_MAX_PREPARED_XACTS = 1 << 0,	/* include max_prepared_xacts */
+	GMB_NUM_AUXILIARY_PROCS = 1 << 1	/* include NUM_AUXILIARY_PROCS */
+} GMBOption;
+
 /* in utils/init/postinit.c */
 extern void pg_split_opts(char **argv, int *argcp, const char *optstr);
 extern void InitializeMaxBackends(void);
+extern int GetMaxBackends(uint32 flags);
+extern void SetMaxBackends(int max_backends);
 extern void InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 						 Oid useroid, char *out_dbname, bool override_allow_connections);
 extern void BaseInit(void);
-- 
2.16.6



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-09 20:13  Greg Sabino Mullane <[email protected]>
  parent: Bossart, Nathan <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Greg Sabino Mullane @ 2021-08-09 20:13 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On Sat, Aug 7, 2021 at 2:01 PM Bossart, Nathan <[email protected]> wrote:

> Here is a rebased version of the patch.
>

Giving this a review.

Patch applies cleanly and `make check` works as of
e12694523e7e4482a052236f12d3d8b58be9a22c

Overall looks very nice and tucks MaxBackends safely away.
I have a few suggestions:

> + size = add_size(size, mul_size(GetMaxBackends(0),
sizeof(BTOneVacInfo)));

The use of GetMaxBackends(0) looks weird - can we add another constant in
there
for the "default" case? Or just have GetMaxBackends() work?

> + GMB_MAX_PREPARED_XACTS = 1 << 0, /* include max_prepared_xacts */
s/include/add in/

>  +typedef enum GMBOption
> +{
> + GMB_MAX_PREPARED_XACTS = 1 << 0, /* include max_prepared_xacts */
> + GMB_NUM_AUXILIARY_PROCS = 1 << 1 /* include NUM_AUXILIARY_PROCS */
> +} GMBOption;

Is a typedef enum really needed? As opposed to something like this style:

#define WL_LATCH_SET         (1 << 0)
#define WL_SOCKET_READABLE   (1 << 1)
#define WL_SOCKET_WRITEABLE  (1 << 2)

> -  (MaxBackends + max_prepared_xacts + 1));
> +  (GetMaxBackends(GMB_MAX_PREPARED_XACTS) + 1));

This is a little confusing - there is no indication to the reader that this
is an additive function. Perhaps a little more intuitive name:

> +  (GetMaxBackends(GMB_PLUS_MAX_PREPARED_XACTS) + 1));

However, the more I went through this patch, the more the GetMaxBackends(0)
nagged at me. The vast majority of the calls are with "0". I'd argue for
just having no arguments at all, which removes a bit of code and actually
makes things like this easier to read:

Original change:
> - uint32  TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS +
max_prepared_xacts;
> + uint32  TotalProcs = GetMaxBackends(GMB_NUM_AUXILIARY_PROCS |
GMB_MAX_PREPARED_XACTS);

Versus:
> - uint32  TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS +
max_prepared_xacts;
> + uint32  TotalProcs = GetMaxBackends() + NUM_AUXILIARY_PROCS +
max_prepared_xacts;

> + * This must be called after modules have had the change to alter GUCs in
> + * shared_preload_libraries, and before shared memory size is determined.
s/change/chance/;

> +void
> +SetMaxBackends(int max_backends)
> +{
> + if (MaxBackendsInitialized)
> +  elog(ERROR, "MaxBackends already initialized");

Is this going to get tripped by a call from restore_backend_variables?

Cheers,
Greg

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-10 00:22  Bossart, Nathan <[email protected]>
  parent: Greg Sabino Mullane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-10 00:22 UTC (permalink / raw)
  To: Greg Sabino Mullane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On 8/9/21, 1:14 PM, "Greg Sabino Mullane" <[email protected]> wrote:
> Giving this a review.

Thanks for your review.

> However, the more I went through this patch, the more the GetMaxBackends(0) nagged at me. The vast majority of the calls are with "0". I'd argue for just having no arguments at all, which removes a bit of code and actually makes things like this easier to read:

I agree.  The argument is nonzero in less than half of the calls to
GetMaxBackends(), and I'm not sure it adds a whole lot of value in the
first place.  I removed the argument in v3.

>> + * This must be called after modules have had the change to alter GUCs in
>> + * shared_preload_libraries, and before shared memory size is determined.
> s/change/chance/;

I fixed this in v3.

>> +void
>> +SetMaxBackends(int max_backends)
>> +{
>> + if (MaxBackendsInitialized)
>> +  elog(ERROR, "MaxBackends already initialized");
>
> Is this going to get tripped by a call from restore_backend_variables?

I ran 'make check-world' with EXEC_BACKEND with no problems, so I
don't think so.

Nathan



Attachments:

  [application/octet-stream] v3-0001-Disallow-external-access-to-MaxBackends.patch (29.6K, ../../[email protected]/2-v3-0001-Disallow-external-access-to-MaxBackends.patch)
  download | inline diff:
From d76d6b308f12cdd3e09d5b1d43a734ef4805b06e Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Tue, 10 Aug 2021 00:05:00 +0000
Subject: [PATCH v3 1/1] Disallow external access to MaxBackends.

Presently, MaxBackends is externally visible, but it may still be
uninitialized in places where it would be convenient to use (e.g.,
_PG_init()).  This change makes MaxBackends static to postinit.c to
disallow such direct access.  Instead, MaxBackends should now be
accessed via GetMaxBackends().

Separately, adjust the comments about needing to register
background workers before initializing MaxBackends.  Since
6bc8ef0b, InitializeMaxBackends() has used max_worker_processes
instead of tallying up the number of registered background workers,
so background worker registration is no longer a prerequisite.  The
ordering of this logic is still useful for allowing libraries to
adjust GUCs, so the comments have been updated to mention that use-
case.
---
 src/backend/access/nbtree/nbtutils.c        |  4 +--
 src/backend/access/transam/multixact.c      |  2 +-
 src/backend/access/transam/twophase.c       |  2 +-
 src/backend/commands/async.c                | 10 ++++----
 src/backend/libpq/pqcomm.c                  |  2 +-
 src/backend/postmaster/auxprocess.c         |  2 +-
 src/backend/postmaster/postmaster.c         | 14 +++++------
 src/backend/storage/ipc/dsm.c               |  2 +-
 src/backend/storage/ipc/procarray.c         |  2 +-
 src/backend/storage/ipc/procsignal.c        |  2 +-
 src/backend/storage/ipc/sinvaladt.c         |  4 +--
 src/backend/storage/lmgr/deadlock.c         | 31 ++++++++++++-----------
 src/backend/storage/lmgr/lock.c             | 20 +++++++--------
 src/backend/storage/lmgr/predicate.c        | 10 ++++----
 src/backend/storage/lmgr/proc.c             | 16 ++++++------
 src/backend/utils/activity/backend_status.c | 10 ++++----
 src/backend/utils/adt/lockfuncs.c           |  4 +--
 src/backend/utils/init/postinit.c           | 39 +++++++++++++++++++++--------
 src/include/miscadmin.h                     |  3 ++-
 19 files changed, 99 insertions(+), 80 deletions(-)

diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index d524310723..427a3b57dd 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -2065,7 +2065,7 @@ BTreeShmemSize(void)
 	Size		size;
 
 	size = offsetof(BTVacInfo, vacuums);
-	size = add_size(size, mul_size(MaxBackends, sizeof(BTOneVacInfo)));
+	size = add_size(size, mul_size(GetMaxBackends(), sizeof(BTOneVacInfo)));
 	return size;
 }
 
@@ -2094,7 +2094,7 @@ BTreeShmemInit(void)
 		btvacinfo->cycle_ctr = (BTCycleId) time(NULL);
 
 		btvacinfo->num_vacuums = 0;
-		btvacinfo->max_vacuums = MaxBackends;
+		btvacinfo->max_vacuums = GetMaxBackends();
 	}
 	else
 		Assert(found);
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index e6c70ed0bc..2eed5fbd01 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -285,7 +285,7 @@ typedef struct MultiXactStateData
  * Last element of OldestMemberMXactId and OldestVisibleMXactId arrays.
  * Valid elements are (1..MaxOldestSlot); element 0 is never used.
  */
-#define MaxOldestSlot	(MaxBackends + max_prepared_xacts)
+#define MaxOldestSlot	(GetMaxBackends() + max_prepared_xacts)
 
 /* Pointers to the state data in shared memory */
 static MultiXactStateData *MultiXactState;
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 6d3efb49a4..10ce1538a6 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -293,7 +293,7 @@ TwoPhaseShmemInit(void)
 			 * prepared transaction. Currently multixact.c uses that
 			 * technique.
 			 */
-			gxacts[i].dummyBackendId = MaxBackends + 1 + i;
+			gxacts[i].dummyBackendId = GetMaxBackends() + 1 + i;
 		}
 	}
 	else
diff --git a/src/backend/commands/async.c b/src/backend/commands/async.c
index 4b16fb5682..b2d5548c8d 100644
--- a/src/backend/commands/async.c
+++ b/src/backend/commands/async.c
@@ -511,7 +511,7 @@ AsyncShmemSize(void)
 	Size		size;
 
 	/* This had better match AsyncShmemInit */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends() + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	size = add_size(size, SimpleLruShmemSize(NUM_NOTIFY_BUFFERS, 0));
@@ -534,7 +534,7 @@ AsyncShmemInit(void)
 	 * The used entries in the backend[] array run from 1 to MaxBackends; the
 	 * zero'th entry is unused but must be allocated.
 	 */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends() + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	asyncQueueControl = (AsyncQueueControl *)
@@ -549,7 +549,7 @@ AsyncShmemInit(void)
 		QUEUE_FIRST_LISTENER = InvalidBackendId;
 		asyncQueueControl->lastQueueFillWarn = 0;
 		/* zero'th entry won't be used, but let's initialize it anyway */
-		for (int i = 0; i <= MaxBackends; i++)
+		for (int i = 0; i <= GetMaxBackends(); i++)
 		{
 			QUEUE_BACKEND_PID(i) = InvalidPid;
 			QUEUE_BACKEND_DBOID(i) = InvalidOid;
@@ -1685,8 +1685,8 @@ SignalBackends(void)
 	 * preallocate the arrays?	But in practice this is only run in trivial
 	 * transactions, so there should surely be space available.
 	 */
-	pids = (int32 *) palloc(MaxBackends * sizeof(int32));
-	ids = (BackendId *) palloc(MaxBackends * sizeof(BackendId));
+	pids = (int32 *) palloc(GetMaxBackends() * sizeof(int32));
+	ids = (BackendId *) palloc(GetMaxBackends() * sizeof(BackendId));
 	count = 0;
 
 	LWLockAcquire(NotifyQueueLock, LW_EXCLUSIVE);
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 89a5f901aa..00e1d9c0b3 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -556,7 +556,7 @@ StreamServerPort(int family, const char *hostName, unsigned short portNumber,
 		 * intended to provide a clamp on the request on platforms where an
 		 * overly large request provokes a kernel error (are there any?).
 		 */
-		maxconn = MaxBackends * 2;
+		maxconn = GetMaxBackends() * 2;
 		if (maxconn > PG_SOMAXCONN)
 			maxconn = PG_SOMAXCONN;
 
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 7452f908b2..201560181f 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -116,7 +116,7 @@ AuxiliaryProcessMain(AuxProcType auxtype)
 	 * This will need rethinking if we ever want more than one of a particular
 	 * auxiliary process type.
 	 */
-	ProcSignalInit(MaxBackends + MyAuxProcType + 1);
+	ProcSignalInit(GetMaxBackends() + MyAuxProcType + 1);
 
 	/*
 	 * Auxiliary processes don't run transactions, but they may need a
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index fc0bc8d99e..f7688a86a1 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -997,10 +997,8 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher.  It's probably a good idea to call it before
+	 * any modules had a chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1021,8 +1019,8 @@ PostmasterMain(int argc, char *argv[])
 #endif
 
 	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
+	 * Now that loadable modules have had their chance to alter any GUCs,
+	 * calculate MaxBackends.
 	 */
 	InitializeMaxBackends();
 
@@ -6187,7 +6185,7 @@ save_backend_variables(BackendParameters *param, Port *port,
 	param->query_id_enabled = query_id_enabled;
 	param->max_safe_fds = max_safe_fds;
 
-	param->MaxBackends = MaxBackends;
+	param->MaxBackends = GetMaxBackends();
 
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
@@ -6421,7 +6419,7 @@ restore_backend_variables(BackendParameters *param, Port *port)
 	query_id_enabled = param->query_id_enabled;
 	max_safe_fds = param->max_safe_fds;
 
-	MaxBackends = param->MaxBackends;
+	SetMaxBackends(param->MaxBackends);
 
 #ifdef WIN32
 	PostmasterHandle = param->PostmasterHandle;
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index b461a5f7e9..84ad54a82a 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -165,7 +165,7 @@ dsm_postmaster_startup(PGShmemHeader *shim)
 
 	/* Determine size for new control segment. */
 	maxitems = PG_DYNSHMEM_FIXED_SLOTS
-		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * MaxBackends;
+		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * GetMaxBackends();
 	elog(DEBUG2, "dynamic shared memory system will support %u segments",
 		 maxitems);
 	segsize = dsm_control_bytes_needed(maxitems);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index c7816fcfb3..8506f12372 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -365,7 +365,7 @@ ProcArrayShmemSize(void)
 	Size		size;
 
 	/* Size of the ProcArray structure itself */
-#define PROCARRAY_MAXPROCS	(MaxBackends + max_prepared_xacts)
+#define PROCARRAY_MAXPROCS	(GetMaxBackends() + max_prepared_xacts)
 
 	size = offsetof(ProcArrayStruct, pgprocnos);
 	size = add_size(size, mul_size(sizeof(int), PROCARRAY_MAXPROCS));
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index defb75aa26..b4fd8010ea 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -85,7 +85,7 @@ typedef struct
  * possible auxiliary process type.  (This scheme assumes there is not
  * more than one of any auxiliary process type at a time.)
  */
-#define NumProcSignalSlots	(MaxBackends + NUM_AUXPROCTYPES)
+#define NumProcSignalSlots	(GetMaxBackends() + NUM_AUXPROCTYPES)
 
 /* Check whether the relevant type bit is set in the flags. */
 #define BARRIER_SHOULD_CHECK(flags, type) \
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index 946bd8e3cb..2a3bd0216d 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -205,7 +205,7 @@ SInvalShmemSize(void)
 	Size		size;
 
 	size = offsetof(SISeg, procState);
-	size = add_size(size, mul_size(sizeof(ProcState), MaxBackends));
+	size = add_size(size, mul_size(sizeof(ProcState), GetMaxBackends()));
 
 	return size;
 }
@@ -231,7 +231,7 @@ CreateSharedInvalidationState(void)
 	shmInvalBuffer->maxMsgNum = 0;
 	shmInvalBuffer->nextThreshold = CLEANUP_MIN;
 	shmInvalBuffer->lastBackend = 0;
-	shmInvalBuffer->maxBackends = MaxBackends;
+	shmInvalBuffer->maxBackends = GetMaxBackends();
 	SpinLockInit(&shmInvalBuffer->msgnumLock);
 
 	/* The buffer[] array is initially all unused, so we need not fill it */
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index 67733c0d1a..6c540b0c0d 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -143,6 +143,7 @@ void
 InitDeadLockChecking(void)
 {
 	MemoryContext oldcxt;
+	int max_backends = GetMaxBackends();
 
 	/* Make sure allocations are permanent */
 	oldcxt = MemoryContextSwitchTo(TopMemoryContext);
@@ -151,16 +152,16 @@ InitDeadLockChecking(void)
 	 * FindLockCycle needs at most MaxBackends entries in visitedProcs[] and
 	 * deadlockDetails[].
 	 */
-	visitedProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
-	deadlockDetails = (DEADLOCK_INFO *) palloc(MaxBackends * sizeof(DEADLOCK_INFO));
+	visitedProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
+	deadlockDetails = (DEADLOCK_INFO *) palloc(max_backends * sizeof(DEADLOCK_INFO));
 
 	/*
 	 * TopoSort needs to consider at most MaxBackends wait-queue entries, and
 	 * it needn't run concurrently with FindLockCycle.
 	 */
 	topoProcs = visitedProcs;	/* re-use this space */
-	beforeConstraints = (int *) palloc(MaxBackends * sizeof(int));
-	afterConstraints = (int *) palloc(MaxBackends * sizeof(int));
+	beforeConstraints = (int *) palloc(max_backends * sizeof(int));
+	afterConstraints = (int *) palloc(max_backends * sizeof(int));
 
 	/*
 	 * We need to consider rearranging at most MaxBackends/2 wait queues
@@ -169,8 +170,8 @@ InitDeadLockChecking(void)
 	 * MaxBackends total waiters.
 	 */
 	waitOrders = (WAIT_ORDER *)
-		palloc((MaxBackends / 2) * sizeof(WAIT_ORDER));
-	waitOrderProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
+		palloc((max_backends / 2) * sizeof(WAIT_ORDER));
+	waitOrderProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
 
 	/*
 	 * Allow at most MaxBackends distinct constraints in a configuration. (Is
@@ -180,7 +181,7 @@ InitDeadLockChecking(void)
 	 * limits the maximum recursion depth of DeadLockCheckRecurse. Making it
 	 * really big might potentially allow a stack-overflow problem.
 	 */
-	maxCurConstraints = MaxBackends;
+	maxCurConstraints = max_backends;
 	curConstraints = (EDGE *) palloc(maxCurConstraints * sizeof(EDGE));
 
 	/*
@@ -191,7 +192,7 @@ InitDeadLockChecking(void)
 	 * last MaxBackends entries in possibleConstraints[] are reserved as
 	 * output workspace for FindLockCycle.
 	 */
-	maxPossibleConstraints = MaxBackends * 4;
+	maxPossibleConstraints = max_backends * 4;
 	possibleConstraints =
 		(EDGE *) palloc(maxPossibleConstraints * sizeof(EDGE));
 
@@ -327,7 +328,7 @@ DeadLockCheckRecurse(PGPROC *proc)
 	if (nCurConstraints >= maxCurConstraints)
 		return true;			/* out of room for active constraints? */
 	oldPossibleConstraints = nPossibleConstraints;
-	if (nPossibleConstraints + nEdges + MaxBackends <= maxPossibleConstraints)
+	if (nPossibleConstraints + nEdges + GetMaxBackends() <= maxPossibleConstraints)
 	{
 		/* We can save the edge list in possibleConstraints[] */
 		nPossibleConstraints += nEdges;
@@ -388,7 +389,7 @@ TestConfiguration(PGPROC *startProc)
 	/*
 	 * Make sure we have room for FindLockCycle's output.
 	 */
-	if (nPossibleConstraints + MaxBackends > maxPossibleConstraints)
+	if (nPossibleConstraints + GetMaxBackends() > maxPossibleConstraints)
 		return -1;
 
 	/*
@@ -486,7 +487,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 				 * record total length of cycle --- outer levels will now fill
 				 * deadlockDetails[]
 				 */
-				Assert(depth <= MaxBackends);
+				Assert(depth <= GetMaxBackends());
 				nDeadlockDetails = depth;
 
 				return true;
@@ -500,7 +501,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 		}
 	}
 	/* Mark proc as seen */
-	Assert(nVisitedProcs < MaxBackends);
+	Assert(nVisitedProcs < GetMaxBackends());
 	visitedProcs[nVisitedProcs++] = checkProc;
 
 	/*
@@ -698,7 +699,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends());
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -771,7 +772,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends());
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -834,7 +835,7 @@ ExpandConstraints(EDGE *constraints,
 		waitOrders[nWaitOrders].procs = waitOrderProcs + nWaitOrderProcs;
 		waitOrders[nWaitOrders].nProcs = lock->waitProcs.size;
 		nWaitOrderProcs += lock->waitProcs.size;
-		Assert(nWaitOrderProcs <= MaxBackends);
+		Assert(nWaitOrderProcs <= GetMaxBackends());
 
 		/*
 		 * Do the topo sort.  TopoSort need not examine constraints after this
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..b1c78c1401 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -55,7 +55,7 @@
 int			max_locks_per_xact; /* set by guc.c */
 
 #define NLOCKENTS() \
-	mul_size(max_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_locks_per_xact, add_size(GetMaxBackends(), max_prepared_xacts))
 
 
 /*
@@ -2938,12 +2938,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 			vxids = (VirtualTransactionId *)
 				MemoryContextAlloc(TopMemoryContext,
 								   sizeof(VirtualTransactionId) *
-								   (MaxBackends + max_prepared_xacts + 1));
+								   (GetMaxBackends() + max_prepared_xacts + 1));
 	}
 	else
 		vxids = (VirtualTransactionId *)
 			palloc0(sizeof(VirtualTransactionId) *
-					(MaxBackends + max_prepared_xacts + 1));
+					(GetMaxBackends() + max_prepared_xacts + 1));
 
 	/* Compute hash code and partition lock, and look up conflicting modes. */
 	hashcode = LockTagHashCode(locktag);
@@ -3100,7 +3100,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 
 	LWLockRelease(partitionLock);
 
-	if (count > MaxBackends + max_prepared_xacts)	/* should never happen */
+	if (count > GetMaxBackends() + max_prepared_xacts)	/* should never happen */
 		elog(PANIC, "too many conflicting locks found");
 
 	vxids[count].backendId = InvalidBackendId;
@@ -3651,7 +3651,7 @@ GetLockStatusData(void)
 	data = (LockData *) palloc(sizeof(LockData));
 
 	/* Guess how much space we'll need. */
-	els = MaxBackends;
+	els = GetMaxBackends();
 	el = 0;
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * els);
 
@@ -3685,7 +3685,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends();
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3717,7 +3717,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends();
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3846,7 +3846,7 @@ GetBlockerStatusData(int blocked_pid)
 	 * for the procs[] array; the other two could need enlargement, though.)
 	 */
 	data->nprocs = data->nlocks = data->npids = 0;
-	data->maxprocs = data->maxlocks = data->maxpids = MaxBackends;
+	data->maxprocs = data->maxlocks = data->maxpids = GetMaxBackends();
 	data->procs = (BlockedProcData *) palloc(sizeof(BlockedProcData) * data->maxprocs);
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * data->maxlocks);
 	data->waiter_pids = (int *) palloc(sizeof(int) * data->maxpids);
@@ -3949,7 +3949,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 		if (data->nlocks >= data->maxlocks)
 		{
-			data->maxlocks += MaxBackends;
+			data->maxlocks += GetMaxBackends();
 			data->locks = (LockInstanceData *)
 				repalloc(data->locks, sizeof(LockInstanceData) * data->maxlocks);
 		}
@@ -3978,7 +3978,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 	if (queue_size > data->maxpids - data->npids)
 	{
-		data->maxpids = Max(data->maxpids + MaxBackends,
+		data->maxpids = Max(data->maxpids + GetMaxBackends(),
 							data->npids + queue_size);
 		data->waiter_pids = (int *) repalloc(data->waiter_pids,
 											 sizeof(int) * data->maxpids);
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 56267bdc3c..3ffb74e1f7 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -257,7 +257,7 @@
 	(&MainLWLockArray[PREDICATELOCK_MANAGER_LWLOCK_OFFSET + (i)].lock)
 
 #define NPREDICATELOCKTARGETENTS() \
-	mul_size(max_predicate_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_predicate_locks_per_xact, add_size(GetMaxBackends(), max_prepared_xacts))
 
 #define SxactIsOnFinishedList(sxact) (!SHMQueueIsDetached(&((sxact)->finishedLink)))
 
@@ -1222,7 +1222,7 @@ InitPredicateLocks(void)
 	 * Compute size for serializable transaction hashtable. Note these
 	 * calculations must agree with PredicateLockShmemSize!
 	 */
-	max_table_size = (MaxBackends + max_prepared_xacts);
+	max_table_size = (GetMaxBackends() + max_prepared_xacts);
 
 	/*
 	 * Allocate a list to hold information on transactions participating in
@@ -1374,7 +1374,7 @@ PredicateLockShmemSize(void)
 	size = add_size(size, size / 10);
 
 	/* transaction list */
-	max_table_size = MaxBackends + max_prepared_xacts;
+	max_table_size = GetMaxBackends() + max_prepared_xacts;
 	max_table_size *= 10;
 	size = add_size(size, PredXactListDataSize);
 	size = add_size(size, mul_size((Size) max_table_size,
@@ -1905,7 +1905,7 @@ GetSerializableTransactionSnapshotInt(Snapshot snapshot,
 	{
 		++(PredXact->WritableSxactCount);
 		Assert(PredXact->WritableSxactCount <=
-			   (MaxBackends + max_prepared_xacts));
+			   (GetMaxBackends() + max_prepared_xacts));
 	}
 
 	MySerializableXact = sxact;
@@ -5107,7 +5107,7 @@ predicatelock_twophase_recover(TransactionId xid, uint16 info,
 		{
 			++(PredXact->WritableSxactCount);
 			Assert(PredXact->WritableSxactCount <=
-				   (MaxBackends + max_prepared_xacts));
+				   (GetMaxBackends() + max_prepared_xacts));
 		}
 
 		/*
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index b7d9da0aa9..9046ccec2d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -103,7 +103,7 @@ ProcGlobalShmemSize(void)
 {
 	Size		size = 0;
 	Size		TotalProcs =
-	add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+	add_size(GetMaxBackends(), add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
 
 	/* ProcGlobal */
 	size = add_size(size, sizeof(PROC_HDR));
@@ -127,7 +127,7 @@ ProcGlobalSemas(void)
 	 * We need a sema per backend (including autovacuum), plus one for each
 	 * auxiliary process.
 	 */
-	return MaxBackends + NUM_AUXILIARY_PROCS;
+	return GetMaxBackends() + NUM_AUXILIARY_PROCS;
 }
 
 /*
@@ -162,7 +162,7 @@ InitProcGlobal(void)
 	int			i,
 				j;
 	bool		found;
-	uint32		TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
+	uint32		TotalProcs = GetMaxBackends() + NUM_AUXILIARY_PROCS + max_prepared_xacts;
 
 	/* Create the ProcGlobal shared structure */
 	ProcGlobal = (PROC_HDR *)
@@ -197,7 +197,7 @@ InitProcGlobal(void)
 	MemSet(procs, 0, TotalProcs * sizeof(PGPROC));
 	ProcGlobal->allProcs = procs;
 	/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
-	ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+	ProcGlobal->allProcCount = GetMaxBackends() + NUM_AUXILIARY_PROCS;
 
 	/*
 	 * Allocate arrays mirroring PGPROC fields in a dense manner. See
@@ -223,7 +223,7 @@ InitProcGlobal(void)
 		 * dummy PGPROCs don't need these though - they're never associated
 		 * with a real process
 		 */
-		if (i < MaxBackends + NUM_AUXILIARY_PROCS)
+		if (i < GetMaxBackends() + NUM_AUXILIARY_PROCS)
 		{
 			procs[i].sem = PGSemaphoreCreate();
 			InitSharedLatch(&(procs[i].procLatch));
@@ -260,7 +260,7 @@ InitProcGlobal(void)
 			ProcGlobal->bgworkerFreeProcs = &procs[i];
 			procs[i].procgloballist = &ProcGlobal->bgworkerFreeProcs;
 		}
-		else if (i < MaxBackends)
+		else if (i < GetMaxBackends())
 		{
 			/* PGPROC for walsender, add to walsenderFreeProcs list */
 			procs[i].links.next = (SHM_QUEUE *) ProcGlobal->walsenderFreeProcs;
@@ -288,8 +288,8 @@ InitProcGlobal(void)
 	 * Save pointers to the blocks of PGPROC structures reserved for auxiliary
 	 * processes and prepared transactions.
 	 */
-	AuxiliaryProcs = &procs[MaxBackends];
-	PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+	AuxiliaryProcs = &procs[GetMaxBackends()];
+	PreparedXactProcs = &procs[GetMaxBackends() + NUM_AUXILIARY_PROCS];
 
 	/* Create ProcStructLock spinlock, too */
 	ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 2901f9f5a9..7a1b4eebd7 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -35,7 +35,7 @@
  * includes autovacuum workers and background workers as well.
  * ----------
  */
-#define NumBackendStatSlots (MaxBackends + NUM_AUXPROCTYPES)
+#define NumBackendStatSlots (GetMaxBackends() + NUM_AUXPROCTYPES)
 
 
 /* ----------
@@ -251,7 +251,7 @@ pgstat_beinit(void)
 	/* Initialize MyBEEntry */
 	if (MyBackendId != InvalidBackendId)
 	{
-		Assert(MyBackendId >= 1 && MyBackendId <= MaxBackends);
+		Assert(MyBackendId >= 1 && MyBackendId <= GetMaxBackends());
 		MyBEEntry = &BackendStatusArray[MyBackendId - 1];
 	}
 	else
@@ -267,7 +267,7 @@ pgstat_beinit(void)
 		 * MaxBackends + AuxBackendType + 1 as the index of the slot for an
 		 * auxiliary process.
 		 */
-		MyBEEntry = &BackendStatusArray[MaxBackends + MyAuxProcType];
+		MyBEEntry = &BackendStatusArray[GetMaxBackends() + MyAuxProcType];
 	}
 
 	/* Set up a process-exit hook to clean up */
@@ -892,7 +892,7 @@ pgstat_get_backend_current_activity(int pid, bool checkUser)
 	int			i;
 
 	beentry = BackendStatusArray;
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= GetMaxBackends(); i++)
 	{
 		/*
 		 * Although we expect the target backend's entry to be stable, that
@@ -978,7 +978,7 @@ pgstat_get_crashed_backend_activity(int pid, char *buffer, int buflen)
 	if (beentry == NULL || BackendActivityBuffer == NULL)
 		return NULL;
 
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= GetMaxBackends(); i++)
 	{
 		if (beentry->st_procpid == pid)
 		{
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 5dc0a5882c..f905824215 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -561,11 +561,11 @@ pg_safe_snapshot_blocking_pids(PG_FUNCTION_ARGS)
 	Datum	   *blocker_datums;
 
 	/* A buffer big enough for any possible blocker list without truncation */
-	blockers = (int *) palloc(MaxBackends * sizeof(int));
+	blockers = (int *) palloc(GetMaxBackends() * sizeof(int));
 
 	/* Collect a snapshot of processes waited for by GetSafeSnapshot */
 	num_blockers =
-		GetSafeSnapshotBlockingPids(blocked_pid, blockers, MaxBackends);
+		GetSafeSnapshotBlockingPids(blocked_pid, blockers, GetMaxBackends());
 
 	/* Convert int array to Datum array */
 	if (num_blockers > 0)
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 5089dd43ae..ba413be4f8 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -25,6 +25,7 @@
 #include "access/session.h"
 #include "access/sysattr.h"
 #include "access/tableam.h"
+#include "access/twophase.h"
 #include "access/xact.h"
 #include "access/xlog.h"
 #include "catalog/catalog.h"
@@ -63,6 +64,9 @@
 #include "utils/syscache.h"
 #include "utils/timeout.h"
 
+static int MaxBackends = 0;
+static int MaxBackendsInitialized = false;
+
 static HeapTuple GetDatabaseTuple(const char *dbname);
 static HeapTuple GetDatabaseTupleByOid(Oid dboid);
 static void PerformAuthentication(Port *port);
@@ -476,9 +480,8 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called after modules have had the chance to alter GUCs in
+ * shared_preload_libraries, and before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
@@ -488,15 +491,31 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 void
 InitializeMaxBackends(void)
 {
-	Assert(MaxBackends == 0);
-
 	/* the extra unit accounts for the autovacuum launcher */
-	MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
-		max_worker_processes + max_wal_senders;
+	SetMaxBackends(MaxConnections + autovacuum_max_workers + 1 +
+		max_worker_processes + max_wal_senders);
+}
 
-	/* internal error because the values were all checked previously */
-	if (MaxBackends > MAX_BACKENDS)
+int
+GetMaxBackends(void)
+{
+	if (!MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends not yet initialized");
+
+	return MaxBackends;
+}
+
+void
+SetMaxBackends(int max_backends)
+{
+	if (MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends already initialized");
+
+	if (max_backends > MAX_BACKENDS)
 		elog(ERROR, "too many backends configured");
+
+	MaxBackends = max_backends;
+	MaxBackendsInitialized = true;
 }
 
 /*
@@ -596,7 +615,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 
 	SharedInvalBackendInit(false);
 
-	if (MyBackendId > MaxBackends || MyBackendId <= 0)
+	if (MyBackendId > GetMaxBackends() || MyBackendId <= 0)
 		elog(FATAL, "bad backend ID: %d", MyBackendId);
 
 	/* Now that we have a BackendId, we can participate in ProcSignal */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 2e2e9a364a..fd0977ab05 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -172,7 +172,6 @@ extern PGDLLIMPORT char *DataDir;
 extern PGDLLIMPORT int data_directory_mode;
 
 extern PGDLLIMPORT int NBuffers;
-extern PGDLLIMPORT int MaxBackends;
 extern PGDLLIMPORT int MaxConnections;
 extern PGDLLIMPORT int max_worker_processes;
 extern PGDLLIMPORT int max_parallel_workers;
@@ -455,6 +454,8 @@ extern AuxProcType MyAuxProcType;
 /* in utils/init/postinit.c */
 extern void pg_split_opts(char **argv, int *argcp, const char *optstr);
 extern void InitializeMaxBackends(void);
+extern int GetMaxBackends(void);
+extern void SetMaxBackends(int max_backends);
 extern void InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 						 Oid useroid, char *out_dbname, bool override_allow_connections);
 extern void BaseInit(void);
-- 
2.16.6



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-11 13:41  Greg Sabino Mullane <[email protected]>
  parent: Bossart, Nathan <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Greg Sabino Mullane @ 2021-08-11 13:41 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On Mon, Aug 9, 2021 at 8:22 PM Bossart, Nathan <[email protected]> wrote:

> > Is this going to get tripped by a call from restore_backend_variables?
>
> I ran 'make check-world' with EXEC_BACKEND with no problems, so I
> don't think so.
>

v3 looks good, but I'm still not sure how to test the bit mentioned above.
I'm not familiar with this part of the code (SubPostmasterMain etc.), but
running make check-world with EXEC_BACKEND does not seem to execute that
code, as I added exit(1) to restore_backend_variables() and the tests still
ran fine. Further digging shows that even though the #ifdef EXEC_BACKEND
path is triggered, no --fork argument was being passed. Is there something
else one needs to provide to force that --fork (see line 189 of
src/backend/main/main.c) when testing?

Cheers,
Greg

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-11 14:08  Tom Lane <[email protected]>
  parent: Greg Sabino Mullane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2021-08-11 14:08 UTC (permalink / raw)
  To: Greg Sabino Mullane <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

Greg Sabino Mullane <[email protected]> writes:
> v3 looks good, but I'm still not sure how to test the bit mentioned above.
> I'm not familiar with this part of the code (SubPostmasterMain etc.), but
> running make check-world with EXEC_BACKEND does not seem to execute that
> code, as I added exit(1) to restore_backend_variables() and the tests still
> ran fine.

You must not have enabled EXEC_BACKEND properly.  It's a compile-time
#define that affects multiple modules, so it's easy to get wrong.
The way I usually turn it on is

	make distclean
	./configure ... options of choice ...
	edit src/include/pg_config.h, add "#define EXEC_BACKEND" line
	make, install, test

In this way the setting is persistent till the next distclean/configure
cycle.

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-12 15:41  Greg Sabino Mullane <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Greg Sabino Mullane @ 2021-08-12 15:41 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Bossart, Nathan <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; [email protected] <[email protected]>

On Wed, Aug 11, 2021 at 10:08 AM Tom Lane <[email protected]> wrote:

> You must not have enabled EXEC_BACKEND properly.  It's a compile-time
> #define that affects multiple modules, so it's easy to get wrong.
> The way I usually turn it on is
>

Thank you. I was able to get it all working, and withdraw any objections to
that bit of the patch. :)

Cheers,
Greg


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* RE: make MaxBackends available in _PG_init
@ 2021-08-15 08:04  [email protected] <[email protected]>
  parent: Greg Sabino Mullane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: [email protected] @ 2021-08-15 08:04 UTC (permalink / raw)
  To: Bossart, Nathan <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

I noticed that in v3-0001-Disallow-external-access-to-MaxBackends.patch, there are some modifications like:

> -		for (int i = 0; i <= MaxBackends; i++)
> +		for (int i = 0; i <= GetMaxBackends(); i++)

I don't think calling function GetMaxBackends() in the for loop is a good idea.  
How about use a temp variable to save the return value of function GetMaxBackends() ?

Regards,
Shenhao Wang


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2021-08-16 04:02  Bossart, Nathan <[email protected]>
  parent: [email protected] <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Bossart, Nathan @ 2021-08-16 04:02 UTC (permalink / raw)
  To: [email protected] <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On 8/15/21, 1:05 AM, "[email protected]" <[email protected]> wrote:
> I don't think calling function GetMaxBackends() in the for loop is a good idea.
> How about use a temp variable to save the return value of function GetMaxBackends() ?

I did this in v4.  There may be a couple of remaining places that call
GetMaxBackends() several times, but the function should be relatively
inexpensive.

Nathan



Attachments:

  [application/octet-stream] v4-0001-Disallow-external-access-to-MaxBackends.patch (30.0K, ../../[email protected]/2-v4-0001-Disallow-external-access-to-MaxBackends.patch)
  download | inline diff:
From 902007fd67a9aa4f8aac767ac9bfda88070a8cdd Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Mon, 16 Aug 2021 02:59:32 +0000
Subject: [PATCH v4 1/1] Disallow external access to MaxBackends.

Presently, MaxBackends is externally visible, but it may still be
uninitialized in places where it would be convenient to use (e.g.,
_PG_init()).  This change makes MaxBackends static to postinit.c to
disallow such direct access.  Instead, MaxBackends should now be
accessed via GetMaxBackends().

Separately, adjust the comments about needing to register
background workers before initializing MaxBackends.  Since
6bc8ef0b, InitializeMaxBackends() has used max_worker_processes
instead of tallying up the number of registered background workers,
so background worker registration is no longer a prerequisite.  The
ordering of this logic is still useful for allowing libraries to
adjust GUCs, so the comments have been updated to mention that use-
case.
---
 src/backend/access/nbtree/nbtutils.c        |  4 +--
 src/backend/access/transam/multixact.c      |  2 +-
 src/backend/access/transam/twophase.c       |  2 +-
 src/backend/commands/async.c                | 11 ++++----
 src/backend/libpq/pqcomm.c                  |  2 +-
 src/backend/postmaster/auxprocess.c         |  2 +-
 src/backend/postmaster/postmaster.c         | 14 +++++------
 src/backend/storage/ipc/dsm.c               |  2 +-
 src/backend/storage/ipc/procarray.c         |  2 +-
 src/backend/storage/ipc/procsignal.c        |  2 +-
 src/backend/storage/ipc/sinvaladt.c         |  4 +--
 src/backend/storage/lmgr/deadlock.c         | 31 ++++++++++++-----------
 src/backend/storage/lmgr/lock.c             | 20 +++++++--------
 src/backend/storage/lmgr/predicate.c        | 10 ++++----
 src/backend/storage/lmgr/proc.c             | 16 ++++++------
 src/backend/utils/activity/backend_status.c | 12 +++++----
 src/backend/utils/adt/lockfuncs.c           |  4 +--
 src/backend/utils/init/postinit.c           | 39 +++++++++++++++++++++--------
 src/include/miscadmin.h                     |  3 ++-
 19 files changed, 102 insertions(+), 80 deletions(-)

diff --git a/src/backend/access/nbtree/nbtutils.c b/src/backend/access/nbtree/nbtutils.c
index d524310723..427a3b57dd 100644
--- a/src/backend/access/nbtree/nbtutils.c
+++ b/src/backend/access/nbtree/nbtutils.c
@@ -2065,7 +2065,7 @@ BTreeShmemSize(void)
 	Size		size;
 
 	size = offsetof(BTVacInfo, vacuums);
-	size = add_size(size, mul_size(MaxBackends, sizeof(BTOneVacInfo)));
+	size = add_size(size, mul_size(GetMaxBackends(), sizeof(BTOneVacInfo)));
 	return size;
 }
 
@@ -2094,7 +2094,7 @@ BTreeShmemInit(void)
 		btvacinfo->cycle_ctr = (BTCycleId) time(NULL);
 
 		btvacinfo->num_vacuums = 0;
-		btvacinfo->max_vacuums = MaxBackends;
+		btvacinfo->max_vacuums = GetMaxBackends();
 	}
 	else
 		Assert(found);
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index e6c70ed0bc..2eed5fbd01 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -285,7 +285,7 @@ typedef struct MultiXactStateData
  * Last element of OldestMemberMXactId and OldestVisibleMXactId arrays.
  * Valid elements are (1..MaxOldestSlot); element 0 is never used.
  */
-#define MaxOldestSlot	(MaxBackends + max_prepared_xacts)
+#define MaxOldestSlot	(GetMaxBackends() + max_prepared_xacts)
 
 /* Pointers to the state data in shared memory */
 static MultiXactStateData *MultiXactState;
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2156de187c..5f9ab7ee47 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -293,7 +293,7 @@ TwoPhaseShmemInit(void)
 			 * prepared transaction. Currently multixact.c uses that
 			 * technique.
 			 */
-			gxacts[i].dummyBackendId = MaxBackends + 1 + i;
+			gxacts[i].dummyBackendId = GetMaxBackends() + 1 + i;
 		}
 	}
 	else
diff --git a/src/backend/commands/async.c b/src/backend/commands/async.c
index 4b16fb5682..2d18edc648 100644
--- a/src/backend/commands/async.c
+++ b/src/backend/commands/async.c
@@ -511,7 +511,7 @@ AsyncShmemSize(void)
 	Size		size;
 
 	/* This had better match AsyncShmemInit */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(GetMaxBackends() + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	size = add_size(size, SimpleLruShmemSize(NUM_NOTIFY_BUFFERS, 0));
@@ -527,6 +527,7 @@ AsyncShmemInit(void)
 {
 	bool		found;
 	Size		size;
+	int			max_backends = GetMaxBackends();
 
 	/*
 	 * Create or attach to the AsyncQueueControl structure.
@@ -534,7 +535,7 @@ AsyncShmemInit(void)
 	 * The used entries in the backend[] array run from 1 to MaxBackends; the
 	 * zero'th entry is unused but must be allocated.
 	 */
-	size = mul_size(MaxBackends + 1, sizeof(QueueBackendStatus));
+	size = mul_size(max_backends + 1, sizeof(QueueBackendStatus));
 	size = add_size(size, offsetof(AsyncQueueControl, backend));
 
 	asyncQueueControl = (AsyncQueueControl *)
@@ -549,7 +550,7 @@ AsyncShmemInit(void)
 		QUEUE_FIRST_LISTENER = InvalidBackendId;
 		asyncQueueControl->lastQueueFillWarn = 0;
 		/* zero'th entry won't be used, but let's initialize it anyway */
-		for (int i = 0; i <= MaxBackends; i++)
+		for (int i = 0; i <= max_backends; i++)
 		{
 			QUEUE_BACKEND_PID(i) = InvalidPid;
 			QUEUE_BACKEND_DBOID(i) = InvalidOid;
@@ -1685,8 +1686,8 @@ SignalBackends(void)
 	 * preallocate the arrays?	But in practice this is only run in trivial
 	 * transactions, so there should surely be space available.
 	 */
-	pids = (int32 *) palloc(MaxBackends * sizeof(int32));
-	ids = (BackendId *) palloc(MaxBackends * sizeof(BackendId));
+	pids = (int32 *) palloc(GetMaxBackends() * sizeof(int32));
+	ids = (BackendId *) palloc(GetMaxBackends() * sizeof(BackendId));
 	count = 0;
 
 	LWLockAcquire(NotifyQueueLock, LW_EXCLUSIVE);
diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c
index 89a5f901aa..00e1d9c0b3 100644
--- a/src/backend/libpq/pqcomm.c
+++ b/src/backend/libpq/pqcomm.c
@@ -556,7 +556,7 @@ StreamServerPort(int family, const char *hostName, unsigned short portNumber,
 		 * intended to provide a clamp on the request on platforms where an
 		 * overly large request provokes a kernel error (are there any?).
 		 */
-		maxconn = MaxBackends * 2;
+		maxconn = GetMaxBackends() * 2;
 		if (maxconn > PG_SOMAXCONN)
 			maxconn = PG_SOMAXCONN;
 
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index 7452f908b2..201560181f 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -116,7 +116,7 @@ AuxiliaryProcessMain(AuxProcType auxtype)
 	 * This will need rethinking if we ever want more than one of a particular
 	 * auxiliary process type.
 	 */
-	ProcSignalInit(MaxBackends + MyAuxProcType + 1);
+	ProcSignalInit(GetMaxBackends() + MyAuxProcType + 1);
 
 	/*
 	 * Auxiliary processes don't run transactions, but they may need a
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 9c2c98614a..4328fcff83 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -997,10 +997,8 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher.  It's probably a good idea to call it before
+	 * any modules had a chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1021,8 +1019,8 @@ PostmasterMain(int argc, char *argv[])
 #endif
 
 	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
+	 * Now that loadable modules have had their chance to alter any GUCs,
+	 * calculate MaxBackends.
 	 */
 	InitializeMaxBackends();
 
@@ -6182,7 +6180,7 @@ save_backend_variables(BackendParameters *param, Port *port,
 	param->query_id_enabled = query_id_enabled;
 	param->max_safe_fds = max_safe_fds;
 
-	param->MaxBackends = MaxBackends;
+	param->MaxBackends = GetMaxBackends();
 
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
@@ -6416,7 +6414,7 @@ restore_backend_variables(BackendParameters *param, Port *port)
 	query_id_enabled = param->query_id_enabled;
 	max_safe_fds = param->max_safe_fds;
 
-	MaxBackends = param->MaxBackends;
+	SetMaxBackends(param->MaxBackends);
 
 #ifdef WIN32
 	PostmasterHandle = param->PostmasterHandle;
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index b461a5f7e9..84ad54a82a 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -165,7 +165,7 @@ dsm_postmaster_startup(PGShmemHeader *shim)
 
 	/* Determine size for new control segment. */
 	maxitems = PG_DYNSHMEM_FIXED_SLOTS
-		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * MaxBackends;
+		+ PG_DYNSHMEM_SLOTS_PER_BACKEND * GetMaxBackends();
 	elog(DEBUG2, "dynamic shared memory system will support %u segments",
 		 maxitems);
 	segsize = dsm_control_bytes_needed(maxitems);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index c7816fcfb3..8506f12372 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -365,7 +365,7 @@ ProcArrayShmemSize(void)
 	Size		size;
 
 	/* Size of the ProcArray structure itself */
-#define PROCARRAY_MAXPROCS	(MaxBackends + max_prepared_xacts)
+#define PROCARRAY_MAXPROCS	(GetMaxBackends() + max_prepared_xacts)
 
 	size = offsetof(ProcArrayStruct, pgprocnos);
 	size = add_size(size, mul_size(sizeof(int), PROCARRAY_MAXPROCS));
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index defb75aa26..b4fd8010ea 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -85,7 +85,7 @@ typedef struct
  * possible auxiliary process type.  (This scheme assumes there is not
  * more than one of any auxiliary process type at a time.)
  */
-#define NumProcSignalSlots	(MaxBackends + NUM_AUXPROCTYPES)
+#define NumProcSignalSlots	(GetMaxBackends() + NUM_AUXPROCTYPES)
 
 /* Check whether the relevant type bit is set in the flags. */
 #define BARRIER_SHOULD_CHECK(flags, type) \
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index 946bd8e3cb..2a3bd0216d 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -205,7 +205,7 @@ SInvalShmemSize(void)
 	Size		size;
 
 	size = offsetof(SISeg, procState);
-	size = add_size(size, mul_size(sizeof(ProcState), MaxBackends));
+	size = add_size(size, mul_size(sizeof(ProcState), GetMaxBackends()));
 
 	return size;
 }
@@ -231,7 +231,7 @@ CreateSharedInvalidationState(void)
 	shmInvalBuffer->maxMsgNum = 0;
 	shmInvalBuffer->nextThreshold = CLEANUP_MIN;
 	shmInvalBuffer->lastBackend = 0;
-	shmInvalBuffer->maxBackends = MaxBackends;
+	shmInvalBuffer->maxBackends = GetMaxBackends();
 	SpinLockInit(&shmInvalBuffer->msgnumLock);
 
 	/* The buffer[] array is initially all unused, so we need not fill it */
diff --git a/src/backend/storage/lmgr/deadlock.c b/src/backend/storage/lmgr/deadlock.c
index 67733c0d1a..6c540b0c0d 100644
--- a/src/backend/storage/lmgr/deadlock.c
+++ b/src/backend/storage/lmgr/deadlock.c
@@ -143,6 +143,7 @@ void
 InitDeadLockChecking(void)
 {
 	MemoryContext oldcxt;
+	int max_backends = GetMaxBackends();
 
 	/* Make sure allocations are permanent */
 	oldcxt = MemoryContextSwitchTo(TopMemoryContext);
@@ -151,16 +152,16 @@ InitDeadLockChecking(void)
 	 * FindLockCycle needs at most MaxBackends entries in visitedProcs[] and
 	 * deadlockDetails[].
 	 */
-	visitedProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
-	deadlockDetails = (DEADLOCK_INFO *) palloc(MaxBackends * sizeof(DEADLOCK_INFO));
+	visitedProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
+	deadlockDetails = (DEADLOCK_INFO *) palloc(max_backends * sizeof(DEADLOCK_INFO));
 
 	/*
 	 * TopoSort needs to consider at most MaxBackends wait-queue entries, and
 	 * it needn't run concurrently with FindLockCycle.
 	 */
 	topoProcs = visitedProcs;	/* re-use this space */
-	beforeConstraints = (int *) palloc(MaxBackends * sizeof(int));
-	afterConstraints = (int *) palloc(MaxBackends * sizeof(int));
+	beforeConstraints = (int *) palloc(max_backends * sizeof(int));
+	afterConstraints = (int *) palloc(max_backends * sizeof(int));
 
 	/*
 	 * We need to consider rearranging at most MaxBackends/2 wait queues
@@ -169,8 +170,8 @@ InitDeadLockChecking(void)
 	 * MaxBackends total waiters.
 	 */
 	waitOrders = (WAIT_ORDER *)
-		palloc((MaxBackends / 2) * sizeof(WAIT_ORDER));
-	waitOrderProcs = (PGPROC **) palloc(MaxBackends * sizeof(PGPROC *));
+		palloc((max_backends / 2) * sizeof(WAIT_ORDER));
+	waitOrderProcs = (PGPROC **) palloc(max_backends * sizeof(PGPROC *));
 
 	/*
 	 * Allow at most MaxBackends distinct constraints in a configuration. (Is
@@ -180,7 +181,7 @@ InitDeadLockChecking(void)
 	 * limits the maximum recursion depth of DeadLockCheckRecurse. Making it
 	 * really big might potentially allow a stack-overflow problem.
 	 */
-	maxCurConstraints = MaxBackends;
+	maxCurConstraints = max_backends;
 	curConstraints = (EDGE *) palloc(maxCurConstraints * sizeof(EDGE));
 
 	/*
@@ -191,7 +192,7 @@ InitDeadLockChecking(void)
 	 * last MaxBackends entries in possibleConstraints[] are reserved as
 	 * output workspace for FindLockCycle.
 	 */
-	maxPossibleConstraints = MaxBackends * 4;
+	maxPossibleConstraints = max_backends * 4;
 	possibleConstraints =
 		(EDGE *) palloc(maxPossibleConstraints * sizeof(EDGE));
 
@@ -327,7 +328,7 @@ DeadLockCheckRecurse(PGPROC *proc)
 	if (nCurConstraints >= maxCurConstraints)
 		return true;			/* out of room for active constraints? */
 	oldPossibleConstraints = nPossibleConstraints;
-	if (nPossibleConstraints + nEdges + MaxBackends <= maxPossibleConstraints)
+	if (nPossibleConstraints + nEdges + GetMaxBackends() <= maxPossibleConstraints)
 	{
 		/* We can save the edge list in possibleConstraints[] */
 		nPossibleConstraints += nEdges;
@@ -388,7 +389,7 @@ TestConfiguration(PGPROC *startProc)
 	/*
 	 * Make sure we have room for FindLockCycle's output.
 	 */
-	if (nPossibleConstraints + MaxBackends > maxPossibleConstraints)
+	if (nPossibleConstraints + GetMaxBackends() > maxPossibleConstraints)
 		return -1;
 
 	/*
@@ -486,7 +487,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 				 * record total length of cycle --- outer levels will now fill
 				 * deadlockDetails[]
 				 */
-				Assert(depth <= MaxBackends);
+				Assert(depth <= GetMaxBackends());
 				nDeadlockDetails = depth;
 
 				return true;
@@ -500,7 +501,7 @@ FindLockCycleRecurse(PGPROC *checkProc,
 		}
 	}
 	/* Mark proc as seen */
-	Assert(nVisitedProcs < MaxBackends);
+	Assert(nVisitedProcs < GetMaxBackends());
 	visitedProcs[nVisitedProcs++] = checkProc;
 
 	/*
@@ -698,7 +699,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends());
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -771,7 +772,7 @@ FindLockCycleRecurseMember(PGPROC *checkProc,
 					/*
 					 * Add this edge to the list of soft edges in the cycle
 					 */
-					Assert(*nSoftEdges < MaxBackends);
+					Assert(*nSoftEdges < GetMaxBackends());
 					softEdges[*nSoftEdges].waiter = checkProcLeader;
 					softEdges[*nSoftEdges].blocker = leader;
 					softEdges[*nSoftEdges].lock = lock;
@@ -834,7 +835,7 @@ ExpandConstraints(EDGE *constraints,
 		waitOrders[nWaitOrders].procs = waitOrderProcs + nWaitOrderProcs;
 		waitOrders[nWaitOrders].nProcs = lock->waitProcs.size;
 		nWaitOrderProcs += lock->waitProcs.size;
-		Assert(nWaitOrderProcs <= MaxBackends);
+		Assert(nWaitOrderProcs <= GetMaxBackends());
 
 		/*
 		 * Do the topo sort.  TopoSort need not examine constraints after this
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..b1c78c1401 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -55,7 +55,7 @@
 int			max_locks_per_xact; /* set by guc.c */
 
 #define NLOCKENTS() \
-	mul_size(max_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_locks_per_xact, add_size(GetMaxBackends(), max_prepared_xacts))
 
 
 /*
@@ -2938,12 +2938,12 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 			vxids = (VirtualTransactionId *)
 				MemoryContextAlloc(TopMemoryContext,
 								   sizeof(VirtualTransactionId) *
-								   (MaxBackends + max_prepared_xacts + 1));
+								   (GetMaxBackends() + max_prepared_xacts + 1));
 	}
 	else
 		vxids = (VirtualTransactionId *)
 			palloc0(sizeof(VirtualTransactionId) *
-					(MaxBackends + max_prepared_xacts + 1));
+					(GetMaxBackends() + max_prepared_xacts + 1));
 
 	/* Compute hash code and partition lock, and look up conflicting modes. */
 	hashcode = LockTagHashCode(locktag);
@@ -3100,7 +3100,7 @@ GetLockConflicts(const LOCKTAG *locktag, LOCKMODE lockmode, int *countp)
 
 	LWLockRelease(partitionLock);
 
-	if (count > MaxBackends + max_prepared_xacts)	/* should never happen */
+	if (count > GetMaxBackends() + max_prepared_xacts)	/* should never happen */
 		elog(PANIC, "too many conflicting locks found");
 
 	vxids[count].backendId = InvalidBackendId;
@@ -3651,7 +3651,7 @@ GetLockStatusData(void)
 	data = (LockData *) palloc(sizeof(LockData));
 
 	/* Guess how much space we'll need. */
-	els = MaxBackends;
+	els = GetMaxBackends();
 	el = 0;
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * els);
 
@@ -3685,7 +3685,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends();
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3717,7 +3717,7 @@ GetLockStatusData(void)
 
 			if (el >= els)
 			{
-				els += MaxBackends;
+				els += GetMaxBackends();
 				data->locks = (LockInstanceData *)
 					repalloc(data->locks, sizeof(LockInstanceData) * els);
 			}
@@ -3846,7 +3846,7 @@ GetBlockerStatusData(int blocked_pid)
 	 * for the procs[] array; the other two could need enlargement, though.)
 	 */
 	data->nprocs = data->nlocks = data->npids = 0;
-	data->maxprocs = data->maxlocks = data->maxpids = MaxBackends;
+	data->maxprocs = data->maxlocks = data->maxpids = GetMaxBackends();
 	data->procs = (BlockedProcData *) palloc(sizeof(BlockedProcData) * data->maxprocs);
 	data->locks = (LockInstanceData *) palloc(sizeof(LockInstanceData) * data->maxlocks);
 	data->waiter_pids = (int *) palloc(sizeof(int) * data->maxpids);
@@ -3949,7 +3949,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 		if (data->nlocks >= data->maxlocks)
 		{
-			data->maxlocks += MaxBackends;
+			data->maxlocks += GetMaxBackends();
 			data->locks = (LockInstanceData *)
 				repalloc(data->locks, sizeof(LockInstanceData) * data->maxlocks);
 		}
@@ -3978,7 +3978,7 @@ GetSingleProcBlockerStatusData(PGPROC *blocked_proc, BlockedProcsData *data)
 
 	if (queue_size > data->maxpids - data->npids)
 	{
-		data->maxpids = Max(data->maxpids + MaxBackends,
+		data->maxpids = Max(data->maxpids + GetMaxBackends(),
 							data->npids + queue_size);
 		data->waiter_pids = (int *) repalloc(data->waiter_pids,
 											 sizeof(int) * data->maxpids);
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 56267bdc3c..3ffb74e1f7 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -257,7 +257,7 @@
 	(&MainLWLockArray[PREDICATELOCK_MANAGER_LWLOCK_OFFSET + (i)].lock)
 
 #define NPREDICATELOCKTARGETENTS() \
-	mul_size(max_predicate_locks_per_xact, add_size(MaxBackends, max_prepared_xacts))
+	mul_size(max_predicate_locks_per_xact, add_size(GetMaxBackends(), max_prepared_xacts))
 
 #define SxactIsOnFinishedList(sxact) (!SHMQueueIsDetached(&((sxact)->finishedLink)))
 
@@ -1222,7 +1222,7 @@ InitPredicateLocks(void)
 	 * Compute size for serializable transaction hashtable. Note these
 	 * calculations must agree with PredicateLockShmemSize!
 	 */
-	max_table_size = (MaxBackends + max_prepared_xacts);
+	max_table_size = (GetMaxBackends() + max_prepared_xacts);
 
 	/*
 	 * Allocate a list to hold information on transactions participating in
@@ -1374,7 +1374,7 @@ PredicateLockShmemSize(void)
 	size = add_size(size, size / 10);
 
 	/* transaction list */
-	max_table_size = MaxBackends + max_prepared_xacts;
+	max_table_size = GetMaxBackends() + max_prepared_xacts;
 	max_table_size *= 10;
 	size = add_size(size, PredXactListDataSize);
 	size = add_size(size, mul_size((Size) max_table_size,
@@ -1905,7 +1905,7 @@ GetSerializableTransactionSnapshotInt(Snapshot snapshot,
 	{
 		++(PredXact->WritableSxactCount);
 		Assert(PredXact->WritableSxactCount <=
-			   (MaxBackends + max_prepared_xacts));
+			   (GetMaxBackends() + max_prepared_xacts));
 	}
 
 	MySerializableXact = sxact;
@@ -5107,7 +5107,7 @@ predicatelock_twophase_recover(TransactionId xid, uint16 info,
 		{
 			++(PredXact->WritableSxactCount);
 			Assert(PredXact->WritableSxactCount <=
-				   (MaxBackends + max_prepared_xacts));
+				   (GetMaxBackends() + max_prepared_xacts));
 		}
 
 		/*
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index b7d9da0aa9..9046ccec2d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -103,7 +103,7 @@ ProcGlobalShmemSize(void)
 {
 	Size		size = 0;
 	Size		TotalProcs =
-	add_size(MaxBackends, add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
+	add_size(GetMaxBackends(), add_size(NUM_AUXILIARY_PROCS, max_prepared_xacts));
 
 	/* ProcGlobal */
 	size = add_size(size, sizeof(PROC_HDR));
@@ -127,7 +127,7 @@ ProcGlobalSemas(void)
 	 * We need a sema per backend (including autovacuum), plus one for each
 	 * auxiliary process.
 	 */
-	return MaxBackends + NUM_AUXILIARY_PROCS;
+	return GetMaxBackends() + NUM_AUXILIARY_PROCS;
 }
 
 /*
@@ -162,7 +162,7 @@ InitProcGlobal(void)
 	int			i,
 				j;
 	bool		found;
-	uint32		TotalProcs = MaxBackends + NUM_AUXILIARY_PROCS + max_prepared_xacts;
+	uint32		TotalProcs = GetMaxBackends() + NUM_AUXILIARY_PROCS + max_prepared_xacts;
 
 	/* Create the ProcGlobal shared structure */
 	ProcGlobal = (PROC_HDR *)
@@ -197,7 +197,7 @@ InitProcGlobal(void)
 	MemSet(procs, 0, TotalProcs * sizeof(PGPROC));
 	ProcGlobal->allProcs = procs;
 	/* XXX allProcCount isn't really all of them; it excludes prepared xacts */
-	ProcGlobal->allProcCount = MaxBackends + NUM_AUXILIARY_PROCS;
+	ProcGlobal->allProcCount = GetMaxBackends() + NUM_AUXILIARY_PROCS;
 
 	/*
 	 * Allocate arrays mirroring PGPROC fields in a dense manner. See
@@ -223,7 +223,7 @@ InitProcGlobal(void)
 		 * dummy PGPROCs don't need these though - they're never associated
 		 * with a real process
 		 */
-		if (i < MaxBackends + NUM_AUXILIARY_PROCS)
+		if (i < GetMaxBackends() + NUM_AUXILIARY_PROCS)
 		{
 			procs[i].sem = PGSemaphoreCreate();
 			InitSharedLatch(&(procs[i].procLatch));
@@ -260,7 +260,7 @@ InitProcGlobal(void)
 			ProcGlobal->bgworkerFreeProcs = &procs[i];
 			procs[i].procgloballist = &ProcGlobal->bgworkerFreeProcs;
 		}
-		else if (i < MaxBackends)
+		else if (i < GetMaxBackends())
 		{
 			/* PGPROC for walsender, add to walsenderFreeProcs list */
 			procs[i].links.next = (SHM_QUEUE *) ProcGlobal->walsenderFreeProcs;
@@ -288,8 +288,8 @@ InitProcGlobal(void)
 	 * Save pointers to the blocks of PGPROC structures reserved for auxiliary
 	 * processes and prepared transactions.
 	 */
-	AuxiliaryProcs = &procs[MaxBackends];
-	PreparedXactProcs = &procs[MaxBackends + NUM_AUXILIARY_PROCS];
+	AuxiliaryProcs = &procs[GetMaxBackends()];
+	PreparedXactProcs = &procs[GetMaxBackends() + NUM_AUXILIARY_PROCS];
 
 	/* Create ProcStructLock spinlock, too */
 	ProcStructLock = (slock_t *) ShmemAlloc(sizeof(slock_t));
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index 2901f9f5a9..3c9389b208 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -35,7 +35,7 @@
  * includes autovacuum workers and background workers as well.
  * ----------
  */
-#define NumBackendStatSlots (MaxBackends + NUM_AUXPROCTYPES)
+#define NumBackendStatSlots (GetMaxBackends() + NUM_AUXPROCTYPES)
 
 
 /* ----------
@@ -251,7 +251,7 @@ pgstat_beinit(void)
 	/* Initialize MyBEEntry */
 	if (MyBackendId != InvalidBackendId)
 	{
-		Assert(MyBackendId >= 1 && MyBackendId <= MaxBackends);
+		Assert(MyBackendId >= 1 && MyBackendId <= GetMaxBackends());
 		MyBEEntry = &BackendStatusArray[MyBackendId - 1];
 	}
 	else
@@ -267,7 +267,7 @@ pgstat_beinit(void)
 		 * MaxBackends + AuxBackendType + 1 as the index of the slot for an
 		 * auxiliary process.
 		 */
-		MyBEEntry = &BackendStatusArray[MaxBackends + MyAuxProcType];
+		MyBEEntry = &BackendStatusArray[GetMaxBackends() + MyAuxProcType];
 	}
 
 	/* Set up a process-exit hook to clean up */
@@ -890,9 +890,10 @@ pgstat_get_backend_current_activity(int pid, bool checkUser)
 {
 	PgBackendStatus *beentry;
 	int			i;
+	int			max_backends = GetMaxBackends();
 
 	beentry = BackendStatusArray;
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= max_backends; i++)
 	{
 		/*
 		 * Although we expect the target backend's entry to be stable, that
@@ -968,6 +969,7 @@ pgstat_get_crashed_backend_activity(int pid, char *buffer, int buflen)
 {
 	volatile PgBackendStatus *beentry;
 	int			i;
+	int			max_backends = GetMaxBackends();
 
 	beentry = BackendStatusArray;
 
@@ -978,7 +980,7 @@ pgstat_get_crashed_backend_activity(int pid, char *buffer, int buflen)
 	if (beentry == NULL || BackendActivityBuffer == NULL)
 		return NULL;
 
-	for (i = 1; i <= MaxBackends; i++)
+	for (i = 1; i <= max_backends; i++)
 	{
 		if (beentry->st_procpid == pid)
 		{
diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c
index 5dc0a5882c..f905824215 100644
--- a/src/backend/utils/adt/lockfuncs.c
+++ b/src/backend/utils/adt/lockfuncs.c
@@ -561,11 +561,11 @@ pg_safe_snapshot_blocking_pids(PG_FUNCTION_ARGS)
 	Datum	   *blocker_datums;
 
 	/* A buffer big enough for any possible blocker list without truncation */
-	blockers = (int *) palloc(MaxBackends * sizeof(int));
+	blockers = (int *) palloc(GetMaxBackends() * sizeof(int));
 
 	/* Collect a snapshot of processes waited for by GetSafeSnapshot */
 	num_blockers =
-		GetSafeSnapshotBlockingPids(blocked_pid, blockers, MaxBackends);
+		GetSafeSnapshotBlockingPids(blocked_pid, blockers, GetMaxBackends());
 
 	/* Convert int array to Datum array */
 	if (num_blockers > 0)
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 78bc64671e..d30e664985 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -25,6 +25,7 @@
 #include "access/session.h"
 #include "access/sysattr.h"
 #include "access/tableam.h"
+#include "access/twophase.h"
 #include "access/xact.h"
 #include "access/xlog.h"
 #include "catalog/catalog.h"
@@ -63,6 +64,9 @@
 #include "utils/syscache.h"
 #include "utils/timeout.h"
 
+static int MaxBackends = 0;
+static int MaxBackendsInitialized = false;
+
 static HeapTuple GetDatabaseTuple(const char *dbname);
 static HeapTuple GetDatabaseTupleByOid(Oid dboid);
 static void PerformAuthentication(Port *port);
@@ -476,9 +480,8 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called after modules have had the chance to alter GUCs in
+ * shared_preload_libraries, and before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
@@ -488,15 +491,31 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 void
 InitializeMaxBackends(void)
 {
-	Assert(MaxBackends == 0);
-
 	/* the extra unit accounts for the autovacuum launcher */
-	MaxBackends = MaxConnections + autovacuum_max_workers + 1 +
-		max_worker_processes + max_wal_senders;
+	SetMaxBackends(MaxConnections + autovacuum_max_workers + 1 +
+		max_worker_processes + max_wal_senders);
+}
 
-	/* internal error because the values were all checked previously */
-	if (MaxBackends > MAX_BACKENDS)
+int
+GetMaxBackends(void)
+{
+	if (!MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends not yet initialized");
+
+	return MaxBackends;
+}
+
+void
+SetMaxBackends(int max_backends)
+{
+	if (MaxBackendsInitialized)
+		elog(ERROR, "MaxBackends already initialized");
+
+	if (max_backends > MAX_BACKENDS)
 		elog(ERROR, "too many backends configured");
+
+	MaxBackends = max_backends;
+	MaxBackendsInitialized = true;
 }
 
 /*
@@ -596,7 +615,7 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 
 	SharedInvalBackendInit(false);
 
-	if (MyBackendId > MaxBackends || MyBackendId <= 0)
+	if (MyBackendId > GetMaxBackends() || MyBackendId <= 0)
 		elog(FATAL, "bad backend ID: %d", MyBackendId);
 
 	/* Now that we have a BackendId, we can participate in ProcSignal */
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 2e2e9a364a..fd0977ab05 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -172,7 +172,6 @@ extern PGDLLIMPORT char *DataDir;
 extern PGDLLIMPORT int data_directory_mode;
 
 extern PGDLLIMPORT int NBuffers;
-extern PGDLLIMPORT int MaxBackends;
 extern PGDLLIMPORT int MaxConnections;
 extern PGDLLIMPORT int max_worker_processes;
 extern PGDLLIMPORT int max_parallel_workers;
@@ -455,6 +454,8 @@ extern AuxProcType MyAuxProcType;
 /* in utils/init/postinit.c */
 extern void pg_split_opts(char **argv, int *argcp, const char *optstr);
 extern void InitializeMaxBackends(void);
+extern int GetMaxBackends(void);
+extern void SetMaxBackends(int max_backends);
 extern void InitPostgres(const char *in_dbname, Oid dboid, const char *username,
 						 Oid useroid, char *out_dbname, bool override_allow_connections);
 extern void BaseInit(void);
-- 
2.16.6



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-23 04:52  Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-23 04:52 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Michael Paquier <[email protected]>; Robert Haas <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

Sorry for showing up this late, but I'm a bit confused with the new situation.

Unless I'm missing something, the new situation is that the system is supposed
to prevent access to MaxBackends during s_p_l_pg_init, for reasons I totally
agree with, but without doing anything for extensions that actually need to
access it at that time.  So what are extensions supposed to do now if they do
need the information during their _PG_init() / RequestAddinShmemSpace()?

Note that I have two of such extensions.  They actually only need it to give
access to the queryid in the background workers since it's otherwise not
available, but there's at least pg_wait_sampling extension that needs the value
to request shmem for other needs.

One funny note is that my extensions aren't using MaxBackends but instead
compute it based on MaxConnections, autovacuum_max_workers and so on.  So those
two extensions aren't currently broken, or more accurately not more than they
previously were.  Is there any reasoning to not protect all the variables that
contribute to MaxBackends?

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-23 12:32  Robert Haas <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-03-23 12:32 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Wed, Mar 23, 2022 at 12:53 AM Julien Rouhaud <[email protected]> wrote:
> Unless I'm missing something, the new situation is that the system is supposed
> to prevent access to MaxBackends during s_p_l_pg_init, for reasons I totally
> agree with, but without doing anything for extensions that actually need to
> access it at that time.  So what are extensions supposed to do now if they do
> need the information during their _PG_init() / RequestAddinShmemSpace()?

Well, the conclusion upthread was that extensions might change the
values of those GUCs from _PG_init(). If that's a real thing, then
what you're asking for here is impossible, because the final value is
indeterminate until all such extensions have finished twiddling those
the GUCs. On the other hand, it's definitely intended that extensions
should RequestAddinShmemSpace() from _PG_init(), and wanting to size
that memory based on MaxBackends is totally reasonable. Do we need to
add another function, alongside _PG_init(), that gets called after
MaxBackends is determined and before it's too late to
RequestAddinShmemSpace()?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-23 13:03  Julien Rouhaud <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-23 13:03 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Wed, Mar 23, 2022 at 08:32:39AM -0400, Robert Haas wrote:
> On Wed, Mar 23, 2022 at 12:53 AM Julien Rouhaud <[email protected]> wrote:
> > Unless I'm missing something, the new situation is that the system is supposed
> > to prevent access to MaxBackends during s_p_l_pg_init, for reasons I totally
> > agree with, but without doing anything for extensions that actually need to
> > access it at that time.  So what are extensions supposed to do now if they do
> > need the information during their _PG_init() / RequestAddinShmemSpace()?
> 
> Well, the conclusion upthread was that extensions might change the
> values of those GUCs from _PG_init(). If that's a real thing, then
> what you're asking for here is impossible, because the final value is
> indeterminate until all such extensions have finished twiddling those
> the GUCs. On the other hand, it's definitely intended that extensions
> should RequestAddinShmemSpace() from _PG_init(), and wanting to size
> that memory based on MaxBackends is totally reasonable. Do we need to
> add another function, alongside _PG_init(), that gets called after
> MaxBackends is determined and before it's too late to
> RequestAddinShmemSpace()?

Yes, I don't see how we can support such extensions without an additional hook,
probably called right after InitializeMaxBackends() since at least
InitializeShmemGUCs() will need to know about those extra memory needs.

I'm not sure how to prevent third party code from messing with the gucs in it,
but a clear indication in the hook comment should probably be enough.  It's not
like it's hard for third-party code author to break something anyway.

And should we do something about MaxConnections, autovacuum_max_workers,
max_worker_processes and max_wal_senders too?  It's unlikely that I'm the only
one who's computing my own MaxBackends to workaround the previous 0 value that
was available during _PG_init, and I'm not sure that everyone will notice such
a new hook.





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-24 20:20  Nathan Bossart <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-03-24 20:20 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Robert Haas <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Wed, Mar 23, 2022 at 09:03:18PM +0800, Julien Rouhaud wrote:
> On Wed, Mar 23, 2022 at 08:32:39AM -0400, Robert Haas wrote:
>> Well, the conclusion upthread was that extensions might change the
>> values of those GUCs from _PG_init(). If that's a real thing, then
>> what you're asking for here is impossible, because the final value is
>> indeterminate until all such extensions have finished twiddling those
>> the GUCs. On the other hand, it's definitely intended that extensions
>> should RequestAddinShmemSpace() from _PG_init(), and wanting to size
>> that memory based on MaxBackends is totally reasonable. Do we need to
>> add another function, alongside _PG_init(), that gets called after
>> MaxBackends is determined and before it's too late to
>> RequestAddinShmemSpace()?
> 
> Yes, I don't see how we can support such extensions without an additional hook,
> probably called right after InitializeMaxBackends() since at least
> InitializeShmemGUCs() will need to know about those extra memory needs.

Another possibility could be to add a hook that is called _before_
_PG_init() where libraries are permitted to adjust GUCs.  After the library
is loaded, we first call this _PG_change_GUCs() function, then we
initialize MaxBackends, and then we finally call _PG_init().  This way,
extensions would have access to MaxBackends within _PG_init(), and if an
extension really needed to alter GUCs, іt could define this new function.

> I'm not sure how to prevent third party code from messing with the gucs in it,
> but a clear indication in the hook comment should probably be enough.  It's not
> like it's hard for third-party code author to break something anyway.

ERROR-ing in SetConfigOption() might be another way to dissuade folks from
messing with GUCs.  This is obviously not a perfect solution.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-24 20:27  Robert Haas <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-03-24 20:27 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Thu, Mar 24, 2022 at 4:20 PM Nathan Bossart <[email protected]> wrote:
> Another possibility could be to add a hook that is called _before_
> _PG_init() where libraries are permitted to adjust GUCs.  After the library
> is loaded, we first call this _PG_change_GUCs() function, then we
> initialize MaxBackends, and then we finally call _PG_init().  This way,
> extensions would have access to MaxBackends within _PG_init(), and if an
> extension really needed to alter GUCs, іt could define this new function.

Yeah, I think this might be better.

-- 
Robert Haas
EDB: http://www.enterprisedb.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-25 02:39  Julien Rouhaud <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-25 02:39 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Thu, Mar 24, 2022 at 04:27:36PM -0400, Robert Haas wrote:
> On Thu, Mar 24, 2022 at 4:20 PM Nathan Bossart <[email protected]> wrote:
> > Another possibility could be to add a hook that is called _before_
> > _PG_init() where libraries are permitted to adjust GUCs.  After the library
> > is loaded, we first call this _PG_change_GUCs() function, then we
> > initialize MaxBackends, and then we finally call _PG_init().  This way,
> > extensions would have access to MaxBackends within _PG_init(), and if an
> > extension really needed to alter GUCs, іt could define this new function.
> 
> Yeah, I think this might be better.

Well, if it's before _PG_init() then it won't be a new hook but a new symbol
that has to be handled with dlsym.

But it seems to be that the bigger problem is that this approach won't fix
anything unless we can prevent third-party code from messing with GUCs after
that point as we could otherwise have discrepancies between GetMaxBackends()
and the underlying GUCs until every single extension that wants to change GUC
is modified uses this new symbol.  And I also don't see how we could force that
unless we have a better GUC API that doesn't entirely relies on strings,
otherwise a simple thing like "allow 5 extra bgworkers" is going to be really
painful.

A new hook after _PG_init() sure leaves the burden to the few extension that
needs to allocate shmem based on MaxBackends, but there probably isn't that
much (I have a few dozens locally cloned and found only 1 apart from mine, and
I would be happy to take care of those), and it also means that they have a
chance to do something correct even if other extensions messing with GUCs
aren't fixed.

Arguably, you could certainly try to change GUCs in that new hook, but it
wouldn't be any different from doing so in shmem_startup_hook so I don't think
it's really a problem.

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-25 03:11  Julien Rouhaud <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-25 03:11 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Fri, Mar 25, 2022 at 10:39:51AM +0800, Julien Rouhaud wrote:
> On Thu, Mar 24, 2022 at 04:27:36PM -0400, Robert Haas wrote:
> > On Thu, Mar 24, 2022 at 4:20 PM Nathan Bossart <[email protected]> wrote:
> > > Another possibility could be to add a hook that is called _before_
> > > _PG_init() where libraries are permitted to adjust GUCs.  After the library
> > > is loaded, we first call this _PG_change_GUCs() function, then we
> > > initialize MaxBackends, and then we finally call _PG_init().  This way,
> > > extensions would have access to MaxBackends within _PG_init(), and if an
> > > extension really needed to alter GUCs, іt could define this new function.
> > 
> > Yeah, I think this might be better.
> 
> Well, if it's before _PG_init() then it won't be a new hook but a new symbol
> that has to be handled with dlsym.
> 
> But it seems to be that the bigger problem is that this approach won't fix
> anything unless we can prevent third-party code from messing with GUCs after
> that point as we could otherwise have discrepancies between GetMaxBackends()
> and the underlying GUCs until every single extension that wants to change GUC
> is modified uses this new symbol.  And I also don't see how we could force that
> unless we have a better GUC API that doesn't entirely relies on strings,
> otherwise a simple thing like "allow 5 extra bgworkers" is going to be really
> painful.
> 
> A new hook after _PG_init() sure leaves the burden to the few extension that
> needs to allocate shmem based on MaxBackends, but there probably isn't that
> much (I have a few dozens locally cloned and found only 1 apart from mine, and
> I would be happy to take care of those), and it also means that they have a
> chance to do something correct even if other extensions messing with GUCs
> aren't fixed.
> 
> Arguably, you could certainly try to change GUCs in that new hook, but it
> wouldn't be any different from doing so in shmem_startup_hook so I don't think
> it's really a problem.

As an example, here's a POC for a new shmem_request_hook hook after _PG_init().
With it I could easily fix pg_wait_sampling shmem allocation (and checked that
it's indeed requesting the correct size).


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-25 05:08  Michael Paquier <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Michael Paquier @ 2022-03-25 05:08 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Fri, Mar 25, 2022 at 11:11:46AM +0800, Julien Rouhaud wrote:
> As an example, here's a POC for a new shmem_request_hook hook after _PG_init().
> With it I could easily fix pg_wait_sampling shmem allocation (and checked that
> it's indeed requesting the correct size).

Are you sure that the end of a release cycle is the good moment to
begin designing new hooks?  Anything added is something we are going
to need supporting moving forward.  My brain is telling me that we
ought to revisit the business with GetMaxBackends() properly instead,
and perhaps revert that.

This solution makes me uneasy from the start (already stated
upthread), because it is not a solution to the original problem, just
a safeguard that handles one small-ish portion of the whole parameter
calculation cycle.
--
Michael


Attachments:

  [application/pgp-signature] signature.asc (833B, ../../Yj1OOR%[email protected]/2-signature.asc)
  download

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-25 06:35  Julien Rouhaud <[email protected]>
  parent: Michael Paquier <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-25 06:35 UTC (permalink / raw)
  To: Michael Paquier <[email protected]>; +Cc: Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Fri, Mar 25, 2022 at 02:08:09PM +0900, Michael Paquier wrote:
> On Fri, Mar 25, 2022 at 11:11:46AM +0800, Julien Rouhaud wrote:
> > As an example, here's a POC for a new shmem_request_hook hook after _PG_init().
> > With it I could easily fix pg_wait_sampling shmem allocation (and checked that
> > it's indeed requesting the correct size).
> 
> Are you sure that the end of a release cycle is the good moment to
> begin designing new hooks?  Anything added is something we are going
> to need supporting moving forward.  My brain is telling me that we
> ought to revisit the business with GetMaxBackends() properly instead,
> and perhaps revert that.

I agree, and as I mentioned in my original email I don't think that the
committed patch is actually adding something on which we can really build on.
So I'm also in favor of reverting, as it seems like be a better option in the
long run to have a clean and broader solution.





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-25 22:23  Andres Freund <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Andres Freund @ 2022-03-25 22:23 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Michael Paquier <[email protected]>; Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

On 2022-03-25 14:35:42 +0800, Julien Rouhaud wrote:
> On Fri, Mar 25, 2022 at 02:08:09PM +0900, Michael Paquier wrote:
> > On Fri, Mar 25, 2022 at 11:11:46AM +0800, Julien Rouhaud wrote:
> > > As an example, here's a POC for a new shmem_request_hook hook after _PG_init().
> > > With it I could easily fix pg_wait_sampling shmem allocation (and checked that
> > > it's indeed requesting the correct size).
> > 
> > Are you sure that the end of a release cycle is the good moment to
> > begin designing new hooks?  Anything added is something we are going
> > to need supporting moving forward.  My brain is telling me that we
> > ought to revisit the business with GetMaxBackends() properly instead,
> > and perhaps revert that.
> 
> I agree, and as I mentioned in my original email I don't think that the
> committed patch is actually adding something on which we can really build on.
> So I'm also in favor of reverting, as it seems like be a better option in the
> long run to have a clean and broader solution.

I don't really understand. The issue that started this thread was bugs in
extensions due to accessing MaxBackends before it is initialized - which the
patch prevents. The stuff that you're complaining about / designing here
doesn't seem related to that. I like the idea of the hooks etc, but I fail to
see why we "ought to revisit the business with GetMaxBackends()"?

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-26 07:22  Julien Rouhaud <[email protected]>
  parent: Andres Freund <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-26 07:22 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Michael Paquier <[email protected]>; Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

On Fri, Mar 25, 2022 at 03:23:17PM -0700, Andres Freund wrote:
>
> I don't really understand. The issue that started this thread was bugs in
> extensions due to accessing MaxBackends before it is initialized - which the
> patch prevents.

Well, the patch prevents accessing a 0-valued MaxBackends but doesn't do
anything to solve the original complaint.  It's not like extensions won't need
to access that information during _PG_init anymore.

> The stuff that you're complaining about / designing here
> doesn't seem related to that. I like the idea of the hooks etc, but I fail to
> see why we "ought to revisit the business with GetMaxBackends()"?

Because this GetMaxBackends() doesn't solve the problem nor brings any
infrastructure that could be reused to solve it.

I think that the root issue could be rephrased with:

Can we initialize MaxBackends earlier so that _PG_init() can see it because
maintaining MaxConnections + autovacuum_max_workers + 1 + max_worker_processes
+ max_wal_senders is troublesome.

And indeed, any third party code that previously needed to access what
MaxBackends is supposed to store should already be using that formula, and
the new GetMaxBackends() doesn't do anything about it.  So all extensions will
be as broken as before, except the few that were using MaxBackends without
realizing it's 0.  And if those exist (there's actually one) they're not that
broken, probably because MaxBackend is only used to request additional shmem,
with wanted value small enough so that it's compensated by the extra 100kB
shmem postgres allocates.

Since all those underlying GUCs shouldn't be accessible, we need some more
general infrastructure that would work for those too on top of a way to access
MaxBackends when extensions needs it.

Note that the only use case I'm aware of is for RequestAddinShmemSpace, so a
hook after _PG_init like the example shmem_request_hook would be enough for the
latter, but maybe there are more use cases for which it wouldn't.

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-26 17:23  Andres Freund <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 2 replies; 94+ messages in thread

From: Andres Freund @ 2022-03-26 17:23 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Michael Paquier <[email protected]>; Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

On 2022-03-26 15:22:03 +0800, Julien Rouhaud wrote:
> On Fri, Mar 25, 2022 at 03:23:17PM -0700, Andres Freund wrote:
> >
> > I don't really understand. The issue that started this thread was bugs in
> > extensions due to accessing MaxBackends before it is initialized - which the
> > patch prevents.
> 
> Well, the patch prevents accessing a 0-valued MaxBackends but doesn't do
> anything to solve the original complaint.  It's not like extensions won't need
> to access that information during _PG_init anymore.

It resolves the pretty common bug that an extension breaks once it's used via
s_p_l instead of loaded on-demand because MaxBackends isn't initialized in the
s_p_l case.


> And indeed, any third party code that previously needed to access what
> MaxBackends is supposed to store should already be using that formula, and
> the new GetMaxBackends() doesn't do anything about it.

It couldn't rely on MaxBackends before. It can't rely on GetMaxBackends()
now. You can see why I think that what you want is unrelated to the
introduction of GetMaxBackends().

If we introduce a separate hook that allows to influence things like
max_connections or whatnot we'd *even more* need a way to verify whether it's
legal to access MaxBackends in that moment.


> So all extensions will be as broken as before, except the few that were
> using MaxBackends without realizing it's 0. And if those exist (there's
> actually one) they're not that broken, probably because MaxBackend is only
> used to request additional shmem, with wanted value small enough so that
> it's compensated by the extra 100kB shmem postgres allocates.

I don't think it's rare at all.

Greetings,

Andres Freund





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-27 04:15  Julien Rouhaud <[email protected]>
  parent: Andres Freund <[email protected]>
  1 sibling, 0 replies; 94+ messages in thread

From: Julien Rouhaud @ 2022-03-27 04:15 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Michael Paquier <[email protected]>; Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Sat, Mar 26, 2022 at 10:23:16AM -0700, Andres Freund wrote:
>
> On 2022-03-26 15:22:03 +0800, Julien Rouhaud wrote:
> > On Fri, Mar 25, 2022 at 03:23:17PM -0700, Andres Freund wrote:
> > >
> > > I don't really understand. The issue that started this thread was bugs in
> > > extensions due to accessing MaxBackends before it is initialized - which the
> > > patch prevents.
> > 
> > Well, the patch prevents accessing a 0-valued MaxBackends but doesn't do
> > anything to solve the original complaint.  It's not like extensions won't need
> > to access that information during _PG_init anymore.
> 
> It resolves the pretty common bug that an extension breaks once it's used via
> s_p_l instead of loaded on-demand because MaxBackends isn't initialized in the
> s_p_l case.

I can hear the argument.  However I don't know any extension that relies on
MaxBackends and doesn't need to be in s_p_l, and unless I'm missing something
no one provided such an example, only people needing the value for
RequestAddinShmemSpace().

> > And indeed, any third party code that previously needed to access what
> > MaxBackends is supposed to store should already be using that formula, and
> > the new GetMaxBackends() doesn't do anything about it.
> 
> It couldn't rely on MaxBackends before. It can't rely on GetMaxBackends()
> now. You can see why I think that what you want is unrelated to the
> introduction of GetMaxBackends().

Sure, but code also couldn't really rely on MaxConnections or any other similar
GUCs and yet nothing is done for that, and the chosen approach doesn't help for
that.

The only difference is that those GUCs are less broken, as in the value is
likely to be valid more often, and closer to the final value when it's not.
But still broken.

I think GetMaxBackends() is more likely to force authors to rely on computing
the value using the underlying GUCs, since there's nothing else that can be
done.  And if that's an acceptable answer, why aren't we computing MaxBackends
before and after processing s_p_l?

> If we introduce a separate hook that allows to influence things like
> max_connections or whatnot we'd *even more* need a way to verify whether it's
> legal to access MaxBackends in that moment.

Again, I'm not opposed to validating that MaxBackends is valid when third-party
code is accessing it, quite the opposite.  I'm opposed to do so *only* for
MaxBackends, and without providing a way for third-party code to access that
value when they need it, which is for all the cases I know (after more digging
that's now 5 extensions) for RequestAddinShmemSpace().

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-29 16:22  Robert Haas <[email protected]>
  parent: Andres Freund <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-03-29 16:22 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Nathan Bossart <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Sat, Mar 26, 2022 at 1:23 PM Andres Freund <[email protected]> wrote:
> > And indeed, any third party code that previously needed to access what
> > MaxBackends is supposed to store should already be using that formula, and
> > the new GetMaxBackends() doesn't do anything about it.
>
> It couldn't rely on MaxBackends before. It can't rely on GetMaxBackends()
> now. You can see why I think that what you want is unrelated to the
> introduction of GetMaxBackends().

It's not, though, because the original proposal was to change things
around so that the value of MaxBackends would have been reliable in
_PG_init(). If we'd done that, then extensions that are using it in
_PG_init() would have gone from being buggy to being not-buggy. But
since you advocated against that change, we instead committed
something that caused them to go from being buggy to failing outright.
That's pretty painful for people with such extensions. And IMHO, it's
*much* more legitimate to want to size a data structure based on the
value of MaxBackends than it is for extensions to override GUC values.
If we can make the latter use case work in a sane way, OK, although I
have my doubts about how sane it really is, but it can't be at the
expense of telling extensions that have been (incorrectly) using
MaxBackends in _PG_init() that we're throwing them under the bus.

IMHO, the proper thing to do if certain GUC values are required for an
extension to work is to put that information in the documentation and
error out at an appropriate point if the user does not follow the
directions. Then this issue does not arise. But there's no reasonable
workaround for being unable to size data structures based on
MaxBackends.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-03-30 16:30  Nathan Bossart <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-03-30 16:30 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Tue, Mar 29, 2022 at 12:22:21PM -0400, Robert Haas wrote:
> It's not, though, because the original proposal was to change things
> around so that the value of MaxBackends would have been reliable in
> _PG_init(). If we'd done that, then extensions that are using it in
> _PG_init() would have gone from being buggy to being not-buggy. But
> since you advocated against that change, we instead committed
> something that caused them to go from being buggy to failing outright.
> That's pretty painful for people with such extensions. And IMHO, it's
> *much* more legitimate to want to size a data structure based on the
> value of MaxBackends than it is for extensions to override GUC values.
> If we can make the latter use case work in a sane way, OK, although I
> have my doubts about how sane it really is, but it can't be at the
> expense of telling extensions that have been (incorrectly) using
> MaxBackends in _PG_init() that we're throwing them under the bus.
> 
> IMHO, the proper thing to do if certain GUC values are required for an
> extension to work is to put that information in the documentation and
> error out at an appropriate point if the user does not follow the
> directions. Then this issue does not arise. But there's no reasonable
> workaround for being unable to size data structures based on
> MaxBackends.

FWIW I would be on board with reverting all the GetMaxBackends() stuff if
we made the value available in _PG_init() and stopped supporting GUC
overrides by extensions (e.g., ERROR-ing in SetConfigOption() when
process_shared_preload_libraries_in_progress is true).

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-09 13:24  Julien Rouhaud <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 4 replies; 94+ messages in thread

From: Julien Rouhaud @ 2022-04-09 13:24 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Wed, Mar 30, 2022 at 09:30:51AM -0700, Nathan Bossart wrote:
> On Tue, Mar 29, 2022 at 12:22:21PM -0400, Robert Haas wrote:
> > It's not, though, because the original proposal was to change things
> > around so that the value of MaxBackends would have been reliable in
> > _PG_init(). If we'd done that, then extensions that are using it in
> > _PG_init() would have gone from being buggy to being not-buggy. But
> > since you advocated against that change, we instead committed
> > something that caused them to go from being buggy to failing outright.
> > That's pretty painful for people with such extensions. And IMHO, it's
> > *much* more legitimate to want to size a data structure based on the
> > value of MaxBackends than it is for extensions to override GUC values.
> > If we can make the latter use case work in a sane way, OK, although I
> > have my doubts about how sane it really is, but it can't be at the
> > expense of telling extensions that have been (incorrectly) using
> > MaxBackends in _PG_init() that we're throwing them under the bus.
> > 
> > IMHO, the proper thing to do if certain GUC values are required for an
> > extension to work is to put that information in the documentation and
> > error out at an appropriate point if the user does not follow the
> > directions. Then this issue does not arise. But there's no reasonable
> > workaround for being unable to size data structures based on
> > MaxBackends.
> 
> FWIW I would be on board with reverting all the GetMaxBackends() stuff if
> we made the value available in _PG_init() and stopped supporting GUC
> overrides by extensions (e.g., ERROR-ing in SetConfigOption() when
> process_shared_preload_libraries_in_progress is true).

Yeah I would prefer this approach too, although it couldn't prevent extension
from directly modifying the underlying variables so I don't know how effective
it would be.

On the bright side, I see that citus is using SetConfigOption() to increase
max_prepared_transactions [1].  That's the only extension mentioned in that
thread that does modify some GUC, and this GUC was already marked as
PGDLLIMPORT since 2017 so it probably wasn't done that way for Windows
compatibility reason.

In the meantime, should we add an open item?

[1] https://github.com/citusdata/citus/blob/master/src/backend/distributed/transaction/transaction_manag...





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-10 23:34  Michael Paquier <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  3 siblings, 0 replies; 94+ messages in thread

From: Michael Paquier @ 2022-04-10 23:34 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Sat, Apr 09, 2022 at 09:24:42PM +0800, Julien Rouhaud wrote:
> Yeah I would prefer this approach too, although it couldn't prevent extension
> from directly modifying the underlying variables so I don't know how effective
> it would be.
> 
> On the bright side, I see that citus is using SetConfigOption() to increase
> max_prepared_transactions [1].  That's the only extension mentioned in that
> thread that does modify some GUC, and this GUC was already marked as
> PGDLLIMPORT since 2017 so it probably wasn't done that way for Windows
> compatibility reason.
> 
> In the meantime, should we add an open item?

Yes, I think that we need to discuss more the matter here, so added one.
--
Michael


Attachments:

  [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
  download

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-11 15:36  Julien Rouhaud <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  3 siblings, 0 replies; 94+ messages in thread

From: Julien Rouhaud @ 2022-04-11 15:36 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

On Mon, Apr 11, 2022 at 10:47:17AM -0400, Robert Haas wrote:
> On Sat, Apr 9, 2022 at 9:24 AM Julien Rouhaud <[email protected]> wrote:
>
> > On the bright side, I see that citus is using SetConfigOption() to increase
> > max_prepared_transactions [1].  That's the only extension mentioned in that
> > thread that does modify some GUC, and this GUC was already marked as
> > PGDLLIMPORT since 2017 so it probably wasn't done that way for Windows
> > compatibility reason.
>
> I don't quite understand this, but I think if we want to support this
> kind of thing it needs a separate hook.

My point was that even if we say we don't support this, we have no way to
actually fully enforce it, as the variables are exported.  So it's good that
the only know case is using the GUC API, since we can enforce this one.





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 05:08  Julien Rouhaud <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  3 siblings, 0 replies; 94+ messages in thread

From: Julien Rouhaud @ 2022-04-12 05:08 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

Hi,

On Mon, Apr 11, 2022 at 02:14:35PM -0700, Nathan Bossart wrote:
> On Mon, Apr 11, 2022 at 01:44:42PM -0700, Nathan Bossart wrote:
> > On Mon, Apr 11, 2022 at 04:36:36PM -0400, Robert Haas wrote:
> >> If we throw an error while defining_custom_guc is true, how will it
> >> ever again become false?
> > 
> > Ah, I knew I was forgetting something this morning.
> > 
> > It won't, but the only place it is presently needed is when loading
> > shared_preload_libraries, so I believe startup will fail anyway.  However,
> > I can see defining_custom_guc being used elsewhere, so that is probably not
> > good enough.  Another approach could be to add a static
> > set_config_option_internal() function with a boolean argument to indicate
> > whether it is being used while defining a custom GUC.  I'll adjust 0003
> > with that approach unless a better idea prevails.
> 
> Here's a new patch set.  I've only changed 0003 as described above.  My
> apologies for the unnecessary round trip.

It looks sensible to me, although I think I would apply 0003 before 0002.

+   if (process_shared_preload_libraries_in_progress &&
+       !allow_when_loading_preload_libs)
+       ereport(ERROR,
+               (errcode(ERRCODE_CANT_CHANGE_RUNTIME_PARAM),
+                errmsg("cannot change parameters while loading "
+                       "\"shared_preload_libraries\"")));
+

I think the error message should mention at least which GUC is being modified.





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 18:03  Nathan Bossart <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  3 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-12 18:03 UTC (permalink / raw)
  To: Andres Freund <[email protected]>; +Cc: Robert Haas <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 10:44:27AM -0700, Andres Freund wrote:
> On 2022-04-11 14:14:35 -0700, Nathan Bossart wrote:
>> Here's a new patch set.  I've only changed 0003 as described above.  My
>> apologies for the unnecessary round trip.
> 
> ISTM we shouldn't apply 0002, 0003 to master before we've branches 15 off..

That seems reasonable to me.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 19:12  Robert Haas <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 2 replies; 94+ messages in thread

From: Robert Haas @ 2022-04-12 19:12 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 2:03 PM Nathan Bossart <[email protected]> wrote:
> On Tue, Apr 12, 2022 at 10:44:27AM -0700, Andres Freund wrote:
> > On 2022-04-11 14:14:35 -0700, Nathan Bossart wrote:
> >> Here's a new patch set.  I've only changed 0003 as described above.  My
> >> apologies for the unnecessary round trip.
> >
> > ISTM we shouldn't apply 0002, 0003 to master before we've branches 15 off..
>
> That seems reasonable to me.

Gah, I hate putting this off for another year, but I guess I'm also
not convinced that 0003 has the right idea, so maybe it's for the
best. Here's what's bugging me: 0003 decrees that _PG_init() is the
Wrong Place to Adjust GUC Settings. Now, that begs the question: what
is the right place to adjust GUC settings? I argued upthread that
extensions shouldn't be overriding values from postgresql.conf at all,
but on further reflection I think there's at least one legitimate use
case for this sort of thing: some kind of auto-tuning module that, for
example, looks at how much memory you have and sets shared_buffers or
work_mem or something based on the result. It's arguable how well
something like this can actually work, but we probably shouldn't try
to prevent people from doing it. I'm a little less clear why Citus
thinks this is an appropriate thing for it to do, vs. telling the user
to configure a useful value, and I'd be curious to hear an explanation
around that.

But if there's even one use case where adjusting GUCs at this phase is
reasonable, then 0003 isn't really good enough. We need an 0004 that
provides a new hook in a place where such changes can safely be made.

Meanwhile, committed 0001.

--
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 19:21  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-12 19:21 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> Gah, I hate putting this off for another year, but I guess I'm also
> not convinced that 0003 has the right idea, so maybe it's for the
> best. Here's what's bugging me: 0003 decrees that _PG_init() is the
> Wrong Place to Adjust GUC Settings. Now, that begs the question: what
> is the right place to adjust GUC settings? I argued upthread that
> extensions shouldn't be overriding values from postgresql.conf at all,
> but on further reflection I think there's at least one legitimate use
> case for this sort of thing: some kind of auto-tuning module that, for
> example, looks at how much memory you have and sets shared_buffers or
> work_mem or something based on the result.

Yeah.  It's a very long way from "changing shared memory sizes here
doesn't work" to "no extension is allowed to change any GUC at load
time".  A counterexample is that it's not clear why an extension
shouldn't be allowed to define a GUC using DefineCustomXXXVariable
and then immediately set it to some other value.  I think trying to
forbid that across-the-board is going to mostly result in breaking
a lot of cases that are perfectly safe.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 19:30  Robert Haas <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-12 19:30 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 3:22 PM Tom Lane <[email protected]> wrote:
> Yeah.  It's a very long way from "changing shared memory sizes here
> doesn't work" to "no extension is allowed to change any GUC at load
> time".  A counterexample is that it's not clear why an extension
> shouldn't be allowed to define a GUC using DefineCustomXXXVariable
> and then immediately set it to some other value.  I think trying to
> forbid that across-the-board is going to mostly result in breaking
> a lot of cases that are perfectly safe.

Hmm, that's an interesting case which I hadn't considered. It seems
like sort of a lame thing to do, because why wouldn't you just create
the GUC with the right initial value instead of modifying it after the
fact? But maybe there's some use case in which it makes sense to do it
that way. A read-only GUC that advertises some calculated value,
perhaps?

-- 
Robert Haas
EDB: http://www.enterprisedb.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 19:49  Nathan Bossart <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 0 replies; 94+ messages in thread

From: Nathan Bossart @ 2022-04-12 19:49 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; Tom Lane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 03:12:42PM -0400, Robert Haas wrote:
> But if there's even one use case where adjusting GUCs at this phase is
> reasonable, then 0003 isn't really good enough. We need an 0004 that
> provides a new hook in a place where such changes can safely be made.

I think that is doable.  IMO it should be ѕomething like _PG_change_GUCs()
that is called before _PG_init().  The other option is to add a hook called
after _PG_init() where MaxBackends is available (in which case we likely
want GetMaxBackends() again).  Thoughts?

> Meanwhile, committed 0001.

Thanks.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 19:59  Nathan Bossart <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-12 19:59 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 03:30:23PM -0400, Robert Haas wrote:
> On Tue, Apr 12, 2022 at 3:22 PM Tom Lane <[email protected]> wrote:
>> Yeah.  It's a very long way from "changing shared memory sizes here
>> doesn't work" to "no extension is allowed to change any GUC at load
>> time".  A counterexample is that it's not clear why an extension
>> shouldn't be allowed to define a GUC using DefineCustomXXXVariable
>> and then immediately set it to some other value.  I think trying to
>> forbid that across-the-board is going to mostly result in breaking
>> a lot of cases that are perfectly safe.
> 
> Hmm, that's an interesting case which I hadn't considered. It seems
> like sort of a lame thing to do, because why wouldn't you just create
> the GUC with the right initial value instead of modifying it after the
> fact? But maybe there's some use case in which it makes sense to do it
> that way. A read-only GUC that advertises some calculated value,
> perhaps?

I think it'd be reasonable to allow changing custom GUCs in _PG_init().
Those aren't going to impact things like MaxBackends.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 20:58  Tom Lane <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 2 replies; 94+ messages in thread

From: Tom Lane @ 2022-04-12 20:58 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Nathan Bossart <[email protected]> writes:
> I think it'd be reasonable to allow changing custom GUCs in _PG_init().
> Those aren't going to impact things like MaxBackends.

It's reasonable to allow changing *most* GUCs in _PG_init(); let us
please not break cases that work fine today.

It seems after a bit of reflection that what we ought to do is identify
the subset of PGC_POSTMASTER GUCs that feed into shared memory sizing,
mark those with a new GUC flag, and not allow them to be changed after
shared memory is set up.  (An alternative to a new flag bit is to
subdivide the PGC_POSTMASTER context into two values, but that seems
like it'd be far more invasive.  A flag bit needn't be user-visible.)
Then, what we'd need to do is arrange for shared-preload modules to
receive their _PG_init call before shared memory creation (when they
can still twiddle such GUCs) and then another callback later.

			regards, tom lane

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 21:07  Nathan Bossart <[email protected]>
  parent: Tom Lane <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-12 21:07 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 04:58:42PM -0400, Tom Lane wrote:
> Nathan Bossart <[email protected]> writes:
>> I think it'd be reasonable to allow changing custom GUCs in _PG_init().
>> Those aren't going to impact things like MaxBackends.
> 
> It's reasonable to allow changing *most* GUCs in _PG_init(); let us
> please not break cases that work fine today.

Right, this is a fair point.

> It seems after a bit of reflection that what we ought to do is identify
> the subset of PGC_POSTMASTER GUCs that feed into shared memory sizing,
> mark those with a new GUC flag, and not allow them to be changed after
> shared memory is set up.  (An alternative to a new flag bit is to
> subdivide the PGC_POSTMASTER context into two values, but that seems
> like it'd be far more invasive.  A flag bit needn't be user-visible.)
> Then, what we'd need to do is arrange for shared-preload modules to
> receive their _PG_init call before shared memory creation (when they
> can still twiddle such GUCs) and then another callback later.

If we allow changing GUCs in _PG_init() and provide another hook where
MaxBackends will be initialized, do we need to introduce another GUC flag,
or can we get away with just blocking all GUC changes when the new hook is
called?  I'm slightly hesitant to add a GUC flag that will need to manually
maintained.  Wouldn't it be easily forgotten?

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 21:14  Robert Haas <[email protected]>
  parent: Tom Lane <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-12 21:14 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 4:58 PM Tom Lane <[email protected]> wrote:
> It's reasonable to allow changing *most* GUCs in _PG_init(); let us
> please not break cases that work fine today.
>
> It seems after a bit of reflection that what we ought to do is identify
> the subset of PGC_POSTMASTER GUCs that feed into shared memory sizing,
> mark those with a new GUC flag, and not allow them to be changed after
> shared memory is set up.  (An alternative to a new flag bit is to
> subdivide the PGC_POSTMASTER context into two values, but that seems
> like it'd be far more invasive.  A flag bit needn't be user-visible.)
> Then, what we'd need to do is arrange for shared-preload modules to
> receive their _PG_init call before shared memory creation (when they
> can still twiddle such GUCs) and then another callback later.

I dunno, I feel like this is over-engineered. An extra hook wouldn't
really affect anyone who doesn't need to use it, and I doubt that
adapting to it would create much pain for anyone who is knowledgeable
enough to be writing these kinds of extensions in the first place.
This design is something everyone who adds a GUC from now until the
end of the project needs to know about for the benefit of a tiny
number of extension authors who weren't really going to have a big
problem anyway.

-- 
Robert Haas
EDB: http://www.enterprisedb.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 21:43  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-12 21:43 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> On Tue, Apr 12, 2022 at 4:58 PM Tom Lane <[email protected]> wrote:
>> It seems after a bit of reflection that what we ought to do is identify
>> the subset of PGC_POSTMASTER GUCs that feed into shared memory sizing,
>> mark those with a new GUC flag, and not allow them to be changed after
>> shared memory is set up.

> I dunno, I feel like this is over-engineered.

It probably is.  I'm just offering this as a solution if people want to
insist on a mechanism to prevent unsafe GUC changes.  If we drop the
idea of trying to forcibly prevent extensions from Doing It Wrong,
then we don't need this, only the extra hook.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 21:46  Tom Lane <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-12 21:46 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Nathan Bossart <[email protected]> writes:
> If we allow changing GUCs in _PG_init() and provide another hook where
> MaxBackends will be initialized, do we need to introduce another GUC flag,
> or can we get away with just blocking all GUC changes when the new hook is
> called?  I'm slightly hesitant to add a GUC flag that will need to manually
> maintained.  Wouldn't it be easily forgotten?

On the whole I think Robert's got the right idea: we do not really
need an enforcement mechanism at all.  People who are writing
extensions that do this sort of thing will learn how to do it right.
(It's not like there's not a thousand other things they have to get
right.)

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-12 22:26  Nathan Bossart <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-12 22:26 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 05:46:36PM -0400, Tom Lane wrote:
> Nathan Bossart <[email protected]> writes:
>> If we allow changing GUCs in _PG_init() and provide another hook where
>> MaxBackends will be initialized, do we need to introduce another GUC flag,
>> or can we get away with just blocking all GUC changes when the new hook is
>> called?  I'm slightly hesitant to add a GUC flag that will need to manually
>> maintained.  Wouldn't it be easily forgotten?
> 
> On the whole I think Robert's got the right idea: we do not really
> need an enforcement mechanism at all.  People who are writing
> extensions that do this sort of thing will learn how to do it right.
> (It's not like there's not a thousand other things they have to get
> right.)

Okay.  So maybe we only need the attached patches.  0001 is just 5ecd018
un-reverted, and 0002 is Julien's patch from a few weeks ago with some
minor edits.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com


Attachments:

  [text/x-diff] v3-0001-Fix-comments-about-bgworker-registration-before-M.patch (3.0K, ../../20220412222616.GA2113503@nathanxps13/2-v3-0001-Fix-comments-about-bgworker-registration-before-M.patch)
  download | inline diff:
From 4c5ebca537ffdfbf61079a82b18ce7bc97222c69 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Tue, 12 Apr 2022 14:57:00 -0700
Subject: [PATCH v3 1/2] Fix comments about bgworker registration before
 MaxBackends initialization

Since 6bc8ef0b, InitializeMaxBackends() has used max_worker_processes
instead of adapting MaxBackends to the number of background workers
registered by modules loaded in shared_preload_libraries (at this time,
bgworkers were only static, but gained dynamic capabilities as a matter
of supporting parallel queries meaning that a control cap was
necessary).

Some comments referred to the past registration logic, making them
confusing and incorrect, so fix these.

Some of the out-of-core modules that could be loaded in this path
sometimes like to manipulate dynamically some of the resource-related
GUCs for their own needs, this commit adds a note about that.

Author: Nathan Bossart
Discussion: https://postgr.es/m/20220127181815.GA551692@nathanxps13
---
 src/backend/postmaster/postmaster.c | 10 ++++------
 src/backend/utils/init/postinit.c   |  5 ++---
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 3dcaf8a4a5..d57fefa9a8 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1005,10 +1005,8 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher.  It's probably a good idea to call this
+	 * before any modules had a chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1029,8 +1027,8 @@ PostmasterMain(int argc, char *argv[])
 #endif
 
 	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
+	 * Now that loadable modules have had their chance to alter any GUCs,
+	 * calculate MaxBackends.
 	 */
 	InitializeMaxBackends();
 
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 9139fe895c..a28612b375 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -538,9 +538,8 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called after modules have had the chance to alter GUCs in
+ * shared_preload_libraries and before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
-- 
2.25.1



  [text/x-diff] v3-0002-Add-a-new-shmem_request_hook-hook.patch (2.9K, ../../20220412222616.GA2113503@nathanxps13/3-v3-0002-Add-a-new-shmem_request_hook-hook.patch)
  download | inline diff:
From 862a5b9a8a27995157a8b71b8f3ce09bf6ed17c5 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <[email protected]>
Date: Fri, 25 Mar 2022 11:05:24 +0800
Subject: [PATCH v3 2/2] Add a new shmem_request_hook hook.

Currently, preloaded libraries are expected to request additional
shared memory in _PG_init().  However, it is not unusal for such
requests to depend on MaxBackends, which won't be initialized at
that time.  This introduces a new hook where modules can use
MaxBackends to request additional shared memory.

Author: Julien Rouhaud
Discussion: https://postgr.es/m/20220412210112.GA2065815%40nathanxps13
---
 src/backend/storage/ipc/ipci.c   | 15 +++++++++++++++
 src/include/storage/ipc.h        |  2 ++
 src/tools/pgindent/typedefs.list |  1 +
 3 files changed, 18 insertions(+)

diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 75e456360b..fdd72175d5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
 /* GUCs */
 int			shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
 
+shmem_request_hook_type shmem_request_hook = NULL;
 shmem_startup_hook_type shmem_startup_hook = NULL;
 
 static Size total_addin_request = 0;
@@ -152,6 +153,20 @@ CalculateShmemSize(int *num_semaphores)
 	size = add_size(size, ShmemBackendArraySize());
 #endif
 
+	/*
+	 * Final chance for modules to request additional shared memory.
+	 *
+	 * Ordinarily, modules call RequestAddinShmemSpace() in _PG_init(), but it
+	 * is not unusal for such requests to depend on MaxBackends, which won't be
+	 * initialized at that time.  This hook provides an opportunity for modules
+	 * to use MaxBackends when requesting shared memory.
+	 */
+	if (addin_request_allowed && shmem_request_hook)
+	{
+		Assert(MaxBackends > 0);
+		shmem_request_hook();
+	}
+
 	/* freeze the addin request size and include it */
 	addin_request_allowed = false;
 	size = add_size(size, total_addin_request);
diff --git a/src/include/storage/ipc.h b/src/include/storage/ipc.h
index fade4dbe63..5f2c6683db 100644
--- a/src/include/storage/ipc.h
+++ b/src/include/storage/ipc.h
@@ -19,6 +19,7 @@
 #define IPC_H
 
 typedef void (*pg_on_exit_callback) (int code, Datum arg);
+typedef void (*shmem_request_hook_type) (void);
 typedef void (*shmem_startup_hook_type) (void);
 
 /*----------
@@ -75,6 +76,7 @@ extern void on_exit_reset(void);
 extern void check_on_shmem_exit_lists_are_empty(void);
 
 /* ipci.c */
+extern PGDLLIMPORT shmem_request_hook_type shmem_request_hook;
 extern PGDLLIMPORT shmem_startup_hook_type shmem_startup_hook;
 
 extern Size CalculateShmemSize(int *num_semaphores);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 87ee7bf866..71a97654e0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3549,6 +3549,7 @@ shm_mq_result
 shm_toc
 shm_toc_entry
 shm_toc_estimator
+shmem_request_hook_type
 shmem_startup_hook_type
 sig_atomic_t
 sigjmp_buf
-- 
2.25.1



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 07:54  Michael Paquier <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Michael Paquier @ 2022-04-13 07:54 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 05:43:17PM -0400, Tom Lane wrote:
> It probably is.  I'm just offering this as a solution if people want to
> insist on a mechanism to prevent unsafe GUC changes.  If we drop the
> idea of trying to forcibly prevent extensions from Doing It Wrong,
> then we don't need this, only the extra hook.

FWIW, I'd be fine with a simple solution like what Julien has been
proposing with the extra hook, rather than a GUC to enforce all that.
That may be nice in the long-run, but the potential benefits may not
be completely clear, either, after a closer read of this thread.
--
Michael


Attachments:

  [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
  download

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 13:16  Robert Haas <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-13 13:16 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 12, 2022 at 6:26 PM Nathan Bossart <[email protected]> wrote:
> Okay.  So maybe we only need the attached patches.  0001 is just 5ecd018
> un-reverted, and 0002 is Julien's patch from a few weeks ago with some
> minor edits.

Hmm. I suppose I was thinking that we'd go the other way around: move
the call to InitializeMaxBackends() earlier, as proposed previously,
and add a hook to allow extensions to get control before that point.
The reason that I like that approach is that I suspect that it's more
common for extensions to want to size shared memory data structures
than it is for them to want to change GUCs, and therefore I thought
that approach would fix things for the most amount of people with the
least amount of code change. But it seems like maybe Tom thinks I'm
incorrect about the relative frequency of those things, so I don't
know.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 13:27  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-13 13:27 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> Hmm. I suppose I was thinking that we'd go the other way around: move
> the call to InitializeMaxBackends() earlier, as proposed previously,
> and add a hook to allow extensions to get control before that point.
> The reason that I like that approach is that I suspect that it's more
> common for extensions to want to size shared memory data structures
> than it is for them to want to change GUCs, and therefore I thought
> that approach would fix things for the most amount of people with the
> least amount of code change. But it seems like maybe Tom thinks I'm
> incorrect about the relative frequency of those things, so I don't
> know.

Maybe I'm missing something, but I figured we'd keep the _PG_init
calls where they are to minimize side-effects, and then add an optional
hook just before/after shared memory size is determined.  Cases that
work well now continue to work well, and cases that don't work so
well can be fixed by making use of the hook.  In particular you
can still do RequestAddinShmemSpace() in _PG_init as long as the
request size doesn't depend on factors that other extensions might
change.  If you're doing something funny then you might need to
postpone RequestAddinShmemSpace till the new hook call.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 14:22  Robert Haas <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-13 14:22 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, Apr 13, 2022 at 9:27 AM Tom Lane <[email protected]> wrote:
> Robert Haas <[email protected]> writes:
> > Hmm. I suppose I was thinking that we'd go the other way around: move
> > the call to InitializeMaxBackends() earlier, as proposed previously,
> > and add a hook to allow extensions to get control before that point.
> > The reason that I like that approach is that I suspect that it's more
> > common for extensions to want to size shared memory data structures
> > than it is for them to want to change GUCs, and therefore I thought
> > that approach would fix things for the most amount of people with the
> > least amount of code change. But it seems like maybe Tom thinks I'm
> > incorrect about the relative frequency of those things, so I don't
> > know.
>
> Maybe I'm missing something, but I figured we'd keep the _PG_init
> calls where they are to minimize side-effects, and then add an optional
> hook just before/after shared memory size is determined.  Cases that
> work well now continue to work well, and cases that don't work so
> well can be fixed by making use of the hook.  In particular you
> can still do RequestAddinShmemSpace() in _PG_init as long as the
> request size doesn't depend on factors that other extensions might
> change.  If you're doing something funny then you might need to
> postpone RequestAddinShmemSpace till the new hook call.

I wouldn't say that approach is wrong. I think we're just prioritizing
different things. I think that the pattern of wanting to request
shared memory in _PG_init() is common, and there are lots of
extensions that already do it that way, and if we pursue this plan,
then all of those extensions really ought to be updated to use the new
hook, including pg_prewarm and pg_stat_statements. They may continue
to work if they don't, but they'll be doing something that is not
really best practice any more but will happen to work until you make
your request for shared memory dependent on the wrong thing, or until
you load up another module at the same time that tweaks some GUC upon
which your calculation depends (imagine an autotuner that adjusts
pg_stat_statements.max at startup time). So I think we'll be
inflicting subtle bugs on extension authors that will never really get
fully sorted out.

Now, on the other hand, I think that the pattern of changing GUCs in
_PG_init() is comparatively uncommon. I don't believe that we have any
in-core code that does it, and the only out-of-core code that does to
my knowledge is Citus. So if we subtly change the best practices
around setting GUCs in _PG_init() hooks, I think that's going to
affect a vastly smaller number of extensions. Either way, we're
changing best practices without really creating a hard break, but I'd
rather move the wood for the small number of extensions that are
tweaking GUCs in _PG_init() than the larger number of extensions (or
so I suppose) that are requesting shared memory there.

Now, we could also decide to create a hard compatibility break to
force extension authors to update. I think that could be done in one
of two ways. One possibility is to refuse SetConfigOption() in
_PG_init(); anyone who wants to do that has to use the new set-a-GUC
hook we simultaneously add. The other is to refuse
RequestAddinShmemSpace() and RequestNamedLWLockTranche() in
_PG_init(); anyone who wants to do that has to use the new
request-shmem-resources hook that we simultaneously add. I tend to
think that the latter is a slightly more principled option, because I
doubt that refusing SetConfigOption() is a bullet-proof defense
against, well, I don't know, anything really. You can hackily modify
GUCs by other means, and you can do bad math with things that aren't
GUCs. I am however pretty sure that if we refuse to provide shmem
space unless you request it from a new add-shmem hook, that will be
pretty water-tight, and it doesn't seem like it should be a big issue
for extension authors to move that code to some new hook we invent.

So in the end I see basically four possibilities here:

A. No hard compatibility break, but invent a set-a-GUC hook that runs earlier.
B. No hard compatibility break, but invent a request-shmem hook that runs later.
C. Invent a set-a-GUC hook that runs earlier AND refuse to set GUCs in _PG_init.
D. Invent a request-shmem hook that runs later AND refuse to accept
shmem requests in _PG_init.

My preferred options are A or D for the reasons explained above. It
seems like you prefer B.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 14:43  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-13 14:43 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> I wouldn't say that approach is wrong. I think we're just prioritizing
> different things. I think that the pattern of wanting to request
> shared memory in _PG_init() is common, and there are lots of
> extensions that already do it that way, and if we pursue this plan,
> then all of those extensions really ought to be updated to use the new
> hook, including pg_prewarm and pg_stat_statements. They may continue
> to work if they don't, but they'll be doing something that is not
> really best practice any more but will happen to work until you make
> your request for shared memory dependent on the wrong thing, or until
> you load up another module at the same time that tweaks some GUC upon
> which your calculation depends (imagine an autotuner that adjusts
> pg_stat_statements.max at startup time). So I think we'll be
> inflicting subtle bugs on extension authors that will never really get
> fully sorted out.

Yeah, there is something to be said for preventing subtle interactions
between extensions.

> So in the end I see basically four possibilities here:

> A. No hard compatibility break, but invent a set-a-GUC hook that runs earlier.
> B. No hard compatibility break, but invent a request-shmem hook that runs later.
> C. Invent a set-a-GUC hook that runs earlier AND refuse to set GUCs in _PG_init.
> D. Invent a request-shmem hook that runs later AND refuse to accept
> shmem requests in _PG_init.

I dislike A for the reason I already stated: _PG_init should be the
first code we run in an extension.  Not doing that is just too hacky
for words.  While B breaks the least stuff in the short run, I agree
that it leaves extension authors on the hook to avoid unpleasant
interactions, and that the only way they can be sure to do so is
to move their shmem requests to the new hook.  So if we're willing
to accept a hard compatibility break to prevent such bugs, then
I too prefer D to C.  What I'm not quite convinced about is whether
the problem is big enough to warrant a compatibility break.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 15:25  Robert Haas <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-13 15:25 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, Apr 13, 2022 at 10:43 AM Tom Lane <[email protected]> wrote:
> Yeah, there is something to be said for preventing subtle interactions
> between extensions.
>
> > So in the end I see basically four possibilities here:
>
> > A. No hard compatibility break, but invent a set-a-GUC hook that runs earlier.
> > B. No hard compatibility break, but invent a request-shmem hook that runs later.
> > C. Invent a set-a-GUC hook that runs earlier AND refuse to set GUCs in _PG_init.
> > D. Invent a request-shmem hook that runs later AND refuse to accept
> > shmem requests in _PG_init.
>
> I dislike A for the reason I already stated: _PG_init should be the
> first code we run in an extension.  Not doing that is just too hacky
> for words.

That seems like a fair position, even if I don't really understand why
it would be as bad as "too hacky for words."

> While B breaks the least stuff in the short run, I agree
> that it leaves extension authors on the hook to avoid unpleasant
> interactions, and that the only way they can be sure to do so is
> to move their shmem requests to the new hook.  So if we're willing
> to accept a hard compatibility break to prevent such bugs, then
> I too prefer D to C.  What I'm not quite convinced about is whether
> the problem is big enough to warrant a compatibility break.

It's sort of a philosophical question. How do you measure the size of
such a problem? What units do you even use for such a size
measurement? How big does it have to be to justify a compatibility
break? Presumably it depends on the size of the compatibility break,
which is also not subject to any sort of objective measurement.

I guess my feeling about this is that _PG_init() is sort of a grab
bag, and that's not very good. You're supposed to register new GUCs
there, and request shared memory space, and install any hook functions
you want to use, and maybe some other stuff. But why is it appropriate
for all of those things to happen at the same time? I think it pretty
clearly isn't, and that's why you see _PG_init() functions testing
process_shared_preload_libraries_in_progress, and why
RequestAddinShmemSpace() is a no-op if it's not the right time for
such things. To me, all of that is just a sign that the system is
badly designed. Imagine if someone proposed to make XLogInsert() or
SetConfigOption() or LockBuffer() sometimes just return without doing
anything. We would just call those functions in places where those
actions weren't appropriate, and the function would just do nothing
silently without signalling an error. Surely such a proposal would be
shot down as an awful idea, and the only reason the _PG_init() case is
any different is because it's not new. But it doesn't seem to me that
it's any better of an idea here than it would be there.

And under proposal D we'd actually be fixing that, because we'd have a
hook that is the right place to request shared memory and we'd
complain if those functions were called from anywhere else.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 15:39  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-13 15:39 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> I guess my feeling about this is that _PG_init() is sort of a grab
> bag, and that's not very good. You're supposed to register new GUCs
> there, and request shared memory space, and install any hook functions
> you want to use, and maybe some other stuff. But why is it appropriate
> for all of those things to happen at the same time? I think it pretty
> clearly isn't, and that's why you see _PG_init() functions testing
> process_shared_preload_libraries_in_progress, and why
> RequestAddinShmemSpace() is a no-op if it's not the right time for
> such things. To me, all of that is just a sign that the system is
> badly designed.

Hmm, that's a good point.  If we can replace "RequestAddinShmemSpace
does nothing if called at the wrong time" with "RequestAddinShmemSpace
throws error if called at the wrong time", that seems like a pretty
clear improvement in robustness.

Is there anything else typically done in _PG_init that has to be
conditional on process_shared_preload_libraries_in_progress?  I recall
something about reserving LWLocks, which probably should get the same
treatment.  If we can get to a point where _PG_init shouldn't need to
care about process_shared_preload_libraries_in_progress, because all
the stuff that would care is moved to this new hook, then that would
be very clear cleanup.

			regards, tom lane

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 15:50  Robert Haas <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-13 15:50 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, Apr 13, 2022 at 11:39 AM Tom Lane <[email protected]> wrote:
> Is there anything else typically done in _PG_init that has to be
> conditional on process_shared_preload_libraries_in_progress?  I recall
> something about reserving LWLocks, which probably should get the same
> treatment.  If we can get to a point where _PG_init shouldn't need to
> care about process_shared_preload_libraries_in_progress, because all
> the stuff that would care is moved to this new hook, then that would
> be very clear cleanup.

What's a little wonky right now is that it's fairly common for
extensions to just return straight-off without doing *anything at all*
if !process_shared_preload_libraries_in_progress. See
pg_stat_statements for example. It seems kind of strange to me to make
registering GUCs dependent on
process_shared_preload_libraries_in_progress; why shouldn't that just
happen always? It's understandable that we don't want to install the
hook functions if we're not being loaded from shared_preload_libaries,
though.

It may be too much to hope that we're going to completely get rid of
process_shared_preload_libraries_in_progress tests. But even given
that, I think having a request-shared-mem hook makes sense and would
make things cleaner than they are today. Maybe in the future we'll end
up with other hooks as well, like a "this is the place to register
GUCs" hook and a perhaps a "this is the place to register you custom
rmgr" hook. I'm not really sure that we need those and I don't want to
make things complicated just for the heck of it, but it's not shocking
that different operations need to happen at different times in the
startup sequence, and I don't think we should be afraid to split up
the monolithic _PG_init() into as many separate things as are required
to make it work right.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 16:05  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-13 16:05 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> What's a little wonky right now is that it's fairly common for
> extensions to just return straight-off without doing *anything at all*
> if !process_shared_preload_libraries_in_progress. See
> pg_stat_statements for example. It seems kind of strange to me to make
> registering GUCs dependent on
> process_shared_preload_libraries_in_progress; why shouldn't that just
> happen always? It's understandable that we don't want to install the
> hook functions if we're not being loaded from shared_preload_libaries,
> though.

Yeah, I was just investigating that.  The problem that pg_stat_statements
has is that it wants to create PGC_POSTMASTER GUCs, which is something
that guc.c specifically forbids:

    /*
     * Only allow custom PGC_POSTMASTER variables to be created during shared
     * library preload; any later than that, we can't ensure that the value
     * doesn't change after startup.  This is a fatal elog if it happens; just
     * erroring out isn't safe because we don't know what the calling loadable
     * module might already have hooked into.
     */
    if (context == PGC_POSTMASTER &&
        !process_shared_preload_libraries_in_progress)
        elog(FATAL, "cannot create PGC_POSTMASTER variables after startup");

The key reason guc.c does that is that if it allowed the case, then there
would be no guarantee that a "PGC_POSTMASTER" GUC has the same value in
every backend of the cluster, which'd likely break most use-cases for
such a GUC (it'd certainly break pg_stat_statements, which assumes that
the local setting of that GUC reflects the size of its shmem area).
Perhaps we can improve on that situation with some more thought, but
I'm not very clear on how.

> It may be too much to hope that we're going to completely get rid of
> process_shared_preload_libraries_in_progress tests.

Perhaps, but this is definitely an area that could stand to have some
serious design thought put into it.  You're quite right that it's
largely a mess today, but I think proposals like "forbid setting GUCs
during _PG_init" would add to the mess rather than clean things up.

			regards, tom lane

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-13 18:30  Nathan Bossart <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-13 18:30 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, Apr 13, 2022 at 12:05:08PM -0400, Tom Lane wrote:
> Robert Haas <[email protected]> writes:
>> It may be too much to hope that we're going to completely get rid of
>> process_shared_preload_libraries_in_progress tests.
> 
> Perhaps, but this is definitely an area that could stand to have some
> serious design thought put into it.  You're quite right that it's
> largely a mess today, but I think proposals like "forbid setting GUCs
> during _PG_init" would add to the mess rather than clean things up.

+1.  The simplest design might be to just make a separate preload hook.
_PG_init() would install hooks, and then this preload hook would do
anything that needs to happen when the library is loaded via s_p_l.
However, I think we still want a new shmem request hook where MaxBackends
is initialized.

If we do move forward with the shmem request hook, do we want to disallow
shmem requests anywhere else, or should we just leave it up to extension
authors to do the right thing?  Many shmem requests in _PG_init() are
probably okay unless they depend on MaxBackends or another GUC that someone
might change.  Given that, I think I currently prefer the latter (option B
from upthread).

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-14 05:50  Julien Rouhaud <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-04-14 05:50 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, Apr 13, 2022 at 11:30:40AM -0700, Nathan Bossart wrote:
> On Wed, Apr 13, 2022 at 12:05:08PM -0400, Tom Lane wrote:
> > Robert Haas <[email protected]> writes:
> >> It may be too much to hope that we're going to completely get rid of
> >> process_shared_preload_libraries_in_progress tests.
> > 
> > Perhaps, but this is definitely an area that could stand to have some
> > serious design thought put into it.  You're quite right that it's
> > largely a mess today, but I think proposals like "forbid setting GUCs
> > during _PG_init" would add to the mess rather than clean things up.
> 
> +1.  The simplest design might be to just make a separate preload hook.
> _PG_init() would install hooks, and then this preload hook would do
> anything that needs to happen when the library is loaded via s_p_l.
> However, I think we still want a new shmem request hook where MaxBackends
> is initialized.

Yeah this one seems needed no matter what.

> If we do move forward with the shmem request hook, do we want to disallow
> shmem requests anywhere else, or should we just leave it up to extension
> authors to do the right thing?  Many shmem requests in _PG_init() are
> probably okay unless they depend on MaxBackends or another GUC that someone
> might change.  Given that, I think I currently prefer the latter (option B
> from upthread).

I'd be in favor of a hard break.  There are already multiple extensions that
relies on non final value of GUCs to size their shmem request.  And as an
extension author it can be hard to realize that, as those extensions work just
fine until someone wants to try it with some other extension that changes some
GUC.  Forcing shmem request in a new hook will make sure that it's *always*
correct, and that only requires very minimal work on the extension side.





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-14 16:22  Nathan Bossart <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-14 16:22 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Tom Lane <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Thu, Apr 14, 2022 at 01:50:24PM +0800, Julien Rouhaud wrote:
> On Wed, Apr 13, 2022 at 11:30:40AM -0700, Nathan Bossart wrote:
>> If we do move forward with the shmem request hook, do we want to disallow
>> shmem requests anywhere else, or should we just leave it up to extension
>> authors to do the right thing?  Many shmem requests in _PG_init() are
>> probably okay unless they depend on MaxBackends or another GUC that someone
>> might change.  Given that, I think I currently prefer the latter (option B
>> from upthread).
> 
> I'd be in favor of a hard break.  There are already multiple extensions that
> relies on non final value of GUCs to size their shmem request.  And as an
> extension author it can be hard to realize that, as those extensions work just
> fine until someone wants to try it with some other extension that changes some
> GUC.  Forcing shmem request in a new hook will make sure that it's *always*
> correct, and that only requires very minimal work on the extension side.

Yeah, this is a good point.  If we're okay with breaking existing
extensions like this, I will work on a patch.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-14 16:32  Robert Haas <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-04-14 16:32 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Thu, Apr 14, 2022 at 12:22 PM Nathan Bossart
<[email protected]> wrote:
> > I'd be in favor of a hard break.  There are already multiple extensions that
> > relies on non final value of GUCs to size their shmem request.  And as an
> > extension author it can be hard to realize that, as those extensions work just
> > fine until someone wants to try it with some other extension that changes some
> > GUC.  Forcing shmem request in a new hook will make sure that it's *always*
> > correct, and that only requires very minimal work on the extension side.
>
> Yeah, this is a good point.  If we're okay with breaking existing
> extensions like this, I will work on a patch.

I tend to think it's a good idea. It's not great to inconvenience
extension authors, but I think that with the kind of design we are
talking about here it will be pretty straightforward to see how to
update your extension: move the code that request shmem resources into
the new hook. If you want to be compatible with older PostgreSQL
releases by conditional compilation, conditionally call that function
from _PG_init() when PG_VERSION_NUM < whatever.

Compare that with the current situation, where things seem to mostly
work but sometimes don't, and it's not quite clear what to do about
it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-14 16:39  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-14 16:39 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> On Thu, Apr 14, 2022 at 12:22 PM Nathan Bossart
> <[email protected]> wrote:
>>> I'd be in favor of a hard break.

>> Yeah, this is a good point.  If we're okay with breaking existing
>> extensions like this, I will work on a patch.

> I tend to think it's a good idea.

I've come around to that view as well.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-18 22:52  Nathan Bossart <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-04-18 22:52 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Thu, Apr 14, 2022 at 12:39:46PM -0400, Tom Lane wrote:
> Robert Haas <[email protected]> writes:
>> On Thu, Apr 14, 2022 at 12:22 PM Nathan Bossart
>> <[email protected]> wrote:
>>>> I'd be in favor of a hard break.
> 
>>> Yeah, this is a good point.  If we're okay with breaking existing
>>> extensions like this, I will work on a patch.
> 
>> I tend to think it's a good idea.
> 
> I've come around to that view as well.

Here is a new patch set that introduces the aforementioned "hard break."

I noticed that requests for more LWLocks follow a similar pattern as
regular shared memory requests, and I figured that we would want to do
something similar for those, but I wasn't sure exactly how to proceed.  I
saw two options: 1) use shmem_request_hook for both regular requests and
LWLock requests or 2) introduce an lwlock_request_hook.  My instinct was
that option 1 was preferable, but AFAICT this requires introducing a new
external variable for inspecting whether the request is made at a valid
time.  This would be similar to
process_shared_preload_libraries_in_progress, which I believe means a
determined extension author could easily hack around the request
restrictions.  I thought option 2 added too much machinery to work around
this problem.  For now, I haven't made any changes for LWLock requests.
What are your thoughts?

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com


Attachments:

  [text/x-diff] v4-0001-Fix-comments-about-bgworker-registration-before-M.patch (3.0K, ../../20220418225244.GA2387491@nathanxps13/2-v4-0001-Fix-comments-about-bgworker-registration-before-M.patch)
  download | inline diff:
From 55af4473761a8bcfe8c2ff940a5df4625a40653a Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Tue, 12 Apr 2022 14:57:00 -0700
Subject: [PATCH v4 1/2] Fix comments about bgworker registration before
 MaxBackends initialization

Since 6bc8ef0b, InitializeMaxBackends() has used max_worker_processes
instead of adapting MaxBackends to the number of background workers
registered by modules loaded in shared_preload_libraries (at this time,
bgworkers were only static, but gained dynamic capabilities as a matter
of supporting parallel queries meaning that a control cap was
necessary).

Some comments referred to the past registration logic, making them
confusing and incorrect, so fix these.

Some of the out-of-core modules that could be loaded in this path
sometimes like to manipulate dynamically some of the resource-related
GUCs for their own needs, this commit adds a note about that.

Author: Nathan Bossart
Discussion: https://postgr.es/m/20220127181815.GA551692@nathanxps13
---
 src/backend/postmaster/postmaster.c | 10 ++++------
 src/backend/utils/init/postinit.c   |  5 ++---
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 964a56dec4..ce4007bb2c 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1005,10 +1005,8 @@ PostmasterMain(int argc, char *argv[])
 	LocalProcessControlFile(false);
 
 	/*
-	 * Register the apply launcher.  Since it registers a background worker,
-	 * it needs to be called before InitializeMaxBackends(), and it's probably
-	 * a good idea to call it before any modules had chance to take the
-	 * background worker slots.
+	 * Register the apply launcher.  It's probably a good idea to call this
+	 * before any modules had a chance to take the background worker slots.
 	 */
 	ApplyLauncherRegister();
 
@@ -1029,8 +1027,8 @@ PostmasterMain(int argc, char *argv[])
 #endif
 
 	/*
-	 * Now that loadable modules have had their chance to register background
-	 * workers, calculate MaxBackends.
+	 * Now that loadable modules have had their chance to alter any GUCs,
+	 * calculate MaxBackends.
 	 */
 	InitializeMaxBackends();
 
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 9139fe895c..a28612b375 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -538,9 +538,8 @@ pg_split_opts(char **argv, int *argcp, const char *optstr)
 /*
  * Initialize MaxBackends value from config options.
  *
- * This must be called after modules have had the chance to register background
- * workers in shared_preload_libraries, and before shared memory size is
- * determined.
+ * This must be called after modules have had the chance to alter GUCs in
+ * shared_preload_libraries and before shared memory size is determined.
  *
  * Note that in EXEC_BACKEND environment, the value is passed down from
  * postmaster to subprocesses via BackendParameters in SubPostmasterMain; only
-- 
2.25.1



  [text/x-diff] v4-0002-Add-a-new-shmem_request_hook-hook.patch (10.4K, ../../20220418225244.GA2387491@nathanxps13/3-v4-0002-Add-a-new-shmem_request_hook-hook.patch)
  download | inline diff:
From 135c1363d0ff2aaa1886d4ed59016c3b610c375a Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Mon, 18 Apr 2022 15:25:37 -0700
Subject: [PATCH v4 2/2] Add a new shmem_request_hook hook.

Currently, preloaded libraries are expected to request additional
shared memory in _PG_init().  However, it is not unusal for such
requests to depend on MaxBackends, which won't be initialized at
that time.  Such requests could also depend on GUCs that other
modules might change.  This introduces a new hook where modules can
safely use MaxBackends and GUCs to request additional shared
memory.

Furthermore, this change restricts shared memory requests by
preloaded libraries to this hook.  Previously, libraries could
request additional shared memory until the size of the main shared
memory segment was calculated.  Besides decoupling a common library
task from _PG_init(), this ensures that shared memory requests are
only allowed when MaxBackends is initialized and GUCs should not be
changed.  Unlike before, we no longer silently ignore requests
received at invalid times.  Instead, we ERROR if someone tries to
request additional shared memory outside of the hook.

Authors: Julien Rouhaud, Nathan Bossart
Discussion: https://postgr.es/m/20220412210112.GA2065815%40nathanxps13
---
 contrib/pg_prewarm/autoprewarm.c              | 27 ++++++++++++-
 .../pg_stat_statements/pg_stat_statements.c   | 27 +++++++++----
 src/backend/postmaster/postmaster.c           |  5 +++
 src/backend/storage/ipc/ipci.c                | 39 +++++++++++++------
 src/include/storage/ipc.h                     |  2 +
 src/include/storage/shmem.h                   |  1 +
 src/tools/pgindent/typedefs.list              |  1 +
 7 files changed, 81 insertions(+), 21 deletions(-)

diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c
index 45e012a63a..14345d060a 100644
--- a/contrib/pg_prewarm/autoprewarm.c
+++ b/contrib/pg_prewarm/autoprewarm.c
@@ -83,6 +83,7 @@ typedef struct AutoPrewarmSharedState
 } AutoPrewarmSharedState;
 
 void		_PG_init(void);
+void		_PG_fini(void);
 void		autoprewarm_main(Datum main_arg);
 void		autoprewarm_database_main(Datum main_arg);
 
@@ -96,6 +97,8 @@ static void apw_start_database_worker(void);
 static bool apw_init_shmem(void);
 static void apw_detach_shmem(int code, Datum arg);
 static int	apw_compare_blockinfo(const void *p, const void *q);
+static void autoprewarm_shmem_request(void);
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
 
 /* Pointer to shared-memory state. */
 static AutoPrewarmSharedState *apw_state = NULL;
@@ -139,13 +142,35 @@ _PG_init(void)
 
 	MarkGUCPrefixReserved("pg_prewarm");
 
-	RequestAddinShmemSpace(MAXALIGN(sizeof(AutoPrewarmSharedState)));
+	prev_shmem_request_hook = shmem_request_hook;
+	shmem_request_hook = autoprewarm_shmem_request;
 
 	/* Register autoprewarm worker, if enabled. */
 	if (autoprewarm)
 		apw_start_leader_worker();
 }
 
+/*
+ * Module unload callback.
+ */
+void
+_PG_fini(void)
+{
+	shmem_request_hook = prev_shmem_request_hook;
+}
+
+/*
+ * Requests any additional shared memory required for autoprewarm.
+ */
+static void
+autoprewarm_shmem_request(void)
+{
+	if (prev_shmem_request_hook)
+		prev_shmem_request_hook();
+
+	RequestAddinShmemSpace(MAXALIGN(sizeof(AutoPrewarmSharedState)));
+}
+
 /*
  * Main entry point for the leader autoprewarm process.  Per-database workers
  * have a separate entry point.
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index df2ce63790..87b75d779e 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -252,6 +252,7 @@ static int	exec_nested_level = 0;
 static int	plan_nested_level = 0;
 
 /* Saved hook values in case of unload */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
 static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
 static post_parse_analyze_hook_type prev_post_parse_analyze_hook = NULL;
 static planner_hook_type prev_planner_hook = NULL;
@@ -317,6 +318,7 @@ PG_FUNCTION_INFO_V1(pg_stat_statements_1_10);
 PG_FUNCTION_INFO_V1(pg_stat_statements);
 PG_FUNCTION_INFO_V1(pg_stat_statements_info);
 
+static void pgss_shmem_request(void);
 static void pgss_shmem_startup(void);
 static void pgss_shmem_shutdown(int code, Datum arg);
 static void pgss_post_parse_analyze(ParseState *pstate, Query *query,
@@ -452,17 +454,11 @@ _PG_init(void)
 
 	MarkGUCPrefixReserved("pg_stat_statements");
 
-	/*
-	 * Request additional shared resources.  (These are no-ops if we're not in
-	 * the postmaster process.)  We'll allocate or attach to the shared
-	 * resources in pgss_shmem_startup().
-	 */
-	RequestAddinShmemSpace(pgss_memsize());
-	RequestNamedLWLockTranche("pg_stat_statements", 1);
-
 	/*
 	 * Install hooks.
 	 */
+	prev_shmem_request_hook = shmem_request_hook;
+	shmem_request_hook = pgss_shmem_request;
 	prev_shmem_startup_hook = shmem_startup_hook;
 	shmem_startup_hook = pgss_shmem_startup;
 	prev_post_parse_analyze_hook = post_parse_analyze_hook;
@@ -488,6 +484,7 @@ void
 _PG_fini(void)
 {
 	/* Uninstall hooks. */
+	shmem_request_hook = prev_shmem_request_hook;
 	shmem_startup_hook = prev_shmem_startup_hook;
 	post_parse_analyze_hook = prev_post_parse_analyze_hook;
 	planner_hook = prev_planner_hook;
@@ -498,6 +495,20 @@ _PG_fini(void)
 	ProcessUtility_hook = prev_ProcessUtility;
 }
 
+/*
+ * shmem_request hook: request additional shared resources.  We'll allocate or
+ * attach to the shared resources in pgss_shmem_startup().
+ */
+static void
+pgss_shmem_request(void)
+{
+	if (prev_shmem_request_hook)
+		prev_shmem_request_hook();
+
+	RequestAddinShmemSpace(pgss_memsize());
+	RequestNamedLWLockTranche("pg_stat_statements", 1);
+}
+
 /*
  * shmem_startup hook: allocate or attach to shared memory,
  * then load any pre-existing statistics from file.
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index ce4007bb2c..57663ddc6a 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1032,6 +1032,11 @@ PostmasterMain(int argc, char *argv[])
 	 */
 	InitializeMaxBackends();
 
+	/*
+	 * Give preloaded libraries a chance to request additional shared memory.
+	 */
+	ProcessShmemRequests();
+
 	/*
 	 * Now that loadable modules have had their chance to request additional
 	 * shared memory, determine the value of any runtime-computed GUCs that
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 75e456360b..117aacba7d 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,28 +52,45 @@
 /* GUCs */
 int			shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
 
+shmem_request_hook_type shmem_request_hook = NULL;
 shmem_startup_hook_type shmem_startup_hook = NULL;
 
 static Size total_addin_request = 0;
-static bool addin_request_allowed = true;
+static bool addin_request_allowed = false;
 
+/*
+ * ProcessShmemRequests
+ *
+ * Calls to RequestAddinShmemSpace() by preloaded libraries are only allowed in
+ * the shmem_request_hook.
+ */
+void
+ProcessShmemRequests(void)
+{
+	Assert(MaxBackends > 0);
+
+	addin_request_allowed = true;
+
+	if (shmem_request_hook)
+		shmem_request_hook();
+
+	addin_request_allowed = false;
+}
 
 /*
  * RequestAddinShmemSpace
  *		Request that extra shmem space be allocated for use by
  *		a loadable module.
  *
- * This is only useful if called from the _PG_init hook of a library that
- * is loaded into the postmaster via shared_preload_libraries.  Once
- * shared memory has been allocated, calls will be ignored.  (We could
- * raise an error, but it seems better to make it a no-op, so that
- * libraries containing such calls can be reloaded if needed.)
+ * This may only be called via the shmem_request_hook of a library that is
+ * loaded into the postmaster via shared_preload_libraries.  Calls from
+ * elsewhere will ERROR.
  */
 void
 RequestAddinShmemSpace(Size size)
 {
 	if (IsUnderPostmaster || !addin_request_allowed)
-		return;					/* too late */
+		elog(ERROR, "cannot request additional shared memory outside shmem_request_hook");
 	total_addin_request = add_size(total_addin_request, size);
 }
 
@@ -83,9 +100,6 @@ RequestAddinShmemSpace(Size size)
  *
  * If num_semaphores is not NULL, it will be set to the number of semaphores
  * required.
- *
- * Note that this function freezes the additional shared memory request size
- * from loadable modules.
  */
 Size
 CalculateShmemSize(int *num_semaphores)
@@ -93,6 +107,8 @@ CalculateShmemSize(int *num_semaphores)
 	Size		size;
 	int			numSemas;
 
+	Assert(!addin_request_allowed);
+
 	/* Compute number of semaphores we'll need */
 	numSemas = ProcGlobalSemas();
 	numSemas += SpinlockSemas();
@@ -152,8 +168,7 @@ CalculateShmemSize(int *num_semaphores)
 	size = add_size(size, ShmemBackendArraySize());
 #endif
 
-	/* freeze the addin request size and include it */
-	addin_request_allowed = false;
+	/* include additional requested shmem from preload libraries */
 	size = add_size(size, total_addin_request);
 
 	/* might as well round it off to a multiple of a typical page size */
diff --git a/src/include/storage/ipc.h b/src/include/storage/ipc.h
index fade4dbe63..5f2c6683db 100644
--- a/src/include/storage/ipc.h
+++ b/src/include/storage/ipc.h
@@ -19,6 +19,7 @@
 #define IPC_H
 
 typedef void (*pg_on_exit_callback) (int code, Datum arg);
+typedef void (*shmem_request_hook_type) (void);
 typedef void (*shmem_startup_hook_type) (void);
 
 /*----------
@@ -75,6 +76,7 @@ extern void on_exit_reset(void);
 extern void check_on_shmem_exit_lists_are_empty(void);
 
 /* ipci.c */
+extern PGDLLIMPORT shmem_request_hook_type shmem_request_hook;
 extern PGDLLIMPORT shmem_startup_hook_type shmem_startup_hook;
 
 extern Size CalculateShmemSize(int *num_semaphores);
diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index de9e7c6e73..bdb0acfedf 100644
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -46,6 +46,7 @@ extern Size add_size(Size s1, Size s2);
 extern Size mul_size(Size s1, Size s2);
 
 /* ipci.c */
+extern void ProcessShmemRequests(void);
 extern void RequestAddinShmemSpace(Size size);
 
 /* size constants for the shmem index table */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 87ee7bf866..71a97654e0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3549,6 +3549,7 @@ shm_mq_result
 shm_toc
 shm_toc_entry
 shm_toc_estimator
+shmem_request_hook_type
 shmem_startup_hook_type
 sig_atomic_t
 sigjmp_buf
-- 
2.25.1



^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-18 23:33  Tom Lane <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-04-18 23:33 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Nathan Bossart <[email protected]> writes:
> I noticed that requests for more LWLocks follow a similar pattern as
> regular shared memory requests, and I figured that we would want to do
> something similar for those, but I wasn't sure exactly how to proceed.  I
> saw two options: 1) use shmem_request_hook for both regular requests and
> LWLock requests or 2) introduce an lwlock_request_hook.  My instinct was
> that option 1 was preferable,

Yeah, I agree, which says that maybe the hook name needs to be something
else (not that I have a good proposal).

> but AFAICT this requires introducing a new
> external variable for inspecting whether the request is made at a valid
> time.

Uh, why?  It'd be the core code's responsibility to place the hook
call at a point where you could do both.

			regards, tom lane






^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-04-19 09:49  Julien Rouhaud <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 2 replies; 94+ messages in thread

From: Julien Rouhaud @ 2022-04-19 09:49 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Hi,

On Mon, Apr 18, 2022 at 08:17:04PM -0400, Tom Lane wrote:
> Nathan Bossart <[email protected]> writes:
> > I'm looking for a clean way to ERROR if someone attempts to call
> > RequestAddinShmemSpace() or RequestNamedLWLockTranche() outside of the
> > hook.  Currently, we are using static variables in ipci.c and lwlock.c to
> > silently ignore invalid requests.  I could add a new 'extern bool' called
> > 'process_shmem_requests_in_progress', but extensions could easily hack
> > around that to allow requests in _PG_init().  Maybe I am overthinking all
> > this and that is good enough.
> 
> If they do that and it breaks something, that's their fault not ours.
> (It's not like there's not $BIGNUM ways for a C-language module to
> break the backend, anyway.)

Agreed.  Similarly the process_shared_preload_libraries_in_progress flag could
be modified by extension, and that wouldn't be any better.

> BTW, I'd make such errors FATAL, as it's unlikely that we can recover
> cleanly from an error during initialization of a loadable module.
> The module's likely to be only partially initialized/hooked in.

While at it, should we make process_shmem_requests_in_progress true when the
new hook is called?  The hook should only be called when that's the case, and
extension authors may feel like asserting it.





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 06:35  Anton A. Melnikov <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  1 sibling, 0 replies; 94+ messages in thread

From: Anton A. Melnikov @ 2022-05-06 06:35 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; Julien Rouhaud <[email protected]>; +Cc: Tom Lane <[email protected]>; Robert Haas <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Hello!

On 19.04.2022 18:46, Nathan Bossart wrote:
> Okay, I did it this way in v5.
> 

Recently i ran into a problem that it would be preferable in our 
extended pg_stat_statements to use MaxBackEnds in _PG_Init() but it's 
equal to zero here.
And i was very happy to find this patch and this thread.  It was not 
only very interesting and informative for me but also solves the current 
problem. Patch is applied cleanly on current master. Tests under Linux 
and Windows were successful.
I've tried this patch with our extended pg_stat_statements and checked 
that MaxBackends has the correct value during extra shmem allocating. 
Thanks a lot!

+1 for this patch.

I have a little doubt about the comment in postmaster.c: "Now that 
loadable modules have had their chance to alter any GUCs". Perhaps this 
comment is too general. Not sure that it is possible to change any 
arbitrary GUC here.
And maybe not bad to add Assert(size > 0) in RequestAddinShmemSpace() to 
see that the size calculation in contrib was wrong.

With best regards,

-- 
Anton A. Melnikov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 13:51  Robert Haas <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-05-06 13:51 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, Apr 19, 2022 at 11:47 AM Nathan Bossart
<[email protected]> wrote:
> Okay, I did it this way in v5.

I pushed 0001. Regarding 0002, I think there's no point in adding a
_PG_fini(). The code to call _PG_fini() has been surrounded by #ifdef
NOT_USED forever, and that seems unlikely to change any time soon as
long as we stick with these stupid hook designs where there's
effectively a linked list of hooks, but you can't actually access the
list because spread across a bunch of random static variables in each
module. I think we ought to go through all the hooks that are being
used in this kind of way and replace them with a system where there's
an explicit List of functions to call, and instead of this lame stuff
like:

+       prev_shmem_request_hook = shmem_request_hook;
+       shmem_request_hook = autoprewarm_shmem_request;

You instead do:

shmem_request_functions = lappend(shmem_request_functions,
autoprewarm_shmem_request);

Or maybe wrap that up in an API, like:

add_shmem_request_function(autoprewarm_shmem_request);

Then it would be easy to have corresponding a corresponding remove function.

For shmem request, it would probably make sense to ditch the part
where each hook function calls the next hook function and instead just
have the calling code loop over the list and call them one by one. For
things like the executor start/run/end hooks, that wouldn't work, but
you could have something like
invoke_next_executor_start_hook_function(args).

I don't think we should try to make this kind of change now - it seems
to me that what you've done here is consistent with existing practice
and we might as well commit it more or less like this for now. But
perhaps for v16 or some future release we can think about redoing a
bunch of hooks this way. There would be some migration pain for
extension authors for sure, but then we might get to a point where
extensions can be cleanly unloaded in at least some circumstances,
instead of continuing to write silly _PG_fini() functions that
couldn't possibly ever work.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 13:57  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-05-06 13:57 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> perhaps for v16 or some future release we can think about redoing a
> bunch of hooks this way. There would be some migration pain for
> extension authors for sure, but then we might get to a point where
> extensions can be cleanly unloaded in at least some circumstances,
> instead of continuing to write silly _PG_fini() functions that
> couldn't possibly ever work.

I agree that _PG_fini functions as they stand are worthless.
What I'm not getting is why we should care enough about that
to break just about everybody's extension.  Even if unloading
extensions were feasible, who would bother?

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 14:10  Robert Haas <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-05-06 14:10 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 6, 2022 at 9:57 AM Tom Lane <[email protected]> wrote:
> Robert Haas <[email protected]> writes:
> > perhaps for v16 or some future release we can think about redoing a
> > bunch of hooks this way. There would be some migration pain for
> > extension authors for sure, but then we might get to a point where
> > extensions can be cleanly unloaded in at least some circumstances,
> > instead of continuing to write silly _PG_fini() functions that
> > couldn't possibly ever work.
>
> I agree that _PG_fini functions as they stand are worthless.
> What I'm not getting is why we should care enough about that
> to break just about everybody's extension.  Even if unloading
> extensions were feasible, who would bother?

Well, if we think that, then we ought to remove the NOT_USED code and
all the random _PG_fini() stuff that's still floating around.

I don't actually have a clear answer to whether it's a useful thing to
be able to unload modules. If the module is just providing a bunch of
SQL-callable functions, it probably isn't. If it's modifying the
behavior of your session, it might be. Now, it could also have an
"off" switch in the form of a GUC, and probably should - but you'd
probably save at least a few cycles by detaching from the hooks rather
than just still getting called and doing nothing, and maybe that's
worth something. Whether it's worth enough to justify making changes
that will affect extensions, I'm not sure.

IOW, I don't really know what we ought to do here, but I think that
maintaining a vestigial system that has never worked and can never
work is not it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 14:43  Tom Lane <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Tom Lane @ 2022-05-06 14:43 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

Robert Haas <[email protected]> writes:
> On Fri, May 6, 2022 at 9:57 AM Tom Lane <[email protected]> wrote:
>> I agree that _PG_fini functions as they stand are worthless.
>> What I'm not getting is why we should care enough about that
>> to break just about everybody's extension.  Even if unloading
>> extensions were feasible, who would bother?

> Well, if we think that, then we ought to remove the NOT_USED code and
> all the random _PG_fini() stuff that's still floating around.

I think that's exactly what we should do, if it bugs you that stuff
is just sitting there.  I see no prospect that we'll ever make it
work, because the question of unhooking from hooks is just the tip
of the iceberg.  As an example, what should happen with any custom
GUCs the module has defined?  Dropping their values might not be
very nice, but if we leave them around then the next LOAD (if any)
will see a conflict.  Another fun question is whether it's ever
safe to unload a module that was preloaded by the postmaster.

In short, this seems like a can of very wriggly worms, with not
a lot of benefit that would ensue from opening it.

			regards, tom lane

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 15:27  Nathan Bossart <[email protected]>
  parent: Tom Lane <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-05-06 15:27 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 06, 2022 at 10:43:21AM -0400, Tom Lane wrote:
> Robert Haas <[email protected]> writes:
>> On Fri, May 6, 2022 at 9:57 AM Tom Lane <[email protected]> wrote:
>>> I agree that _PG_fini functions as they stand are worthless.
>>> What I'm not getting is why we should care enough about that
>>> to break just about everybody's extension.  Even if unloading
>>> extensions were feasible, who would bother?
> 
>> Well, if we think that, then we ought to remove the NOT_USED code and
>> all the random _PG_fini() stuff that's still floating around.
> 
> I think that's exactly what we should do, if it bugs you that stuff
> is just sitting there.  I see no prospect that we'll ever make it
> work, because the question of unhooking from hooks is just the tip
> of the iceberg.  As an example, what should happen with any custom
> GUCs the module has defined?  Dropping their values might not be
> very nice, but if we leave them around then the next LOAD (if any)
> will see a conflict.  Another fun question is whether it's ever
> safe to unload a module that was preloaded by the postmaster.
> 
> In short, this seems like a can of very wriggly worms, with not
> a lot of benefit that would ensue from opening it.

+1, I'll put together a new patch set.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 21:15  Nathan Bossart <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-05-06 21:15 UTC (permalink / raw)
  To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 06, 2022 at 08:27:11AM -0700, Nathan Bossart wrote:
> +1, I'll put together a new patch set.

As promised...

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-06 23:51  Robert Haas <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-05-06 23:51 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 6, 2022 at 5:15 PM Nathan Bossart <[email protected]> wrote:
> On Fri, May 06, 2022 at 08:27:11AM -0700, Nathan Bossart wrote:
> > +1, I'll put together a new patch set.
>
> As promised...

Looks reasonable to me.

Anyone else have thoughts?

-- 
Robert Haas
EDB: http://www.enterprisedb.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-10 08:55  Michael Paquier <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Michael Paquier @ 2022-05-10 08:55 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 06, 2022 at 07:51:43PM -0400, Robert Haas wrote:
> On Fri, May 6, 2022 at 5:15 PM Nathan Bossart <[email protected]> wrote:
>> On Fri, May 06, 2022 at 08:27:11AM -0700, Nathan Bossart wrote:
>> > +1, I'll put together a new patch set.
>>
>> As promised...
> 
> Looks reasonable to me.

0001 looks sensible seen from here with this approach.

> Anyone else have thoughts?

I agree that removing support for the unloading part would be nice to
clean up now on HEAD.  Note that 0002 is missing the removal of one
reference to _PG_fini in xfunc.sgml (<primary> markup).
--
Michael


Attachments:

  [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
  download

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-10 15:56  Nathan Bossart <[email protected]>
  parent: Michael Paquier <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-05-10 15:56 UTC (permalink / raw)
  To: Michael Paquier <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, May 10, 2022 at 05:55:12PM +0900, Michael Paquier wrote:
> I agree that removing support for the unloading part would be nice to
> clean up now on HEAD.  Note that 0002 is missing the removal of one
> reference to _PG_fini in xfunc.sgml (<primary> markup).

Oops, sorry about that.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-11 02:18  Michael Paquier <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Michael Paquier @ 2022-05-11 02:18 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Tue, May 10, 2022 at 08:56:28AM -0700, Nathan Bossart wrote:
> On Tue, May 10, 2022 at 05:55:12PM +0900, Michael Paquier wrote:
>> I agree that removing support for the unloading part would be nice to
>> clean up now on HEAD.  Note that 0002 is missing the removal of one
>> reference to _PG_fini in xfunc.sgml (<primary> markup).
> 
> Oops, sorry about that.

0002 looks pretty good from here.  If it were me, I would apply 0002
first to avoid the extra tweak in pg_stat_statements with _PG_fini in
0001.

Regarding 0001, I have spotted an extra issue in the documentation.
xfunc.sgml has a section called "Shared Memory and LWLocks" about the
use of RequestNamedLWLockTranche() and RequestAddinShmemSpace() called
from _PG_init(), which would now be wrong as we'd need the hook.  IMO,
the docs should mention the registration of the hook in _PG_init(),
the APIs involved, and should mention to look at pg_stat_statements.c
for a code sample (we tend to avoid the addition of sample code in the
docs lately as we habe no way to check their compilation
automatically).

Robert, are you planning to look at what's proposed and push these?
FWIW, I am familiar enough with the problems at hand to do this in
time for beta1 (aka by tomorrow to give it room in the buildfarm), but
I don't want to step on your feet, either.
--
Michael

Attachments:

  [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
  download

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-11 04:12  Nathan Bossart <[email protected]>
  parent: Michael Paquier <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-05-11 04:12 UTC (permalink / raw)
  To: Michael Paquier <[email protected]>; +Cc: Robert Haas <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, May 11, 2022 at 11:18:29AM +0900, Michael Paquier wrote:
> 0002 looks pretty good from here.  If it were me, I would apply 0002
> first to avoid the extra tweak in pg_stat_statements with _PG_fini in
> 0001.

I swapped 0001 and 0002 in v8.

> Regarding 0001, I have spotted an extra issue in the documentation.
> xfunc.sgml has a section called "Shared Memory and LWLocks" about the
> use of RequestNamedLWLockTranche() and RequestAddinShmemSpace() called
> from _PG_init(), which would now be wrong as we'd need the hook.  IMO,
> the docs should mention the registration of the hook in _PG_init(),
> the APIs involved, and should mention to look at pg_stat_statements.c
> for a code sample (we tend to avoid the addition of sample code in the
> docs lately as we habe no way to check their compilation
> automatically).

Ah, good catch.  I adjusted the docs as you suggested.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com


^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-11 20:18  Robert Haas <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Robert Haas @ 2022-05-11 20:18 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; +Cc: Michael Paquier <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, May 11, 2022 at 12:12 AM Nathan Bossart
<[email protected]> wrote:
> I swapped 0001 and 0002 in v8.
>
> Ah, good catch.  I adjusted the docs as you suggested.

OK, I have committed 0001 now.

-- 
Robert Haas
EDB: http://www.enterprisedb.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-11 21:01  Nathan Bossart <[email protected]>
  parent: Robert Haas <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Nathan Bossart @ 2022-05-11 21:01 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Wed, May 11, 2022 at 04:18:10PM -0400, Robert Haas wrote:
> OK, I have committed 0001 now.

Thanks!

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-12 15:51  Anton A. Melnikov <[email protected]>
  parent: Nathan Bossart <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Anton A. Melnikov @ 2022-05-12 15:51 UTC (permalink / raw)
  To: Nathan Bossart <[email protected]>; Robert Haas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On 12.05.2022 00:01, Nathan Bossart wrote:
> On Wed, May 11, 2022 at 04:18:10PM -0400, Robert Haas wrote:
>> OK, I have committed 0001 now.
> 
> Thanks!
> 

Maybe remove the first part from the patchset?
Because now the Patch Tester is giving apply error for the first part 
and can't test the other.
http://cfbot.cputube.org/patch_38_3614.log

With best regards,
-- 
Anton A. Melnikov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-13 00:15  Michael Paquier <[email protected]>
  parent: Anton A. Melnikov <[email protected]>
  0 siblings, 1 reply; 94+ messages in thread

From: Michael Paquier @ 2022-05-13 00:15 UTC (permalink / raw)
  To: Anton A. Melnikov <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Robert Haas <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Thu, May 12, 2022 at 06:51:59PM +0300, Anton A. Melnikov wrote:
> Maybe remove the first part from the patchset?
> Because now the Patch Tester is giving apply error for the first part and
> can't test the other.
> http://cfbot.cputube.org/patch_38_3614.log

Yep.  As simple as the attached.
--
Michael


Attachments:

  [text/x-diff] v9-0001-Add-a-new-shmem_request_hook-hook.patch (14.0K, ../../Yn2jE%[email protected]/2-v9-0001-Add-a-new-shmem_request_hook-hook.patch)
  download | inline diff:
From d69844e34049d8c38b1e05fb75104469b2d123e1 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <[email protected]>
Date: Mon, 18 Apr 2022 15:25:37 -0700
Subject: [PATCH v9] Add a new shmem_request_hook hook.

Currently, preloaded libraries are expected to request additional
shared memory and LWLocks in _PG_init().  However, it is not unusal
for such requests to depend on MaxBackends, which won't be
initialized at that time.  Such requests could also depend on GUCs
that other modules might change.  This introduces a new hook where
modules can safely use MaxBackends and GUCs to request additional
shared memory and LWLocks.

Furthermore, this change restricts requests for shared memory and
LWLocks to this hook.  Previously, libraries could make requests
until the size of the main shared memory segment was calculated.
Unlike before, we no longer silently ignore requests received at
invalid times.  Instead, we FATAL if someone tries to request
additional shared memory or LWLocks outside of the hook.

Authors: Julien Rouhaud, Nathan Bossart
Discussion: https://postgr.es/m/20220412210112.GA2065815%40nathanxps13
---
 src/include/miscadmin.h                       |  5 ++++
 src/backend/postmaster/postmaster.c           |  5 ++++
 src/backend/storage/ipc/ipci.c                | 20 +++++---------
 src/backend/storage/lmgr/lwlock.c             | 18 ++++---------
 src/backend/utils/init/miscinit.c             | 15 +++++++++++
 doc/src/sgml/xfunc.sgml                       | 12 +++++++--
 contrib/pg_prewarm/autoprewarm.c              | 17 +++++++++++-
 .../pg_stat_statements/pg_stat_statements.c   | 26 +++++++++++++------
 src/tools/pgindent/typedefs.list              |  1 +
 9 files changed, 81 insertions(+), 38 deletions(-)

diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 53fd168d93..0af130fbc5 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -465,6 +465,7 @@ extern void BaseInit(void);
 extern PGDLLIMPORT bool IgnoreSystemIndexes;
 extern PGDLLIMPORT bool process_shared_preload_libraries_in_progress;
 extern PGDLLIMPORT bool process_shared_preload_libraries_done;
+extern PGDLLIMPORT bool process_shmem_requests_in_progress;
 extern PGDLLIMPORT char *session_preload_libraries_string;
 extern PGDLLIMPORT char *shared_preload_libraries_string;
 extern PGDLLIMPORT char *local_preload_libraries_string;
@@ -478,9 +479,13 @@ extern bool RecheckDataDirLockFile(void);
 extern void ValidatePgVersion(const char *path);
 extern void process_shared_preload_libraries(void);
 extern void process_session_preload_libraries(void);
+extern void process_shmem_requests(void);
 extern void pg_bindtextdomain(const char *domain);
 extern bool has_rolreplication(Oid roleid);
 
+typedef void (*shmem_request_hook_type) (void);
+extern PGDLLIMPORT shmem_request_hook_type shmem_request_hook;
+
 /* in executor/nodeHash.c */
 extern size_t get_hash_memory_limit(void);
 
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index bf591f048d..3b73e26956 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1042,6 +1042,11 @@ PostmasterMain(int argc, char *argv[])
 	 */
 	InitializeMaxBackends();
 
+	/*
+	 * Give preloaded libraries a chance to request additional shared memory.
+	 */
+	process_shmem_requests();
+
 	/*
 	 * Now that loadable modules have had their chance to request additional
 	 * shared memory, determine the value of any runtime-computed GUCs that
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 75e456360b..26372d95b3 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -55,25 +55,21 @@ int			shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
 shmem_startup_hook_type shmem_startup_hook = NULL;
 
 static Size total_addin_request = 0;
-static bool addin_request_allowed = true;
-
 
 /*
  * RequestAddinShmemSpace
  *		Request that extra shmem space be allocated for use by
  *		a loadable module.
  *
- * This is only useful if called from the _PG_init hook of a library that
- * is loaded into the postmaster via shared_preload_libraries.  Once
- * shared memory has been allocated, calls will be ignored.  (We could
- * raise an error, but it seems better to make it a no-op, so that
- * libraries containing such calls can be reloaded if needed.)
+ * This may only be called via the shmem_request_hook of a library that is
+ * loaded into the postmaster via shared_preload_libraries.  Calls from
+ * elsewhere will fail.
  */
 void
 RequestAddinShmemSpace(Size size)
 {
-	if (IsUnderPostmaster || !addin_request_allowed)
-		return;					/* too late */
+	if (!process_shmem_requests_in_progress)
+		elog(FATAL, "cannot request additional shared memory outside shmem_request_hook");
 	total_addin_request = add_size(total_addin_request, size);
 }
 
@@ -83,9 +79,6 @@ RequestAddinShmemSpace(Size size)
  *
  * If num_semaphores is not NULL, it will be set to the number of semaphores
  * required.
- *
- * Note that this function freezes the additional shared memory request size
- * from loadable modules.
  */
 Size
 CalculateShmemSize(int *num_semaphores)
@@ -152,8 +145,7 @@ CalculateShmemSize(int *num_semaphores)
 	size = add_size(size, ShmemBackendArraySize());
 #endif
 
-	/* freeze the addin request size and include it */
-	addin_request_allowed = false;
+	/* include additional requested shmem from preload libraries */
 	size = add_size(size, total_addin_request);
 
 	/* might as well round it off to a multiple of a typical page size */
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index fef462b110..8aef909037 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -243,8 +243,6 @@ int			NamedLWLockTrancheRequests = 0;
 /* points to data in shared memory: */
 NamedLWLockTranche *NamedLWLockTrancheArray = NULL;
 
-static bool lock_named_request_allowed = true;
-
 static void InitializeLWLocks(void);
 static inline void LWLockReportWaitStart(LWLock *lock);
 static inline void LWLockReportWaitEnd(void);
@@ -458,9 +456,6 @@ LWLockShmemSize(void)
 	for (i = 0; i < NamedLWLockTrancheRequests; i++)
 		size = add_size(size, strlen(NamedLWLockTrancheRequestArray[i].tranche_name) + 1);
 
-	/* Disallow adding any more named tranches. */
-	lock_named_request_allowed = false;
-
 	return size;
 }
 
@@ -691,12 +686,9 @@ LWLockRegisterTranche(int tranche_id, const char *tranche_name)
  *		Request that extra LWLocks be allocated during postmaster
  *		startup.
  *
- * This is only useful for extensions if called from the _PG_init hook
- * of a library that is loaded into the postmaster via
- * shared_preload_libraries.  Once shared memory has been allocated, calls
- * will be ignored.  (We could raise an error, but it seems better to make
- * it a no-op, so that libraries containing such calls can be reloaded if
- * needed.)
+ * This may only be called via the shmem_request_hook of a library that is
+ * loaded into the postmaster via shared_preload_libraries.  Calls from
+ * elsewhere will fail.
  *
  * The tranche name will be user-visible as a wait event name, so try to
  * use a name that fits the style for those.
@@ -706,8 +698,8 @@ RequestNamedLWLockTranche(const char *tranche_name, int num_lwlocks)
 {
 	NamedLWLockTrancheRequest *request;
 
-	if (IsUnderPostmaster || !lock_named_request_allowed)
-		return;					/* too late */
+	if (!process_shmem_requests_in_progress)
+		elog(FATAL, "cannot request additional LWLocks outside shmem_request_hook");
 
 	if (NamedLWLockTrancheRequestArray == NULL)
 	{
diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c
index 6ed584e114..ec6a61594a 100644
--- a/src/backend/utils/init/miscinit.c
+++ b/src/backend/utils/init/miscinit.c
@@ -1618,6 +1618,9 @@ char	   *local_preload_libraries_string = NULL;
 bool		process_shared_preload_libraries_in_progress = false;
 bool		process_shared_preload_libraries_done = false;
 
+shmem_request_hook_type shmem_request_hook = NULL;
+bool		process_shmem_requests_in_progress = false;
+
 /*
  * load the shared libraries listed in 'libraries'
  *
@@ -1701,6 +1704,18 @@ process_session_preload_libraries(void)
 				   true);
 }
 
+/*
+ * process any shared memory requests from preloaded libraries
+ */
+void
+process_shmem_requests(void)
+{
+	process_shmem_requests_in_progress = true;
+	if (shmem_request_hook)
+		shmem_request_hook();
+	process_shmem_requests_in_progress = false;
+}
+
 void
 pg_bindtextdomain(const char *domain)
 {
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index fbe718e3c2..3b0adc0704 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3402,22 +3402,30 @@ CREATE FUNCTION make_array(anyelement) RETURNS anyarray
      startup.  The add-in's shared library must be preloaded by specifying
      it in
      <xref linkend="guc-shared-preload-libraries"/><indexterm><primary>shared_preload_libraries</primary></indexterm>.
+     The shared library should register a <literal>shmem_request_hook</literal>
+     in its <function>_PG_init</function> function.  This
+     <literal>shmem_request_hook</literal> can reserve LWLocks or shared memory.
      Shared memory is reserved by calling:
 <programlisting>
 void RequestAddinShmemSpace(int size)
 </programlisting>
-     from your <function>_PG_init</function> function.
+     from your <literal>shmem_request_hook</literal>.
     </para>
     <para>
      LWLocks are reserved by calling:
 <programlisting>
 void RequestNamedLWLockTranche(const char *tranche_name, int num_lwlocks)
 </programlisting>
-     from <function>_PG_init</function>.  This will ensure that an array of
+     from your <literal>shmem_request_hook</literal>.  This will ensure that an array of
      <literal>num_lwlocks</literal> LWLocks is available under the name
      <literal>tranche_name</literal>.  Use <function>GetNamedLWLockTranche</function>
      to get a pointer to this array.
     </para>
+    <para>
+     An example of a <literal>shmem_request_hook</literal> can be found in
+     <filename>contrib/pg_stat_statements/pg_stat_statements.c</filename> in the
+     <productname>PostgreSQL</productname> source tree.
+    </para>
     <para>
      To avoid possible race-conditions, each backend should use the LWLock
      <function>AddinShmemInitLock</function> when connecting to and initializing
diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c
index 45e012a63a..c0c4f5d9ca 100644
--- a/contrib/pg_prewarm/autoprewarm.c
+++ b/contrib/pg_prewarm/autoprewarm.c
@@ -96,6 +96,8 @@ static void apw_start_database_worker(void);
 static bool apw_init_shmem(void);
 static void apw_detach_shmem(int code, Datum arg);
 static int	apw_compare_blockinfo(const void *p, const void *q);
+static void autoprewarm_shmem_request(void);
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
 
 /* Pointer to shared-memory state. */
 static AutoPrewarmSharedState *apw_state = NULL;
@@ -139,13 +141,26 @@ _PG_init(void)
 
 	MarkGUCPrefixReserved("pg_prewarm");
 
-	RequestAddinShmemSpace(MAXALIGN(sizeof(AutoPrewarmSharedState)));
+	prev_shmem_request_hook = shmem_request_hook;
+	shmem_request_hook = autoprewarm_shmem_request;
 
 	/* Register autoprewarm worker, if enabled. */
 	if (autoprewarm)
 		apw_start_leader_worker();
 }
 
+/*
+ * Requests any additional shared memory required for autoprewarm.
+ */
+static void
+autoprewarm_shmem_request(void)
+{
+	if (prev_shmem_request_hook)
+		prev_shmem_request_hook();
+
+	RequestAddinShmemSpace(MAXALIGN(sizeof(AutoPrewarmSharedState)));
+}
+
 /*
  * Main entry point for the leader autoprewarm process.  Per-database workers
  * have a separate entry point.
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 5c8b9ff943..768cedd91a 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -252,6 +252,7 @@ static int	exec_nested_level = 0;
 static int	plan_nested_level = 0;
 
 /* Saved hook values in case of unload */
+static shmem_request_hook_type prev_shmem_request_hook = NULL;
 static shmem_startup_hook_type prev_shmem_startup_hook = NULL;
 static post_parse_analyze_hook_type prev_post_parse_analyze_hook = NULL;
 static planner_hook_type prev_planner_hook = NULL;
@@ -316,6 +317,7 @@ PG_FUNCTION_INFO_V1(pg_stat_statements_1_10);
 PG_FUNCTION_INFO_V1(pg_stat_statements);
 PG_FUNCTION_INFO_V1(pg_stat_statements_info);
 
+static void pgss_shmem_request(void);
 static void pgss_shmem_startup(void);
 static void pgss_shmem_shutdown(int code, Datum arg);
 static void pgss_post_parse_analyze(ParseState *pstate, Query *query,
@@ -451,17 +453,11 @@ _PG_init(void)
 
 	MarkGUCPrefixReserved("pg_stat_statements");
 
-	/*
-	 * Request additional shared resources.  (These are no-ops if we're not in
-	 * the postmaster process.)  We'll allocate or attach to the shared
-	 * resources in pgss_shmem_startup().
-	 */
-	RequestAddinShmemSpace(pgss_memsize());
-	RequestNamedLWLockTranche("pg_stat_statements", 1);
-
 	/*
 	 * Install hooks.
 	 */
+	prev_shmem_request_hook = shmem_request_hook;
+	shmem_request_hook = pgss_shmem_request;
 	prev_shmem_startup_hook = shmem_startup_hook;
 	shmem_startup_hook = pgss_shmem_startup;
 	prev_post_parse_analyze_hook = post_parse_analyze_hook;
@@ -480,6 +476,20 @@ _PG_init(void)
 	ProcessUtility_hook = pgss_ProcessUtility;
 }
 
+/*
+ * shmem_request hook: request additional shared resources.  We'll allocate or
+ * attach to the shared resources in pgss_shmem_startup().
+ */
+static void
+pgss_shmem_request(void)
+{
+	if (prev_shmem_request_hook)
+		prev_shmem_request_hook();
+
+	RequestAddinShmemSpace(pgss_memsize());
+	RequestNamedLWLockTranche("pg_stat_statements", 1);
+}
+
 /*
  * shmem_startup hook: allocate or attach to shared memory,
  * then load any pre-existing statistics from file.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dd1214977a..4fb746930a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3651,6 +3651,7 @@ shm_mq_result
 shm_toc
 shm_toc_entry
 shm_toc_estimator
+shmem_request_hook_type
 shmem_startup_hook_type
 sig_atomic_t
 sigjmp_buf
-- 
2.36.0



  [application/pgp-signature] signature.asc (833B, ../../Yn2jE%[email protected]/3-signature.asc)
  download

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-13 13:49  Robert Haas <[email protected]>
  parent: Michael Paquier <[email protected]>
  0 siblings, 2 replies; 94+ messages in thread

From: Robert Haas @ 2022-05-13 13:49 UTC (permalink / raw)
  To: Michael Paquier <[email protected]>; +Cc: Anton A. Melnikov <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Thu, May 12, 2022 at 8:15 PM Michael Paquier <[email protected]> wrote:
> On Thu, May 12, 2022 at 06:51:59PM +0300, Anton A. Melnikov wrote:
> > Maybe remove the first part from the patchset?
> > Because now the Patch Tester is giving apply error for the first part and
> > can't test the other.
> > http://cfbot.cputube.org/patch_38_3614.log
>
> Yep.  As simple as the attached.

Committed.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-13 15:35  Nathan Bossart <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 0 replies; 94+ messages in thread

From: Nathan Bossart @ 2022-05-13 15:35 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Anton A. Melnikov <[email protected]>; Tom Lane <[email protected]>; Julien Rouhaud <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 13, 2022 at 09:49:54AM -0400, Robert Haas wrote:
> Committed.

Thanks!

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-15 12:58  Julien Rouhaud <[email protected]>
  parent: Robert Haas <[email protected]>
  1 sibling, 1 reply; 94+ messages in thread

From: Julien Rouhaud @ 2022-05-15 12:58 UTC (permalink / raw)
  To: Robert Haas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Anton A. Melnikov <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Fri, May 13, 2022 at 09:49:54AM -0400, Robert Haas wrote:
>
> Committed.

Thanks!

For the record I submitted patches or pull requests this weekend for all
repositories I was aware of to fix the compatibility with this patch, hoping
that it will save some time to the packagers when they will take care of the
beta.

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* Re: make MaxBackends available in _PG_init
@ 2022-05-16 13:25  Robert Haas <[email protected]>
  parent: Julien Rouhaud <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Robert Haas @ 2022-05-16 13:25 UTC (permalink / raw)
  To: Julien Rouhaud <[email protected]>; +Cc: Michael Paquier <[email protected]>; Anton A. Melnikov <[email protected]>; Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]>; Bossart, Nathan <[email protected]>; Fujii Masao <[email protected]>; [email protected] <[email protected]>; Bharath Rupireddy <[email protected]>; Greg Sabino Mullane <[email protected]>; [email protected] <[email protected]>

On Sun, May 15, 2022 at 8:58 AM Julien Rouhaud <[email protected]> wrote:
> For the record I submitted patches or pull requests this weekend for all
> repositories I was aware of to fix the compatibility with this patch, hoping
> that it will save some time to the packagers when they will take care of the
> beta.

Nice!

-- 
Robert Haas
EDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 94+ messages in thread

* [PATCH v1] solaris: Use versioning scripts instead of -Bsymbolic
@ 2022-08-23 08:50  Andres Freund <[email protected]>
  0 siblings, 0 replies; 94+ messages in thread

From: Andres Freund @ 2022-08-23 08:50 UTC (permalink / raw)

-Bsymbolic causes a lot of "ld: warning: symbol referencing errors"
warnings. It's quite easy to add the symbol versioning script, we just need a
slightly different parameter name.

Discussion: https://postgr.es/m/[email protected]
---
 src/Makefile.shlib | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/Makefile.shlib b/src/Makefile.shlib
index 2af6192f0f3..3202b2e67de 100644
--- a/src/Makefile.shlib
+++ b/src/Makefile.shlib
@@ -188,10 +188,15 @@ ifeq ($(PORTNAME), linux)
 endif
 
 ifeq ($(PORTNAME), solaris)
-  LINK.shared		= $(COMPILER) -shared -Wl,-Bsymbolic
+  LINK.shared		= $(COMPILER) -shared
   ifdef soname
     LINK.shared	+= -Wl,-soname,$(soname)
   endif
+  BUILD.exports		= ( echo '{ global:'; $(AWK) '/^[^\#]/ {printf "%s;\n",$$1}' $<; echo ' local: *; };' ) >$@
+  exports_file		= $(SHLIB_EXPORTS:%.txt=%.list)
+  ifneq (,$(exports_file))
+    LINK.shared		+= -Wl,-M$(exports_file)
+  endif
 endif
 
 ifeq ($(PORTNAME), cygwin)
-- 
2.37.0.3.g30cc8d0f14


--aym2ahtpoerge4dw--





^ permalink  raw  reply  [nested|flat] 94+ messages in thread

end of thread, other threads:[~2022-08-23 08:50 UTC | newest]

Thread overview: 94+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-04-04 15:06 [PATCH 1/2] Refactor checks for deleted GiST pages. Heikki Linnakangas <[email protected]>
2019-04-04 15:06 [PATCH 1/2] Refactor checks for deleted GiST pages. Heikki Linnakangas <[email protected]>
2019-07-22 12:57 [PATCH 1/2] Refactor checks for deleted GiST pages. Heikki Linnakangas <[email protected]>
2020-09-21 03:43 make MaxBackends available in _PG_init Wang, Shenhao <[email protected]>
2020-09-21 11:52 ` Re: make MaxBackends available in _PG_init Bharath Rupireddy <[email protected]>
2021-08-02 18:18   ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-02 20:06     ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2021-08-02 20:27       ` Re: make MaxBackends available in _PG_init Alvaro Herrera <[email protected]>
2021-08-02 20:37       ` Re: make MaxBackends available in _PG_init Andres Freund <[email protected]>
2021-08-02 21:57         ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-02 22:11           ` Re: make MaxBackends available in _PG_init Andres Freund <[email protected]>
2021-08-02 22:35             ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-02 22:42               ` Re: make MaxBackends available in _PG_init Andres Freund <[email protected]>
2021-08-02 22:58                 ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-03 00:19                   ` RE: make MaxBackends available in _PG_init [email protected] <[email protected]>
2021-08-03 15:25         ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2021-08-03 23:13           ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-07 18:01           ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-09 20:13             ` Re: make MaxBackends available in _PG_init Greg Sabino Mullane <[email protected]>
2021-08-10 00:22               ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2021-08-11 13:41                 ` Re: make MaxBackends available in _PG_init Greg Sabino Mullane <[email protected]>
2021-08-11 14:08                   ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2021-08-12 15:41                     ` Re: make MaxBackends available in _PG_init Greg Sabino Mullane <[email protected]>
2021-08-15 08:04                       ` RE: make MaxBackends available in _PG_init [email protected] <[email protected]>
2021-08-16 04:02                         ` Re: make MaxBackends available in _PG_init Bossart, Nathan <[email protected]>
2022-03-23 04:52 Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-23 12:32 ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-03-23 13:03   ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-24 20:20     ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-03-24 20:27       ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-03-25 02:39         ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-25 03:11           ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-25 05:08             ` Re: make MaxBackends available in _PG_init Michael Paquier <[email protected]>
2022-03-25 06:35               ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-25 22:23                 ` Re: make MaxBackends available in _PG_init Andres Freund <[email protected]>
2022-03-26 07:22                   ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-26 17:23                     ` Re: make MaxBackends available in _PG_init Andres Freund <[email protected]>
2022-03-27 04:15                       ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-03-29 16:22                       ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-03-30 16:30                         ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-09 13:24                           ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-04-10 23:34                             ` Re: make MaxBackends available in _PG_init Michael Paquier <[email protected]>
2022-04-11 15:36                             ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-04-12 05:08                             ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-04-12 18:03                             ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-12 19:12                               ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-12 19:21                                 ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-12 19:30                                   ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-12 19:59                                     ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-12 20:58                                       ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-12 21:07                                         ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-12 21:46                                           ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-12 22:26                                             ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-13 13:16                                               ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-13 13:27                                                 ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-13 14:22                                                   ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-13 14:43                                                     ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-13 15:25                                                       ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-13 15:39                                                         ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-13 15:50                                                           ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-13 16:05                                                             ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-13 18:30                                                               ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-14 05:50                                                                 ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-04-14 16:22                                                                   ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-14 16:32                                                                     ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-14 16:39                                                                       ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-18 22:52                                                                         ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-04-18 23:33                                                                           ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-19 09:49                                                                             ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-05-06 06:35                                                                               ` Re: make MaxBackends available in _PG_init Anton A. Melnikov <[email protected]>
2022-05-06 13:51                                                                               ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-05-06 13:57                                                                                 ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-05-06 14:10                                                                                   ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-05-06 14:43                                                                                     ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-05-06 15:27                                                                                       ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-05-06 21:15                                                                                         ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-05-06 23:51                                                                                           ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-05-10 08:55                                                                                             ` Re: make MaxBackends available in _PG_init Michael Paquier <[email protected]>
2022-05-10 15:56                                                                                               ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-05-11 02:18                                                                                                 ` Re: make MaxBackends available in _PG_init Michael Paquier <[email protected]>
2022-05-11 04:12                                                                                                   ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-05-11 20:18                                                                                                     ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-05-11 21:01                                                                                                       ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-05-12 15:51                                                                                                         ` Re: make MaxBackends available in _PG_init Anton A. Melnikov <[email protected]>
2022-05-13 00:15                                                                                                           ` Re: make MaxBackends available in _PG_init Michael Paquier <[email protected]>
2022-05-13 13:49                                                                                                             ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-05-13 15:35                                                                                                               ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-05-15 12:58                                                                                                               ` Re: make MaxBackends available in _PG_init Julien Rouhaud <[email protected]>
2022-05-16 13:25                                                                                                                 ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-12 21:14                                         ` Re: make MaxBackends available in _PG_init Robert Haas <[email protected]>
2022-04-12 21:43                                           ` Re: make MaxBackends available in _PG_init Tom Lane <[email protected]>
2022-04-13 07:54                                             ` Re: make MaxBackends available in _PG_init Michael Paquier <[email protected]>
2022-04-12 19:49                                 ` Re: make MaxBackends available in _PG_init Nathan Bossart <[email protected]>
2022-08-23 08:50 [PATCH v1] solaris: Use versioning scripts instead of -Bsymbolic Andres Freund <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox