Date: Fri, 27 Aug 2021 12:38:13 -0700
From: Andres Freund <andres@anarazel.de>
To: "Bossart, Nathan" <bossartn@amazon.com>
Cc: Magnus Hagander <magnus@hagander.net>,
	Michael Paquier <michael@paquier.xyz>,
	Mark Dilger <mark.dilger@enterprisedb.com>,
	Don Seiler <don@seiler.us>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: Estimating HugePages Requirements?
Message-ID: <20210827193813.oqo5lamvyzahs35o@alap3.anarazel.de>
References: 
 <CAHJZqBBLHFNs6it-fcJ6LEUXeC5t73soR3h50zUSFpg7894qfQ@mail.gmail.com>
 <F2772387-CE0F-46BF-B5F1-CC55516EB885@amazon.com>
 <73A4E25F-1039-4FC9-8E6E-0A00230BDFB8@enterprisedb.com>
 <92BB93C3-1AA8-4548-96F7-062C4AF09182@amazon.com>
 <20210810034205.r72j5uouynepj4f2@alap3.anarazel.de>
 <0545F7B3-70C0-4DE8-8C85-EAFE6113B7EE@amazon.com>
 <YSiKS58gSXDpj6NR@paquier.xyz>
 <CABUevEwGHcsmp4rgZdcbinZOdEs_cSCabihDvRty=0zz1H95kw@mail.gmail.com>
 <20210827184648.d3jbv73pgyccqwkz@alap3.anarazel.de>
 <071DC95B-4CA8-4D2B-BD24-1A66B3EFFE28@amazon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <071DC95B-4CA8-4D2B-BD24-1A66B3EFFE28@amazon.com>
Archived-At: 
 <https://www.postgresql.org/message-id/20210827193813.oqo5lamvyzahs35o%40alap3.anarazel.de>
Precedence: bulk

Hi,

On 2021-08-27 19:27:18 +0000, Bossart, Nathan wrote:
> +     <varlistentry>
> +      <term><option>--output-shmem</option></term>
> +      <listitem>
> +       <para>
> +        Prints the amount of shared memory required for the given
> +        configuration and exits.  This can be used on a running server, but
> +        the return value reflects the amount of shared memory needed based
> +        on the current invocation.  It does not return the amount of shared
> +        memory in use by the running server.  This must be the first
> +        argument on the command line.
> +       </para>
> +
> +       <para>
> +        This option is useful for determining the number of huge pages
> +        needed for the server.  For more information, see
> +        <xref linkend="linux-huge-pages"/>.
> +       </para>
> +      </listitem>
> +     </varlistentry>
> +

One thing I wonder is if this wouldn't better be dealt with in a more generic
way. While this is the most problematic runtime computed GUC, it's not the
only one. What if we introduced a new shared_memory_size GUC, and made
--describe-config output it? Perhaps adding --describe-config=guc-name?

I also wonder if we should output the number of hugepages needed instead of
the "raw" bytes of shared memory. The whole business about figuring out the
huge page size, dividing the shared memory size by that and then rounding up
could be removed in that case. Due to huge_page_size it's not even immediately
obvious which huge page size one should use...


> diff --git a/src/backend/main/main.c b/src/backend/main/main.c
> index 3a2a0d598c..c141ae3d1c 100644
> --- a/src/backend/main/main.c
> +++ b/src/backend/main/main.c
> @@ -182,9 +182,11 @@ main(int argc, char *argv[])
>  	 */
>  
>  	if (argc > 1 && strcmp(argv[1], "--check") == 0)
> -		BootstrapModeMain(argc, argv, true);
> +		BootstrapModeMain(argc, argv, true, false);
> +	else if (argc > 1 && strcmp(argv[1], "--output-shmem") == 0)
> +		BootstrapModeMain(argc, argv, false, true);
>  	else if (argc > 1 && strcmp(argv[1], "--boot") == 0)
> -		BootstrapModeMain(argc, argv, false);
> +		BootstrapModeMain(argc, argv, false, false);
>  #ifdef EXEC_BACKEND
>  	else if (argc > 1 && strncmp(argv[1], "--fork", 6) == 0)
>  		SubPostmasterMain(argc, argv);

help() needs an update too.


> diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
> index 3e4ec53a97..b225b1ee70 100644
> --- a/src/backend/storage/ipc/ipci.c
> +++ b/src/backend/storage/ipc/ipci.c
> @@ -75,6 +75,87 @@ RequestAddinShmemSpace(Size size)
>  	total_addin_request = add_size(total_addin_request, size);
>  }
>  
> +/*
> + * CalculateShmemSize
> + *		Calculates the amount of shared memory and number of semaphores needed.
> + *
> + * If num_semaphores is not NULL, it will be set to the number of semaphores
> + * required.
> + *
> + * Note that this function freezes the additional shared memory request size
> + * from loadable modules.
> + */
> +Size
> +CalculateShmemSize(int *num_semaphores)
> +{

Can you split this into a separate commit? It feels fairy uncontroversial to
me, so I think we could just apply it soon?

Greetings,

Andres Freund