public inbox for [email protected]
help / color / mirror / Atom feedWindows default locale vs initdb
44+ messages / 12 participants
[nested] [flat]
* Windows default locale vs initdb
@ 2021-04-19 05:42 Thomas Munro <[email protected]>
2021-04-19 08:52 ` Re: Windows default locale vs initdb Pavel Stehule <[email protected]>
2021-04-19 18:16 ` Re: Windows default locale vs initdb Peter Eisentraut <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
0 siblings, 3 replies; 44+ messages in thread
From: Thomas Munro @ 2021-04-19 05:42 UTC (permalink / raw)
To: pgsql-hackers
Hi,
Moving this topic into its own thread from the one about collation
versions, because it concerns pre-existing problems, and that thread
is long.
Currently initdb sets up template databases with old-style Windows
locale names reported by the OS, and they seem to have caused us quite
a few problems over the years:
db29620d "Work around Windows locale name with non-ASCII character."
aa1d2fc5 "Another attempt at fixing Windows Norwegian locale."
db477b69 "Deal with yet another issue related to "Norwegian (Bokmål)"..."
9f12a3b9 "Tolerate version lookup failure for old style Windows locale..."
... and probably more, and also various threads about , for example,
"German_German.1252" vs "German_Switzerland.1252" which seem to get
confused or badly canonicalised or rejected somewhere in the mix.
I hadn't focused on any of that before, being a non-Windows-user, but
the entire contents of win32setlocale.c supports the theory that
Windows' manual meant what it said when it said[1]:
"We do not recommend this form for locale strings embedded in
code or serialized to storage, because these strings are more likely
to be changed by an operating system update than the locale name
form."
I suppose that was the only form available at the time the code was
written, so there was no choice. The question we asked ourselves
multiple times in the other thread was how we're supposed to get to
the modern BCP 47 form when creating the template databases. It looks
like one possibility, since Vista, is to call
GetUserDefaultLocaleName()[2], which doesn't appear to have been
discussed before on this list. That doesn't allow you to ask for the
default for each individual category, but I don't know if that is even
a concept for Windows user settings. It may be that some of the other
nearby functions give a better answer for some reason. But one thing
is clear from a test that someone kindly ran for me: it reports
standardised strings like "en-NZ", not strings like "English_New
Zealand.1252".
No patch, but I wondered if any Windows hackers have any feedback on
relative sanity of trying to fix all these problems this way.
[1] https://docs.microsoft.com/en-us/cpp/c-runtime-library/locale-names-languages-and-country-region-str...
[2] https://docs.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getuserdefaultlocalename
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2021-04-19 08:52 ` Pavel Stehule <[email protected]>
2021-04-19 10:52 ` Re: Windows default locale vs initdb Andrew Dunstan <[email protected]>
2 siblings, 1 reply; 44+ messages in thread
From: Pavel Stehule @ 2021-04-19 08:52 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
po 19. 4. 2021 v 7:43 odesílatel Thomas Munro <[email protected]>
napsal:
> Hi,
>
> Moving this topic into its own thread from the one about collation
> versions, because it concerns pre-existing problems, and that thread
> is long.
>
> Currently initdb sets up template databases with old-style Windows
> locale names reported by the OS, and they seem to have caused us quite
> a few problems over the years:
>
> db29620d "Work around Windows locale name with non-ASCII character."
> aa1d2fc5 "Another attempt at fixing Windows Norwegian locale."
> db477b69 "Deal with yet another issue related to "Norwegian (Bokmål)"..."
> 9f12a3b9 "Tolerate version lookup failure for old style Windows locale..."
>
> ... and probably more, and also various threads about , for example,
> "German_German.1252" vs "German_Switzerland.1252" which seem to get
> confused or badly canonicalised or rejected somewhere in the mix.
>
> I hadn't focused on any of that before, being a non-Windows-user, but
> the entire contents of win32setlocale.c supports the theory that
> Windows' manual meant what it said when it said[1]:
>
> "We do not recommend this form for locale strings embedded in
> code or serialized to storage, because these strings are more likely
> to be changed by an operating system update than the locale name
> form."
>
> I suppose that was the only form available at the time the code was
> written, so there was no choice. The question we asked ourselves
> multiple times in the other thread was how we're supposed to get to
> the modern BCP 47 form when creating the template databases. It looks
> like one possibility, since Vista, is to call
> GetUserDefaultLocaleName()[2], which doesn't appear to have been
> discussed before on this list. That doesn't allow you to ask for the
> default for each individual category, but I don't know if that is even
> a concept for Windows user settings. It may be that some of the other
> nearby functions give a better answer for some reason. But one thing
> is clear from a test that someone kindly ran for me: it reports
> standardised strings like "en-NZ", not strings like "English_New
> Zealand.1252".
>
> No patch, but I wondered if any Windows hackers have any feedback on
> relative sanity of trying to fix all these problems this way.
>
Last weekend I talked with one user about one interesting (and messing)
issue. They needed to create a new database with Czech collation on Azure
SAS. There was not any entry in pg_collation for Czech language. The reply
from Microsoft support was to use CREATE DATABASE xxx TEMPLATE 'template0'
ENCODING 'utf8' LOCALE 'cs_CZ.UTF8' and it was working.
Regards
Pavel
> [1]
> https://docs.microsoft.com/en-us/cpp/c-runtime-library/locale-names-languages-and-country-region-str...
> [2]
> https://docs.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-getuserdefaultlocalename
>
>
>
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-04-19 08:52 ` Re: Windows default locale vs initdb Pavel Stehule <[email protected]>
@ 2021-04-19 10:52 ` Andrew Dunstan <[email protected]>
2021-04-19 10:57 ` Re: Windows default locale vs initdb Pavel Stehule <[email protected]>
2021-04-19 14:26 ` Re: Windows default locale vs initdb Dave Page <[email protected]>
0 siblings, 2 replies; 44+ messages in thread
From: Andrew Dunstan @ 2021-04-19 10:52 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
On Mon, Apr 19, 2021 at 4:53 AM Pavel Stehule <[email protected]>
wrote:
>
>
> po 19. 4. 2021 v 7:43 odesílatel Thomas Munro <[email protected]>
> napsal:
>
>> Hi,
>>
>> Moving this topic into its own thread from the one about collation
>> versions, because it concerns pre-existing problems, and that thread
>> is long.
>>
>> Currently initdb sets up template databases with old-style Windows
>> locale names reported by the OS, and they seem to have caused us quite
>> a few problems over the years:
>>
>> db29620d "Work around Windows locale name with non-ASCII character."
>> aa1d2fc5 "Another attempt at fixing Windows Norwegian locale."
>> db477b69 "Deal with yet another issue related to "Norwegian (Bokmål)"..."
>> 9f12a3b9 "Tolerate version lookup failure for old style Windows locale..."
>>
>> ... and probably more, and also various threads about , for example,
>> "German_German.1252" vs "German_Switzerland.1252" which seem to get
>> confused or badly canonicalised or rejected somewhere in the mix.
>>
>> I hadn't focused on any of that before, being a non-Windows-user, but
>> the entire contents of win32setlocale.c supports the theory that
>> Windows' manual meant what it said when it said[1]:
>>
>> "We do not recommend this form for locale strings embedded in
>> code or serialized to storage, because these strings are more likely
>> to be changed by an operating system update than the locale name
>> form."
>>
>> I suppose that was the only form available at the time the code was
>> written, so there was no choice. The question we asked ourselves
>> multiple times in the other thread was how we're supposed to get to
>> the modern BCP 47 form when creating the template databases. It looks
>> like one possibility, since Vista, is to call
>> GetUserDefaultLocaleName()[2], which doesn't appear to have been
>> discussed before on this list. That doesn't allow you to ask for the
>> default for each individual category, but I don't know if that is even
>> a concept for Windows user settings. It may be that some of the other
>> nearby functions give a better answer for some reason. But one thing
>> is clear from a test that someone kindly ran for me: it reports
>> standardised strings like "en-NZ", not strings like "English_New
>> Zealand.1252".
>>
>> No patch, but I wondered if any Windows hackers have any feedback on
>> relative sanity of trying to fix all these problems this way.
>>
>
> Last weekend I talked with one user about one interesting (and messing)
> issue. They needed to create a new database with Czech collation on Azure
> SAS. There was not any entry in pg_collation for Czech language. The reply
> from Microsoft support was to use CREATE DATABASE xxx TEMPLATE 'template0'
> ENCODING 'utf8' LOCALE 'cs_CZ.UTF8' and it was working.
>
>
>
My understanding from Microsoft staff at conferences is that Azure's
PostgreSQL SAS runs on linux, not WIndows.
cheers
andrew
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-04-19 08:52 ` Re: Windows default locale vs initdb Pavel Stehule <[email protected]>
2021-04-19 10:52 ` Re: Windows default locale vs initdb Andrew Dunstan <[email protected]>
@ 2021-04-19 10:57 ` Pavel Stehule <[email protected]>
1 sibling, 0 replies; 44+ messages in thread
From: Pavel Stehule @ 2021-04-19 10:57 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
po 19. 4. 2021 v 12:52 odesílatel Andrew Dunstan <[email protected]>
napsal:
>
>
> On Mon, Apr 19, 2021 at 4:53 AM Pavel Stehule <[email protected]>
> wrote:
>
>>
>>
>> po 19. 4. 2021 v 7:43 odesílatel Thomas Munro <[email protected]>
>> napsal:
>>
>>> Hi,
>>>
>>> Moving this topic into its own thread from the one about collation
>>> versions, because it concerns pre-existing problems, and that thread
>>> is long.
>>>
>>> Currently initdb sets up template databases with old-style Windows
>>> locale names reported by the OS, and they seem to have caused us quite
>>> a few problems over the years:
>>>
>>> db29620d "Work around Windows locale name with non-ASCII character."
>>> aa1d2fc5 "Another attempt at fixing Windows Norwegian locale."
>>> db477b69 "Deal with yet another issue related to "Norwegian (Bokmål)"..."
>>> 9f12a3b9 "Tolerate version lookup failure for old style Windows
>>> locale..."
>>>
>>> ... and probably more, and also various threads about , for example,
>>> "German_German.1252" vs "German_Switzerland.1252" which seem to get
>>> confused or badly canonicalised or rejected somewhere in the mix.
>>>
>>> I hadn't focused on any of that before, being a non-Windows-user, but
>>> the entire contents of win32setlocale.c supports the theory that
>>> Windows' manual meant what it said when it said[1]:
>>>
>>> "We do not recommend this form for locale strings embedded in
>>> code or serialized to storage, because these strings are more likely
>>> to be changed by an operating system update than the locale name
>>> form."
>>>
>>> I suppose that was the only form available at the time the code was
>>> written, so there was no choice. The question we asked ourselves
>>> multiple times in the other thread was how we're supposed to get to
>>> the modern BCP 47 form when creating the template databases. It looks
>>> like one possibility, since Vista, is to call
>>> GetUserDefaultLocaleName()[2], which doesn't appear to have been
>>> discussed before on this list. That doesn't allow you to ask for the
>>> default for each individual category, but I don't know if that is even
>>> a concept for Windows user settings. It may be that some of the other
>>> nearby functions give a better answer for some reason. But one thing
>>> is clear from a test that someone kindly ran for me: it reports
>>> standardised strings like "en-NZ", not strings like "English_New
>>> Zealand.1252".
>>>
>>> No patch, but I wondered if any Windows hackers have any feedback on
>>> relative sanity of trying to fix all these problems this way.
>>>
>>
>> Last weekend I talked with one user about one interesting (and messing)
>> issue. They needed to create a new database with Czech collation on Azure
>> SAS. There was not any entry in pg_collation for Czech language. The reply
>> from Microsoft support was to use CREATE DATABASE xxx TEMPLATE 'template0'
>> ENCODING 'utf8' LOCALE 'cs_CZ.UTF8' and it was working.
>>
>>
>>
> My understanding from Microsoft staff at conferences is that Azure's
> PostgreSQL SAS runs on linux, not WIndows.
>
I had different informations, but still there was something wrong because
no czech locales was in pg_collation
>
> cheers
>
> andrew
>
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-04-19 08:52 ` Re: Windows default locale vs initdb Pavel Stehule <[email protected]>
2021-04-19 10:52 ` Re: Windows default locale vs initdb Andrew Dunstan <[email protected]>
@ 2021-04-19 14:26 ` Dave Page <[email protected]>
2021-04-19 16:28 ` Re: Windows default locale vs initdb Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 44+ messages in thread
From: Dave Page @ 2021-04-19 14:26 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On Mon, Apr 19, 2021 at 11:52 AM Andrew Dunstan <[email protected]> wrote:
>
> My understanding from Microsoft staff at conferences is that Azure's
> PostgreSQL SAS runs on linux, not WIndows.
>
This is from a regular Azure Database for PostgreSQL single server:
postgres=> select version();
version
------------------------------------------------------------
PostgreSQL 11.6, compiled by Visual C++ build 1800, 64-bit
(1 row)
And this is from the new Flexible Server preview:
postgres=> select version();
version
-----------------------------------------------------------------------------------------------------------------
PostgreSQL 12.6 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu
5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609, 64-bit
(1 row)
So I guess it's a case of "it depends".
--
Dave Page
Blog: https://pgsnake.blogspot.com
Twitter: @pgsnake
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-04-19 08:52 ` Re: Windows default locale vs initdb Pavel Stehule <[email protected]>
2021-04-19 10:52 ` Re: Windows default locale vs initdb Andrew Dunstan <[email protected]>
2021-04-19 14:26 ` Re: Windows default locale vs initdb Dave Page <[email protected]>
@ 2021-04-19 16:28 ` Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 44+ messages in thread
From: Andrew Dunstan @ 2021-04-19 16:28 UTC (permalink / raw)
To: Dave Page <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers
On 4/19/21 10:26 AM, Dave Page wrote:
>
>
> On Mon, Apr 19, 2021 at 11:52 AM Andrew Dunstan <[email protected]
> <mailto:[email protected]>> wrote:
>
>
> My understanding from Microsoft staff at conferences is that
> Azure's PostgreSQL SAS runs on linux, not WIndows.
>
>
> This is from a regular Azure Database for PostgreSQL single server:
>
> postgres=> select version();
> version
> ------------------------------------------------------------
> PostgreSQL 11.6, compiled by Visual C++ build 1800, 64-bit
> (1 row)
>
> And this is from the new Flexible Server preview:
>
> postgres=> select version();
> version
>
> -----------------------------------------------------------------------------------------------------------------
> PostgreSQL 12.6 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu
> 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609, 64-bit
> (1 row)
>
> So I guess it's a case of "it depends".
>
Good to know. A year or two back at more than one conference I tried to enlist some of these folks in helping us with Windows PostgreSQL and their reply was that they knew nothing about it because they were on Linux :-) I guess things change over time.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2021-04-19 18:16 ` Peter Eisentraut <[email protected]>
2 siblings, 0 replies; 44+ messages in thread
From: Peter Eisentraut @ 2021-04-19 18:16 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; pgsql-hackers
On 19.04.21 07:42, Thomas Munro wrote:
> It looks
> like one possibility, since Vista, is to call
> GetUserDefaultLocaleName()[2], which doesn't appear to have been
> discussed before on this list. That doesn't allow you to ask for the
> default for each individual category, but I don't know if that is even
> a concept for Windows user settings.
pg_newlocale_from_collation() doesn't support collcollate != collctype
on Windows anyway, so that wouldn't be an issue.
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2021-05-16 04:29 ` Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2 siblings, 1 reply; 44+ messages in thread
From: Noah Misch @ 2021-05-16 04:29 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
On Mon, Apr 19, 2021 at 05:42:51PM +1200, Thomas Munro wrote:
> Currently initdb sets up template databases with old-style Windows
> locale names reported by the OS, and they seem to have caused us quite
> a few problems over the years:
>
> db29620d "Work around Windows locale name with non-ASCII character."
> aa1d2fc5 "Another attempt at fixing Windows Norwegian locale."
> db477b69 "Deal with yet another issue related to "Norwegian (Bokmål)"..."
> 9f12a3b9 "Tolerate version lookup failure for old style Windows locale..."
> I suppose that was the only form available at the time the code was
> written, so there was no choice.
Right.
> The question we asked ourselves
> multiple times in the other thread was how we're supposed to get to
> the modern BCP 47 form when creating the template databases. It looks
> like one possibility, since Vista, is to call
> GetUserDefaultLocaleName()[2]
> No patch, but I wondered if any Windows hackers have any feedback on
> relative sanity of trying to fix all these problems this way.
Sounds reasonable. If PostgreSQL v15 would otherwise run on Windows Server
2003 R2, this is a good time to let that support end.
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
@ 2021-12-15 10:32 ` Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Juan José Santamaría Flecha @ 2021-12-15 10:32 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Thomas Munro <[email protected]>; pgsql-hackers
On Sun, May 16, 2021 at 6:29 AM Noah Misch <[email protected]> wrote:
> On Mon, Apr 19, 2021 at 05:42:51PM +1200, Thomas Munro wrote:
>
> > The question we asked ourselves
> > multiple times in the other thread was how we're supposed to get to
> > the modern BCP 47 form when creating the template databases. It looks
> > like one possibility, since Vista, is to call
> > GetUserDefaultLocaleName()[2]
>
> > No patch, but I wondered if any Windows hackers have any feedback on
> > relative sanity of trying to fix all these problems this way.
>
> Sounds reasonable. If PostgreSQL v15 would otherwise run on Windows Server
> 2003 R2, this is a good time to let that support end.
>
> The value returned by GetUserDefaultLocaleName() is a system configured
parameter, independent of what you set with setlocale(). It might be
reasonable for initdb but not for a backend in most cases.
You can get the locale POSIX-ish name using GetLocaleInfoEx(), but this is
no longer recommended, because using LCIDs is no longer recommended [1].
Although, this would work for legacy locales. Please find attached a POC
patch showing this approach.
[1] https://docs.microsoft.com/en-us/globalization/locale/locale-names
Regards,
Juan José Santamaría Flecha
Attachments:
[application/octet-stream] 0001-POC-Make-Windows-locale-POSIX-looking.patch (1.1K, ../../CAC+AXB2c7icYiXJCwv+4=Di=xZft0NxT=5LP3BcjFnF8ZuBhAg@mail.gmail.com/3-0001-POC-Make-Windows-locale-POSIX-looking.patch)
download | inline diff:
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 403adb0..23bf991 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -2040,6 +2040,28 @@ check_locale_name(int category, const char *locale, char **canonname)
/* set the locale with setlocale, to see if it accepts it. */
res = setlocale(category, locale);
+#if defined(WIN32) && _WIN32_WINNT >= 0x0600
+ {
+ wchar_t localename[LOCALE_NAME_MAX_LENGTH];
+ char localebuf[LOCALE_NAME_MAX_LENGTH];
+ LCID lcid;
+ char *hyphen;
+
+ if (MultiByteToWideChar(CP_ACP, 0, res, -1, localename,
+ LOCALE_NAME_MAX_LENGTH) &&
+ GetLocaleInfoEx(localename, LOCALE_RETURN_NUMBER | LOCALE_ILANGUAGE,
+ (LPWSTR)&lcid, sizeof(lcid)) &&
+ LCIDToLocaleName(lcid, localename, LOCALE_NAME_MAX_LENGTH, 0) &&
+ WideCharToMultiByte(CP_ACP, 0, localename, -1, localebuf,
+ LOCALE_NAME_MAX_LENGTH, NULL, NULL))
+ {
+ hyphen = strchr(localebuf, '-');
+ if (hyphen)
+ *hyphen = '_';
+ res = pg_strdup(localebuf);
+ }
+ }
+#endif
/* save canonical name if requested. */
if (res && canonname)
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
@ 2022-07-18 22:58 ` Thomas Munro <[email protected]>
2022-07-19 02:46 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-20 08:34 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
0 siblings, 2 replies; 44+ messages in thread
From: Thomas Munro @ 2022-07-18 22:58 UTC (permalink / raw)
To: Juan José Santamaría Flecha <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Wed, Dec 15, 2021 at 11:32 PM Juan José Santamaría Flecha
<[email protected]> wrote:
> On Sun, May 16, 2021 at 6:29 AM Noah Misch <[email protected]> wrote:
>> On Mon, Apr 19, 2021 at 05:42:51PM +1200, Thomas Munro wrote:
>> > The question we asked ourselves
>> > multiple times in the other thread was how we're supposed to get to
>> > the modern BCP 47 form when creating the template databases. It looks
>> > like one possibility, since Vista, is to call
>> > GetUserDefaultLocaleName()[2]
>>
>> > No patch, but I wondered if any Windows hackers have any feedback on
>> > relative sanity of trying to fix all these problems this way.
>>
>> Sounds reasonable. If PostgreSQL v15 would otherwise run on Windows Server
>> 2003 R2, this is a good time to let that support end.
>>
> The value returned by GetUserDefaultLocaleName() is a system configured parameter, independent of what you set with setlocale(). It might be reasonable for initdb but not for a backend in most cases.
Agreed. Only for initdb, and only if you didn't specify a locale name
on the command line.
> You can get the locale POSIX-ish name using GetLocaleInfoEx(), but this is no longer recommended, because using LCIDs is no longer recommended [1]. Although, this would work for legacy locales. Please find attached a POC patch showing this approach.
Now that museum-grade Windows has been defenestrated, we are free to
call GetUserDefaultLocaleName(). Here's a patch.
One thing you did in your patch that I disagree with, I think, was to
convert a BCP 47 name to a POSIX name early, that is, s/-/_/. I think
we should use the locale name exactly as Windows (really, under the
covers, ICU) spells it. There is only one place in the tree today
that really wants a POSIX locale name, and that's LC_MESSAGES,
accessed by GNU gettext, not Windows. We already had code to cope
with that.
I think we should also convert to POSIX format when making the
collname in your pg_import_system_collations() proposal, so that
COLLATE "en_US" works (= a SQL identifier), but that's another
thread[1]. I don't think we should do it in collcollate or
datcollate, which is a string for the OS to interpret.
With my garbage collector hat on, I would like to rip out all of the
support for traditional locale names, eventually. Deleting kludgy
code is easy and fun -- 0002 is a first swing at that -- but there
remains an important unanswered question. How should someone
pg_upgrade a "English_Canada.1521" cluster if we now reject that name?
We'd need to do a conversion to "en-CA", or somehow tell the user to.
Hmmmm.
[1] https://www.postgresql.org/message-id/flat/CAC%2BAXB0WFjJGL1n33bRv8wsnV-3PZD0A7kkjJ2KjPH0dOWqQdg%40m...
Attachments:
[text/x-patch] 0001-Default-to-BCP-47-locale-in-initdb-on-Windows.patch (3.5K, ../../CA+hUKGKrGP3BhS+h1zdLYRRvhtbyaz4bhjUyPktAQfH=uS2JXw@mail.gmail.com/2-0001-Default-to-BCP-47-locale-in-initdb-on-Windows.patch)
download | inline diff:
From d6d677fd185242590f0f716cf69d09e735122ff7 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 19 Jul 2022 06:31:17 +1200
Subject: [PATCH 1/2] Default to BCP 47 locale in initdb on Windows.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Avoid selecting traditional Windows locale names written with English
words, because they are unstable and not recommended for use in
databases. Since setlocale() returns such names, instead use
GetUserDefaultLocaleName() if the user didn't provide an explicit
locale.
Also update the documentation to recommend BCP 47 over the traditional
names when providing explicit values to initdb.
Reviewed-by: Juan José Santamaría Flecha <[email protected]>
Discussion: https://postgr.es/m/CA%2BhUKGJ%3DXThErgAQRoqfCy1bKPxXVuF0%3D2zDbB%2BSxDs59pv7Fw%40mail.gmail.com
---
doc/src/sgml/charset.sgml | 10 ++++++++--
src/bin/initdb/initdb.c | 28 +++++++++++++++++++++++++++-
2 files changed, 35 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index 445fd175d8..22e33f0f57 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -83,8 +83,14 @@ initdb --locale=sv_SE
system under what names depends on what was provided by the operating
system vendor and what was installed. On most Unix systems, the command
<literal>locale -a</literal> will provide a list of available locales.
- Windows uses more verbose locale names, such as <literal>German_Germany</literal>
- or <literal>Swedish_Sweden.1252</literal>, but the principles are the same.
+ </para>
+
+ <para>
+ Windows uses BCP 47 language tags.
+ For example, <literal>sv-SE</literal> represents Swedish as spoken in Sweden.
+ Windows also supports more verbose locale names based on English words,
+ such as <literal>German_Germany</literal> or <literal>Swedish_Sweden.1252</literal>,
+ but these are not recommended.
</para>
<para>
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 89b888eaa5..57c5ecf3cf 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -59,6 +59,10 @@
#include "sys/mman.h"
#endif
+#ifdef WIN32
+#include <winnls.h>
+#endif
+
#include "access/xlog_internal.h"
#include "catalog/pg_authid_d.h"
#include "catalog/pg_class_d.h" /* pgrminclude ignore */
@@ -2022,7 +2026,27 @@ check_locale_name(int category, const char *locale, char **canonname)
/* for setlocale() call */
if (!locale)
- locale = "";
+ {
+#ifdef WIN32
+ wchar_t wide_name[LOCALE_NAME_MAX_LENGTH];
+ char name[LOCALE_NAME_MAX_LENGTH];
+
+ /* use Windows API to find the default in BCP47 format */
+ if (GetUserDefaultLocaleName(wide_name, LOCALE_NAME_MAX_LENGTH) == 0)
+ pg_fatal("failed to get default locale name, error %d",
+ GetLastError());
+ if (WideCharToMultiByte(CP_ACP, 0, wide_name, -1, name,
+ LOCALE_NAME_MAX_LENGTH, NULL, NULL) == 0)
+ pg_fatal("failed to convert locale name, error %d",
+ GetLastError());
+ locale = pg_strdup(name);
+#else
+ /* use environment to find the default */
+ locale = pg_strdup("");
+#endif
+ }
+ else
+ locale = pg_strdup(locale);
/* set the locale with setlocale, to see if it accepts it. */
res = setlocale(category, locale);
@@ -2054,6 +2078,8 @@ check_locale_name(int category, const char *locale, char **canonname)
pg_fatal("invalid locale settings; check LANG and LC_* environment variables");
}
}
+
+ free(locale);
}
/*
--
2.30.2
[text/x-patch] 0002-Remove-support-for-old-Windows-locale-names.patch (19.6K, ../../CA+hUKGKrGP3BhS+h1zdLYRRvhtbyaz4bhjUyPktAQfH=uS2JXw@mail.gmail.com/3-0002-Remove-support-for-old-Windows-locale-names.patch)
download | inline diff:
From a440c6675d92682782f8a10667eec30d35e6fd9c Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 19 Jul 2022 08:53:08 +1200
Subject: [PATCH 2/2] Remove support for old Windows locale names.
We now use BCP 47 locale names by default and also advise those for
explicit use. Remove support for munging the old unstable and
unsystematic English word-style locale names.
XXX When could we do this?
XXX How are you supposed to pg_upgrade a system with datcollate =
"English_Canada" to "en-CA"?
---
configure | 6 -
configure.ac | 1 -
doc/src/sgml/charset.sgml | 2 +-
src/backend/utils/adt/pg_locale.c | 235 ++----------------------------
src/include/port/win32_port.h | 9 --
src/port/win32setlocale.c | 193 ------------------------
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 18 insertions(+), 430 deletions(-)
delete mode 100644 src/port/win32setlocale.c
diff --git a/configure b/configure
index a4f4d321fb..5b7704352d 100755
--- a/configure
+++ b/configure
@@ -17174,12 +17174,6 @@ esac
;;
esac
- case " $LIBOBJS " in
- *" win32setlocale.$ac_objext "* ) ;;
- *) LIBOBJS="$LIBOBJS win32setlocale.$ac_objext"
- ;;
-esac
-
case " $LIBOBJS " in
*" win32stat.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
diff --git a/configure.ac b/configure.ac
index 5bd29a4d2f..358580e6d6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1995,7 +1995,6 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32ntdll)
AC_LIBOBJ(win32security)
- AC_LIBOBJ(win32setlocale)
AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index 22e33f0f57..e02d7e0020 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -90,7 +90,7 @@ initdb --locale=sv_SE
For example, <literal>sv-SE</literal> represents Swedish as spoken in Sweden.
Windows also supports more verbose locale names based on English words,
such as <literal>German_Germany</literal> or <literal>Swedish_Sweden.1252</literal>,
- but these are not recommended.
+ but these should not be used in PostgreSQL.
</para>
<para>
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 607a4b7340..044f62de2b 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -118,7 +118,7 @@ static HTAB *collation_cache = NULL;
#if defined(WIN32) && defined(LC_MESSAGES)
-static char *IsoLocaleName(const char *); /* MSVC specific */
+static char *PosixLocaleName(const char *);
#endif
#ifdef USE_ICU
@@ -204,10 +204,7 @@ pg_perm_setlocale(int category, const char *locale)
case LC_MESSAGES:
envvar = "LC_MESSAGES";
#ifdef WIN32
- result = IsoLocaleName(locale);
- if (result == NULL)
- result = (char *) locale;
- elog(DEBUG3, "IsoLocaleName() executed; locale: \"%s\"", result);
+ locale = PosixLocaleName(locale);
#endif /* WIN32 */
break;
#endif /* LC_MESSAGES */
@@ -905,218 +902,35 @@ cache_locale_time(void)
#if defined(WIN32) && defined(LC_MESSAGES)
/*
- * Convert a Windows setlocale() argument to a Unix-style one.
+ * Convert a Windows BCP 47 locale name to a POSIX one.
*
* Regardless of platform, we install message catalogs under a Unix-style
* LL[_CC][.ENCODING][@VARIANT] naming convention. Only LC_MESSAGES settings
* following that style will elicit localized interface strings.
*
- * Before Visual Studio 2012 (msvcr110.dll), Windows setlocale() accepted "C"
- * (but not "c") and strings of the form <Language>[_<Country>][.<CodePage>],
- * case-insensitive. setlocale() returns the fully-qualified form; for
- * example, setlocale("thaI") returns "Thai_Thailand.874". Internally,
- * setlocale() and _create_locale() select a "locale identifier"[1] and store
- * it in an undocumented _locale_t field. From that LCID, we can retrieve the
- * ISO 639 language and the ISO 3166 country. Character encoding does not
- * matter, because the server and client encodings govern that.
- *
- * Windows Vista introduced the "locale name" concept[2], closely following
- * RFC 4646. Locale identifiers are now deprecated. Starting with Visual
- * Studio 2012, setlocale() accepts locale names in addition to the strings it
- * accepted historically. It does not standardize them; setlocale("Th-tH")
- * returns "Th-tH". setlocale(category, "") still returns a traditional
- * string. Furthermore, msvcr110.dll changed the undocumented _locale_t
- * content to carry locale names instead of locale identifiers.
- *
- * Visual Studio 2015 should still be able to do the same as Visual Studio
- * 2012, but the declaration of locale_name is missing in _locale_t, causing
- * this code compilation to fail, hence this falls back instead on to
- * enumerating all system locales by using EnumSystemLocalesEx to find the
- * required locale name. If the input argument is in Unix-style then we can
- * get ISO Locale name directly by using GetLocaleInfoEx() with LCType as
- * LOCALE_SNAME.
- *
- * MinGW headers declare _create_locale(), but msvcrt.dll lacks that symbol in
- * releases before Windows 8. IsoLocaleName() always fails in a MinGW-built
- * postgres.exe, so only Unix-style values of the lc_messages GUC can elicit
- * localized messages. In particular, every lc_messages setting that initdb
- * can select automatically will yield only C-locale messages. XXX This could
- * be fixed by running the fully-qualified locale name through a lookup table.
- *
- * This function returns a pointer to a static buffer bearing the converted
- * name or NULL if conversion fails.
- *
- * [1] https://docs.microsoft.com/en-us/windows/win32/intl/locale-identifiers
- * [2] https://docs.microsoft.com/en-us/windows/win32/intl/locale-names
- */
-
-/*
- * Callback function for EnumSystemLocalesEx() in get_iso_localename().
- *
- * This function enumerates all system locales, searching for one that matches
- * an input with the format: <Language>[_<Country>], e.g.
- * English[_United States]
- *
- * The input is a three wchar_t array as an LPARAM. The first element is the
- * locale_name we want to match, the second element is an allocated buffer
- * where the Unix-style locale is copied if a match is found, and the third
- * element is the search status, 1 if a match was found, 0 otherwise.
+ * Historically, verbose, but unsystematic and unstable names like
+ * "Thai_Thailand.874" were supported, but now only BCP 47 input is expected.
+ * That means we just need to be able to convert "en-US" to "en_US".
*/
-static BOOL CALLBACK
-search_locale_enum(LPWSTR pStr, DWORD dwFlags, LPARAM lparam)
-{
- wchar_t test_locale[LOCALE_NAME_MAX_LENGTH];
- wchar_t **argv;
-
- (void) (dwFlags);
-
- argv = (wchar_t **) lparam;
- *argv[2] = (wchar_t) 0;
-
- memset(test_locale, 0, sizeof(test_locale));
-
- /* Get the name of the <Language> in English */
- if (GetLocaleInfoEx(pStr, LOCALE_SENGLISHLANGUAGENAME,
- test_locale, LOCALE_NAME_MAX_LENGTH))
- {
- /*
- * If the enumerated locale does not have a hyphen ("en") OR the
- * lc_message input does not have an underscore ("English"), we only
- * need to compare the <Language> tags.
- */
- if (wcsrchr(pStr, '-') == NULL || wcsrchr(argv[0], '_') == NULL)
- {
- if (_wcsicmp(argv[0], test_locale) == 0)
- {
- wcscpy(argv[1], pStr);
- *argv[2] = (wchar_t) 1;
- return FALSE;
- }
- }
-
- /*
- * We have to compare a full <Language>_<Country> tag, so we append
- * the underscore and name of the country/region in English, e.g.
- * "English_United States".
- */
- else
- {
- size_t len;
-
- wcscat(test_locale, L"_");
- len = wcslen(test_locale);
- if (GetLocaleInfoEx(pStr, LOCALE_SENGLISHCOUNTRYNAME,
- test_locale + len,
- LOCALE_NAME_MAX_LENGTH - len))
- {
- if (_wcsicmp(argv[0], test_locale) == 0)
- {
- wcscpy(argv[1], pStr);
- *argv[2] = (wchar_t) 1;
- return FALSE;
- }
- }
- }
- }
-
- return TRUE;
-}
-
-/*
- * This function converts a Windows locale name to an ISO formatted version
- * for Visual Studio 2015 or greater.
- *
- * Returns NULL, if no valid conversion was found.
- */
-static char *
-get_iso_localename(const char *winlocname)
-{
- wchar_t wc_locale_name[LOCALE_NAME_MAX_LENGTH];
- wchar_t buffer[LOCALE_NAME_MAX_LENGTH];
- static char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
- char *period;
- int len;
- int ret_val;
-
- /*
- * Valid locales have the following syntax:
- * <Language>[_<Country>[.<CodePage>]]
- *
- * GetLocaleInfoEx can only take locale name without code-page and for the
- * purpose of this API the code-page doesn't matter.
- */
- period = strchr(winlocname, '.');
- if (period != NULL)
- len = period - winlocname;
- else
- len = pg_mbstrlen(winlocname);
-
- memset(wc_locale_name, 0, sizeof(wc_locale_name));
- memset(buffer, 0, sizeof(buffer));
- MultiByteToWideChar(CP_ACP, 0, winlocname, len, wc_locale_name,
- LOCALE_NAME_MAX_LENGTH);
-
- /*
- * If the lc_messages is already a Unix-style string, we have a direct
- * match with LOCALE_SNAME, e.g. en-US, en_US.
- */
- ret_val = GetLocaleInfoEx(wc_locale_name, LOCALE_SNAME, (LPWSTR) &buffer,
- LOCALE_NAME_MAX_LENGTH);
- if (!ret_val)
- {
- /*
- * Search for a locale in the system that matches language and country
- * name.
- */
- wchar_t *argv[3];
-
- argv[0] = wc_locale_name;
- argv[1] = buffer;
- argv[2] = (wchar_t *) &ret_val;
- EnumSystemLocalesEx(search_locale_enum, LOCALE_WINDOWS, (LPARAM) argv,
- NULL);
- }
-
- if (ret_val)
- {
- size_t rc;
- char *hyphen;
-
- /* Locale names use only ASCII, any conversion locale suffices. */
- rc = wchar2char(iso_lc_messages, buffer, sizeof(iso_lc_messages), NULL);
- if (rc == -1 || rc == sizeof(iso_lc_messages))
- return NULL;
-
- /*
- * Simply replace the hyphen with an underscore. See comments in
- * IsoLocaleName.
- */
- hyphen = strchr(iso_lc_messages, '-');
- if (hyphen)
- *hyphen = '_';
- return iso_lc_messages;
- }
-
- return NULL;
-}
-
static char *
-IsoLocaleName(const char *winlocname)
+PosixLocaleName(const char *winlocname)
{
-#if defined(_MSC_VER)
- static char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
+ char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
+ char *hyphen;
if (pg_strcasecmp("c", winlocname) == 0 ||
pg_strcasecmp("posix", winlocname) == 0)
{
strcpy(iso_lc_messages, "C");
- return iso_lc_messages;
}
else
- return get_iso_localename(winlocname);
-
-#endif /* defined(_MSC_VER) */
- return NULL; /* Not supported on this version of msvc/mingw */
+ {
+ strlcpy(iso_lc_messages, winlocname, sizeof(iso_lc_messages));
+ hypen = strchr(iso_lc_messages, '-');
+ if (hyphen)
+ *hyphen = '_';
+ }
+ return pg_strcpy(iso_lc_messages);
}
#endif /* WIN32 && LC_MESSAGES */
@@ -1680,33 +1494,16 @@ get_collation_actual_version(char collprovider, const char *collcollate)
ereport(ERROR,
(errmsg("could not load locale \"%s\"", collcollate)));
#elif defined(WIN32)
- /*
- * If we are targeting Windows Vista and above, we can ask for a name
- * given a collation name (earlier versions required a location code
- * that we don't have).
- */
NLSVERSIONINFOEX version = {sizeof(NLSVERSIONINFOEX)};
WCHAR wide_collcollate[LOCALE_NAME_MAX_LENGTH];
MultiByteToWideChar(CP_ACP, 0, collcollate, -1, wide_collcollate,
LOCALE_NAME_MAX_LENGTH);
if (!GetNLSVersionEx(COMPARE_STRING, wide_collcollate, &version))
- {
- /*
- * GetNLSVersionEx() wants a language tag such as "en-US", not a
- * locale name like "English_United States.1252". Until those
- * values can be prevented from entering the system, or 100%
- * reliably converted to the more useful tag format, tolerate the
- * resulting error and report that we have no version data.
- */
- if (GetLastError() == ERROR_INVALID_PARAMETER)
- return NULL;
-
ereport(ERROR,
(errmsg("could not get collation version for locale \"%s\": error code %lu",
collcollate,
GetLastError())));
- }
collversion = psprintf("%ld.%ld,%ld.%ld",
(version.dwNLSVersion >> 8) & 0xFFFF,
version.dwNLSVersion & 0xFF,
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 5121c0c626..13ed144a8f 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -437,15 +437,6 @@ extern int _pgstat64(const char *name, struct stat *buf);
#undef setlocale
#endif
-/*
- * Define our own wrapper macro around setlocale() to work around bugs in
- * Windows' native setlocale() function.
- */
-extern char *pgwin32_setlocale(int category, const char *locale);
-
-#define setlocale(a,b) pgwin32_setlocale(a,b)
-
-
/* In backend/port/win32/signal.c */
extern PGDLLIMPORT volatile int pg_signal_queue;
extern PGDLLIMPORT int pg_signal_mask;
diff --git a/src/port/win32setlocale.c b/src/port/win32setlocale.c
deleted file mode 100644
index aadd09a4e9..0000000000
--- a/src/port/win32setlocale.c
+++ /dev/null
@@ -1,193 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * win32setlocale.c
- * Wrapper to work around bugs in Windows setlocale() implementation
- *
- * Copyright (c) 2011-2022, PostgreSQL Global Development Group
- *
- * IDENTIFICATION
- * src/port/win32setlocale.c
- *
- *
- * The setlocale() function in Windows is broken in two ways. First, it
- * has a problem with locale names that have a dot in the country name. For
- * example:
- *
- * "Chinese (Traditional)_Hong Kong S.A.R..950"
- *
- * For some reason, setlocale() doesn't accept that as argument, even though
- * setlocale(LC_ALL, NULL) returns exactly that. Fortunately, it accepts
- * various alternative names for such countries, so to work around the broken
- * setlocale() function, we map the troublemaking locale names to accepted
- * aliases, before calling setlocale().
- *
- * The second problem is that the locale name for "Norwegian (Bokmål)"
- * contains a non-ASCII character. That's problematic, because it's not clear
- * what encoding the locale name itself is supposed to be in, when you
- * haven't yet set a locale. Also, it causes problems when the cluster
- * contains databases with different encodings, as the locale name is stored
- * in the pg_database system catalog. To work around that, when setlocale()
- * returns that locale name, map it to a pure-ASCII alias for the same
- * locale.
- *-------------------------------------------------------------------------
- */
-
-#include "c.h"
-
-#undef setlocale
-
-struct locale_map
-{
- /*
- * String in locale name to replace. Can be a single string (end is NULL),
- * or separate start and end strings. If two strings are given, the locale
- * name must contain both of them, and everything between them is
- * replaced. This is used for a poor-man's regexp search, allowing
- * replacement of "start.*end".
- */
- const char *locale_name_start;
- const char *locale_name_end;
-
- const char *replacement; /* string to replace the match with */
-};
-
-/*
- * Mappings applied before calling setlocale(), to the argument.
- */
-static const struct locale_map locale_map_argument[] = {
- /*
- * "HKG" is listed here:
- * http://msdn.microsoft.com/en-us/library/cdax410z%28v=vs.71%29.aspx
- * (Country/Region Strings).
- *
- * "ARE" is the ISO-3166 three-letter code for U.A.E. It is not on the
- * above list, but seems to work anyway.
- */
- {"Hong Kong S.A.R.", NULL, "HKG"},
- {"U.A.E.", NULL, "ARE"},
-
- /*
- * The ISO-3166 country code for Macau S.A.R. is MAC, but Windows doesn't
- * seem to recognize that. And Macau isn't listed in the table of accepted
- * abbreviations linked above. Fortunately, "ZHM" seems to be accepted as
- * an alias for "Chinese (Traditional)_Macau S.A.R..950". I'm not sure
- * where "ZHM" comes from, must be some legacy naming scheme. But hey, it
- * works.
- *
- * Note that unlike HKG and ARE, ZHM is an alias for the *whole* locale
- * name, not just the country part.
- *
- * Some versions of Windows spell it "Macau", others "Macao".
- */
- {"Chinese (Traditional)_Macau S.A.R..950", NULL, "ZHM"},
- {"Chinese_Macau S.A.R..950", NULL, "ZHM"},
- {"Chinese (Traditional)_Macao S.A.R..950", NULL, "ZHM"},
- {"Chinese_Macao S.A.R..950", NULL, "ZHM"},
- {NULL, NULL, NULL}
-};
-
-/*
- * Mappings applied after calling setlocale(), to its return value.
- */
-static const struct locale_map locale_map_result[] = {
- /*
- * "Norwegian (Bokmål)" locale name contains the a-ring character.
- * Map it to a pure-ASCII alias.
- *
- * It's not clear what encoding setlocale() uses when it returns the
- * locale name, so to play it safe, we search for "Norwegian (Bok*l)".
- *
- * Just to make life even more complicated, some versions of Windows spell
- * the locale name without parentheses. Translate that too.
- */
- {"Norwegian (Bokm", "l)_Norway", "Norwegian_Norway"},
- {"Norwegian Bokm", "l_Norway", "Norwegian_Norway"},
- {NULL, NULL, NULL}
-};
-
-#define MAX_LOCALE_NAME_LEN 100
-
-static const char *
-map_locale(const struct locale_map *map, const char *locale)
-{
- static char aliasbuf[MAX_LOCALE_NAME_LEN];
- int i;
-
- /* Check if the locale name matches any of the problematic ones. */
- for (i = 0; map[i].locale_name_start != NULL; i++)
- {
- const char *needle_start = map[i].locale_name_start;
- const char *needle_end = map[i].locale_name_end;
- const char *replacement = map[i].replacement;
- char *match;
- char *match_start = NULL;
- char *match_end = NULL;
-
- match = strstr(locale, needle_start);
- if (match)
- {
- /*
- * Found a match for the first part. If this was a two-part
- * replacement, find the second part.
- */
- match_start = match;
- if (needle_end)
- {
- match = strstr(match_start + strlen(needle_start), needle_end);
- if (match)
- match_end = match + strlen(needle_end);
- else
- match_start = NULL;
- }
- else
- match_end = match_start + strlen(needle_start);
- }
-
- if (match_start)
- {
- /* Found a match. Replace the matched string. */
- int matchpos = match_start - locale;
- int replacementlen = strlen(replacement);
- char *rest = match_end;
- int restlen = strlen(rest);
-
- /* check that the result fits in the static buffer */
- if (matchpos + replacementlen + restlen + 1 > MAX_LOCALE_NAME_LEN)
- return NULL;
-
- memcpy(&aliasbuf[0], &locale[0], matchpos);
- memcpy(&aliasbuf[matchpos], replacement, replacementlen);
- /* includes null terminator */
- memcpy(&aliasbuf[matchpos + replacementlen], rest, restlen + 1);
-
- return aliasbuf;
- }
- }
-
- /* no match, just return the original string */
- return locale;
-}
-
-char *
-pgwin32_setlocale(int category, const char *locale)
-{
- const char *argument;
- char *result;
-
- if (locale == NULL)
- argument = NULL;
- else
- argument = map_locale(locale_map_argument, locale);
-
- /* Call the real setlocale() function */
- result = setlocale(category, argument);
-
- /*
- * setlocale() is specified to return a "char *" that the caller is
- * forbidden to modify, so casting away the "const" is innocuous.
- */
- if (result)
- result = unconstify(char *, map_locale(locale_map_result, result));
-
- return result;
-}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index e4feda10fd..1bc35ef926 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -108,7 +108,7 @@ sub mkvcbuild
pqsignal.c mkdtemp.c qsort.c qsort_arg.c bsearch_arg.c quotes.c system.c
strerror.c tar.c
win32env.c win32error.c win32ntdll.c
- win32security.c win32setlocale.c win32stat.c);
+ win32security.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.30.2
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2022-07-19 02:46 ` Thomas Munro <[email protected]>
2022-07-20 10:26 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
1 sibling, 1 reply; 44+ messages in thread
From: Thomas Munro @ 2022-07-19 02:46 UTC (permalink / raw)
To: Juan José Santamaría Flecha <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Tue, Jul 19, 2022 at 10:58 AM Thomas Munro <[email protected]> wrote:
> Here's a patch.
I added this to the next commitfest, and cfbot promptly told me about
some warnings I needed to fix. That'll teach me to post a patch
tested with "ci-os-only: windows". Looking more closely at some error
messages that report GetLastError() where I'd mixed up %d and %lu, I
see also that I didn't quite follow existing conventions for wording
when reporting Windows error numbers, so I fixed that too.
In the "startcreate" step on CI you can see that it says:
The database cluster will be initialized with locale "en-US".
The default database encoding has accordingly been set to "WIN1252".
The default text search configuration will be set to "english".
As for whether "accordingly" still applies, by the logic of of
win32_langinfo()... Windows still considers WIN1252 to be the default
ANSI code page for "en-US", though it'd work with UTF-8 too. I'm not
sure what to make of that. The goal here was to give Windows users
good defaults, but WIN1252 is probably not what most people actually
want. Hmph.
Attachments:
[text/x-patch] v2-0001-Default-to-BCP-47-locale-in-initdb-on-Windows.patch (3.8K, ../../CA+hUKGK3zubzq_JZ8MU=zzjOLKKgdq_Knyf_Bra042nqj8gf3g@mail.gmail.com/2-v2-0001-Default-to-BCP-47-locale-in-initdb-on-Windows.patch)
download | inline diff:
From 95f2684150e2938f2e555d16bbed4295a6dad279 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 19 Jul 2022 06:31:17 +1200
Subject: [PATCH v2 1/2] Default to BCP 47 locale in initdb on Windows.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Avoid selecting traditional Windows locale names written with English
words, because they are unstable and not recommended for use in
databases. Since setlocale() returns such names, on Windows use
GetUserDefaultLocaleName() if the user didn't provide an explicit
locale.
Also update the documentation to recommend BCP 47 over the traditional
names when providing explicit values to initdb.
Reviewed-by: Juan José Santamaría Flecha <[email protected]>
Discussion: https://postgr.es/m/CA%2BhUKGJ%3DXThErgAQRoqfCy1bKPxXVuF0%3D2zDbB%2BSxDs59pv7Fw%40mail.gmail.com
---
doc/src/sgml/charset.sgml | 10 ++++++++--
src/bin/initdb/initdb.c | 31 +++++++++++++++++++++++++++++--
2 files changed, 37 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index 445fd175d8..b656ca489f 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -83,8 +83,14 @@ initdb --locale=sv_SE
system under what names depends on what was provided by the operating
system vendor and what was installed. On most Unix systems, the command
<literal>locale -a</literal> will provide a list of available locales.
- Windows uses more verbose locale names, such as <literal>German_Germany</literal>
- or <literal>Swedish_Sweden.1252</literal>, but the principles are the same.
+ </para>
+
+ <para>
+ Windows uses BCP 47 language tags, like ICU.
+ For example, <literal>sv-SE</literal> represents Swedish as spoken in Sweden.
+ Windows also supports more verbose locale names based on English words,
+ such as <literal>German_Germany</literal> or <literal>Swedish_Sweden.1252</literal>,
+ but these are not recommended.
</para>
<para>
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 89b888eaa5..3af08b7b99 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -59,6 +59,10 @@
#include "sys/mman.h"
#endif
+#ifdef WIN32
+#include <winnls.h>
+#endif
+
#include "access/xlog_internal.h"
#include "catalog/pg_authid_d.h"
#include "catalog/pg_class_d.h" /* pgrminclude ignore */
@@ -2007,6 +2011,7 @@ locale_date_order(const char *locale)
static void
check_locale_name(int category, const char *locale, char **canonname)
{
+ char *locale_copy;
char *save;
char *res;
@@ -2022,10 +2027,30 @@ check_locale_name(int category, const char *locale, char **canonname)
/* for setlocale() call */
if (!locale)
- locale = "";
+ {
+#ifdef WIN32
+ wchar_t wide_name[LOCALE_NAME_MAX_LENGTH];
+ char name[LOCALE_NAME_MAX_LENGTH];
+
+ /* use Windows API to find the default in BCP47 format */
+ if (GetUserDefaultLocaleName(wide_name, LOCALE_NAME_MAX_LENGTH) == 0)
+ pg_fatal("failed to get default locale name: error code %lu",
+ GetLastError());
+ if (WideCharToMultiByte(CP_ACP, 0, wide_name, -1, name,
+ LOCALE_NAME_MAX_LENGTH, NULL, NULL) == 0)
+ pg_fatal("failed to convert locale name: error code %lu",
+ GetLastError());
+ locale_copy = pg_strdup(name);
+#else
+ /* use environment to find the default */
+ locale_copy = pg_strdup("");
+#endif
+ }
+ else
+ locale_copy = pg_strdup(locale);
/* set the locale with setlocale, to see if it accepts it. */
- res = setlocale(category, locale);
+ res = setlocale(category, locale_copy);
/* save canonical name if requested. */
if (res && canonname)
@@ -2054,6 +2079,8 @@ check_locale_name(int category, const char *locale, char **canonname)
pg_fatal("invalid locale settings; check LANG and LC_* environment variables");
}
}
+
+ free(locale_copy);
}
/*
--
2.35.1
[text/x-patch] v2-0002-Remove-support-for-old-Windows-locale-names.patch (19.7K, ../../CA+hUKGK3zubzq_JZ8MU=zzjOLKKgdq_Knyf_Bra042nqj8gf3g@mail.gmail.com/3-v2-0002-Remove-support-for-old-Windows-locale-names.patch)
download | inline diff:
From 1e0b75b4c8958397a8e660fa0b8759f1da78a753 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 19 Jul 2022 08:53:08 +1200
Subject: [PATCH v2 2/2] Remove support for old Windows locale names.
We now use BCP 47 locale names by default and also advise those for
explicit use. Remove support for munging the old unstable and
unsystematic English word-style locale names.
XXX When could we do this?
XXX How are you supposed to pg_upgrade a system with datcollate =
"English_Canada" to "en-CA"?
XXX There are more traces to remove in win32_langinfo()
---
configure | 6 -
configure.ac | 1 -
doc/src/sgml/charset.sgml | 2 +-
src/backend/utils/adt/pg_locale.c | 235 ++----------------------------
src/include/port/win32_port.h | 9 --
src/port/win32setlocale.c | 193 ------------------------
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 18 insertions(+), 430 deletions(-)
delete mode 100644 src/port/win32setlocale.c
diff --git a/configure b/configure
index a4f4d321fb..5b7704352d 100755
--- a/configure
+++ b/configure
@@ -17174,12 +17174,6 @@ esac
;;
esac
- case " $LIBOBJS " in
- *" win32setlocale.$ac_objext "* ) ;;
- *) LIBOBJS="$LIBOBJS win32setlocale.$ac_objext"
- ;;
-esac
-
case " $LIBOBJS " in
*" win32stat.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
diff --git a/configure.ac b/configure.ac
index 5bd29a4d2f..358580e6d6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1995,7 +1995,6 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32ntdll)
AC_LIBOBJ(win32security)
- AC_LIBOBJ(win32setlocale)
AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index b656ca489f..3c12f3f344 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -90,7 +90,7 @@ initdb --locale=sv_SE
For example, <literal>sv-SE</literal> represents Swedish as spoken in Sweden.
Windows also supports more verbose locale names based on English words,
such as <literal>German_Germany</literal> or <literal>Swedish_Sweden.1252</literal>,
- but these are not recommended.
+ but these should not be used in PostgreSQL.
</para>
<para>
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 607a4b7340..044f62de2b 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -118,7 +118,7 @@ static HTAB *collation_cache = NULL;
#if defined(WIN32) && defined(LC_MESSAGES)
-static char *IsoLocaleName(const char *); /* MSVC specific */
+static char *PosixLocaleName(const char *);
#endif
#ifdef USE_ICU
@@ -204,10 +204,7 @@ pg_perm_setlocale(int category, const char *locale)
case LC_MESSAGES:
envvar = "LC_MESSAGES";
#ifdef WIN32
- result = IsoLocaleName(locale);
- if (result == NULL)
- result = (char *) locale;
- elog(DEBUG3, "IsoLocaleName() executed; locale: \"%s\"", result);
+ locale = PosixLocaleName(locale);
#endif /* WIN32 */
break;
#endif /* LC_MESSAGES */
@@ -905,218 +902,35 @@ cache_locale_time(void)
#if defined(WIN32) && defined(LC_MESSAGES)
/*
- * Convert a Windows setlocale() argument to a Unix-style one.
+ * Convert a Windows BCP 47 locale name to a POSIX one.
*
* Regardless of platform, we install message catalogs under a Unix-style
* LL[_CC][.ENCODING][@VARIANT] naming convention. Only LC_MESSAGES settings
* following that style will elicit localized interface strings.
*
- * Before Visual Studio 2012 (msvcr110.dll), Windows setlocale() accepted "C"
- * (but not "c") and strings of the form <Language>[_<Country>][.<CodePage>],
- * case-insensitive. setlocale() returns the fully-qualified form; for
- * example, setlocale("thaI") returns "Thai_Thailand.874". Internally,
- * setlocale() and _create_locale() select a "locale identifier"[1] and store
- * it in an undocumented _locale_t field. From that LCID, we can retrieve the
- * ISO 639 language and the ISO 3166 country. Character encoding does not
- * matter, because the server and client encodings govern that.
- *
- * Windows Vista introduced the "locale name" concept[2], closely following
- * RFC 4646. Locale identifiers are now deprecated. Starting with Visual
- * Studio 2012, setlocale() accepts locale names in addition to the strings it
- * accepted historically. It does not standardize them; setlocale("Th-tH")
- * returns "Th-tH". setlocale(category, "") still returns a traditional
- * string. Furthermore, msvcr110.dll changed the undocumented _locale_t
- * content to carry locale names instead of locale identifiers.
- *
- * Visual Studio 2015 should still be able to do the same as Visual Studio
- * 2012, but the declaration of locale_name is missing in _locale_t, causing
- * this code compilation to fail, hence this falls back instead on to
- * enumerating all system locales by using EnumSystemLocalesEx to find the
- * required locale name. If the input argument is in Unix-style then we can
- * get ISO Locale name directly by using GetLocaleInfoEx() with LCType as
- * LOCALE_SNAME.
- *
- * MinGW headers declare _create_locale(), but msvcrt.dll lacks that symbol in
- * releases before Windows 8. IsoLocaleName() always fails in a MinGW-built
- * postgres.exe, so only Unix-style values of the lc_messages GUC can elicit
- * localized messages. In particular, every lc_messages setting that initdb
- * can select automatically will yield only C-locale messages. XXX This could
- * be fixed by running the fully-qualified locale name through a lookup table.
- *
- * This function returns a pointer to a static buffer bearing the converted
- * name or NULL if conversion fails.
- *
- * [1] https://docs.microsoft.com/en-us/windows/win32/intl/locale-identifiers
- * [2] https://docs.microsoft.com/en-us/windows/win32/intl/locale-names
- */
-
-/*
- * Callback function for EnumSystemLocalesEx() in get_iso_localename().
- *
- * This function enumerates all system locales, searching for one that matches
- * an input with the format: <Language>[_<Country>], e.g.
- * English[_United States]
- *
- * The input is a three wchar_t array as an LPARAM. The first element is the
- * locale_name we want to match, the second element is an allocated buffer
- * where the Unix-style locale is copied if a match is found, and the third
- * element is the search status, 1 if a match was found, 0 otherwise.
+ * Historically, verbose, but unsystematic and unstable names like
+ * "Thai_Thailand.874" were supported, but now only BCP 47 input is expected.
+ * That means we just need to be able to convert "en-US" to "en_US".
*/
-static BOOL CALLBACK
-search_locale_enum(LPWSTR pStr, DWORD dwFlags, LPARAM lparam)
-{
- wchar_t test_locale[LOCALE_NAME_MAX_LENGTH];
- wchar_t **argv;
-
- (void) (dwFlags);
-
- argv = (wchar_t **) lparam;
- *argv[2] = (wchar_t) 0;
-
- memset(test_locale, 0, sizeof(test_locale));
-
- /* Get the name of the <Language> in English */
- if (GetLocaleInfoEx(pStr, LOCALE_SENGLISHLANGUAGENAME,
- test_locale, LOCALE_NAME_MAX_LENGTH))
- {
- /*
- * If the enumerated locale does not have a hyphen ("en") OR the
- * lc_message input does not have an underscore ("English"), we only
- * need to compare the <Language> tags.
- */
- if (wcsrchr(pStr, '-') == NULL || wcsrchr(argv[0], '_') == NULL)
- {
- if (_wcsicmp(argv[0], test_locale) == 0)
- {
- wcscpy(argv[1], pStr);
- *argv[2] = (wchar_t) 1;
- return FALSE;
- }
- }
-
- /*
- * We have to compare a full <Language>_<Country> tag, so we append
- * the underscore and name of the country/region in English, e.g.
- * "English_United States".
- */
- else
- {
- size_t len;
-
- wcscat(test_locale, L"_");
- len = wcslen(test_locale);
- if (GetLocaleInfoEx(pStr, LOCALE_SENGLISHCOUNTRYNAME,
- test_locale + len,
- LOCALE_NAME_MAX_LENGTH - len))
- {
- if (_wcsicmp(argv[0], test_locale) == 0)
- {
- wcscpy(argv[1], pStr);
- *argv[2] = (wchar_t) 1;
- return FALSE;
- }
- }
- }
- }
-
- return TRUE;
-}
-
-/*
- * This function converts a Windows locale name to an ISO formatted version
- * for Visual Studio 2015 or greater.
- *
- * Returns NULL, if no valid conversion was found.
- */
-static char *
-get_iso_localename(const char *winlocname)
-{
- wchar_t wc_locale_name[LOCALE_NAME_MAX_LENGTH];
- wchar_t buffer[LOCALE_NAME_MAX_LENGTH];
- static char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
- char *period;
- int len;
- int ret_val;
-
- /*
- * Valid locales have the following syntax:
- * <Language>[_<Country>[.<CodePage>]]
- *
- * GetLocaleInfoEx can only take locale name without code-page and for the
- * purpose of this API the code-page doesn't matter.
- */
- period = strchr(winlocname, '.');
- if (period != NULL)
- len = period - winlocname;
- else
- len = pg_mbstrlen(winlocname);
-
- memset(wc_locale_name, 0, sizeof(wc_locale_name));
- memset(buffer, 0, sizeof(buffer));
- MultiByteToWideChar(CP_ACP, 0, winlocname, len, wc_locale_name,
- LOCALE_NAME_MAX_LENGTH);
-
- /*
- * If the lc_messages is already a Unix-style string, we have a direct
- * match with LOCALE_SNAME, e.g. en-US, en_US.
- */
- ret_val = GetLocaleInfoEx(wc_locale_name, LOCALE_SNAME, (LPWSTR) &buffer,
- LOCALE_NAME_MAX_LENGTH);
- if (!ret_val)
- {
- /*
- * Search for a locale in the system that matches language and country
- * name.
- */
- wchar_t *argv[3];
-
- argv[0] = wc_locale_name;
- argv[1] = buffer;
- argv[2] = (wchar_t *) &ret_val;
- EnumSystemLocalesEx(search_locale_enum, LOCALE_WINDOWS, (LPARAM) argv,
- NULL);
- }
-
- if (ret_val)
- {
- size_t rc;
- char *hyphen;
-
- /* Locale names use only ASCII, any conversion locale suffices. */
- rc = wchar2char(iso_lc_messages, buffer, sizeof(iso_lc_messages), NULL);
- if (rc == -1 || rc == sizeof(iso_lc_messages))
- return NULL;
-
- /*
- * Simply replace the hyphen with an underscore. See comments in
- * IsoLocaleName.
- */
- hyphen = strchr(iso_lc_messages, '-');
- if (hyphen)
- *hyphen = '_';
- return iso_lc_messages;
- }
-
- return NULL;
-}
-
static char *
-IsoLocaleName(const char *winlocname)
+PosixLocaleName(const char *winlocname)
{
-#if defined(_MSC_VER)
- static char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
+ char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
+ char *hyphen;
if (pg_strcasecmp("c", winlocname) == 0 ||
pg_strcasecmp("posix", winlocname) == 0)
{
strcpy(iso_lc_messages, "C");
- return iso_lc_messages;
}
else
- return get_iso_localename(winlocname);
-
-#endif /* defined(_MSC_VER) */
- return NULL; /* Not supported on this version of msvc/mingw */
+ {
+ strlcpy(iso_lc_messages, winlocname, sizeof(iso_lc_messages));
+ hypen = strchr(iso_lc_messages, '-');
+ if (hyphen)
+ *hyphen = '_';
+ }
+ return pg_strcpy(iso_lc_messages);
}
#endif /* WIN32 && LC_MESSAGES */
@@ -1680,33 +1494,16 @@ get_collation_actual_version(char collprovider, const char *collcollate)
ereport(ERROR,
(errmsg("could not load locale \"%s\"", collcollate)));
#elif defined(WIN32)
- /*
- * If we are targeting Windows Vista and above, we can ask for a name
- * given a collation name (earlier versions required a location code
- * that we don't have).
- */
NLSVERSIONINFOEX version = {sizeof(NLSVERSIONINFOEX)};
WCHAR wide_collcollate[LOCALE_NAME_MAX_LENGTH];
MultiByteToWideChar(CP_ACP, 0, collcollate, -1, wide_collcollate,
LOCALE_NAME_MAX_LENGTH);
if (!GetNLSVersionEx(COMPARE_STRING, wide_collcollate, &version))
- {
- /*
- * GetNLSVersionEx() wants a language tag such as "en-US", not a
- * locale name like "English_United States.1252". Until those
- * values can be prevented from entering the system, or 100%
- * reliably converted to the more useful tag format, tolerate the
- * resulting error and report that we have no version data.
- */
- if (GetLastError() == ERROR_INVALID_PARAMETER)
- return NULL;
-
ereport(ERROR,
(errmsg("could not get collation version for locale \"%s\": error code %lu",
collcollate,
GetLastError())));
- }
collversion = psprintf("%ld.%ld,%ld.%ld",
(version.dwNLSVersion >> 8) & 0xFFFF,
version.dwNLSVersion & 0xFF,
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 5121c0c626..13ed144a8f 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -437,15 +437,6 @@ extern int _pgstat64(const char *name, struct stat *buf);
#undef setlocale
#endif
-/*
- * Define our own wrapper macro around setlocale() to work around bugs in
- * Windows' native setlocale() function.
- */
-extern char *pgwin32_setlocale(int category, const char *locale);
-
-#define setlocale(a,b) pgwin32_setlocale(a,b)
-
-
/* In backend/port/win32/signal.c */
extern PGDLLIMPORT volatile int pg_signal_queue;
extern PGDLLIMPORT int pg_signal_mask;
diff --git a/src/port/win32setlocale.c b/src/port/win32setlocale.c
deleted file mode 100644
index aadd09a4e9..0000000000
--- a/src/port/win32setlocale.c
+++ /dev/null
@@ -1,193 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * win32setlocale.c
- * Wrapper to work around bugs in Windows setlocale() implementation
- *
- * Copyright (c) 2011-2022, PostgreSQL Global Development Group
- *
- * IDENTIFICATION
- * src/port/win32setlocale.c
- *
- *
- * The setlocale() function in Windows is broken in two ways. First, it
- * has a problem with locale names that have a dot in the country name. For
- * example:
- *
- * "Chinese (Traditional)_Hong Kong S.A.R..950"
- *
- * For some reason, setlocale() doesn't accept that as argument, even though
- * setlocale(LC_ALL, NULL) returns exactly that. Fortunately, it accepts
- * various alternative names for such countries, so to work around the broken
- * setlocale() function, we map the troublemaking locale names to accepted
- * aliases, before calling setlocale().
- *
- * The second problem is that the locale name for "Norwegian (Bokmål)"
- * contains a non-ASCII character. That's problematic, because it's not clear
- * what encoding the locale name itself is supposed to be in, when you
- * haven't yet set a locale. Also, it causes problems when the cluster
- * contains databases with different encodings, as the locale name is stored
- * in the pg_database system catalog. To work around that, when setlocale()
- * returns that locale name, map it to a pure-ASCII alias for the same
- * locale.
- *-------------------------------------------------------------------------
- */
-
-#include "c.h"
-
-#undef setlocale
-
-struct locale_map
-{
- /*
- * String in locale name to replace. Can be a single string (end is NULL),
- * or separate start and end strings. If two strings are given, the locale
- * name must contain both of them, and everything between them is
- * replaced. This is used for a poor-man's regexp search, allowing
- * replacement of "start.*end".
- */
- const char *locale_name_start;
- const char *locale_name_end;
-
- const char *replacement; /* string to replace the match with */
-};
-
-/*
- * Mappings applied before calling setlocale(), to the argument.
- */
-static const struct locale_map locale_map_argument[] = {
- /*
- * "HKG" is listed here:
- * http://msdn.microsoft.com/en-us/library/cdax410z%28v=vs.71%29.aspx
- * (Country/Region Strings).
- *
- * "ARE" is the ISO-3166 three-letter code for U.A.E. It is not on the
- * above list, but seems to work anyway.
- */
- {"Hong Kong S.A.R.", NULL, "HKG"},
- {"U.A.E.", NULL, "ARE"},
-
- /*
- * The ISO-3166 country code for Macau S.A.R. is MAC, but Windows doesn't
- * seem to recognize that. And Macau isn't listed in the table of accepted
- * abbreviations linked above. Fortunately, "ZHM" seems to be accepted as
- * an alias for "Chinese (Traditional)_Macau S.A.R..950". I'm not sure
- * where "ZHM" comes from, must be some legacy naming scheme. But hey, it
- * works.
- *
- * Note that unlike HKG and ARE, ZHM is an alias for the *whole* locale
- * name, not just the country part.
- *
- * Some versions of Windows spell it "Macau", others "Macao".
- */
- {"Chinese (Traditional)_Macau S.A.R..950", NULL, "ZHM"},
- {"Chinese_Macau S.A.R..950", NULL, "ZHM"},
- {"Chinese (Traditional)_Macao S.A.R..950", NULL, "ZHM"},
- {"Chinese_Macao S.A.R..950", NULL, "ZHM"},
- {NULL, NULL, NULL}
-};
-
-/*
- * Mappings applied after calling setlocale(), to its return value.
- */
-static const struct locale_map locale_map_result[] = {
- /*
- * "Norwegian (Bokmål)" locale name contains the a-ring character.
- * Map it to a pure-ASCII alias.
- *
- * It's not clear what encoding setlocale() uses when it returns the
- * locale name, so to play it safe, we search for "Norwegian (Bok*l)".
- *
- * Just to make life even more complicated, some versions of Windows spell
- * the locale name without parentheses. Translate that too.
- */
- {"Norwegian (Bokm", "l)_Norway", "Norwegian_Norway"},
- {"Norwegian Bokm", "l_Norway", "Norwegian_Norway"},
- {NULL, NULL, NULL}
-};
-
-#define MAX_LOCALE_NAME_LEN 100
-
-static const char *
-map_locale(const struct locale_map *map, const char *locale)
-{
- static char aliasbuf[MAX_LOCALE_NAME_LEN];
- int i;
-
- /* Check if the locale name matches any of the problematic ones. */
- for (i = 0; map[i].locale_name_start != NULL; i++)
- {
- const char *needle_start = map[i].locale_name_start;
- const char *needle_end = map[i].locale_name_end;
- const char *replacement = map[i].replacement;
- char *match;
- char *match_start = NULL;
- char *match_end = NULL;
-
- match = strstr(locale, needle_start);
- if (match)
- {
- /*
- * Found a match for the first part. If this was a two-part
- * replacement, find the second part.
- */
- match_start = match;
- if (needle_end)
- {
- match = strstr(match_start + strlen(needle_start), needle_end);
- if (match)
- match_end = match + strlen(needle_end);
- else
- match_start = NULL;
- }
- else
- match_end = match_start + strlen(needle_start);
- }
-
- if (match_start)
- {
- /* Found a match. Replace the matched string. */
- int matchpos = match_start - locale;
- int replacementlen = strlen(replacement);
- char *rest = match_end;
- int restlen = strlen(rest);
-
- /* check that the result fits in the static buffer */
- if (matchpos + replacementlen + restlen + 1 > MAX_LOCALE_NAME_LEN)
- return NULL;
-
- memcpy(&aliasbuf[0], &locale[0], matchpos);
- memcpy(&aliasbuf[matchpos], replacement, replacementlen);
- /* includes null terminator */
- memcpy(&aliasbuf[matchpos + replacementlen], rest, restlen + 1);
-
- return aliasbuf;
- }
- }
-
- /* no match, just return the original string */
- return locale;
-}
-
-char *
-pgwin32_setlocale(int category, const char *locale)
-{
- const char *argument;
- char *result;
-
- if (locale == NULL)
- argument = NULL;
- else
- argument = map_locale(locale_map_argument, locale);
-
- /* Call the real setlocale() function */
- result = setlocale(category, argument);
-
- /*
- * setlocale() is specified to return a "char *" that the caller is
- * forbidden to modify, so casting away the "const" is innocuous.
- */
- if (result)
- result = unconstify(char *, map_locale(locale_map_result, result));
-
- return result;
-}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index e4feda10fd..1bc35ef926 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -108,7 +108,7 @@ sub mkvcbuild
pqsignal.c mkdtemp.c qsort.c qsort_arg.c bsearch_arg.c quotes.c system.c
strerror.c tar.c
win32env.c win32error.c win32ntdll.c
- win32security.c win32setlocale.c win32stat.c);
+ win32security.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.35.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-19 02:46 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2022-07-20 10:26 ` Juan José Santamaría Flecha <[email protected]>
2022-07-20 11:44 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Juan José Santamaría Flecha @ 2022-07-20 10:26 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Tue, Jul 19, 2022 at 4:47 AM Thomas Munro <[email protected]> wrote:
> As for whether "accordingly" still applies, by the logic of of
> win32_langinfo()... Windows still considers WIN1252 to be the default
> ANSI code page for "en-US", though it'd work with UTF-8 too. I'm not
> sure what to make of that. The goal here was to give Windows users
> good defaults, but WIN1252 is probably not what most people actually
> want. Hmph.
>
Still, WIN1252 is not the wrong answer for what we are asking. Even if you
enable UTF-8 support [1], the system will use the current default Windows
ANSI code page (ACP) for the locale and UTF-8 for the code page.
[1]
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-170
Regards,
Juan José Santamaría Flecha
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-19 02:46 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-20 10:26 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
@ 2022-07-20 11:44 ` Thomas Munro <[email protected]>
2022-07-22 11:58 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Thomas Munro @ 2022-07-20 11:44 UTC (permalink / raw)
To: Juan José Santamaría Flecha <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Wed, Jul 20, 2022 at 10:27 PM Juan José Santamaría Flecha
<[email protected]> wrote:
> On Tue, Jul 19, 2022 at 4:47 AM Thomas Munro <[email protected]> wrote:
>> As for whether "accordingly" still applies, by the logic of of
>> win32_langinfo()... Windows still considers WIN1252 to be the default
>> ANSI code page for "en-US", though it'd work with UTF-8 too. I'm not
>> sure what to make of that. The goal here was to give Windows users
>> good defaults, but WIN1252 is probably not what most people actually
>> want. Hmph.
>
>
> Still, WIN1252 is not the wrong answer for what we are asking. Even if you enable UTF-8 support [1], the system will use the current default Windows ANSI code page (ACP) for the locale and UTF-8 for the code page.
I'm still confused about what that means. Suppose we decided to
insist by adding a ".UTF-8" suffix to the name, as that page says we
can now that we're on Windows 10+, when building the default locale
name (see experimental 0002 patch, attached). It initially seemed to
have the right effect:
The database cluster will be initialized with locale "en-US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
But then the Turkish i test in contrib/citext/sql/citext_utf8.sql failed[1]:
SELECT 'i'::citext = 'İ'::citext AS t;
t
---
- t
+ f
(1 row)
About the pg_upgrade problem, maybe it's OK ... existing old format
names should continue to work, but we can still remove the weird code
that does locale name tweaking, right? pg_upgraded databases should
contain fixed names (ie that were fixed by old initdb so should
continue to work), and new clusters will get BCP 47 names.
I don't really know, I was just playing with rough ideas by sending
patches to CI here...
[1] https://cirrus-ci.com/task/6423238052937728
Attachments:
[text/x-patch] v3-0001-Default-to-BCP-47-locale-in-initdb-on-Windows.patch (3.8K, ../../CA+hUKGJZskvCh=Qm75UkHrY6c1QZUuC92Po9rponj1BbLmcMEA@mail.gmail.com/2-v3-0001-Default-to-BCP-47-locale-in-initdb-on-Windows.patch)
download | inline diff:
From b007eb45e575956d5035f4152f72177abddc2762 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 19 Jul 2022 06:31:17 +1200
Subject: [PATCH v3 1/3] Default to BCP 47 locale in initdb on Windows.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Avoid selecting traditional Windows locale names written with English
words, because they are unstable and not recommended for use in
databases. Since setlocale() returns such names, on Windows use
GetUserDefaultLocaleName() if the user didn't provide an explicit
locale.
Also update the documentation to recommend BCP 47 over the traditional
names when providing explicit values to initdb.
Reviewed-by: Juan José Santamaría Flecha <[email protected]>
Discussion: https://postgr.es/m/CA%2BhUKGJ%3DXThErgAQRoqfCy1bKPxXVuF0%3D2zDbB%2BSxDs59pv7Fw%40mail.gmail.com
---
doc/src/sgml/charset.sgml | 10 ++++++++--
src/bin/initdb/initdb.c | 31 +++++++++++++++++++++++++++++--
2 files changed, 37 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index 445fd175d8..b656ca489f 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -83,8 +83,14 @@ initdb --locale=sv_SE
system under what names depends on what was provided by the operating
system vendor and what was installed. On most Unix systems, the command
<literal>locale -a</literal> will provide a list of available locales.
- Windows uses more verbose locale names, such as <literal>German_Germany</literal>
- or <literal>Swedish_Sweden.1252</literal>, but the principles are the same.
+ </para>
+
+ <para>
+ Windows uses BCP 47 language tags, like ICU.
+ For example, <literal>sv-SE</literal> represents Swedish as spoken in Sweden.
+ Windows also supports more verbose locale names based on English words,
+ such as <literal>German_Germany</literal> or <literal>Swedish_Sweden.1252</literal>,
+ but these are not recommended.
</para>
<para>
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 89b888eaa5..3af08b7b99 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -59,6 +59,10 @@
#include "sys/mman.h"
#endif
+#ifdef WIN32
+#include <winnls.h>
+#endif
+
#include "access/xlog_internal.h"
#include "catalog/pg_authid_d.h"
#include "catalog/pg_class_d.h" /* pgrminclude ignore */
@@ -2007,6 +2011,7 @@ locale_date_order(const char *locale)
static void
check_locale_name(int category, const char *locale, char **canonname)
{
+ char *locale_copy;
char *save;
char *res;
@@ -2022,10 +2027,30 @@ check_locale_name(int category, const char *locale, char **canonname)
/* for setlocale() call */
if (!locale)
- locale = "";
+ {
+#ifdef WIN32
+ wchar_t wide_name[LOCALE_NAME_MAX_LENGTH];
+ char name[LOCALE_NAME_MAX_LENGTH];
+
+ /* use Windows API to find the default in BCP47 format */
+ if (GetUserDefaultLocaleName(wide_name, LOCALE_NAME_MAX_LENGTH) == 0)
+ pg_fatal("failed to get default locale name: error code %lu",
+ GetLastError());
+ if (WideCharToMultiByte(CP_ACP, 0, wide_name, -1, name,
+ LOCALE_NAME_MAX_LENGTH, NULL, NULL) == 0)
+ pg_fatal("failed to convert locale name: error code %lu",
+ GetLastError());
+ locale_copy = pg_strdup(name);
+#else
+ /* use environment to find the default */
+ locale_copy = pg_strdup("");
+#endif
+ }
+ else
+ locale_copy = pg_strdup(locale);
/* set the locale with setlocale, to see if it accepts it. */
- res = setlocale(category, locale);
+ res = setlocale(category, locale_copy);
/* save canonical name if requested. */
if (res && canonname)
@@ -2054,6 +2079,8 @@ check_locale_name(int category, const char *locale, char **canonname)
pg_fatal("invalid locale settings; check LANG and LC_* environment variables");
}
}
+
+ free(locale_copy);
}
/*
--
2.30.2
[text/x-patch] v3-0002-Default-to-UTF-8-in-initdb-on-Windows.patch (2.0K, ../../CA+hUKGJZskvCh=Qm75UkHrY6c1QZUuC92Po9rponj1BbLmcMEA@mail.gmail.com/3-v3-0002-Default-to-UTF-8-in-initdb-on-Windows.patch)
download | inline diff:
From 430fda564b3a36f346a672668e5181f43132368e Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Wed, 20 Jul 2022 22:38:50 +1200
Subject: [PATCH v3 2/3] Default to UTF-8 in initdb on Windows.
---
src/backend/utils/adt/pg_locale.c | 13 ++++++++++++-
src/bin/initdb/initdb.c | 4 +++-
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 607a4b7340..6242dc094d 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -1687,8 +1687,19 @@ get_collation_actual_version(char collprovider, const char *collcollate)
*/
NLSVERSIONINFOEX version = {sizeof(NLSVERSIONINFOEX)};
WCHAR wide_collcollate[LOCALE_NAME_MAX_LENGTH];
+ char copy_collcollate[LOCALE_NAME_MAX_LENGTH];
- MultiByteToWideChar(CP_ACP, 0, collcollate, -1, wide_collcollate,
+ /* Trim off encoding, if there is one */
+ strlcpy(copy_collcollate, collcollate, sizeof(copy_collcollate));
+ for (char *p = copy_collcollate; *p; ++p)
+ {
+ if (*p == '.')
+ {
+ *p = 0;
+ break;
+ }
+ }
+ MultiByteToWideChar(CP_ACP, 0, copy_collcollate, -1, wide_collcollate,
LOCALE_NAME_MAX_LENGTH);
if (!GetNLSVersionEx(COMPARE_STRING, wide_collcollate, &version))
{
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 3af08b7b99..1170b80d75 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -2040,7 +2040,9 @@ check_locale_name(int category, const char *locale, char **canonname)
LOCALE_NAME_MAX_LENGTH, NULL, NULL) == 0)
pg_fatal("failed to convert locale name: error code %lu",
GetLastError());
- locale_copy = pg_strdup(name);
+ /* default to UTF-8, no matter what the system default codepage is */
+ locale_copy = pg_malloc(strlen(name) + sizeof(".UTF-8") + 1);
+ sprintf(locale_copy, "%s.UTF-8", name);
#else
/* use environment to find the default */
locale_copy = pg_strdup("");
--
2.30.2
[text/x-patch] v3-0003-Remove-support-for-old-Windows-locale-names.patch (19.6K, ../../CA+hUKGJZskvCh=Qm75UkHrY6c1QZUuC92Po9rponj1BbLmcMEA@mail.gmail.com/4-v3-0003-Remove-support-for-old-Windows-locale-names.patch)
download | inline diff:
From 5281778fbcd20155fd4613732c00e140f7745842 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 19 Jul 2022 08:53:08 +1200
Subject: [PATCH v3 3/3] Remove support for old Windows locale names.
We now use BCP 47 locale names by default and also advise those for
explicit use. Remove support for munging the old unstable and
unsystematic English word-style locale names.
If you explicitly provide a traditional value like "English_United
States.1521", it will still work, but we won't do any kind of cleanup on
the name. This should be enough to work with pg_upgrade'd systems from
before we started using BCP 47.
XXX Is that true?
---
configure | 6 -
configure.ac | 1 -
doc/src/sgml/charset.sgml | 2 +-
src/backend/utils/adt/pg_locale.c | 231 +++---------------------------
src/include/port/win32_port.h | 9 --
src/port/win32setlocale.c | 193 -------------------------
src/tools/msvc/Mkvcbuild.pm | 2 +-
7 files changed, 19 insertions(+), 425 deletions(-)
delete mode 100644 src/port/win32setlocale.c
diff --git a/configure b/configure
index 59fa82b8d7..d0da0c0a40 100755
--- a/configure
+++ b/configure
@@ -17180,12 +17180,6 @@ esac
;;
esac
- case " $LIBOBJS " in
- *" win32setlocale.$ac_objext "* ) ;;
- *) LIBOBJS="$LIBOBJS win32setlocale.$ac_objext"
- ;;
-esac
-
case " $LIBOBJS " in
*" win32stat.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS win32stat.$ac_objext"
diff --git a/configure.ac b/configure.ac
index 612dabf698..c0906429be 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1996,7 +1996,6 @@ if test "$PORTNAME" = "win32"; then
AC_LIBOBJ(win32error)
AC_LIBOBJ(win32ntdll)
AC_LIBOBJ(win32security)
- AC_LIBOBJ(win32setlocale)
AC_LIBOBJ(win32stat)
AC_DEFINE([HAVE_SYMLINK], 1,
[Define to 1 if you have the `symlink' function.])
diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml
index b656ca489f..3c12f3f344 100644
--- a/doc/src/sgml/charset.sgml
+++ b/doc/src/sgml/charset.sgml
@@ -90,7 +90,7 @@ initdb --locale=sv_SE
For example, <literal>sv-SE</literal> represents Swedish as spoken in Sweden.
Windows also supports more verbose locale names based on English words,
such as <literal>German_Germany</literal> or <literal>Swedish_Sweden.1252</literal>,
- but these are not recommended.
+ but these should not be used in PostgreSQL.
</para>
<para>
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 6242dc094d..f7f51a7055 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -118,7 +118,7 @@ static HTAB *collation_cache = NULL;
#if defined(WIN32) && defined(LC_MESSAGES)
-static char *IsoLocaleName(const char *); /* MSVC specific */
+static char *PosixLocaleName(const char *);
#endif
#ifdef USE_ICU
@@ -204,10 +204,7 @@ pg_perm_setlocale(int category, const char *locale)
case LC_MESSAGES:
envvar = "LC_MESSAGES";
#ifdef WIN32
- result = IsoLocaleName(locale);
- if (result == NULL)
- result = (char *) locale;
- elog(DEBUG3, "IsoLocaleName() executed; locale: \"%s\"", result);
+ locale = PosixLocaleName(locale);
#endif /* WIN32 */
break;
#endif /* LC_MESSAGES */
@@ -905,218 +902,35 @@ cache_locale_time(void)
#if defined(WIN32) && defined(LC_MESSAGES)
/*
- * Convert a Windows setlocale() argument to a Unix-style one.
+ * Convert a Windows BCP 47 locale name to a POSIX one.
*
* Regardless of platform, we install message catalogs under a Unix-style
* LL[_CC][.ENCODING][@VARIANT] naming convention. Only LC_MESSAGES settings
* following that style will elicit localized interface strings.
*
- * Before Visual Studio 2012 (msvcr110.dll), Windows setlocale() accepted "C"
- * (but not "c") and strings of the form <Language>[_<Country>][.<CodePage>],
- * case-insensitive. setlocale() returns the fully-qualified form; for
- * example, setlocale("thaI") returns "Thai_Thailand.874". Internally,
- * setlocale() and _create_locale() select a "locale identifier"[1] and store
- * it in an undocumented _locale_t field. From that LCID, we can retrieve the
- * ISO 639 language and the ISO 3166 country. Character encoding does not
- * matter, because the server and client encodings govern that.
- *
- * Windows Vista introduced the "locale name" concept[2], closely following
- * RFC 4646. Locale identifiers are now deprecated. Starting with Visual
- * Studio 2012, setlocale() accepts locale names in addition to the strings it
- * accepted historically. It does not standardize them; setlocale("Th-tH")
- * returns "Th-tH". setlocale(category, "") still returns a traditional
- * string. Furthermore, msvcr110.dll changed the undocumented _locale_t
- * content to carry locale names instead of locale identifiers.
- *
- * Visual Studio 2015 should still be able to do the same as Visual Studio
- * 2012, but the declaration of locale_name is missing in _locale_t, causing
- * this code compilation to fail, hence this falls back instead on to
- * enumerating all system locales by using EnumSystemLocalesEx to find the
- * required locale name. If the input argument is in Unix-style then we can
- * get ISO Locale name directly by using GetLocaleInfoEx() with LCType as
- * LOCALE_SNAME.
- *
- * MinGW headers declare _create_locale(), but msvcrt.dll lacks that symbol in
- * releases before Windows 8. IsoLocaleName() always fails in a MinGW-built
- * postgres.exe, so only Unix-style values of the lc_messages GUC can elicit
- * localized messages. In particular, every lc_messages setting that initdb
- * can select automatically will yield only C-locale messages. XXX This could
- * be fixed by running the fully-qualified locale name through a lookup table.
- *
- * This function returns a pointer to a static buffer bearing the converted
- * name or NULL if conversion fails.
- *
- * [1] https://docs.microsoft.com/en-us/windows/win32/intl/locale-identifiers
- * [2] https://docs.microsoft.com/en-us/windows/win32/intl/locale-names
- */
-
-/*
- * Callback function for EnumSystemLocalesEx() in get_iso_localename().
- *
- * This function enumerates all system locales, searching for one that matches
- * an input with the format: <Language>[_<Country>], e.g.
- * English[_United States]
- *
- * The input is a three wchar_t array as an LPARAM. The first element is the
- * locale_name we want to match, the second element is an allocated buffer
- * where the Unix-style locale is copied if a match is found, and the third
- * element is the search status, 1 if a match was found, 0 otherwise.
- */
-static BOOL CALLBACK
-search_locale_enum(LPWSTR pStr, DWORD dwFlags, LPARAM lparam)
-{
- wchar_t test_locale[LOCALE_NAME_MAX_LENGTH];
- wchar_t **argv;
-
- (void) (dwFlags);
-
- argv = (wchar_t **) lparam;
- *argv[2] = (wchar_t) 0;
-
- memset(test_locale, 0, sizeof(test_locale));
-
- /* Get the name of the <Language> in English */
- if (GetLocaleInfoEx(pStr, LOCALE_SENGLISHLANGUAGENAME,
- test_locale, LOCALE_NAME_MAX_LENGTH))
- {
- /*
- * If the enumerated locale does not have a hyphen ("en") OR the
- * lc_message input does not have an underscore ("English"), we only
- * need to compare the <Language> tags.
- */
- if (wcsrchr(pStr, '-') == NULL || wcsrchr(argv[0], '_') == NULL)
- {
- if (_wcsicmp(argv[0], test_locale) == 0)
- {
- wcscpy(argv[1], pStr);
- *argv[2] = (wchar_t) 1;
- return FALSE;
- }
- }
-
- /*
- * We have to compare a full <Language>_<Country> tag, so we append
- * the underscore and name of the country/region in English, e.g.
- * "English_United States".
- */
- else
- {
- size_t len;
-
- wcscat(test_locale, L"_");
- len = wcslen(test_locale);
- if (GetLocaleInfoEx(pStr, LOCALE_SENGLISHCOUNTRYNAME,
- test_locale + len,
- LOCALE_NAME_MAX_LENGTH - len))
- {
- if (_wcsicmp(argv[0], test_locale) == 0)
- {
- wcscpy(argv[1], pStr);
- *argv[2] = (wchar_t) 1;
- return FALSE;
- }
- }
- }
- }
-
- return TRUE;
-}
-
-/*
- * This function converts a Windows locale name to an ISO formatted version
- * for Visual Studio 2015 or greater.
- *
- * Returns NULL, if no valid conversion was found.
+ * Historically, verbose, but unsystematic and unstable names like
+ * "Thai_Thailand.874" were supported, but now only BCP 47 input is expected.
+ * That means we just need to be able to convert "en-US" to "en_US".
*/
static char *
-get_iso_localename(const char *winlocname)
+PosixLocaleName(const char *winlocname)
{
- wchar_t wc_locale_name[LOCALE_NAME_MAX_LENGTH];
- wchar_t buffer[LOCALE_NAME_MAX_LENGTH];
- static char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
- char *period;
- int len;
- int ret_val;
-
- /*
- * Valid locales have the following syntax:
- * <Language>[_<Country>[.<CodePage>]]
- *
- * GetLocaleInfoEx can only take locale name without code-page and for the
- * purpose of this API the code-page doesn't matter.
- */
- period = strchr(winlocname, '.');
- if (period != NULL)
- len = period - winlocname;
- else
- len = pg_mbstrlen(winlocname);
-
- memset(wc_locale_name, 0, sizeof(wc_locale_name));
- memset(buffer, 0, sizeof(buffer));
- MultiByteToWideChar(CP_ACP, 0, winlocname, len, wc_locale_name,
- LOCALE_NAME_MAX_LENGTH);
-
- /*
- * If the lc_messages is already a Unix-style string, we have a direct
- * match with LOCALE_SNAME, e.g. en-US, en_US.
- */
- ret_val = GetLocaleInfoEx(wc_locale_name, LOCALE_SNAME, (LPWSTR) &buffer,
- LOCALE_NAME_MAX_LENGTH);
- if (!ret_val)
- {
- /*
- * Search for a locale in the system that matches language and country
- * name.
- */
- wchar_t *argv[3];
-
- argv[0] = wc_locale_name;
- argv[1] = buffer;
- argv[2] = (wchar_t *) &ret_val;
- EnumSystemLocalesEx(search_locale_enum, LOCALE_WINDOWS, (LPARAM) argv,
- NULL);
- }
-
- if (ret_val)
- {
- size_t rc;
- char *hyphen;
-
- /* Locale names use only ASCII, any conversion locale suffices. */
- rc = wchar2char(iso_lc_messages, buffer, sizeof(iso_lc_messages), NULL);
- if (rc == -1 || rc == sizeof(iso_lc_messages))
- return NULL;
-
- /*
- * Simply replace the hyphen with an underscore. See comments in
- * IsoLocaleName.
- */
- hyphen = strchr(iso_lc_messages, '-');
- if (hyphen)
- *hyphen = '_';
- return iso_lc_messages;
- }
-
- return NULL;
-}
-
-static char *
-IsoLocaleName(const char *winlocname)
-{
-#if defined(_MSC_VER)
- static char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
+ char iso_lc_messages[LOCALE_NAME_MAX_LENGTH];
+ char *hyphen;
if (pg_strcasecmp("c", winlocname) == 0 ||
pg_strcasecmp("posix", winlocname) == 0)
{
strcpy(iso_lc_messages, "C");
- return iso_lc_messages;
}
else
- return get_iso_localename(winlocname);
-
-#endif /* defined(_MSC_VER) */
- return NULL; /* Not supported on this version of msvc/mingw */
+ {
+ strlcpy(iso_lc_messages, winlocname, sizeof(iso_lc_messages));
+ hypen = strchr(iso_lc_messages, '-');
+ if (hyphen)
+ *hyphen = '_';
+ }
+ return pg_strcpy(iso_lc_messages);
}
#endif /* WIN32 && LC_MESSAGES */
@@ -1680,11 +1494,6 @@ get_collation_actual_version(char collprovider, const char *collcollate)
ereport(ERROR,
(errmsg("could not load locale \"%s\"", collcollate)));
#elif defined(WIN32)
- /*
- * If we are targeting Windows Vista and above, we can ask for a name
- * given a collation name (earlier versions required a location code
- * that we don't have).
- */
NLSVERSIONINFOEX version = {sizeof(NLSVERSIONINFOEX)};
WCHAR wide_collcollate[LOCALE_NAME_MAX_LENGTH];
char copy_collcollate[LOCALE_NAME_MAX_LENGTH];
@@ -1703,13 +1512,7 @@ get_collation_actual_version(char collprovider, const char *collcollate)
LOCALE_NAME_MAX_LENGTH);
if (!GetNLSVersionEx(COMPARE_STRING, wide_collcollate, &version))
{
- /*
- * GetNLSVersionEx() wants a language tag such as "en-US", not a
- * locale name like "English_United States.1252". Until those
- * values can be prevented from entering the system, or 100%
- * reliably converted to the more useful tag format, tolerate the
- * resulting error and report that we have no version data.
- */
+ /* Old style locale names fail here, so ignore. */
if (GetLastError() == ERROR_INVALID_PARAMETER)
return NULL;
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 4de5bf3bf6..a569433811 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -445,15 +445,6 @@ extern int _pgstat64(const char *name, struct stat *buf);
#undef setlocale
#endif
-/*
- * Define our own wrapper macro around setlocale() to work around bugs in
- * Windows' native setlocale() function.
- */
-extern char *pgwin32_setlocale(int category, const char *locale);
-
-#define setlocale(a,b) pgwin32_setlocale(a,b)
-
-
/* In backend/port/win32/signal.c */
extern PGDLLIMPORT volatile int pg_signal_queue;
extern PGDLLIMPORT int pg_signal_mask;
diff --git a/src/port/win32setlocale.c b/src/port/win32setlocale.c
deleted file mode 100644
index aadd09a4e9..0000000000
--- a/src/port/win32setlocale.c
+++ /dev/null
@@ -1,193 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * win32setlocale.c
- * Wrapper to work around bugs in Windows setlocale() implementation
- *
- * Copyright (c) 2011-2022, PostgreSQL Global Development Group
- *
- * IDENTIFICATION
- * src/port/win32setlocale.c
- *
- *
- * The setlocale() function in Windows is broken in two ways. First, it
- * has a problem with locale names that have a dot in the country name. For
- * example:
- *
- * "Chinese (Traditional)_Hong Kong S.A.R..950"
- *
- * For some reason, setlocale() doesn't accept that as argument, even though
- * setlocale(LC_ALL, NULL) returns exactly that. Fortunately, it accepts
- * various alternative names for such countries, so to work around the broken
- * setlocale() function, we map the troublemaking locale names to accepted
- * aliases, before calling setlocale().
- *
- * The second problem is that the locale name for "Norwegian (Bokmål)"
- * contains a non-ASCII character. That's problematic, because it's not clear
- * what encoding the locale name itself is supposed to be in, when you
- * haven't yet set a locale. Also, it causes problems when the cluster
- * contains databases with different encodings, as the locale name is stored
- * in the pg_database system catalog. To work around that, when setlocale()
- * returns that locale name, map it to a pure-ASCII alias for the same
- * locale.
- *-------------------------------------------------------------------------
- */
-
-#include "c.h"
-
-#undef setlocale
-
-struct locale_map
-{
- /*
- * String in locale name to replace. Can be a single string (end is NULL),
- * or separate start and end strings. If two strings are given, the locale
- * name must contain both of them, and everything between them is
- * replaced. This is used for a poor-man's regexp search, allowing
- * replacement of "start.*end".
- */
- const char *locale_name_start;
- const char *locale_name_end;
-
- const char *replacement; /* string to replace the match with */
-};
-
-/*
- * Mappings applied before calling setlocale(), to the argument.
- */
-static const struct locale_map locale_map_argument[] = {
- /*
- * "HKG" is listed here:
- * http://msdn.microsoft.com/en-us/library/cdax410z%28v=vs.71%29.aspx
- * (Country/Region Strings).
- *
- * "ARE" is the ISO-3166 three-letter code for U.A.E. It is not on the
- * above list, but seems to work anyway.
- */
- {"Hong Kong S.A.R.", NULL, "HKG"},
- {"U.A.E.", NULL, "ARE"},
-
- /*
- * The ISO-3166 country code for Macau S.A.R. is MAC, but Windows doesn't
- * seem to recognize that. And Macau isn't listed in the table of accepted
- * abbreviations linked above. Fortunately, "ZHM" seems to be accepted as
- * an alias for "Chinese (Traditional)_Macau S.A.R..950". I'm not sure
- * where "ZHM" comes from, must be some legacy naming scheme. But hey, it
- * works.
- *
- * Note that unlike HKG and ARE, ZHM is an alias for the *whole* locale
- * name, not just the country part.
- *
- * Some versions of Windows spell it "Macau", others "Macao".
- */
- {"Chinese (Traditional)_Macau S.A.R..950", NULL, "ZHM"},
- {"Chinese_Macau S.A.R..950", NULL, "ZHM"},
- {"Chinese (Traditional)_Macao S.A.R..950", NULL, "ZHM"},
- {"Chinese_Macao S.A.R..950", NULL, "ZHM"},
- {NULL, NULL, NULL}
-};
-
-/*
- * Mappings applied after calling setlocale(), to its return value.
- */
-static const struct locale_map locale_map_result[] = {
- /*
- * "Norwegian (Bokmål)" locale name contains the a-ring character.
- * Map it to a pure-ASCII alias.
- *
- * It's not clear what encoding setlocale() uses when it returns the
- * locale name, so to play it safe, we search for "Norwegian (Bok*l)".
- *
- * Just to make life even more complicated, some versions of Windows spell
- * the locale name without parentheses. Translate that too.
- */
- {"Norwegian (Bokm", "l)_Norway", "Norwegian_Norway"},
- {"Norwegian Bokm", "l_Norway", "Norwegian_Norway"},
- {NULL, NULL, NULL}
-};
-
-#define MAX_LOCALE_NAME_LEN 100
-
-static const char *
-map_locale(const struct locale_map *map, const char *locale)
-{
- static char aliasbuf[MAX_LOCALE_NAME_LEN];
- int i;
-
- /* Check if the locale name matches any of the problematic ones. */
- for (i = 0; map[i].locale_name_start != NULL; i++)
- {
- const char *needle_start = map[i].locale_name_start;
- const char *needle_end = map[i].locale_name_end;
- const char *replacement = map[i].replacement;
- char *match;
- char *match_start = NULL;
- char *match_end = NULL;
-
- match = strstr(locale, needle_start);
- if (match)
- {
- /*
- * Found a match for the first part. If this was a two-part
- * replacement, find the second part.
- */
- match_start = match;
- if (needle_end)
- {
- match = strstr(match_start + strlen(needle_start), needle_end);
- if (match)
- match_end = match + strlen(needle_end);
- else
- match_start = NULL;
- }
- else
- match_end = match_start + strlen(needle_start);
- }
-
- if (match_start)
- {
- /* Found a match. Replace the matched string. */
- int matchpos = match_start - locale;
- int replacementlen = strlen(replacement);
- char *rest = match_end;
- int restlen = strlen(rest);
-
- /* check that the result fits in the static buffer */
- if (matchpos + replacementlen + restlen + 1 > MAX_LOCALE_NAME_LEN)
- return NULL;
-
- memcpy(&aliasbuf[0], &locale[0], matchpos);
- memcpy(&aliasbuf[matchpos], replacement, replacementlen);
- /* includes null terminator */
- memcpy(&aliasbuf[matchpos + replacementlen], rest, restlen + 1);
-
- return aliasbuf;
- }
- }
-
- /* no match, just return the original string */
- return locale;
-}
-
-char *
-pgwin32_setlocale(int category, const char *locale)
-{
- const char *argument;
- char *result;
-
- if (locale == NULL)
- argument = NULL;
- else
- argument = map_locale(locale_map_argument, locale);
-
- /* Call the real setlocale() function */
- result = setlocale(category, argument);
-
- /*
- * setlocale() is specified to return a "char *" that the caller is
- * forbidden to modify, so casting away the "const" is innocuous.
- */
- if (result)
- result = unconstify(char *, map_locale(locale_map_result, result));
-
- return result;
-}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index cc7a908d10..fff378d2b3 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -109,7 +109,7 @@ sub mkvcbuild
pqsignal.c mkdtemp.c qsort.c qsort_arg.c bsearch_arg.c quotes.c system.c
strerror.c tar.c
win32env.c win32error.c win32ntdll.c
- win32security.c win32setlocale.c win32stat.c);
+ win32security.c win32stat.c);
push(@pgportfiles, 'strtof.c') if ($vsVersion < '14.00');
--
2.30.2
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-19 02:46 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-20 10:26 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-20 11:44 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2022-07-22 11:58 ` Juan José Santamaría Flecha <[email protected]>
2022-07-29 03:33 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Juan José Santamaría Flecha @ 2022-07-22 11:58 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Wed, Jul 20, 2022 at 1:44 PM Thomas Munro <[email protected]> wrote:
> On Wed, Jul 20, 2022 at 10:27 PM Juan José Santamaría Flecha
> <[email protected]> wrote:
> > Still, WIN1252 is not the wrong answer for what we are asking. Even if
> you enable UTF-8 support [1], the system will use the current default
> Windows ANSI code page (ACP) for the locale and UTF-8 for the code page.
>
> I'm still confused about what that means. Suppose we decided to
> insist by adding a ".UTF-8" suffix to the name, as that page says we
> can now that we're on Windows 10+, when building the default locale
> name (see experimental 0002 patch, attached). It initially seemed to
> have the right effect:
>
> The database cluster will be initialized with locale "en-US.UTF-8".
> The default database encoding has accordingly been set to "UTF8".
> The default text search configuration will be set to "english".
>
> Let me try to explain this using the "Beta: Use Unicode UTF-8 for
worldwide language support" option [1].
- Currently in a system with the language settings of "English_United
States" and that option disabled, when executing initdb you get:
The database cluster will be initialized with locale "English_United
States.1252".
The default database encoding has accordingly been set to "WIN1252".
The default text search configuration will be set to "english".
And as a test for psql:
SET lc_time='tr_tr.utf8';
SET
SELECT to_char('2000-2-01'::date, 'tmmonth');
ERROR: character with byte sequence 0xc5 0x9f in encoding "UTF8" has no
equivalent in encoding "WIN1252"
We get this error even if the database encoding is UTF8, and is caused by
the tr_tr locales being encoded in WIN1254. We can discuss this in another
thread, and I can propose a patch.
- If we enable the UTF-8 support option, then the same test goes as:
The database cluster will be initialized with locale "English_United
States.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
And for psql:
SET lc_time='tr_tr.utf8';
SET
SELECT to_char('2000-2-01'::date, 'tmmonth');
to_char
---------
şubat
(1 row)
In this case the Windows locales are actually UTF8 encoded.
TL;DR; What I want to show through this example is that Windows ACP is not
modified by setlocale(), it can only be done through the Windows registry
and only in recent releases.
> But then the Turkish i test in contrib/citext/sql/citext_utf8.sql
> failed[1]:
>
> SELECT 'i'::citext = 'İ'::citext AS t;
> t
> ---
> - t
> + f
> (1 row)
>
> This is current state of affairs:
- Windows:
SELECT U&'\0131' latin_small_dotless,U&'\0069' latin_small
,U&'\0049' latin_capital, lower(U&'\0049')
,U&'\0130' latin_capital_dotted, lower(U&'\0130');
latin_small_dotless | latin_small | latin_capital | lower |
latin_capital_dotted | lower
---------------------+-------------+---------------+-------+----------------------+-------
ı | i | I | i | İ
| İ
- Linux:
SELECT U&'\0131' latin_small_dotless,U&'\0069' latin_small
,U&'\0049' latin_capital, lower(U&'\0049')
,U&'\0130' latin_capital_dotted, lower(U&'\0130');
latin_small_dotless | latin_small | latin_capital | lower |
latin_capital_dotted | lower
---------------------+-------------+---------------+-------+----------------------+-------
ı | i | I | i | İ
| i
Latin_capital_dotted doesn't have the same lower value.
[1]
https://stackoverflow.com/questions/56419639/what-does-beta-use-unicode-utf-8-for-worldwide-language...
Regards,
Juan José Santamaría Flecha
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-19 02:46 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-20 10:26 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-20 11:44 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-22 11:58 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
@ 2022-07-29 03:33 ` Thomas Munro <[email protected]>
2022-12-23 04:36 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Thomas Munro @ 2022-07-29 03:33 UTC (permalink / raw)
To: Juan José Santamaría Flecha <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Fri, Jul 22, 2022 at 11:59 PM Juan José Santamaría Flecha
<[email protected]> wrote:
> TL;DR; What I want to show through this example is that Windows ACP is not modified by setlocale(), it can only be done through the Windows registry and only in recent releases.
Thanks, that was helpful, and so was that SO link.
So it sounds like I should forget about the v3-0002 patch, but the
v3-0001 and v3-0003 patches might have a future. And it sounds like
we might need to investigate maybe defending ourselves against the ACP
being different than what we expect (ie not matching the database
encoding)? Did I understand correctly that you're looking into that?
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-19 02:46 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-20 10:26 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-20 11:44 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
2022-07-22 11:58 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-29 03:33 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2022-12-23 04:36 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 44+ messages in thread
From: Thomas Munro @ 2022-12-23 04:36 UTC (permalink / raw)
To: Juan José Santamaría Flecha <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Fri, Jul 29, 2022 at 3:33 PM Thomas Munro <[email protected]> wrote:
> On Fri, Jul 22, 2022 at 11:59 PM Juan José Santamaría Flecha
> <[email protected]> wrote:
> > TL;DR; What I want to show through this example is that Windows ACP is not modified by setlocale(), it can only be done through the Windows registry and only in recent releases.
>
> Thanks, that was helpful, and so was that SO link.
>
> So it sounds like I should forget about the v3-0002 patch, but the
> v3-0001 and v3-0003 patches might have a future. And it sounds like
> we might need to investigate maybe defending ourselves against the ACP
> being different than what we expect (ie not matching the database
> encoding)? Did I understand correctly that you're looking into that?
I'm going to withdraw this entry. The sooner we get something like
0001 into a release, the sooner the world will be rid of PostgreSQL
clusters initialised with the bad old locale names that the manual
very clearly tells you not to use for databases.... but I don't
understand this ACP/registry vs database encoding stuff and how it
relates to the use of BCP47 locale names, which puts me off changing
anything until we do.
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Windows default locale vs initdb
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-05-16 04:29 ` Re: Windows default locale vs initdb Noah Misch <[email protected]>
2021-12-15 10:32 ` Re: Windows default locale vs initdb Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Re: Windows default locale vs initdb Thomas Munro <[email protected]>
@ 2022-07-20 08:34 ` Juan José Santamaría Flecha <[email protected]>
1 sibling, 0 replies; 44+ messages in thread
From: Juan José Santamaría Flecha @ 2022-07-20 08:34 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Noah Misch <[email protected]>; pgsql-hackers
On Tue, Jul 19, 2022 at 12:59 AM Thomas Munro <[email protected]>
wrote:
> Now that museum-grade Windows has been defenestrated, we are free to
> call GetUserDefaultLocaleName(). Here's a patch.
>
This LGTM.
>
> I think we should also convert to POSIX format when making the
> collname in your pg_import_system_collations() proposal, so that
> COLLATE "en_US" works (= a SQL identifier), but that's another
> thread[1]. I don't think we should do it in collcollate or
> datcollate, which is a string for the OS to interpret.
>
That thread has been split [1], but that is how the current version behaves.
>
> With my garbage collector hat on, I would like to rip out all of the
> support for traditional locale names, eventually. Deleting kludgy
> code is easy and fun -- 0002 is a first swing at that -- but there
> remains an important unanswered question. How should someone
> pg_upgrade a "English_Canada.1521" cluster if we now reject that name?
> We'd need to do a conversion to "en-CA", or somehow tell the user to.
> Hmmmm.
>
Is there a safe way to do that in pg_upgrade or would we be forcing users
to pg_dump into the new cluster?
[1]
https://www.postgresql.org/message-id/flat/0050ec23-34d9-2765-9015-98c04f0e18ac%40postgrespro.ru
Regards,
Juan José Santamaría Flecha
^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH 4/8] cirrus/002_pg_upgrade: exercise --link and --clone
@ 2022-07-31 00:25 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 44+ messages in thread
From: Justin Pryzby @ 2022-07-31 00:25 UTC (permalink / raw)
This increases code coverage (and maybe accelerates the test).
See also: b059a2409faf5833b3ba7792e247d6466c9e8090
linux,
macos,
//-os-only: freebsd
---
.cirrus.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.cirrus.yml b/.cirrus.yml
index b9a70a05cfa..4cc90484eaf 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -130,22 +130,23 @@ task:
task:
name: FreeBSD - 13 - Meson
env:
CPUS: 4
BUILD_JOBS: 4
TEST_JOBS: 6
CCACHE_DIR: /tmp/ccache_dir
CPPFLAGS: -DRELCACHE_FORCE_RELEASE -DCOPY_PARSE_PLAN_TREES -DWRITE_READ_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -DENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
CFLAGS: -Og -ggdb
+ PG_TEST_PG_UPGRADE_MODE: --link
depends_on: SanityCheck
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*'
compute_engine_instance:
image_project: $IMAGE_PROJECT
image: family/pg-ci-freebsd-13
platform: freebsd
cpu: $CPUS
memory: 4G
disk: 50
@@ -424,22 +425,23 @@ task:
CIRRUS_WORKING_DIR: ${HOME}/pgsql/
CCACHE_DIR: ${HOME}/ccache
HOMEBREW_CACHE: ${HOME}/homebrew-cache
PERL5LIB: ${HOME}/perl5/lib/perl5
CC: ccache cc
CXX: ccache c++
CPPFLAGS: -DRANDOMIZE_ALLOCATED_MEMORY
CFLAGS: -Og -ggdb
CXXFLAGS: -Og -ggdb
+ PG_TEST_PG_UPGRADE_MODE: --clone
depends_on: SanityCheck
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*'
macos_instance:
image: ghcr.io/cirruslabs/macos-ventura-base:latest
sysinfo_script: |
id
uname -a
ulimit -a -H && ulimit -a -S
--
2.34.1
--NvorufYXl2Cnpa+A
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0005-WIP-ci-meson-allow-showing-only-failed-tests.patch"
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
@ 2023-02-01 17:51 Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Ilya Gladyshev @ 2023-02-01 17:51 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
> 1 февр. 2023 г., в 20:27, Matthias van de Meent <[email protected]> написал(а):
>
> On Wed, 1 Feb 2023 at 16:53, Justin Pryzby <[email protected] <mailto:[email protected]>> wrote:
>>
>> On Wed, Feb 01, 2023 at 04:21:35PM +0100, Matthias van de Meent wrote:
>>> On Wed, 1 Feb 2023 at 15:21, Ilya Gladyshev <[email protected]> wrote:
>>>>> 1 февр. 2023 г., в 16:01, Alvaro Herrera <[email protected]> написал(а):
>>>>> Hmm, count_leaf_partitions has to scan pg_inherits and do a syscache
>>>>> lookup for every single element therein ... this sounds slow.
>>>>>
>>>>> In one of the callsites, we already have the partition descriptor
>>>>> available. We could just scan partdesc->is_leaf[] and add one for each
>>>>> 'true' value we see there.
>>>>
>>>> The problem is that partdesc contains only direct children of the table and we need all the children down the inheritance tree to count the total number of leaf partitions in the first callsite.
>>>>
>>>>> In the other callsite, we had the table open just a few lines before the
>>>>> place you call count_leaf_partitions. Maybe we can rejigger things by
>>>>> examining its state before closing it: if relkind is not partitioned we
>>>>> know leaf_partitions=0, and only if partitioned we count leaf partitions.
>>>>> I think that would save some work. I also wonder if it's worth writing
>>>>> a bespoke function for counting leaf partitions rather than relying on
>>>>> find_all_inheritors.
>>>>
>>>> Sure, added this condition to avoid the extra work here.
>>>>
>>>
>>>> When creating an index on a partitioned table, this column is set to
>>>> - the total number of partitions on which the index is to be created.
>>>> + the total number of leaf partitions on which the index is to be created or attached.
>>>
>>> I think we should also add a note about the (now) non-constant nature
>>> of the value, something along the lines of "This value is updated as
>>> we're processing and discovering partitioned tables in the partition
>>> hierarchy".
>>
>> But the TOTAL is constant, right ? Updating the total when being called
>> recursively is the problem these patches fix.
>
> If that's the case, then I'm not seeing the 'fix' part of the patch. I
> thought this patch was fixing the provably incorrect TOTAL value where
> DONE > TOTAL due to the recursive operation overwriting the DONE/TOTAL
> values instead of updating them.
>
> In HEAD we set TOTAL to whatever number partitioned table we're
> currently processing has - regardless of whether we're the top level
> statement.
> With the patch we instead add the number of child relations to that
> count, for which REL_HAS_STORAGE(child) -- or at least, in the v3
> posted by Ilya. Approximately immediately after updating that count we
> recurse to the child relations, and that only returns once it is done
> creating the indexes, so both TOTAL and DONE go up as we process more
> partitions in the hierarchy.
The TOTAL in the patch is set only when processing the top-level parent and it is not updated when we recurse, so yes, it is constant. From v3:
@@ -1219,8 +1243,14 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ if (!OidIsValid(parentIndexId))
+ {
+ int total_parts;
+
+ total_parts = count_leaf_partitions(relationId);
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_parts);
+ }
It is set to the total number of children on all levels of the hierarchy, not just the current one, so the total value doesn’t need to be updated later, because it is set to the correct value from the very beginning.
It is the DONE counter that is updated, and when we attach an index of a partition that is itself a partitioned table (like a2 in your example, if it already had an index created), it will be updated by the number of children of the partition.
@@ -1431,9 +1463,25 @@ DefineIndex(Oid relationId,
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ int attached_parts = 1;
+
+ if (RELKIND_HAS_PARTITIONS(child_relkind))
+ attached_parts = count_leaf_partitions(childRelid);
+
+ /*
+ * If the index was attached, we need to update progress
+ * here, in its parent. For a partitioned index, we need
+ * to mark all of its children that were included in
+ * PROGRESS_CREATEIDX_PARTITIONS_TOTAL as done. If the
+ * index was built by calling DefineIndex() recursively,
+ * the called function is responsible for updating the
+ * progress report for built indexes.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, attached_parts);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
@ 2023-02-01 18:24 ` Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Matthias van de Meent @ 2023-02-01 18:24 UTC (permalink / raw)
To: Ilya Gladyshev <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Wed, 1 Feb 2023 at 18:51, Ilya Gladyshev <[email protected]> wrote:
>
> 1 февр. 2023 г., в 20:27, Matthias van de Meent <[email protected]> написал(а):
>
>> In HEAD we set TOTAL to whatever number partitioned table we're
>> currently processing has - regardless of whether we're the top level
>> statement.
>> With the patch we instead add the number of child relations to that
>> count, for which REL_HAS_STORAGE(child) -- or at least, in the v3
>> posted by Ilya. Approximately immediately after updating that count we
>> recurse to the child relations, and that only returns once it is done
>> creating the indexes, so both TOTAL and DONE go up as we process more
>> partitions in the hierarchy.
>
>
> The TOTAL in the patch is set only when processing the top-level parent and it is not updated when we recurse, so yes, it is constant. From v3:
Ugh, I misread the patch, more specifically count_leaf_partitions and
the !OidIsValid(parentIndexId) condition changes.
You are correct, sorry for the noise.
Kind regards,
Matthias van de Meent
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
@ 2023-02-02 15:18 ` Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-02-02 15:18 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Wed, Feb 01, 2023 at 07:24:48PM +0100, Matthias van de Meent wrote:
> On Wed, 1 Feb 2023 at 18:51, Ilya Gladyshev <[email protected]> wrote:
> >
> > 1 февр. 2023 г., в 20:27, Matthias van de Meent <[email protected]> написал(а):
> >
> >> In HEAD we set TOTAL to whatever number partitioned table we're
> >> currently processing has - regardless of whether we're the top level
> >> statement.
> >> With the patch we instead add the number of child relations to that
> >> count, for which REL_HAS_STORAGE(child) -- or at least, in the v3
> >> posted by Ilya. Approximately immediately after updating that count we
> >> recurse to the child relations, and that only returns once it is done
> >> creating the indexes, so both TOTAL and DONE go up as we process more
> >> partitions in the hierarchy.
> >
> >
> > The TOTAL in the patch is set only when processing the top-level parent and it is not updated when we recurse, so yes, it is constant. From v3:
>
> Ugh, I misread the patch, more specifically count_leaf_partitions and
> the !OidIsValid(parentIndexId) condition changes.
>
> You are correct, sorry for the noise.
That suggests that the comments could've been more clear. I added a
comment suggested by Tomas and adjusted some others and wrote a commit
message. I even ran pgindent for about the 3rd time ever.
002 are my changes as a separate patch, which you could apply to your
local branch.
And 003/4 are assertions that I wrote to demonstrate the problem and the
verify the fixes, but not being proposed for commit.
--
Justin
Attachments:
[text/x-diff] 0001-create-index-progress-increment.patch (6.6K, ../../[email protected]/2-0001-create-index-progress-increment.patch)
download | inline diff:
From 375961e18aaa7ed7b2ebee972ad07c7a38099ef4 Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <[email protected]>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH 1/4] create index progress increment
---
doc/src/sgml/monitoring.sgml | 4 +-
src/backend/commands/indexcmds.c | 64 +++++++++++++++++--
src/backend/utils/activity/backend_progress.c | 25 ++++++++
src/include/utils/backend_progress.h | 1 +
4 files changed, 87 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1756f1a4b67..a911900271c 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6601,7 +6601,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of leaf partitions on which the index is to be created or attached.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6612,7 +6612,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of leaf partitions on which the index has been created or attached.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 16ec0b114e6..936b4e3c1db 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -130,6 +130,30 @@ typedef struct ReindexErrorInfo
char relkind;
} ReindexErrorInfo;
+
+/*
+ * Count the number of direct and indirect leaf partitions, excluding foreign
+ * tables.
+ */
+static int
+count_leaf_partitions(Oid relid)
+{
+ int nleaves = 0;
+ List *childs = find_all_inheritors(relid, NoLock, NULL);
+ ListCell *lc;
+
+ foreach(lc, childs)
+ {
+ Oid partrelid = lfirst_oid(lc);
+
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
+ nleaves++;
+ }
+
+ list_free(childs);
+ return nleaves;
+}
+
/*
* CheckIndexCompatible
* Determine whether an existing index definition is compatible with a
@@ -1219,8 +1243,14 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ if (!OidIsValid(parentIndexId))
+ {
+ int total_parts;
+
+ total_parts = count_leaf_partitions(relationId);
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_parts);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1250,6 +1280,7 @@ DefineIndex(Oid relationId,
{
Oid childRelid = part_oids[i];
Relation childrel;
+ char child_relkind;
Oid child_save_userid;
int child_save_sec_context;
int child_save_nestlevel;
@@ -1259,6 +1290,7 @@ DefineIndex(Oid relationId,
bool found = false;
childrel = table_open(childRelid, lockmode);
+ child_relkind = RelationGetForm(childrel)->relkind;
GetUserIdAndSecContext(&child_save_userid,
&child_save_sec_context);
@@ -1431,9 +1463,25 @@ DefineIndex(Oid relationId,
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ int attached_parts = 1;
+
+ if (RELKIND_HAS_PARTITIONS(child_relkind))
+ attached_parts = count_leaf_partitions(childRelid);
+
+ /*
+ * If the index was attached, we need to update progress
+ * here, in its parent. For a partitioned index, we need
+ * to mark all of its children that were included in
+ * PROGRESS_CREATEIDX_PARTITIONS_TOTAL as done. If the
+ * index was built by calling DefineIndex() recursively,
+ * the called function is responsible for updating the
+ * progress report for built indexes.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, attached_parts);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1484,9 +1532,15 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, we're done. When called recursively
+ * for child tables, the done partition counter is incremented now,
+ * rather than in the caller.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..45f7e7144b4 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,31 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of own backend entry.
+ *-----------
+ */
+void pgstat_progress_incr_param(int index, int64 incr) {
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because current backend is the only process that writes to its own status,
+ * we don't need to do the looping to read the value.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.25.1
[text/x-diff] 0002-s-fix-CREATE-INDEX-progress-report-with-nested-parti.patch (6.3K, ../../[email protected]/3-0002-s-fix-CREATE-INDEX-progress-report-with-nested-parti.patch)
download | inline diff:
From 3d78449f1c3d2fef8e96f4874f13004325026305 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 1 Feb 2023 21:50:21 -0600
Subject: [PATCH 2/4] s!fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, but doesn't count intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, and if the attached index is
partitioned, the counter is incremented to account for each of its leaf
partitions.
Author: Ilya Gladyshev
Reviewed-By: Justin Pryzby, Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 +++++--
src/backend/commands/indexcmds.c | 30 +++++++++++--------
src/backend/utils/activity/backend_progress.c | 13 ++++----
3 files changed, 34 insertions(+), 19 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a911900271c..fa139dcece7 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6601,7 +6601,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of leaf partitions on which the index is to be created or attached.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6612,7 +6615,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of leaf partitions on which the index has been created or attached.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 936b4e3c1db..84c84c41acc 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -144,7 +144,7 @@ count_leaf_partitions(Oid relid)
foreach(lc, childs)
{
- Oid partrelid = lfirst_oid(lc);
+ Oid partrelid = lfirst_oid(lc);
if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
nleaves++;
@@ -1243,9 +1243,13 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
+ /*
+ * Set the total number of partitions at the start of the command,
+ * but don't change it when being called recursively.
+ */
if (!OidIsValid(parentIndexId))
{
- int total_parts;
+ int total_parts;
total_parts = count_leaf_partitions(relationId);
pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
@@ -1465,19 +1469,21 @@ DefineIndex(Oid relationId,
}
else
{
- int attached_parts = 1;
+ int attached_parts;
- if (RELKIND_HAS_PARTITIONS(child_relkind))
- attached_parts = count_leaf_partitions(childRelid);
+ /*
+ * Avoid the overhead of counting partitions when that
+ * can't apply.
+ */
+ attached_parts = RELKIND_HAS_PARTITIONS(child_relkind) ?
+ count_leaf_partitions(childRelid) : 1;
/*
- * If the index was attached, we need to update progress
- * here, in its parent. For a partitioned index, we need
- * to mark all of its children that were included in
- * PROGRESS_CREATEIDX_PARTITIONS_TOTAL as done. If the
- * index was built by calling DefineIndex() recursively,
- * the called function is responsible for updating the
- * progress report for built indexes.
+ * If a pre-existing index was attached, the progress
+ * report is updated here. If the index was partitioned,
+ * all the children that were counted towards
+ * PROGRESS_CREATEIDX_PARTITIONS_TOTAL are counted as
+ * done.
*/
pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, attached_parts);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 45f7e7144b4..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -61,12 +61,14 @@ pgstat_progress_update_param(int index, int64 val)
/*-----------
* pgstat_progress_incr_param() -
*
- * Increment index'th member in st_progress_param[] of own backend entry.
+ * Increment index'th member in st_progress_param[] of the current backend.
*-----------
*/
-void pgstat_progress_incr_param(int index, int64 incr) {
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
volatile PgBackendStatus *beentry = MyBEEntry;
- int64 val;
+ int64 val;
Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
@@ -74,8 +76,9 @@ void pgstat_progress_incr_param(int index, int64 incr) {
return;
/*
- * Because current backend is the only process that writes to its own status,
- * we don't need to do the looping to read the value.
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
*/
val = beentry->st_progress_param[index];
val += incr;
--
2.25.1
[text/x-diff] 0003-assertions-for-progress-reporting.patch (4.0K, ../../[email protected]/4-0003-assertions-for-progress-reporting.patch)
download | inline diff:
From f484d5c9b602f158e86f9507b662b7bbbb4a19b7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 1 Feb 2023 10:23:53 -0600
Subject: [PATCH 3/4] assertions for progress reporting
---
src/backend/utils/activity/backend_progress.c | 83 +++++++++++++++++++
1 file changed, 83 insertions(+)
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 2a9994b98fd..d0566430fab 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
@@ -37,6 +38,84 @@ pgstat_progress_start_command(ProgressCommandType cmdtype, Oid relid)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Check for consistency of progress data (current < total).
+ *
+ * Check during pgstat_progress_updates_*() rather than only from
+ * pgstat_progress_end_command() to catch issues with uninitialized/stale data
+ * from previous progress commands.
+ *
+ * If a command fails due to interrupt or error, the values may be less than
+ * the expected final value.
+ */
+static void
+pgstat_progress_asserts(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ volatile int64 *a = beentry->st_progress_param;
+
+ switch (beentry->st_progress_command)
+ {
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_VACUUMED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_NUM_DEAD_TUPLES] <=
+ a[PROGRESS_VACUUM_MAX_DEAD_TUPLES]);
+ break;
+
+ case PROGRESS_COMMAND_ANALYZE:
+ Assert(a[PROGRESS_ANALYZE_BLOCKS_DONE] <=
+ a[PROGRESS_ANALYZE_BLOCKS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_EXT_STATS_COMPUTED] <=
+ a[PROGRESS_ANALYZE_EXT_STATS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_CHILD_TABLES_DONE] <=
+ a[PROGRESS_ANALYZE_CHILD_TABLES_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ Assert(a[PROGRESS_CLUSTER_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_CLUSTER_TOTAL_HEAP_BLKS]);
+ /* FALLTHROUGH */
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ Assert(a[PROGRESS_CREATEIDX_TUPLES_DONE] <=
+ a[PROGRESS_CREATEIDX_TUPLES_TOTAL]);
+ Assert(a[PROGRESS_CREATEIDX_PARTITIONS_DONE] <=
+ a[PROGRESS_CREATEIDX_PARTITIONS_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ /* progress reporting is optional for these */
+ if (a[PROGRESS_BASEBACKUP_BACKUP_TOTAL] >= 0)
+ {
+ Assert(a[PROGRESS_BASEBACKUP_BACKUP_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_BACKUP_TOTAL]);
+ Assert(a[PROGRESS_BASEBACKUP_TBLSPC_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_TBLSPC_TOTAL]);
+ }
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+#if 0
+ //This currently fails file_fdw tests, since pgstat_prorgress evidently fails
+ // to support simultaneous copy commands, as happens during JOIN.
+ /* bytes progress is not available in all cases */
+ if (a[PROGRESS_COPY_BYTES_TOTAL] > 0)
+ //Assert(a[PROGRESS_COPY_BYTES_PROCESSED] <= a[PROGRESS_COPY_BYTES_TOTAL]);
+ if (a[PROGRESS_COPY_BYTES_PROCESSED] > a[PROGRESS_COPY_BYTES_TOTAL])
+ elog(WARNING, "PROGRESS_COPY_BYTES_PROCESSED %ld %ld",
+ a[PROGRESS_COPY_BYTES_PROCESSED],
+ a[PROGRESS_COPY_BYTES_TOTAL]);
+#endif
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break; /* Do nothing */
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -56,6 +135,8 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
@@ -113,6 +194,8 @@ pgstat_progress_update_multi_param(int nparam, const int *index,
}
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
--
2.25.1
[text/x-diff] 0004-f-also-assert-that-progress-values-don-t-go-backward.patch (8.1K, ../../[email protected]/5-0004-f-also-assert-that-progress-values-don-t-go-backward.patch)
download | inline diff:
From ee27366bf3f7f5fd6a9884462476b4240cdd5c9a Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 21 Jan 2023 21:41:04 -0600
Subject: [PATCH 4/4] f! also assert that progress values don't go backwards
and the total is constant
See also:
https://www.postgresql.org/message-id/CA%2BTgmoYSvEP3weQaCPGf6%2BDXLy2__JbJUYtoXyWP%3DqHcyGbihA%40mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 37 +++++++++
src/backend/commands/analyze.c | 10 ++-
src/backend/storage/lmgr/lmgr.c | 24 +++---
src/backend/utils/activity/backend_progress.c | 77 +++++++++++++++++++
4 files changed, 135 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f38..22661f6a292 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1042,6 +1042,13 @@ lazy_scan_heap(LVRelState *vacrel)
/* Forget the LP_DEAD items that we just vacuumed */
dead_items->num_items = 0;
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -2199,6 +2206,13 @@ lazy_vacuum(LVRelState *vacrel)
{
Assert(!vacrel->do_index_cleanup);
vacrel->dead_items->num_items = 0;
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
return;
}
@@ -2301,6 +2315,13 @@ lazy_vacuum(LVRelState *vacrel)
* vacuum)
*/
vacrel->dead_items->num_items = 0;
+
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
}
/*
@@ -2414,12 +2435,23 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+#if 0
+ int const progress_inds[] = {
+ PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+ };
+ const int64 progress_vals[] = {
+ PROGRESS_VACUUM_PHASE_VACUUM_HEAP,
+ 0,
+ };
+#endif
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
Assert(vacrel->num_index_scans > 0);
/* Report that we are now vacuuming the heap */
+ //pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
@@ -3190,7 +3222,12 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
dead_items->max_items = max_items;
dead_items->num_items = 0;
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
vacrel->dead_items = dead_items;
}
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index c86e690980e..96710b84558 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1145,6 +1145,12 @@ acquire_sample_rows(Relation onerel, int elevel,
TableScanDesc scan;
BlockNumber nblocks;
BlockNumber blksdone = 0;
+ int64 progress_vals[2] = {0};
+ int const progress_inds[2] = {
+ PROGRESS_ANALYZE_BLOCKS_DONE,
+ PROGRESS_ANALYZE_BLOCKS_TOTAL
+ };
+
#ifdef USE_PREFETCH
int prefetch_maximum = 0; /* blocks to prefetch if enabled */
BlockSamplerData prefetch_bs;
@@ -1169,8 +1175,8 @@ acquire_sample_rows(Relation onerel, int elevel,
#endif
/* Report sampling block numbers */
- pgstat_progress_update_param(PROGRESS_ANALYZE_BLOCKS_TOTAL,
- nblocks);
+ progress_vals[1] = nblocks;
+ pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
/* Prepare for sampling rows */
reservoir_init_selection_state(&rstate, targrows);
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index ee9b89a6726..8666d850660 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,6 +912,15 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
int total = 0;
int done = 0;
+ const int index[] = {
+ PROGRESS_WAITFOR_TOTAL,
+ PROGRESS_WAITFOR_DONE,
+ PROGRESS_WAITFOR_CURRENT_PID
+ };
+ const int64 values[] = {
+ 0, 0, 0
+ };
+
/* Done if no locks to wait for */
if (locktags == NIL)
return;
@@ -930,7 +939,10 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
}
if (progress)
+ {
+ pgstat_progress_update_multi_param(3, index, values);
pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+ }
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
@@ -960,19 +972,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
}
}
- if (progress)
- {
- const int index[] = {
- PROGRESS_WAITFOR_TOTAL,
- PROGRESS_WAITFOR_DONE,
- PROGRESS_WAITFOR_CURRENT_PID
- };
- const int64 values[] = {
- 0, 0, 0
- };
+ if (progress)
pgstat_progress_update_multi_param(3, index, values);
- }
list_free_deep(holders);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d0566430fab..707ab929fac 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -116,6 +116,74 @@ pgstat_progress_asserts(void)
}
}
+static void
+pgstat_progress_assert_forward_progress(int command, int index,
+ int64 oldval, int64 newval)
+{
+ switch (command)
+ {
+ case PROGRESS_COMMAND_ANALYZE:
+
+ /*
+ * phase goes backwards for inheritance tables, which are sampled
+ * twice
+ */
+ if (index != PROGRESS_ANALYZE_CURRENT_CHILD_TABLE_RELID &&
+ index != PROGRESS_ANALYZE_PHASE)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_ANALYZE_BLOCKS_TOTAL ||
+ index == PROGRESS_ANALYZE_EXT_STATS_TOTAL ||
+ index == PROGRESS_ANALYZE_CHILD_TABLES_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ if (index != PROGRESS_CLUSTER_INDEX_RELID)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_CLUSTER_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ if (index != PROGRESS_CREATEIDX_INDEX_OID &&
+ index != PROGRESS_CREATEIDX_SUBPHASE &&
+ index != PROGRESS_WAITFOR_CURRENT_PID &&
+ index != PROGRESS_CREATEIDX_ACCESS_METHOD_OID)
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_CREATEIDX_TUPLES_TOTAL ||
+ index == PROGRESS_CREATEIDX_PARTITIONS_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL &&
+ oldval == 0 && newval == -1)
+ return; /* Do nothing: this is the initial "null"
+ * state before the size is estimated */
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL ||
+ index == PROGRESS_BASEBACKUP_TBLSPC_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_COPY_BYTES_TOTAL)
+ Assert(oldval == 0);
+ break;
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_VACUUM_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break;
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -132,6 +200,15 @@ pgstat_progress_update_param(int index, int64 val)
if (!beentry || !pgstat_track_activities)
return;
+ if (index != PROGRESS_SCAN_BLOCKS_DONE)
+ {
+ /* Check that progress does not go backwards */
+ int64 oldval = beentry->st_progress_param[index];
+
+ pgstat_progress_assert_forward_progress(beentry->st_progress_command,
+ index, oldval, val);
+ }
+
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
--
2.25.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-02-08 22:40 ` Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-02-08 22:40 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Thu, Feb 02, 2023 at 09:18:07AM -0600, Justin Pryzby wrote:
> On Wed, Feb 01, 2023 at 07:24:48PM +0100, Matthias van de Meent wrote:
> > On Wed, 1 Feb 2023 at 18:51, Ilya Gladyshev <[email protected]> wrote:
> > > 1 февр. 2023 г., в 20:27, Matthias van de Meent <[email protected]> написал(а):
> > >
> > >> In HEAD we set TOTAL to whatever number partitioned table we're
> > >> currently processing has - regardless of whether we're the top level
> > >> statement.
> > >> With the patch we instead add the number of child relations to that
> > >> count, for which REL_HAS_STORAGE(child) -- or at least, in the v3
> > >> posted by Ilya. Approximately immediately after updating that count we
> > >> recurse to the child relations, and that only returns once it is done
> > >> creating the indexes, so both TOTAL and DONE go up as we process more
> > >> partitions in the hierarchy.
> > >
> > > The TOTAL in the patch is set only when processing the top-level parent and it is not updated when we recurse, so yes, it is constant. From v3:
> >
> > Ugh, I misread the patch, more specifically count_leaf_partitions and
> > the !OidIsValid(parentIndexId) condition changes.
> >
> > You are correct, sorry for the noise.
>
> That suggests that the comments could've been more clear. I added a
> comment suggested by Tomas and adjusted some others and wrote a commit
> message. I even ran pgindent for about the 3rd time ever.
>
> 002 are my changes as a separate patch, which you could apply to your
> local branch.
>
> And 003/4 are assertions that I wrote to demonstrate the problem and the
> verify the fixes, but not being proposed for commit.
That was probably a confusing way to present it - I should've sent the
relative diff as a .txt rather than as patch 002.
This squishes together 001/2 as the main patch.
I believe it's ready.
--
Justin
Attachments:
[text/x-diff] 0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch (8.1K, ../../[email protected]/2-0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch)
download | inline diff:
From ed643acace062cad15cd6ea9dc76a663de9c97d4 Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <[email protected]>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH 1/3] fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, but doesn't count intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, and if the attached index is
partitioned, the counter is incremented to account for each of its leaf
partitions.
Author: Ilya Gladyshev
Reviewed-By: Justin Pryzby, Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 ++-
src/backend/commands/indexcmds.c | 70 +++++++++++++++++--
src/backend/utils/activity/backend_progress.c | 28 ++++++++
src/include/utils/backend_progress.h | 1 +
4 files changed, 102 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1756f1a4b67..fa139dcece7 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6601,7 +6601,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6612,7 +6615,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 16ec0b114e6..84c84c41acc 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -130,6 +130,30 @@ typedef struct ReindexErrorInfo
char relkind;
} ReindexErrorInfo;
+
+/*
+ * Count the number of direct and indirect leaf partitions, excluding foreign
+ * tables.
+ */
+static int
+count_leaf_partitions(Oid relid)
+{
+ int nleaves = 0;
+ List *childs = find_all_inheritors(relid, NoLock, NULL);
+ ListCell *lc;
+
+ foreach(lc, childs)
+ {
+ Oid partrelid = lfirst_oid(lc);
+
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
+ nleaves++;
+ }
+
+ list_free(childs);
+ return nleaves;
+}
+
/*
* CheckIndexCompatible
* Determine whether an existing index definition is compatible with a
@@ -1219,8 +1243,18 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ /*
+ * Set the total number of partitions at the start of the command,
+ * but don't change it when being called recursively.
+ */
+ if (!OidIsValid(parentIndexId))
+ {
+ int total_parts;
+
+ total_parts = count_leaf_partitions(relationId);
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_parts);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1250,6 +1284,7 @@ DefineIndex(Oid relationId,
{
Oid childRelid = part_oids[i];
Relation childrel;
+ char child_relkind;
Oid child_save_userid;
int child_save_sec_context;
int child_save_nestlevel;
@@ -1259,6 +1294,7 @@ DefineIndex(Oid relationId,
bool found = false;
childrel = table_open(childRelid, lockmode);
+ child_relkind = RelationGetForm(childrel)->relkind;
GetUserIdAndSecContext(&child_save_userid,
&child_save_sec_context);
@@ -1431,9 +1467,27 @@ DefineIndex(Oid relationId,
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ int attached_parts;
+
+ /*
+ * Avoid the overhead of counting partitions when that
+ * can't apply.
+ */
+ attached_parts = RELKIND_HAS_PARTITIONS(child_relkind) ?
+ count_leaf_partitions(childRelid) : 1;
+
+ /*
+ * If a pre-existing index was attached, the progress
+ * report is updated here. If the index was partitioned,
+ * all the children that were counted towards
+ * PROGRESS_CREATEIDX_PARTITIONS_TOTAL are counted as
+ * done.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, attached_parts);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1484,9 +1538,15 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, we're done. When called recursively
+ * for child tables, the done partition counter is incremented now,
+ * rather than in the caller.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,34 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of the current backend.
+ *-----------
+ */
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.25.1
[text/x-diff] 0002-assertions-for-progress-reporting.patch (4.0K, ../../[email protected]/3-0002-assertions-for-progress-reporting.patch)
download | inline diff:
From 184bf2d291944c70323c95c99174ab446fc05149 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 1 Feb 2023 10:23:53 -0600
Subject: [PATCH 2/3] assertions for progress reporting
---
src/backend/utils/activity/backend_progress.c | 83 +++++++++++++++++++
1 file changed, 83 insertions(+)
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 2a9994b98fd..a91b5881b04 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
@@ -37,6 +38,84 @@ pgstat_progress_start_command(ProgressCommandType cmdtype, Oid relid)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Check for consistency of progress data (current < total).
+ *
+ * Check during pgstat_progress_updates_*() rather than only from
+ * pgstat_progress_end_command() to catch issues with uninitialized/stale data
+ * from previous progress commands.
+ *
+ * If a command fails due to interrupt or error, the values may be less than
+ * the expected final value.
+ */
+static void
+pgstat_progress_asserts(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ volatile int64 *a = beentry->st_progress_param;
+
+ switch (beentry->st_progress_command)
+ {
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_VACUUMED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_NUM_DEAD_TUPLES] <=
+ a[PROGRESS_VACUUM_MAX_DEAD_TUPLES]);
+ break;
+
+ case PROGRESS_COMMAND_ANALYZE:
+ Assert(a[PROGRESS_ANALYZE_BLOCKS_DONE] <=
+ a[PROGRESS_ANALYZE_BLOCKS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_EXT_STATS_COMPUTED] <=
+ a[PROGRESS_ANALYZE_EXT_STATS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_CHILD_TABLES_DONE] <=
+ a[PROGRESS_ANALYZE_CHILD_TABLES_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ Assert(a[PROGRESS_CLUSTER_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_CLUSTER_TOTAL_HEAP_BLKS]);
+ /* fall through because CLUSTER rebuilds indexes */
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ Assert(a[PROGRESS_CREATEIDX_TUPLES_DONE] <=
+ a[PROGRESS_CREATEIDX_TUPLES_TOTAL]);
+ Assert(a[PROGRESS_CREATEIDX_PARTITIONS_DONE] <=
+ a[PROGRESS_CREATEIDX_PARTITIONS_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ /* progress reporting is optional for these */
+ if (a[PROGRESS_BASEBACKUP_BACKUP_TOTAL] >= 0)
+ {
+ Assert(a[PROGRESS_BASEBACKUP_BACKUP_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_BACKUP_TOTAL]);
+ Assert(a[PROGRESS_BASEBACKUP_TBLSPC_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_TBLSPC_TOTAL]);
+ }
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+#if 0
+ //This currently fails file_fdw tests, since pgstat_prorgress evidently fails
+ // to support simultaneous copy commands, as happens during JOIN.
+ /* bytes progress is not available in all cases */
+ if (a[PROGRESS_COPY_BYTES_TOTAL] > 0)
+ //Assert(a[PROGRESS_COPY_BYTES_PROCESSED] <= a[PROGRESS_COPY_BYTES_TOTAL]);
+ if (a[PROGRESS_COPY_BYTES_PROCESSED] > a[PROGRESS_COPY_BYTES_TOTAL])
+ elog(WARNING, "PROGRESS_COPY_BYTES_PROCESSED %ld %ld",
+ a[PROGRESS_COPY_BYTES_PROCESSED],
+ a[PROGRESS_COPY_BYTES_TOTAL]);
+#endif
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break; /* Do nothing */
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -56,6 +135,8 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
@@ -113,6 +194,8 @@ pgstat_progress_update_multi_param(int nparam, const int *index,
}
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
--
2.25.1
[text/x-diff] 0003-f-also-assert-that-progress-values-don-t-go-backward.patch (8.1K, ../../[email protected]/4-0003-f-also-assert-that-progress-values-don-t-go-backward.patch)
download | inline diff:
From e63816c3683eadb0a107843a2544e127e6c441d6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 21 Jan 2023 21:41:04 -0600
Subject: [PATCH 3/3] f! also assert that progress values don't go backwards
and the total is constant
See also:
https://www.postgresql.org/message-id/CA%2BTgmoYSvEP3weQaCPGf6%2BDXLy2__JbJUYtoXyWP%3DqHcyGbihA%40mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 37 +++++++++
src/backend/commands/analyze.c | 10 ++-
src/backend/storage/lmgr/lmgr.c | 24 +++---
src/backend/utils/activity/backend_progress.c | 77 +++++++++++++++++++
4 files changed, 135 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f38..22661f6a292 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1042,6 +1042,13 @@ lazy_scan_heap(LVRelState *vacrel)
/* Forget the LP_DEAD items that we just vacuumed */
dead_items->num_items = 0;
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -2199,6 +2206,13 @@ lazy_vacuum(LVRelState *vacrel)
{
Assert(!vacrel->do_index_cleanup);
vacrel->dead_items->num_items = 0;
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
return;
}
@@ -2301,6 +2315,13 @@ lazy_vacuum(LVRelState *vacrel)
* vacuum)
*/
vacrel->dead_items->num_items = 0;
+
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
}
/*
@@ -2414,12 +2435,23 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+#if 0
+ int const progress_inds[] = {
+ PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+ };
+ const int64 progress_vals[] = {
+ PROGRESS_VACUUM_PHASE_VACUUM_HEAP,
+ 0,
+ };
+#endif
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
Assert(vacrel->num_index_scans > 0);
/* Report that we are now vacuuming the heap */
+ //pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
@@ -3190,7 +3222,12 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
dead_items->max_items = max_items;
dead_items->num_items = 0;
+ {
+ int const progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
vacrel->dead_items = dead_items;
}
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index c86e690980e..96710b84558 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1145,6 +1145,12 @@ acquire_sample_rows(Relation onerel, int elevel,
TableScanDesc scan;
BlockNumber nblocks;
BlockNumber blksdone = 0;
+ int64 progress_vals[2] = {0};
+ int const progress_inds[2] = {
+ PROGRESS_ANALYZE_BLOCKS_DONE,
+ PROGRESS_ANALYZE_BLOCKS_TOTAL
+ };
+
#ifdef USE_PREFETCH
int prefetch_maximum = 0; /* blocks to prefetch if enabled */
BlockSamplerData prefetch_bs;
@@ -1169,8 +1175,8 @@ acquire_sample_rows(Relation onerel, int elevel,
#endif
/* Report sampling block numbers */
- pgstat_progress_update_param(PROGRESS_ANALYZE_BLOCKS_TOTAL,
- nblocks);
+ progress_vals[1] = nblocks;
+ pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
/* Prepare for sampling rows */
reservoir_init_selection_state(&rstate, targrows);
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index ee9b89a6726..8666d850660 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,6 +912,15 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
int total = 0;
int done = 0;
+ const int index[] = {
+ PROGRESS_WAITFOR_TOTAL,
+ PROGRESS_WAITFOR_DONE,
+ PROGRESS_WAITFOR_CURRENT_PID
+ };
+ const int64 values[] = {
+ 0, 0, 0
+ };
+
/* Done if no locks to wait for */
if (locktags == NIL)
return;
@@ -930,7 +939,10 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
}
if (progress)
+ {
+ pgstat_progress_update_multi_param(3, index, values);
pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+ }
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
@@ -960,19 +972,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
}
}
- if (progress)
- {
- const int index[] = {
- PROGRESS_WAITFOR_TOTAL,
- PROGRESS_WAITFOR_DONE,
- PROGRESS_WAITFOR_CURRENT_PID
- };
- const int64 values[] = {
- 0, 0, 0
- };
+ if (progress)
pgstat_progress_update_multi_param(3, index, values);
- }
list_free_deep(holders);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index a91b5881b04..3de10361222 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -116,6 +116,74 @@ pgstat_progress_asserts(void)
}
}
+static void
+pgstat_progress_assert_forward_progress(int command, int index,
+ int64 oldval, int64 newval)
+{
+ switch (command)
+ {
+ case PROGRESS_COMMAND_ANALYZE:
+
+ /*
+ * phase goes backwards for inheritance tables, which are sampled
+ * twice
+ */
+ if (index != PROGRESS_ANALYZE_CURRENT_CHILD_TABLE_RELID &&
+ index != PROGRESS_ANALYZE_PHASE)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_ANALYZE_BLOCKS_TOTAL ||
+ index == PROGRESS_ANALYZE_EXT_STATS_TOTAL ||
+ index == PROGRESS_ANALYZE_CHILD_TABLES_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ if (index != PROGRESS_CLUSTER_INDEX_RELID)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_CLUSTER_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ if (index != PROGRESS_CREATEIDX_INDEX_OID &&
+ index != PROGRESS_CREATEIDX_SUBPHASE &&
+ index != PROGRESS_WAITFOR_CURRENT_PID &&
+ index != PROGRESS_CREATEIDX_ACCESS_METHOD_OID)
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_CREATEIDX_TUPLES_TOTAL ||
+ index == PROGRESS_CREATEIDX_PARTITIONS_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL &&
+ oldval == 0 && newval == -1)
+ return; /* Do nothing: this is the initial "null"
+ * state before the size is estimated */
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL ||
+ index == PROGRESS_BASEBACKUP_TBLSPC_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_COPY_BYTES_TOTAL)
+ Assert(oldval == 0);
+ break;
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_VACUUM_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break;
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -132,6 +200,15 @@ pgstat_progress_update_param(int index, int64 val)
if (!beentry || !pgstat_track_activities)
return;
+ if (index != PROGRESS_SCAN_BLOCKS_DONE)
+ {
+ /* Check that progress does not go backwards */
+ int64 oldval = beentry->st_progress_param[index];
+
+ pgstat_progress_assert_forward_progress(beentry->st_progress_command,
+ index, oldval, val);
+ }
+
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
--
2.25.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-02-17 00:26 ` Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-02-17 00:26 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Wed, Feb 08, 2023 at 04:40:49PM -0600, Justin Pryzby wrote:
> This squishes together 001/2 as the main patch.
> I believe it's ready.
Update to address a compiler warning in the supplementary patches adding
assertions.
Attachments:
[text/x-diff] 0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch (8.2K, ../../[email protected]/2-0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch)
download | inline diff:
From 71427bf7cd9927af04513ba3fe99e481a8ba1f61 Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <[email protected]>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH 1/3] fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, but doesn't count intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, and if the attached index is
partitioned, the counter is incremented to account for each of its leaf
partitions.
Author: Ilya Gladyshev
Reviewed-By: Justin Pryzby, Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 ++-
src/backend/commands/indexcmds.c | 70 +++++++++++++++++--
src/backend/utils/activity/backend_progress.c | 28 ++++++++
src/include/utils/backend_progress.h | 1 +
4 files changed, 102 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dca50707ad4..945d64ed2fa 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6896,7 +6896,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6907,7 +6910,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 16ec0b114e6..147265ddcca 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -130,6 +130,30 @@ typedef struct ReindexErrorInfo
char relkind;
} ReindexErrorInfo;
+
+/*
+ * Count the number of direct and indirect leaf partitions. Note that this
+ * also excludes foreign tables.
+ */
+static int
+count_leaf_partitions(Oid relid)
+{
+ int nleaves = 0;
+ List *childs = find_all_inheritors(relid, NoLock, NULL);
+ ListCell *lc;
+
+ foreach(lc, childs)
+ {
+ Oid partrelid = lfirst_oid(lc);
+
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
+ nleaves++;
+ }
+
+ list_free(childs);
+ return nleaves;
+}
+
/*
* CheckIndexCompatible
* Determine whether an existing index definition is compatible with a
@@ -1219,8 +1243,18 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ /*
+ * Set the total number of partitions at the start of the command;
+ * don't update it when being called recursively.
+ */
+ if (!OidIsValid(parentIndexId))
+ {
+ int total_parts;
+
+ total_parts = count_leaf_partitions(relationId);
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_parts);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1250,6 +1284,7 @@ DefineIndex(Oid relationId,
{
Oid childRelid = part_oids[i];
Relation childrel;
+ char child_relkind;
Oid child_save_userid;
int child_save_sec_context;
int child_save_nestlevel;
@@ -1259,6 +1294,7 @@ DefineIndex(Oid relationId,
bool found = false;
childrel = table_open(childRelid, lockmode);
+ child_relkind = RelationGetForm(childrel)->relkind;
GetUserIdAndSecContext(&child_save_userid,
&child_save_sec_context);
@@ -1431,9 +1467,27 @@ DefineIndex(Oid relationId,
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ int attached_parts;
+
+ /*
+ * Avoid the overhead of counting partitions when that
+ * can't apply.
+ */
+ attached_parts = RELKIND_HAS_PARTITIONS(child_relkind) ?
+ count_leaf_partitions(childRelid) : 1;
+
+ /*
+ * If a pre-existing index was attached, the progress
+ * report is updated here. If the index was partitioned,
+ * all the children that were counted towards
+ * PROGRESS_CREATEIDX_PARTITIONS_TOTAL are counted as
+ * done.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, attached_parts);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1484,9 +1538,15 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, we're done. When called recursively
+ * for child tables, the done partition counter is incremented now,
+ * rather than in the caller.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,34 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of the current backend.
+ *-----------
+ */
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.34.1
[text/x-diff] 0002-assertions-for-progress-reporting.patch (4.0K, ../../[email protected]/3-0002-assertions-for-progress-reporting.patch)
download | inline diff:
From 30c5da099a6b75d7e4ce860815907326a361b042 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 1 Feb 2023 10:23:53 -0600
Subject: [PATCH 2/3] assertions for progress reporting
---
src/backend/utils/activity/backend_progress.c | 84 +++++++++++++++++++
1 file changed, 84 insertions(+)
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 2a9994b98fd..63f9482b175 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
@@ -37,6 +38,85 @@ pgstat_progress_start_command(ProgressCommandType cmdtype, Oid relid)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Check for consistency of progress data (current < total).
+ *
+ * Check during pgstat_progress_updates_*() rather than only from
+ * pgstat_progress_end_command() to catch issues with uninitialized/stale data
+ * from previous progress commands.
+ *
+ * If a command fails due to interrupt or error, the values may be less than
+ * the expected final value.
+ */
+static void
+pgstat_progress_asserts(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ volatile int64 *a = beentry->st_progress_param;
+
+ switch (beentry->st_progress_command)
+ {
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_VACUUMED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_NUM_DEAD_TUPLES] <=
+ a[PROGRESS_VACUUM_MAX_DEAD_TUPLES]);
+ break;
+
+ case PROGRESS_COMMAND_ANALYZE:
+ Assert(a[PROGRESS_ANALYZE_BLOCKS_DONE] <=
+ a[PROGRESS_ANALYZE_BLOCKS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_EXT_STATS_COMPUTED] <=
+ a[PROGRESS_ANALYZE_EXT_STATS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_CHILD_TABLES_DONE] <=
+ a[PROGRESS_ANALYZE_CHILD_TABLES_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ Assert(a[PROGRESS_CLUSTER_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_CLUSTER_TOTAL_HEAP_BLKS]);
+ /* FALLTHROUGH */
+ /* ..because CLUSTER rebuilds indexes */
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ Assert(a[PROGRESS_CREATEIDX_TUPLES_DONE] <=
+ a[PROGRESS_CREATEIDX_TUPLES_TOTAL]);
+ Assert(a[PROGRESS_CREATEIDX_PARTITIONS_DONE] <=
+ a[PROGRESS_CREATEIDX_PARTITIONS_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ /* progress reporting is optional for these */
+ if (a[PROGRESS_BASEBACKUP_BACKUP_TOTAL] >= 0)
+ {
+ Assert(a[PROGRESS_BASEBACKUP_BACKUP_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_BACKUP_TOTAL]);
+ Assert(a[PROGRESS_BASEBACKUP_TBLSPC_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_TBLSPC_TOTAL]);
+ }
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+#if 0
+ //This currently fails file_fdw tests, since pgstat_prorgress evidently fails
+ // to support simultaneous copy commands, as happens during JOIN.
+ /* bytes progress is not available in all cases */
+ if (a[PROGRESS_COPY_BYTES_TOTAL] > 0)
+ //Assert(a[PROGRESS_COPY_BYTES_PROCESSED] <= a[PROGRESS_COPY_BYTES_TOTAL]);
+ if (a[PROGRESS_COPY_BYTES_PROCESSED] > a[PROGRESS_COPY_BYTES_TOTAL])
+ elog(WARNING, "PROGRESS_COPY_BYTES_PROCESSED %ld %ld",
+ a[PROGRESS_COPY_BYTES_PROCESSED],
+ a[PROGRESS_COPY_BYTES_TOTAL]);
+#endif
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break; /* Do nothing */
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -56,6 +136,8 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
@@ -113,6 +195,8 @@ pgstat_progress_update_multi_param(int nparam, const int *index,
}
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
--
2.34.1
[text/x-diff] 0003-f-also-assert-that-progress-values-don-t-go-backward.patch (8.1K, ../../[email protected]/4-0003-f-also-assert-that-progress-values-don-t-go-backward.patch)
download | inline diff:
From 5ca1f068413a64eeba0a5fb914287f9855ffb9b7 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 21 Jan 2023 21:41:04 -0600
Subject: [PATCH 3/3] f! also assert that progress values don't go backwards
and the total is constant
See also:
https://www.postgresql.org/message-id/CA%2BTgmoYSvEP3weQaCPGf6%2BDXLy2__JbJUYtoXyWP%3DqHcyGbihA%40mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 37 +++++++++
src/backend/commands/analyze.c | 10 ++-
src/backend/storage/lmgr/lmgr.c | 24 +++---
src/backend/utils/activity/backend_progress.c | 77 +++++++++++++++++++
4 files changed, 135 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f38..61cfbf6a17a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1042,6 +1042,13 @@ lazy_scan_heap(LVRelState *vacrel)
/* Forget the LP_DEAD items that we just vacuumed */
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -2199,6 +2206,13 @@ lazy_vacuum(LVRelState *vacrel)
{
Assert(!vacrel->do_index_cleanup);
vacrel->dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
return;
}
@@ -2301,6 +2315,13 @@ lazy_vacuum(LVRelState *vacrel)
* vacuum)
*/
vacrel->dead_items->num_items = 0;
+
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
}
/*
@@ -2414,12 +2435,23 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+#if 0
+ const int progress_inds[] = {
+ PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+ };
+ const int64 progress_vals[] = {
+ PROGRESS_VACUUM_PHASE_VACUUM_HEAP,
+ 0,
+ };
+#endif
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
Assert(vacrel->num_index_scans > 0);
/* Report that we are now vacuuming the heap */
+ //pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
@@ -3190,7 +3222,12 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
dead_items->max_items = max_items;
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
vacrel->dead_items = dead_items;
}
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 65750958bb2..3bfc941aa2c 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1145,6 +1145,12 @@ acquire_sample_rows(Relation onerel, int elevel,
TableScanDesc scan;
BlockNumber nblocks;
BlockNumber blksdone = 0;
+ int64 progress_vals[2] = {0};
+ int const progress_inds[2] = {
+ PROGRESS_ANALYZE_BLOCKS_DONE,
+ PROGRESS_ANALYZE_BLOCKS_TOTAL
+ };
+
#ifdef USE_PREFETCH
int prefetch_maximum = 0; /* blocks to prefetch if enabled */
BlockSamplerData prefetch_bs;
@@ -1169,8 +1175,8 @@ acquire_sample_rows(Relation onerel, int elevel,
#endif
/* Report sampling block numbers */
- pgstat_progress_update_param(PROGRESS_ANALYZE_BLOCKS_TOTAL,
- nblocks);
+ progress_vals[1] = nblocks;
+ pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
/* Prepare for sampling rows */
reservoir_init_selection_state(&rstate, targrows);
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index ee9b89a6726..8666d850660 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,6 +912,15 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
int total = 0;
int done = 0;
+ const int index[] = {
+ PROGRESS_WAITFOR_TOTAL,
+ PROGRESS_WAITFOR_DONE,
+ PROGRESS_WAITFOR_CURRENT_PID
+ };
+ const int64 values[] = {
+ 0, 0, 0
+ };
+
/* Done if no locks to wait for */
if (locktags == NIL)
return;
@@ -930,7 +939,10 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
}
if (progress)
+ {
+ pgstat_progress_update_multi_param(3, index, values);
pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+ }
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
@@ -960,19 +972,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
}
}
- if (progress)
- {
- const int index[] = {
- PROGRESS_WAITFOR_TOTAL,
- PROGRESS_WAITFOR_DONE,
- PROGRESS_WAITFOR_CURRENT_PID
- };
- const int64 values[] = {
- 0, 0, 0
- };
+ if (progress)
pgstat_progress_update_multi_param(3, index, values);
- }
list_free_deep(holders);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 63f9482b175..e4cfde8c723 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -117,6 +117,74 @@ pgstat_progress_asserts(void)
}
}
+static void
+pgstat_progress_assert_forward_progress(int command, int index,
+ int64 oldval, int64 newval)
+{
+ switch (command)
+ {
+ case PROGRESS_COMMAND_ANALYZE:
+
+ /*
+ * phase goes backwards for inheritance tables, which are sampled
+ * twice
+ */
+ if (index != PROGRESS_ANALYZE_CURRENT_CHILD_TABLE_RELID &&
+ index != PROGRESS_ANALYZE_PHASE)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_ANALYZE_BLOCKS_TOTAL ||
+ index == PROGRESS_ANALYZE_EXT_STATS_TOTAL ||
+ index == PROGRESS_ANALYZE_CHILD_TABLES_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ if (index != PROGRESS_CLUSTER_INDEX_RELID)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_CLUSTER_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ if (index != PROGRESS_CREATEIDX_INDEX_OID &&
+ index != PROGRESS_CREATEIDX_SUBPHASE &&
+ index != PROGRESS_WAITFOR_CURRENT_PID &&
+ index != PROGRESS_CREATEIDX_ACCESS_METHOD_OID)
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_CREATEIDX_TUPLES_TOTAL ||
+ index == PROGRESS_CREATEIDX_PARTITIONS_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL &&
+ oldval == 0 && newval == -1)
+ return; /* Do nothing: this is the initial "null"
+ * state before the size is estimated */
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL ||
+ index == PROGRESS_BASEBACKUP_TBLSPC_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_COPY_BYTES_TOTAL)
+ Assert(oldval == 0);
+ break;
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_VACUUM_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break;
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -133,6 +201,15 @@ pgstat_progress_update_param(int index, int64 val)
if (!beentry || !pgstat_track_activities)
return;
+ if (index != PROGRESS_SCAN_BLOCKS_DONE)
+ {
+ /* Check that progress does not go backwards */
+ int64 oldval = beentry->st_progress_param[index];
+
+ pgstat_progress_assert_forward_progress(beentry->st_progress_command,
+ index, oldval, val);
+ }
+
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
--
2.34.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-10 20:36 ` Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-10 20:36 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> Update to address a compiler warning in the supplementary patches adding
> assertions.
I took a look through this. It seems like basically a good solution,
but the count_leaf_partitions() function is bothering me, for two
reasons:
1. It seems like a pretty expensive thing to do. Don't we have the
info at hand somewhere already?
2. Is it really safe to do find_all_inheritors with NoLock? If so,
a comment explaining why would be good.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-12 20:09 ` Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-12 20:09 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Fri, Mar 10, 2023 at 03:36:10PM -0500, Tom Lane wrote:
> Justin Pryzby <[email protected]> writes:
> > Update to address a compiler warning in the supplementary patches adding
> > assertions.
>
> I took a look through this. It seems like basically a good solution,
> but the count_leaf_partitions() function is bothering me, for two
> reasons:
>
> 1. It seems like a pretty expensive thing to do. Don't we have the
> info at hand somewhere already?
I don't know where that would be. We need the list of both direct *and*
indirect partitions. See:
https://www.postgresql.org/message-id/5073D187-4200-4A2D-BAC0-91C657E3C22E%40gmail.com
If it would help to avoid the concern, then I might consider proposing
not to call get_rel_relkind() ...
> 2. Is it really safe to do find_all_inheritors with NoLock? If so,
> a comment explaining why would be good.
In both cases (both for the parent and for case of a partitioned child
with pre-existing indexes being ATTACHed), the table itself is already
locked by DefineIndex():
lockmode = concurrent ? ShareUpdateExclusiveLock : ShareLock;
rel = table_open(relationId, lockmode);
and
childrel = table_open(childRelid, lockmode);
...
table_close(childrel, NoLock);
And, find_all_inheritors() will also have been called by
ProcessUtilitySlow(). Maybe it's sufficient to mention that ?
--
Justin
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-12 20:14 ` Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-12 20:14 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> On Fri, Mar 10, 2023 at 03:36:10PM -0500, Tom Lane wrote:
>> I took a look through this. It seems like basically a good solution,
>> but the count_leaf_partitions() function is bothering me, for two
>> reasons:
> ... find_all_inheritors() will also have been called by
> ProcessUtilitySlow(). Maybe it's sufficient to mention that ?
Hm. Could we get rid of count_leaf_partitions by doing the work in
ProcessUtilitySlow? Or at least passing that OID list forward instead
of recomputing it?
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-12 22:06 ` Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-12 22:06 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Sun, Mar 12, 2023 at 04:14:06PM -0400, Tom Lane wrote:
> Justin Pryzby <[email protected]> writes:
> > On Fri, Mar 10, 2023 at 03:36:10PM -0500, Tom Lane wrote:
> >> I took a look through this. It seems like basically a good solution,
> >> but the count_leaf_partitions() function is bothering me, for two
> >> reasons:
>
> > ... find_all_inheritors() will also have been called by
> > ProcessUtilitySlow(). Maybe it's sufficient to mention that ?
>
> Hm. Could we get rid of count_leaf_partitions by doing the work in
> ProcessUtilitySlow? Or at least passing that OID list forward instead
> of recomputing it?
count_leaf_partitions() is called in two places:
Once to get PROGRESS_CREATEIDX_PARTITIONS_TOTAL. It'd be easy enough to
pass an integer total via IndexStmt (but I think we wanted to avoid
adding anything there, since it's not a part of the statement).
count_leaf_partitions() is also called for sub-partitions, in the case
that a matching "partitioned index" already exists, and the progress
report needs to be incremented by the number of leaves for which indexes
were ATTACHED. We'd need a mapping from OID => npartitions (or to
compile some data structure of all the partitioned partitions). I guess
CreateIndex() could call CreatePartitionDirectory(). But it looks like
that would be *more* expensive.
--
Justin
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-12 22:25 ` Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-12 22:25 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> On Sun, Mar 12, 2023 at 04:14:06PM -0400, Tom Lane wrote:
>> Hm. Could we get rid of count_leaf_partitions by doing the work in
>> ProcessUtilitySlow? Or at least passing that OID list forward instead
>> of recomputing it?
> count_leaf_partitions() is called in two places:
> Once to get PROGRESS_CREATEIDX_PARTITIONS_TOTAL. It'd be easy enough to
> pass an integer total via IndexStmt (but I think we wanted to avoid
> adding anything there, since it's not a part of the statement).
I agree that adding such a field to IndexStmt would be a very bad idea.
However, adding another parameter to DefineIndex doesn't seem like a
problem. Or could we just call pgstat_progress_update_param directly from
ProcessUtilitySlow, after counting the partitions in the existing loop?
> count_leaf_partitions() is also called for sub-partitions, in the case
> that a matching "partitioned index" already exists, and the progress
> report needs to be incremented by the number of leaves for which indexes
> were ATTACHED.
Can't you increment progress by one at the point where the actual attach
happens?
I also wonder whether leaving non-leaf partitions out of the total
is making things more complicated rather than simpler ...
> We'd need a mapping from OID => npartitions (or to
> compile some data structure of all the partitioned partitions). I guess
> CreateIndex() could call CreatePartitionDirectory(). But it looks like
> that would be *more* expensive.
The reason I find this annoying is that the non-optional nature of the
progress reporting mechanism was sold on the basis that it would add
only negligible overhead. Adding extra pass(es) over pg_inherits
breaks that promise. Maybe it's cheap enough to not matter in the
big scheme of things, but we should not be having to make arguments
like that one.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-13 00:15 ` Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-13 00:15 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
I wrote:
> Justin Pryzby <[email protected]> writes:
>> count_leaf_partitions() is also called for sub-partitions, in the case
>> that a matching "partitioned index" already exists, and the progress
>> report needs to be incremented by the number of leaves for which indexes
>> were ATTACHED.
> Can't you increment progress by one at the point where the actual attach
> happens?
Oh, never mind; now I realize that the point is that you didn't ever
iterate over those leaf indexes.
However, consider a thought experiment: assume for whatever reason that
all the actual index builds happen first, then all the cases where you
succeed in attaching a sub-partitioned index happen at the end of the
command. In that case, the percentage-done indicator would go from
some-number to 100% more or less instantly.
What if we simply do nothing at sub-partitioned indexes? Or if that's
slightly too radical, just increase the PARTITIONS_DONE counter by 1?
That would look indistinguishable from the case where all the attaches
happen at the end.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-13 13:50 ` Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-13 13:50 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Sun, Mar 12, 2023 at 06:25:13PM -0400, Tom Lane wrote:
> Justin Pryzby <[email protected]> writes:
> > On Sun, Mar 12, 2023 at 04:14:06PM -0400, Tom Lane wrote:
> >> Hm. Could we get rid of count_leaf_partitions by doing the work in
> >> ProcessUtilitySlow? Or at least passing that OID list forward instead
> >> of recomputing it?
>
> > count_leaf_partitions() is called in two places:
>
> > Once to get PROGRESS_CREATEIDX_PARTITIONS_TOTAL. It'd be easy enough to
> > pass an integer total via IndexStmt (but I think we wanted to avoid
> > adding anything there, since it's not a part of the statement).
>
> I agree that adding such a field to IndexStmt would be a very bad idea.
> However, adding another parameter to DefineIndex doesn't seem like a
> problem.
It's a problem since this is a bug and it's desirable to backpatch a
fix, right ?
> Or could we just call pgstat_progress_update_param directly from
> ProcessUtilitySlow, after counting the partitions in the existing loop?
That'd be fine if it was only needed for TOTAL, but it doesn't handle
the 2nd call to count_leaf_partitions().
On Sun, Mar 12, 2023 at 08:15:28PM -0400, Tom Lane wrote:
> I wrote:
> > Justin Pryzby <[email protected]> writes:
> >> count_leaf_partitions() is also called for sub-partitions, in the case
> >> that a matching "partitioned index" already exists, and the progress
> >> report needs to be incremented by the number of leaves for which indexes
> >> were ATTACHED.
>
> > Can't you increment progress by one at the point where the actual attach
> > happens?
>
> Oh, never mind; now I realize that the point is that you didn't ever
> iterate over those leaf indexes.
>
> However, consider a thought experiment: assume for whatever reason that
> all the actual index builds happen first, then all the cases where you
> succeed in attaching a sub-partitioned index happen at the end of the
> command. In that case, the percentage-done indicator would go from
> some-number to 100% more or less instantly.
>
> What if we simply do nothing at sub-partitioned indexes? Or if that's
> slightly too radical, just increase the PARTITIONS_DONE counter by 1?
> That would look indistinguishable from the case where all the attaches
> happen at the end.
Incrementing by 0 sounds terrible, since someone who has intermediate
partitioned tables is likely to always see 0% done. (It's true that
intermediate partitioned tables don't seem to have been considered by
the original patch, and it's indisputable that progress reporting
currently misbehaves in that case).
And incrementing PARTITIONS_DONE by 1 could lead to bogus progress
reporting with "N_done > N_Total" if an intermediate partitioned table
had no leaf partitions at all. That's one of the problems this thread
is trying to fix (the other being "total changing in the middle of the
command").
Maybe your idea is usable though, since indirect partitioned indexes
*can* be counted correctly during recursion. What's hard to fix is the
case that an index is both *partitioned* and *attached*. Maybe it's
okay to count that case as 0. The consequence is that the command would
end before the progress report got to 100%.
The other option seems to be to define the progress report to count only
*direct* children.
https://www.postgresql.org/message-id/20221213044331.GJ27893%40telsasoft.com
> The reason I find this annoying is that the non-optional nature of the
> progress reporting mechanism was sold on the basis that it would add
> only negligible overhead. Adding extra pass(es) over pg_inherits
> breaks that promise. Maybe it's cheap enough to not matter in the
> big scheme of things, but we should not be having to make arguments
> like that one.
If someone is running a DDL command involving nested partitions, I'm not
so concerned about the cost of additional scans of pg_inherits. They
either have enough data to justify partitioning partitions, or they're
doing something silly.
--
Justin
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-13 14:42 ` Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-13 14:42 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> On Sun, Mar 12, 2023 at 06:25:13PM -0400, Tom Lane wrote:
>> I agree that adding such a field to IndexStmt would be a very bad idea.
>> However, adding another parameter to DefineIndex doesn't seem like a
>> problem.
> It's a problem since this is a bug and it's desirable to backpatch a
> fix, right ?
I do not think this is important enough to justify a back-patch.
> Incrementing by 0 sounds terrible, since someone who has intermediate
> partitioned tables is likely to always see 0% done.
How so? The counter will increase after there's some actual work done,
ie building an index. If there's no indexes to build then it hardly
matters, because the command will complete in very little time.
> And incrementing PARTITIONS_DONE by 1 could lead to bogus progress
> reporting with "N_done > N_Total" if an intermediate partitioned table
> had no leaf partitions at all.
Well, we could fix that if we made TOTAL be the total number of
descendants rather than just the leaves ;-). But I think not
incrementing is probably better.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-14 14:34 ` Justin Pryzby <[email protected]>
2023-03-14 14:46 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
0 siblings, 2 replies; 44+ messages in thread
From: Justin Pryzby @ 2023-03-14 14:34 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Mon, Mar 13, 2023 at 10:42:59AM -0400, Tom Lane wrote:
> Justin Pryzby <[email protected]> writes:
> > On Sun, Mar 12, 2023 at 06:25:13PM -0400, Tom Lane wrote:
> >> I agree that adding such a field to IndexStmt would be a very bad idea.
> >> However, adding another parameter to DefineIndex doesn't seem like a
> >> problem.
>
> > It's a problem since this is a bug and it's desirable to backpatch a
> > fix, right ?
>
> I do not think this is important enough to justify a back-patch.
That's fine with me, but it comes as a surprise, and it might invalidate
earlier discussions, which were working under the constraint of
maintaining a compatible ABI.
> > Incrementing by 0 sounds terrible, since someone who has intermediate
> > partitioned tables is likely to always see 0% done.
>
> How so? The counter will increase after there's some actual work done,
> ie building an index. If there's no indexes to build then it hardly
> matters, because the command will complete in very little time.
I misunderstood your idea as suggesting to skip progress reporting for
*every* intermediate partitioned index, and its leaves. But I guess
what you meant is to skip progress reporting when ATTACHing an
intermediate partitioned index. That seems okay, since 1) intermediate
partitioned tables are probably rare, and 2) ATTACH is fast, so the
effect is indistinguisable from querying the progress report a few
moments later.
The idea would be for:
1) TOTAL to show the number of direct and indirect leaf partitions;
2) update progress while building direct or indirect indexes;
3) ATTACHing intermediate partitioned tables to increment by 0;
4) ATTACHing a direct child should continue to increment by 1,
since that common case already works as expected and shouldn't be
changed.
The only change from the current patch is (3). (1) still calls
count_leaf_partitions(), but only once. I'd prefer that to rearranging
the progress reporting to set the TOTAL in ProcessUtilitySlow().
--
Justin
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-14 14:46 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 44+ messages in thread
From: Tom Lane @ 2023-03-14 14:46 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Ilya Gladyshev <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> The idea would be for:
> 1) TOTAL to show the number of direct and indirect leaf partitions;
> 2) update progress while building direct or indirect indexes;
> 3) ATTACHing intermediate partitioned tables to increment by 0;
> 4) ATTACHing a direct child should continue to increment by 1,
> since that common case already works as expected and shouldn't be
> changed.
OK.
> The only change from the current patch is (3). (1) still calls
> count_leaf_partitions(), but only once. I'd prefer that to rearranging
> the progress reporting to set the TOTAL in ProcessUtilitySlow().
I don't agree with that. find_all_inheritors is fairly expensive
and it seems completely silly to do it twice just to avoid adding
a parameter to DefineIndex.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-14 14:58 ` Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
1 sibling, 1 reply; 44+ messages in thread
From: Ilya Gladyshev @ 2023-03-14 14:58 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
> 14 марта 2023 г., в 18:34, Justin Pryzby <[email protected]> написал(а):
>
> On Mon, Mar 13, 2023 at 10:42:59AM -0400, Tom Lane wrote:
>> Justin Pryzby <[email protected]> writes:
>>> On Sun, Mar 12, 2023 at 06:25:13PM -0400, Tom Lane wrote:
>>>> I agree that adding such a field to IndexStmt would be a very bad idea.
>>>> However, adding another parameter to DefineIndex doesn't seem like a
>>>> problem.
>>
>>> It's a problem since this is a bug and it's desirable to backpatch a
>>> fix, right ?
>>
>> I do not think this is important enough to justify a back-patch.
>
> That's fine with me, but it comes as a surprise, and it might invalidate
> earlier discussions, which were working under the constraint of
> maintaining a compatible ABI.
>
>>> Incrementing by 0 sounds terrible, since someone who has intermediate
>>> partitioned tables is likely to always see 0% done.
>>
>> How so? The counter will increase after there's some actual work done,
>> ie building an index. If there's no indexes to build then it hardly
>> matters, because the command will complete in very little time.
>
> I misunderstood your idea as suggesting to skip progress reporting for
> *every* intermediate partitioned index, and its leaves. But I guess
> what you meant is to skip progress reporting when ATTACHing an
> intermediate partitioned index. That seems okay, since 1) intermediate
> partitioned tables are probably rare, and 2) ATTACH is fast, so the
> effect is indistinguisable from querying the progress report a few
> moments later.
+1 that counting attached partitioned indexes as 0 is fine.
> The idea would be for:
> 1) TOTAL to show the number of direct and indirect leaf partitions;
> 2) update progress while building direct or indirect indexes;
> 3) ATTACHing intermediate partitioned tables to increment by 0;
> 4) ATTACHing a direct child should continue to increment by 1,
> since that common case already works as expected and shouldn't be
> changed.
>
> The only change from the current patch is (3). (1) still calls
> count_leaf_partitions(), but only once. I'd prefer that to rearranging
> the progress reporting to set the TOTAL in ProcessUtilitySlow().
>
> --
> Justin
As for reusing TOTAL calculated outside of DefineIndex, as I can see, ProcessUtilitySlow is not the only call site for DefineIndex (although, I don’t know whether all of them need progress tracking), for instance, there is ALTER TABLE that calls DefineIndex to create index for constraints. So I feel like rearranging progress reporting will result in unnecessary code duplication in those call sites, so passing in an optional parameter seems to be easier here, if we are going to optimize it, after all. Especially if back-patching is a non-issue.
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
@ 2023-03-16 00:07 ` Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-16 00:07 UTC (permalink / raw)
To: Ilya Gladyshev <[email protected]>; +Cc: Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Tue, Mar 14, 2023 at 06:58:14PM +0400, Ilya Gladyshev wrote:
> > The only change from the current patch is (3). (1) still calls
> > count_leaf_partitions(), but only once. I'd prefer that to rearranging
> > the progress reporting to set the TOTAL in ProcessUtilitySlow().
>
> As for reusing TOTAL calculated outside of DefineIndex, as I can see, ProcessUtilitySlow is not the only call site for DefineIndex (although, I don’t know whether all of them need progress tracking), for instance, there is ALTER TABLE that calls DefineIndex to create index for constraints. So I feel like rearranging progress reporting will result in unnecessary code duplication in those call sites, so passing in an optional parameter seems to be easier here, if we are going to optimize it, after all. Especially if back-patching is a non-issue.
Yeah. See attached. I don't like duplicating the loop. Is this really
the right direction to go ?
I haven't verified if the child tables are locked in all the paths which
would call count_leaf_partitions(). But why is it important to lock
them for this? If they weren't locked before, that'd be a pre-existing
problem...
Attachments:
[text/x-diff] 0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch (12.8K, ../../[email protected]/2-0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch)
download | inline diff:
From 1e05b055d1bd21265b68265ddef5e9654629087c Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <[email protected]>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH] fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, but doesn't count intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, unless the attached index is
partitioned, in which case progress report is not updated.
Author: Ilya Gladyshev
Reviewed-By: Justin Pryzby, Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 ++-
src/backend/bootstrap/bootparse.y | 2 +
src/backend/commands/indexcmds.c | 70 +++++++++++++++++--
src/backend/commands/tablecmds.c | 4 +-
src/backend/tcop/utility.c | 8 ++-
src/backend/utils/activity/backend_progress.c | 28 ++++++++
src/include/commands/defrem.h | 1 +
src/include/utils/backend_progress.h | 1 +
8 files changed, 115 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 6249bb50d02..3f891b75541 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6901,7 +6901,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6912,7 +6915,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 86804bb598e..81a1b7bfec3 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -306,6 +306,7 @@ Boot_DeclareIndexStmt:
$4,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
@@ -358,6 +359,7 @@ Boot_DeclareUniqueIndexStmt:
$5,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 16ec0b114e6..62a8f0d6aa2 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -130,6 +130,32 @@ typedef struct ReindexErrorInfo
char relkind;
} ReindexErrorInfo;
+
+/*
+ * Count the number of direct and indirect leaf partitions. Note that this
+ * also excludes foreign tables. This should be consistent with the loop in
+ * ProcessUtilitySlow().
+ * XXX: are the partitions already locked when this is called by other code paths ?
+ */
+static int
+count_leaf_partitions(Oid relid)
+{
+ int nleaves = 0;
+ List *childs = find_all_inheritors(relid, NoLock, NULL);
+ ListCell *lc;
+
+ foreach(lc, childs)
+ {
+ Oid partrelid = lfirst_oid(lc);
+
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
+ nleaves++;
+ }
+
+ list_free(childs);
+ return nleaves;
+}
+
/*
* CheckIndexCompatible
* Determine whether an existing index definition is compatible with a
@@ -518,6 +544,7 @@ WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
* case the caller had better have checked it earlier.
* 'skip_build': make the catalog entries but don't create the index files
* 'quiet': suppress the NOTICE chatter ordinarily provided for constraints.
+ * 'total_leaves': total number of leaf partitions (excluding foreign tables)
*
* Returns the object address of the created index.
*/
@@ -527,6 +554,7 @@ DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_leaves,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
@@ -1219,8 +1247,22 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ /*
+ * Set the total number of partitions at the start of the command;
+ * don't update it when being called recursively.
+ */
+ if (!OidIsValid(parentIndexId))
+ {
+ /*
+ * When called by ProcessUtilitySlow() the number of leaves is
+ * passed in as an optimization.
+ */
+ if (total_leaves < 0)
+ total_leaves = count_leaf_partitions(relationId);
+
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_leaves);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1250,6 +1292,7 @@ DefineIndex(Oid relationId,
{
Oid childRelid = part_oids[i];
Relation childrel;
+ char child_relkind;
Oid child_save_userid;
int child_save_sec_context;
int child_save_nestlevel;
@@ -1259,6 +1302,7 @@ DefineIndex(Oid relationId,
bool found = false;
childrel = table_open(childRelid, lockmode);
+ child_relkind = RelationGetForm(childrel)->relkind;
GetUserIdAndSecContext(&child_save_userid,
&child_save_sec_context);
@@ -1426,14 +1470,24 @@ DefineIndex(Oid relationId,
InvalidOid, /* no predefined OID */
indexRelationId, /* this is our child */
createdConstraintId,
+ -1, /* The progress isn't updated during recursion */
is_alter_table, check_rights, check_not_in_use,
skip_build, quiet);
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ /*
+ * If a pre-existing index was attached, the progress
+ * report is updated here. But if the index is
+ * partitioned, we don't count its partitions, since
+ * that's expensive, and the ATTACH is fast anyway.
+ */
+ if (!RELKIND_HAS_PARTITIONS(child_relkind))
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1484,9 +1538,15 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, we're done. When called recursively
+ * for child tables, the done partition counter is incremented now,
+ * rather than in the caller.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3e2c5f797cd..2ef4491ec8e 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -1216,6 +1216,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
InvalidOid,
RelationGetRelid(idxRel),
constraintOid,
+ -1,
false, false, false, false, false);
index_close(idxRel, AccessShareLock);
@@ -8640,6 +8641,7 @@ ATExecAddIndex(AlteredTableInfo *tab, Relation rel,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ -1,
true, /* is_alter_table */
check_rights,
false, /* check_not_in_use - we did it already */
@@ -18105,7 +18107,7 @@ AttachPartitionEnsureIndexes(Relation rel, Relation attachrel)
&conOid);
DefineIndex(RelationGetRelid(attachrel), stmt, InvalidOid,
RelationGetRelid(idxRel),
- conOid,
+ conOid, -1,
true, false, false, false, false);
}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index c7d9d96b45d..c8317822a64 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1462,6 +1462,7 @@ ProcessUtilitySlow(ParseState *pstate,
Oid relid;
LOCKMODE lockmode;
bool is_alter_table;
+ int total_leaves = 0;
if (stmt->concurrent)
PreventInTransactionBlock(isTopLevel,
@@ -1502,7 +1503,8 @@ ProcessUtilitySlow(ParseState *pstate,
inheritors = find_all_inheritors(relid, lockmode, NULL);
foreach(lc, inheritors)
{
- char relkind = get_rel_relkind(lfirst_oid(lc));
+ Oid partrelid = lfirst_oid(lc);
+ char relkind = get_rel_relkind(partrelid);
if (relkind != RELKIND_RELATION &&
relkind != RELKIND_MATVIEW &&
@@ -1519,6 +1521,9 @@ ProcessUtilitySlow(ParseState *pstate,
stmt->relation->relname),
errdetail("Table \"%s\" contains partitions that are foreign tables.",
stmt->relation->relname)));
+
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
+ total_leaves++;
}
list_free(inheritors);
}
@@ -1545,6 +1550,7 @@ ProcessUtilitySlow(ParseState *pstate,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ total_leaves,
is_alter_table,
true, /* check_rights */
true, /* check_not_in_use */
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,34 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of the current backend.
+ *-----------
+ */
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 4f7f87fc62c..b774b3028ce 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -29,6 +29,7 @@ extern ObjectAddress DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_leaves,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.34.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-16 15:04 ` Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Ilya Gladyshev @ 2023-03-16 15:04 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
> 16 марта 2023 г., в 04:07, Justin Pryzby <[email protected]> написал(а):
>
> On Tue, Mar 14, 2023 at 06:58:14PM +0400, Ilya Gladyshev wrote:
>>> The only change from the current patch is (3). (1) still calls
>>> count_leaf_partitions(), but only once. I'd prefer that to rearranging
>>> the progress reporting to set the TOTAL in ProcessUtilitySlow().
>>
>> As for reusing TOTAL calculated outside of DefineIndex, as I can see, ProcessUtilitySlow is not the only call site for DefineIndex (although, I don’t know whether all of them need progress tracking), for instance, there is ALTER TABLE that calls DefineIndex to create index for constraints. So I feel like rearranging progress reporting will result in unnecessary code duplication in those call sites, so passing in an optional parameter seems to be easier here, if we are going to optimize it, after all. Especially if back-patching is a non-issue.
>
> Yeah. See attached. I don't like duplicating the loop. Is this really
> the right direction to go ?
>
> I haven't verified if the child tables are locked in all the paths which
> would call count_leaf_partitions(). But why is it important to lock
> them for this? If they weren't locked before, that'd be a pre-existing
> problem...
> <0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch>
I’m not sure what the general policy on locking is, but I have checked ALTER TABLE ADD INDEX, and the all the partitions seem to be locked on the first entry to DefineIndex there. All other call sites pass in the parentIndexId, which means the progress tracking machinery will not be initialized, so I think, we don’t need to do locking in count_leaf_partitions().
The approach in the patch looks good to me. Some nitpicks on the patch:
1. There’s an unnecessary second call to get_rel_relkind in ProcessUtilitySlow, we can just use what’s in the variable relkind.
2. We can also combine else and if to have one less nested level like that:
+ else if (!RELKIND_HAS_PARTITIONS(child_relkind))
3. There was a part of the comment saying "If the index was built by calling DefineIndex() recursively, the called function is responsible for updating the progress report for built indexes.", I think it is still useful to have it there.
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
@ 2023-03-21 18:43 ` Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-21 18:43 UTC (permalink / raw)
To: Ilya Gladyshev <[email protected]>; +Cc: Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Thu, Mar 16, 2023 at 07:04:16PM +0400, Ilya Gladyshev wrote:
> > 16 марта 2023 г., в 04:07, Justin Pryzby <[email protected]> написал(а):
> >
> > On Tue, Mar 14, 2023 at 06:58:14PM +0400, Ilya Gladyshev wrote:
> >>> The only change from the current patch is (3). (1) still calls
> >>> count_leaf_partitions(), but only once. I'd prefer that to rearranging
> >>> the progress reporting to set the TOTAL in ProcessUtilitySlow().
> >>
> >> As for reusing TOTAL calculated outside of DefineIndex, as I can see, ProcessUtilitySlow is not the only call site for DefineIndex (although, I don’t know whether all of them need progress tracking), for instance, there is ALTER TABLE that calls DefineIndex to create index for constraints. So I feel like rearranging progress reporting will result in unnecessary code duplication in those call sites, so passing in an optional parameter seems to be easier here, if we are going to optimize it, after all. Especially if back-patching is a non-issue.
> >
> > Yeah. See attached. I don't like duplicating the loop. Is this really
> > the right direction to go ?
> >
> > I haven't verified if the child tables are locked in all the paths which
> > would call count_leaf_partitions(). But why is it important to lock
> > them for this? If they weren't locked before, that'd be a pre-existing
> > problem...
> > <0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch>
>
> I’m not sure what the general policy on locking is, but I have checked ALTER TABLE ADD INDEX, and the all the partitions seem to be locked on the first entry to DefineIndex there. All other call sites pass in the parentIndexId, which means the progress tracking machinery will not be initialized, so I think, we don’t need to do locking in count_leaf_partitions().
> The approach in the patch looks good to me. Some nitpicks on the patch:
> 1. There’s an unnecessary second call to get_rel_relkind in ProcessUtilitySlow, we can just use what’s in the variable relkind.
> 2. We can also combine else and if to have one less nested level like that:
> + else if (!RELKIND_HAS_PARTITIONS(child_relkind))
>
> 3. There was a part of the comment saying "If the index was built by calling DefineIndex() recursively, the called function is responsible for updating the progress report for built indexes.", I think it is still useful to have it there.
Thanks, I addressed (1) and (3). (2) is deliberate, to allow a place to
put the comment which is not specific to the !HAS_PARTITIONS case.
--
Justin
Attachments:
[text/x-diff] 0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch (12.9K, ../../[email protected]/2-0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch)
download | inline diff:
From 90300bc4ca08088de3a0015ff5c1a251537feacc Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <[email protected]>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH 1/3] fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, but doesn't count intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, unless the attached index is
partitioned, in which case progress report is not updated.
Author: Ilya Gladyshev
Reviewed-By: Justin Pryzby, Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 ++-
src/backend/bootstrap/bootparse.y | 2 +
src/backend/commands/indexcmds.c | 74 +++++++++++++++++--
src/backend/commands/tablecmds.c | 4 +-
src/backend/tcop/utility.c | 8 +-
src/backend/utils/activity/backend_progress.c | 28 +++++++
src/include/commands/defrem.h | 1 +
src/include/utils/backend_progress.h | 1 +
8 files changed, 119 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 6249bb50d02..3f891b75541 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6901,7 +6901,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6912,7 +6915,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, but excludes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 86804bb598e..81a1b7bfec3 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -306,6 +306,7 @@ Boot_DeclareIndexStmt:
$4,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
@@ -358,6 +359,7 @@ Boot_DeclareUniqueIndexStmt:
$5,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 16ec0b114e6..2a87c851746 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -130,6 +130,31 @@ typedef struct ReindexErrorInfo
char relkind;
} ReindexErrorInfo;
+
+/*
+ * Count the number of direct and indirect leaf partitions. Note that this
+ * also excludes foreign tables. This should be consistent with the loop in
+ * ProcessUtilitySlow().
+ */
+static int
+count_leaf_partitions(Oid relid)
+{
+ int nleaves = 0;
+ List *childs = find_all_inheritors(relid, NoLock, NULL);
+ ListCell *lc;
+
+ foreach(lc, childs)
+ {
+ Oid partrelid = lfirst_oid(lc);
+
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
+ nleaves++;
+ }
+
+ list_free(childs);
+ return nleaves;
+}
+
/*
* CheckIndexCompatible
* Determine whether an existing index definition is compatible with a
@@ -518,6 +543,7 @@ WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
* case the caller had better have checked it earlier.
* 'skip_build': make the catalog entries but don't create the index files
* 'quiet': suppress the NOTICE chatter ordinarily provided for constraints.
+ * 'total_leaves': total number of leaf partitions (excluding foreign tables)
*
* Returns the object address of the created index.
*/
@@ -527,6 +553,7 @@ DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_leaves,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
@@ -1219,8 +1246,22 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ /*
+ * Set the total number of partitions at the start of the command;
+ * don't update it when being called recursively.
+ */
+ if (!OidIsValid(parentIndexId))
+ {
+ /*
+ * When called by ProcessUtilitySlow(), the number of leaves is
+ * passed in as an optimization.
+ */
+ if (total_leaves < 0)
+ total_leaves = count_leaf_partitions(relationId);
+
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_leaves);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1250,6 +1291,7 @@ DefineIndex(Oid relationId,
{
Oid childRelid = part_oids[i];
Relation childrel;
+ char child_relkind;
Oid child_save_userid;
int child_save_sec_context;
int child_save_nestlevel;
@@ -1259,6 +1301,7 @@ DefineIndex(Oid relationId,
bool found = false;
childrel = table_open(childRelid, lockmode);
+ child_relkind = RelationGetForm(childrel)->relkind;
GetUserIdAndSecContext(&child_save_userid,
&child_save_sec_context);
@@ -1426,14 +1469,29 @@ DefineIndex(Oid relationId,
InvalidOid, /* no predefined OID */
indexRelationId, /* this is our child */
createdConstraintId,
+ -1,
is_alter_table, check_rights, check_not_in_use,
skip_build, quiet);
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ /*
+ * If the index was built by calling DefineIndex()
+ * recursively, the called function is responsible for
+ * updating the progress report for built indexes.
+ *
+ * If a pre-existing index was attached, the progress
+ * report is updated here. But if the index is
+ * partitioned, we don't count its partitions, since that's
+ * expensive, and the ATTACH is fast anyway.
+ */
+
+ if (!RELKIND_HAS_PARTITIONS(child_relkind))
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1484,9 +1542,15 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, we're done. When called recursively
+ * for child tables, the done partition counter is incremented now,
+ * rather than in the caller.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3e2c5f797cd..df2fe468b65 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -1216,6 +1216,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
InvalidOid,
RelationGetRelid(idxRel),
constraintOid,
+ -1,
false, false, false, false, false);
index_close(idxRel, AccessShareLock);
@@ -8640,6 +8641,7 @@ ATExecAddIndex(AlteredTableInfo *tab, Relation rel,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ -1, /* total_leaves unknown */
true, /* is_alter_table */
check_rights,
false, /* check_not_in_use - we did it already */
@@ -18105,7 +18107,7 @@ AttachPartitionEnsureIndexes(Relation rel, Relation attachrel)
&conOid);
DefineIndex(RelationGetRelid(attachrel), stmt, InvalidOid,
RelationGetRelid(idxRel),
- conOid,
+ conOid, -1,
true, false, false, false, false);
}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index eada7353639..bbc3d7ada23 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1465,6 +1465,7 @@ ProcessUtilitySlow(ParseState *pstate,
Oid relid;
LOCKMODE lockmode;
bool is_alter_table;
+ int total_leaves = 0;
if (stmt->concurrent)
PreventInTransactionBlock(isTopLevel,
@@ -1505,7 +1506,8 @@ ProcessUtilitySlow(ParseState *pstate,
inheritors = find_all_inheritors(relid, lockmode, NULL);
foreach(lc, inheritors)
{
- char relkind = get_rel_relkind(lfirst_oid(lc));
+ Oid partrelid = lfirst_oid(lc);
+ char relkind = get_rel_relkind(partrelid);
if (relkind != RELKIND_RELATION &&
relkind != RELKIND_MATVIEW &&
@@ -1522,6 +1524,9 @@ ProcessUtilitySlow(ParseState *pstate,
stmt->relation->relname),
errdetail("Table \"%s\" contains partitions that are foreign tables.",
stmt->relation->relname)));
+
+ if (RELKIND_HAS_STORAGE(relkind))
+ total_leaves++;
}
list_free(inheritors);
}
@@ -1548,6 +1553,7 @@ ProcessUtilitySlow(ParseState *pstate,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ total_leaves,
is_alter_table,
true, /* check_rights */
true, /* check_not_in_use */
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,34 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of the current backend.
+ *-----------
+ */
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 4f7f87fc62c..b774b3028ce 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -29,6 +29,7 @@ extern ObjectAddress DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_leaves,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.34.1
[text/x-diff] 0002-assertions-for-progress-reporting.patch (5.1K, ../../[email protected]/3-0002-assertions-for-progress-reporting.patch)
download | inline diff:
From a3965c2025062c5dc7080c1025e530628a6180cf Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 1 Feb 2023 10:23:53 -0600
Subject: [PATCH 2/3] assertions for progress reporting
---
src/backend/commands/analyze.c | 10 ++-
src/backend/utils/activity/backend_progress.c | 84 +++++++++++++++++++
2 files changed, 92 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 65750958bb2..3bfc941aa2c 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1145,6 +1145,12 @@ acquire_sample_rows(Relation onerel, int elevel,
TableScanDesc scan;
BlockNumber nblocks;
BlockNumber blksdone = 0;
+ int64 progress_vals[2] = {0};
+ int const progress_inds[2] = {
+ PROGRESS_ANALYZE_BLOCKS_DONE,
+ PROGRESS_ANALYZE_BLOCKS_TOTAL
+ };
+
#ifdef USE_PREFETCH
int prefetch_maximum = 0; /* blocks to prefetch if enabled */
BlockSamplerData prefetch_bs;
@@ -1169,8 +1175,8 @@ acquire_sample_rows(Relation onerel, int elevel,
#endif
/* Report sampling block numbers */
- pgstat_progress_update_param(PROGRESS_ANALYZE_BLOCKS_TOTAL,
- nblocks);
+ progress_vals[1] = nblocks;
+ pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
/* Prepare for sampling rows */
reservoir_init_selection_state(&rstate, targrows);
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 2a9994b98fd..63f9482b175 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
@@ -37,6 +38,85 @@ pgstat_progress_start_command(ProgressCommandType cmdtype, Oid relid)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Check for consistency of progress data (current < total).
+ *
+ * Check during pgstat_progress_updates_*() rather than only from
+ * pgstat_progress_end_command() to catch issues with uninitialized/stale data
+ * from previous progress commands.
+ *
+ * If a command fails due to interrupt or error, the values may be less than
+ * the expected final value.
+ */
+static void
+pgstat_progress_asserts(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ volatile int64 *a = beentry->st_progress_param;
+
+ switch (beentry->st_progress_command)
+ {
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_VACUUMED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_NUM_DEAD_TUPLES] <=
+ a[PROGRESS_VACUUM_MAX_DEAD_TUPLES]);
+ break;
+
+ case PROGRESS_COMMAND_ANALYZE:
+ Assert(a[PROGRESS_ANALYZE_BLOCKS_DONE] <=
+ a[PROGRESS_ANALYZE_BLOCKS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_EXT_STATS_COMPUTED] <=
+ a[PROGRESS_ANALYZE_EXT_STATS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_CHILD_TABLES_DONE] <=
+ a[PROGRESS_ANALYZE_CHILD_TABLES_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ Assert(a[PROGRESS_CLUSTER_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_CLUSTER_TOTAL_HEAP_BLKS]);
+ /* FALLTHROUGH */
+ /* ..because CLUSTER rebuilds indexes */
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ Assert(a[PROGRESS_CREATEIDX_TUPLES_DONE] <=
+ a[PROGRESS_CREATEIDX_TUPLES_TOTAL]);
+ Assert(a[PROGRESS_CREATEIDX_PARTITIONS_DONE] <=
+ a[PROGRESS_CREATEIDX_PARTITIONS_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ /* progress reporting is optional for these */
+ if (a[PROGRESS_BASEBACKUP_BACKUP_TOTAL] >= 0)
+ {
+ Assert(a[PROGRESS_BASEBACKUP_BACKUP_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_BACKUP_TOTAL]);
+ Assert(a[PROGRESS_BASEBACKUP_TBLSPC_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_TBLSPC_TOTAL]);
+ }
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+#if 0
+ //This currently fails file_fdw tests, since pgstat_prorgress evidently fails
+ // to support simultaneous copy commands, as happens during JOIN.
+ /* bytes progress is not available in all cases */
+ if (a[PROGRESS_COPY_BYTES_TOTAL] > 0)
+ //Assert(a[PROGRESS_COPY_BYTES_PROCESSED] <= a[PROGRESS_COPY_BYTES_TOTAL]);
+ if (a[PROGRESS_COPY_BYTES_PROCESSED] > a[PROGRESS_COPY_BYTES_TOTAL])
+ elog(WARNING, "PROGRESS_COPY_BYTES_PROCESSED %ld %ld",
+ a[PROGRESS_COPY_BYTES_PROCESSED],
+ a[PROGRESS_COPY_BYTES_TOTAL]);
+#endif
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break; /* Do nothing */
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -56,6 +136,8 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
@@ -113,6 +195,8 @@ pgstat_progress_update_multi_param(int nparam, const int *index,
}
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
--
2.34.1
[text/x-diff] 0003-f-also-assert-that-progress-values-don-t-go-backward.patch (7.5K, ../../[email protected]/4-0003-f-also-assert-that-progress-values-don-t-go-backward.patch)
download | inline diff:
From 84388375d4808d44877fc9e26eacf9219158be2b Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 21 Jan 2023 21:41:04 -0600
Subject: [PATCH 3/3] f! also assert that progress values don't go backwards
and the total is constant
pgstat_progress_update_multi_param() is being abused as a way to update
a value which might otherwise violate the assertion.
See also:
https://www.postgresql.org/message-id/CA%2BTgmoYSvEP3weQaCPGf6%2BDXLy2__JbJUYtoXyWP%3DqHcyGbihA%40mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 37 +++++++++
src/backend/storage/lmgr/lmgr.c | 24 +++---
src/backend/utils/activity/backend_progress.c | 81 +++++++++++++++++++
3 files changed, 130 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f38..61cfbf6a17a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1042,6 +1042,13 @@ lazy_scan_heap(LVRelState *vacrel)
/* Forget the LP_DEAD items that we just vacuumed */
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -2199,6 +2206,13 @@ lazy_vacuum(LVRelState *vacrel)
{
Assert(!vacrel->do_index_cleanup);
vacrel->dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
return;
}
@@ -2301,6 +2315,13 @@ lazy_vacuum(LVRelState *vacrel)
* vacuum)
*/
vacrel->dead_items->num_items = 0;
+
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
}
/*
@@ -2414,12 +2435,23 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+#if 0
+ const int progress_inds[] = {
+ PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+ };
+ const int64 progress_vals[] = {
+ PROGRESS_VACUUM_PHASE_VACUUM_HEAP,
+ 0,
+ };
+#endif
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
Assert(vacrel->num_index_scans > 0);
/* Report that we are now vacuuming the heap */
+ //pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
@@ -3190,7 +3222,12 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
dead_items->max_items = max_items;
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
vacrel->dead_items = dead_items;
}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index ee9b89a6726..a3acdce5ff5 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,6 +912,13 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
int total = 0;
int done = 0;
+ const int progress_index[] = {
+ PROGRESS_WAITFOR_TOTAL,
+ PROGRESS_WAITFOR_DONE,
+ PROGRESS_WAITFOR_CURRENT_PID
+ };
+ const int64 progress_values[] = {0, 0, 0};
+
/* Done if no locks to wait for */
if (locktags == NIL)
return;
@@ -930,7 +937,10 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
}
if (progress)
+ {
+ pgstat_progress_update_multi_param(3, progress_index, progress_values);
pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+ }
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
@@ -960,19 +970,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
}
}
+
if (progress)
- {
- const int index[] = {
- PROGRESS_WAITFOR_TOTAL,
- PROGRESS_WAITFOR_DONE,
- PROGRESS_WAITFOR_CURRENT_PID
- };
- const int64 values[] = {
- 0, 0, 0
- };
-
- pgstat_progress_update_multi_param(3, index, values);
- }
+ pgstat_progress_update_multi_param(3, progress_index, progress_values);
list_free_deep(holders);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 63f9482b175..41ad6884521 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -117,6 +117,78 @@ pgstat_progress_asserts(void)
}
}
+/*
+ * Check that newval >= oldval, and that when total is not being set twice.
+ */
+static void
+pgstat_progress_assert_forward_progress(int command, int index,
+ int64 oldval, int64 newval)
+{
+ switch (command)
+ {
+ case PROGRESS_COMMAND_ANALYZE:
+ /*
+ * phase goes backwards for inheritance tables, which are sampled
+ * twice
+ */
+ if (index != PROGRESS_ANALYZE_CURRENT_CHILD_TABLE_RELID &&
+ index != PROGRESS_ANALYZE_PHASE)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_ANALYZE_BLOCKS_TOTAL ||
+ index == PROGRESS_ANALYZE_EXT_STATS_TOTAL ||
+ index == PROGRESS_ANALYZE_CHILD_TABLES_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ if (index != PROGRESS_CLUSTER_INDEX_RELID)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_CLUSTER_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ if (index != PROGRESS_CREATEIDX_INDEX_OID &&
+ index != PROGRESS_CREATEIDX_SUBPHASE &&
+ index != PROGRESS_WAITFOR_CURRENT_PID &&
+ index != PROGRESS_CREATEIDX_ACCESS_METHOD_OID)
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_CREATEIDX_TUPLES_TOTAL ||
+ index == PROGRESS_CREATEIDX_PARTITIONS_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL &&
+ oldval == 0 && newval == -1)
+ return; /* Do nothing: this is the initial "null"
+ * state before the size is estimated */
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL ||
+ index == PROGRESS_BASEBACKUP_TBLSPC_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_COPY_BYTES_TOTAL)
+ Assert(oldval == 0);
+ break;
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_VACUUM_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ if (index == PROGRESS_VACUUM_MAX_DEAD_TUPLES)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break;
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -133,6 +205,15 @@ pgstat_progress_update_param(int index, int64 val)
if (!beentry || !pgstat_track_activities)
return;
+ if (index != PROGRESS_SCAN_BLOCKS_DONE)
+ {
+ /* Check that progress does not go backwards */
+ int64 oldval = beentry->st_progress_param[index];
+
+ pgstat_progress_assert_forward_progress(beentry->st_progress_command,
+ index, oldval, val);
+ }
+
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
--
2.34.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-23 20:35 ` Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-23 20:35 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
So I'm still pretty desperately unhappy with count_leaf_partitions.
I don't like expending significant cost purely for progress tracking
purposes, I don't like the undocumented assumption that NoLock is
safe, and what's more, if it is safe then we've already traversed
the inheritance tree to lock everything so in principle we could
have the count already. However, it does seem like getting that
knowledge from point A to point B would be a mess in most places.
One thing we could do to reduce the cost (and improve the safety)
is to forget the idea of checking the relkinds and just set the
PARTITIONS_TOTAL count to list_length() of the find_all_inheritors
result. We've already agreed that it's okay if the PARTITIONS_DONE
count never reaches PARTITIONS_TOTAL, so this would just be taking
that idea further. (Or we could increment PARTITIONS_DONE for
non-leaf partitions when we visit them, thus making that TOTAL
more nearly correct.) Furthermore, as things stand it's not hard
for PARTITIONS_TOTAL to be zero --- there's at least one such case
in the regression tests --- and that seems just weird to me.
By the by, this is awful code:
+ if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
Consult the definition of RELKIND_HAS_STORAGE to see why.
But I want to get rid of that rather than fixing it.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-25 02:53 ` Justin Pryzby <[email protected]>
2023-03-25 15:55 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-25 02:53 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Thu, Mar 23, 2023 at 04:35:46PM -0400, Tom Lane wrote:
> So I'm still pretty desperately unhappy with count_leaf_partitions.
> I don't like expending significant cost purely for progress tracking
> purposes, I don't like the undocumented assumption that NoLock is
> safe, and what's more, if it is safe then we've already traversed
> the inheritance tree to lock everything so in principle we could
> have the count already. However, it does seem like getting that
> knowledge from point A to point B would be a mess in most places.
>
> One thing we could do to reduce the cost (and improve the safety)
> is to forget the idea of checking the relkinds and just set the
> PARTITIONS_TOTAL count to list_length() of the find_all_inheritors
> result.
Actually list_length() minus 1 ...
> We've already agreed that it's okay if the PARTITIONS_DONE
> count never reaches PARTITIONS_TOTAL, so this would just be taking
> that idea further. (Or we could increment PARTITIONS_DONE for
> non-leaf partitions when we visit them, thus making that TOTAL
> more nearly correct.)
Yes, I think that's actually more correct. If TOTAL is set without
regard to relkind, then DONE ought to be set the same way.
I updated the documentation to indicate that the counters include the
intermediate partitioned rels, but I wonder if it's better to say
nothing and leave that undefined.
> Furthermore, as things stand it's not hard
> for PARTITIONS_TOTAL to be zero --- there's at least one such case
> in the regression tests --- and that seems just weird to me.
I don't know why it'd seem weird. postgres doesn't create partitions
automatically, so by default there are none. If we create a table but
never load any data, it'll have no partitions. Also, the TOTAL=0 case
won't go away just because we start counting intermediate partitions.
> By the by, this is awful code:
>
> + if (RELKIND_HAS_STORAGE(get_rel_relkind(partrelid)))
>
> Consult the definition of RELKIND_HAS_STORAGE to see why.
> But I want to get rid of that rather than fixing it.
Good point, but I'd burden-shift the blame to RELKIND_HAS_STORAGE().
BTW, I promoted myself to a co-author of the patch. My interest here is
to resolve this hoping to allow the CIC patch to progress.
--
Justin
Attachments:
[text/x-diff] 0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch (11.9K, ../../[email protected]/2-0001-fix-CREATE-INDEX-progress-report-with-nested-partiti.patch)
download | inline diff:
From b48837a934ef1381d26ef62bbaf66f5e52b0cd6d Mon Sep 17 00:00:00 2001
From: Ilya Gladyshev <[email protected]>
Date: Tue, 31 Jan 2023 19:13:07 +0400
Subject: [PATCH 1/3] fix CREATE INDEX progress report with nested partitions
The progress reporting was added in v12 (ab0dfc961) but the original
patch didn't seem to consider the possibility of nested partitioning.
When called recursively, DefineIndex() would clobber the number of
completed partitions, and it was possible to end up with the TOTAL
counter greater than the DONE counter.
This clarifies/re-defines that the progress reporting counts both direct
and indirect children, as well as intermediate partitioned tables:
- The TOTAL counter is set once at the start of the command.
- For indexes which are newly-built, the recursively-called
DefineIndex() increments the DONE counter.
- For pre-existing indexes which are ATTACHed rather than built,
DefineIndex() increments the DONE counter, unless the attached index is
partitioned, in which case progress report is not updated.
Author: Ilya Gladyshev, Justin Pryzby
Reviewed-By: Tomas Vondra, Dean Rasheed, Alvaro Herrera, Matthias van de Meent, Tom Lane
Discussion: https://www.postgresql.org/message-id/flat/a15f904a70924ffa4ca25c3c744cff31e0e6e143.camel%40gmail.com
---
doc/src/sgml/monitoring.sgml | 10 +++-
src/backend/bootstrap/bootparse.y | 2 +
src/backend/commands/indexcmds.c | 60 +++++++++++++++++--
src/backend/commands/tablecmds.c | 4 +-
src/backend/tcop/utility.c | 6 +-
src/backend/utils/activity/backend_progress.c | 28 +++++++++
src/include/commands/defrem.h | 1 +
src/include/utils/backend_progress.h | 1 +
8 files changed, 103 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 7ab4424bf13..cf88dca53f3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6929,7 +6929,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the total number of partitions on which the index is to be created.
+ the total number of partitions on which the index is to be created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, and includes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
@@ -6940,7 +6943,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
</para>
<para>
When creating an index on a partitioned table, this column is set to
- the number of partitions on which the index has been created.
+ the number of partitions on which the index has been created or attached.
+ In the case of intermediate partitioned tables, this includes both
+ direct and indirect partitions, and includes the intermediate
+ partitioned tables themselves.
This field is <literal>0</literal> during a <literal>REINDEX</literal>.
</para></entry>
</row>
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index 86804bb598e..81a1b7bfec3 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -306,6 +306,7 @@ Boot_DeclareIndexStmt:
$4,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
@@ -358,6 +359,7 @@ Boot_DeclareUniqueIndexStmt:
$5,
InvalidOid,
InvalidOid,
+ -1,
false,
false,
false,
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index ff48f44c66f..edb65d4cacb 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -521,6 +521,7 @@ WaitForOlderSnapshots(TransactionId limitXmin, bool progress)
* case the caller had better have checked it earlier.
* 'skip_build': make the catalog entries but don't create the index files
* 'quiet': suppress the NOTICE chatter ordinarily provided for constraints.
+ * 'total_parts': total number of direct and indirect partitions
*
* Returns the object address of the created index.
*/
@@ -530,6 +531,7 @@ DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_parts,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
@@ -1225,8 +1227,30 @@ DefineIndex(Oid relationId,
Relation parentIndex;
TupleDesc parentDesc;
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
- nparts);
+ /*
+ * Set the total number of partitions at the start of the command;
+ * but don't update it when being called recursively.
+ */
+ if (!OidIsValid(parentIndexId))
+ {
+ /*
+ * When called by ProcessUtilitySlow(), the number of
+ * partitions is passed in as an optimization. This should
+ * count partitions the same way. Subtract one since
+ * find_all_inheritors() includes the rel itself.
+ */
+ if (total_parts < 0)
+ {
+ List *childs = find_all_inheritors(relationId,
+ NoLock, NULL);
+
+ total_parts = list_length(childs) - 1;
+ list_free(childs);
+ }
+
+ pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_TOTAL,
+ total_parts);
+ }
/* Make a local copy of partdesc->oids[], just for safety */
memcpy(part_oids, partdesc->oids, sizeof(Oid) * nparts);
@@ -1432,14 +1456,23 @@ DefineIndex(Oid relationId,
InvalidOid, /* no predefined OID */
indexRelationId, /* this is our child */
createdConstraintId,
+ -1,
is_alter_table, check_rights, check_not_in_use,
skip_build, quiet);
SetUserIdAndSecContext(child_save_userid,
child_save_sec_context);
}
+ else
+ {
+ /*
+ * If a pre-existing index was ATTACHed, the progress
+ * report is updated here. A partitioned index is
+ * likewise counted when attached, but not its partitions,
+ * since that's expensive, and ATTACH is fast anyway.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
+ }
- pgstat_progress_update_param(PROGRESS_CREATEIDX_PARTITIONS_DONE,
- i + 1);
free_attrmap(attmap);
}
@@ -1479,6 +1512,16 @@ DefineIndex(Oid relationId,
table_close(rel, NoLock);
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ {
+ /*
+ * Update progress for a partitioned index itself; the
+ * recursively-called function will have updated the counter for
+ * its child indexes.
+ */
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
+ }
+
return address;
}
@@ -1490,9 +1533,16 @@ DefineIndex(Oid relationId,
/* Close the heap and we're done, in the non-concurrent case */
table_close(rel, NoLock);
- /* If this is the top-level index, we're done. */
+ /*
+ * If this is the top-level index, the command is done. When called
+ * recursively for child tables, the done partition counter is
+ * incremented now, rather than in the caller, to provide fine-grained
+ * progress reporting in the case of intermediate partitioning.
+ */
if (!OidIsValid(parentIndexId))
pgstat_progress_end_command();
+ else
+ pgstat_progress_incr_param(PROGRESS_CREATEIDX_PARTITIONS_DONE, 1);
return address;
}
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3e2c5f797cd..cc74d703f91 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -1216,6 +1216,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
InvalidOid,
RelationGetRelid(idxRel),
constraintOid,
+ -1,
false, false, false, false, false);
index_close(idxRel, AccessShareLock);
@@ -8640,6 +8641,7 @@ ATExecAddIndex(AlteredTableInfo *tab, Relation rel,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ -1, /* total_parts unknown */
true, /* is_alter_table */
check_rights,
false, /* check_not_in_use - we did it already */
@@ -18105,7 +18107,7 @@ AttachPartitionEnsureIndexes(Relation rel, Relation attachrel)
&conOid);
DefineIndex(RelationGetRelid(attachrel), stmt, InvalidOid,
RelationGetRelid(idxRel),
- conOid,
+ conOid, -1,
true, false, false, false, false);
}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index eada7353639..2dc92c72cea 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -1465,6 +1465,7 @@ ProcessUtilitySlow(ParseState *pstate,
Oid relid;
LOCKMODE lockmode;
bool is_alter_table;
+ int nparts = 0;
if (stmt->concurrent)
PreventInTransactionBlock(isTopLevel,
@@ -1503,9 +1504,11 @@ ProcessUtilitySlow(ParseState *pstate,
List *inheritors = NIL;
inheritors = find_all_inheritors(relid, lockmode, NULL);
+ nparts = list_length(inheritors) - 1;
foreach(lc, inheritors)
{
- char relkind = get_rel_relkind(lfirst_oid(lc));
+ Oid partrelid = lfirst_oid(lc);
+ char relkind = get_rel_relkind(partrelid);
if (relkind != RELKIND_RELATION &&
relkind != RELKIND_MATVIEW &&
@@ -1548,6 +1551,7 @@ ProcessUtilitySlow(ParseState *pstate,
InvalidOid, /* no predefined OID */
InvalidOid, /* no parent index */
InvalidOid, /* no parent constraint */
+ nparts,
is_alter_table,
true, /* check_rights */
true, /* check_not_in_use */
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index d96af812b19..2a9994b98fd 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -58,6 +58,34 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*-----------
+ * pgstat_progress_incr_param() -
+ *
+ * Increment index'th member in st_progress_param[] of the current backend.
+ *-----------
+ */
+void
+pgstat_progress_incr_param(int index, int64 incr)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ int64 val;
+
+ Assert(index >= 0 && index < PGSTAT_NUM_PROGRESS_PARAM);
+
+ if (!beentry || !pgstat_track_activities)
+ return;
+
+ /*
+ * Because no other process should write to this backend's own status, we
+ * can read its value from shared memory without needing to loop to ensure
+ * its consistency.
+ */
+ val = beentry->st_progress_param[index];
+ val += incr;
+
+ pgstat_progress_update_param(index, val);
+}
+
/*-----------
* pgstat_progress_update_multi_param() -
*
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index 4f7f87fc62c..478203ed4c4 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -29,6 +29,7 @@ extern ObjectAddress DefineIndex(Oid relationId,
Oid indexRelationId,
Oid parentIndexId,
Oid parentConstraintId,
+ int total_parts,
bool is_alter_table,
bool check_rights,
bool check_not_in_use,
diff --git a/src/include/utils/backend_progress.h b/src/include/utils/backend_progress.h
index 005e5d75ab6..a84752ade99 100644
--- a/src/include/utils/backend_progress.h
+++ b/src/include/utils/backend_progress.h
@@ -36,6 +36,7 @@ typedef enum ProgressCommandType
extern void pgstat_progress_start_command(ProgressCommandType cmdtype,
Oid relid);
extern void pgstat_progress_update_param(int index, int64 val);
+extern void pgstat_progress_incr_param(int index, int64 incr);
extern void pgstat_progress_update_multi_param(int nparam, const int *index,
const int64 *val);
extern void pgstat_progress_end_command(void);
--
2.34.1
[text/x-diff] 0002-assertions-for-progress-reporting.patch (5.1K, ../../[email protected]/3-0002-assertions-for-progress-reporting.patch)
download | inline diff:
From b9bfc805aa17e07ab7a92b6478daa9ff09b2b8e6 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Wed, 1 Feb 2023 10:23:53 -0600
Subject: [PATCH 2/3] assertions for progress reporting
---
src/backend/commands/analyze.c | 10 ++-
src/backend/utils/activity/backend_progress.c | 84 +++++++++++++++++++
2 files changed, 92 insertions(+), 2 deletions(-)
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 65750958bb2..3bfc941aa2c 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -1145,6 +1145,12 @@ acquire_sample_rows(Relation onerel, int elevel,
TableScanDesc scan;
BlockNumber nblocks;
BlockNumber blksdone = 0;
+ int64 progress_vals[2] = {0};
+ int const progress_inds[2] = {
+ PROGRESS_ANALYZE_BLOCKS_DONE,
+ PROGRESS_ANALYZE_BLOCKS_TOTAL
+ };
+
#ifdef USE_PREFETCH
int prefetch_maximum = 0; /* blocks to prefetch if enabled */
BlockSamplerData prefetch_bs;
@@ -1169,8 +1175,8 @@ acquire_sample_rows(Relation onerel, int elevel,
#endif
/* Report sampling block numbers */
- pgstat_progress_update_param(PROGRESS_ANALYZE_BLOCKS_TOTAL,
- nblocks);
+ progress_vals[1] = nblocks;
+ pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
/* Prepare for sampling rows */
reservoir_init_selection_state(&rstate, targrows);
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 2a9994b98fd..63f9482b175 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -10,6 +10,7 @@
*/
#include "postgres.h"
+#include "commands/progress.h"
#include "port/atomics.h" /* for memory barriers */
#include "utils/backend_progress.h"
#include "utils/backend_status.h"
@@ -37,6 +38,85 @@ pgstat_progress_start_command(ProgressCommandType cmdtype, Oid relid)
PGSTAT_END_WRITE_ACTIVITY(beentry);
}
+/*
+ * Check for consistency of progress data (current < total).
+ *
+ * Check during pgstat_progress_updates_*() rather than only from
+ * pgstat_progress_end_command() to catch issues with uninitialized/stale data
+ * from previous progress commands.
+ *
+ * If a command fails due to interrupt or error, the values may be less than
+ * the expected final value.
+ */
+static void
+pgstat_progress_asserts(void)
+{
+ volatile PgBackendStatus *beentry = MyBEEntry;
+ volatile int64 *a = beentry->st_progress_param;
+
+ switch (beentry->st_progress_command)
+ {
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_HEAP_BLKS_VACUUMED] <=
+ a[PROGRESS_VACUUM_TOTAL_HEAP_BLKS]);
+ Assert(a[PROGRESS_VACUUM_NUM_DEAD_TUPLES] <=
+ a[PROGRESS_VACUUM_MAX_DEAD_TUPLES]);
+ break;
+
+ case PROGRESS_COMMAND_ANALYZE:
+ Assert(a[PROGRESS_ANALYZE_BLOCKS_DONE] <=
+ a[PROGRESS_ANALYZE_BLOCKS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_EXT_STATS_COMPUTED] <=
+ a[PROGRESS_ANALYZE_EXT_STATS_TOTAL]);
+ Assert(a[PROGRESS_ANALYZE_CHILD_TABLES_DONE] <=
+ a[PROGRESS_ANALYZE_CHILD_TABLES_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ Assert(a[PROGRESS_CLUSTER_HEAP_BLKS_SCANNED] <=
+ a[PROGRESS_CLUSTER_TOTAL_HEAP_BLKS]);
+ /* FALLTHROUGH */
+ /* ..because CLUSTER rebuilds indexes */
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ Assert(a[PROGRESS_CREATEIDX_TUPLES_DONE] <=
+ a[PROGRESS_CREATEIDX_TUPLES_TOTAL]);
+ Assert(a[PROGRESS_CREATEIDX_PARTITIONS_DONE] <=
+ a[PROGRESS_CREATEIDX_PARTITIONS_TOTAL]);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ /* progress reporting is optional for these */
+ if (a[PROGRESS_BASEBACKUP_BACKUP_TOTAL] >= 0)
+ {
+ Assert(a[PROGRESS_BASEBACKUP_BACKUP_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_BACKUP_TOTAL]);
+ Assert(a[PROGRESS_BASEBACKUP_TBLSPC_STREAMED] <=
+ a[PROGRESS_BASEBACKUP_TBLSPC_TOTAL]);
+ }
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+#if 0
+ //This currently fails file_fdw tests, since pgstat_prorgress evidently fails
+ // to support simultaneous copy commands, as happens during JOIN.
+ /* bytes progress is not available in all cases */
+ if (a[PROGRESS_COPY_BYTES_TOTAL] > 0)
+ //Assert(a[PROGRESS_COPY_BYTES_PROCESSED] <= a[PROGRESS_COPY_BYTES_TOTAL]);
+ if (a[PROGRESS_COPY_BYTES_PROCESSED] > a[PROGRESS_COPY_BYTES_TOTAL])
+ elog(WARNING, "PROGRESS_COPY_BYTES_PROCESSED %ld %ld",
+ a[PROGRESS_COPY_BYTES_PROCESSED],
+ a[PROGRESS_COPY_BYTES_TOTAL]);
+#endif
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break; /* Do nothing */
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -56,6 +136,8 @@ pgstat_progress_update_param(int index, int64 val)
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
@@ -113,6 +195,8 @@ pgstat_progress_update_multi_param(int nparam, const int *index,
}
PGSTAT_END_WRITE_ACTIVITY(beentry);
+
+ pgstat_progress_asserts();
}
/*-----------
--
2.34.1
[text/x-diff] 0003-f-also-assert-that-progress-values-don-t-go-backward.patch (7.5K, ../../[email protected]/4-0003-f-also-assert-that-progress-values-don-t-go-backward.patch)
download | inline diff:
From 68403c5123a0b475affbf89626f7364adce1df24 Mon Sep 17 00:00:00 2001
From: Justin Pryzby <[email protected]>
Date: Sat, 21 Jan 2023 21:41:04 -0600
Subject: [PATCH 3/3] f! also assert that progress values don't go backwards
and the total is constant
pgstat_progress_update_multi_param() is being abused as a way to update
a value which might otherwise violate the assertion.
See also:
https://www.postgresql.org/message-id/CA%2BTgmoYSvEP3weQaCPGf6%2BDXLy2__JbJUYtoXyWP%3DqHcyGbihA%40mail.gmail.com
---
src/backend/access/heap/vacuumlazy.c | 37 +++++++++
src/backend/storage/lmgr/lmgr.c | 24 +++---
src/backend/utils/activity/backend_progress.c | 81 +++++++++++++++++++
3 files changed, 130 insertions(+), 12 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f38..61cfbf6a17a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1042,6 +1042,13 @@ lazy_scan_heap(LVRelState *vacrel)
/* Forget the LP_DEAD items that we just vacuumed */
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -2199,6 +2206,13 @@ lazy_vacuum(LVRelState *vacrel)
{
Assert(!vacrel->do_index_cleanup);
vacrel->dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
+
return;
}
@@ -2301,6 +2315,13 @@ lazy_vacuum(LVRelState *vacrel)
* vacuum)
*/
vacrel->dead_items->num_items = 0;
+
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
}
/*
@@ -2414,12 +2435,23 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+#if 0
+ const int progress_inds[] = {
+ PROGRESS_VACUUM_PHASE,
+ PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+ };
+ const int64 progress_vals[] = {
+ PROGRESS_VACUUM_PHASE_VACUUM_HEAP,
+ 0,
+ };
+#endif
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
Assert(vacrel->num_index_scans > 0);
/* Report that we are now vacuuming the heap */
+ //pgstat_progress_update_multi_param(2, progress_inds, progress_vals);
pgstat_progress_update_param(PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_PHASE_VACUUM_HEAP);
@@ -3190,7 +3222,12 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
dead_items->max_items = max_items;
dead_items->num_items = 0;
+ {
+ const int progress_inds[] = {PROGRESS_VACUUM_NUM_DEAD_TUPLES};
+ const int64 progress_vals[] = {0};
+ pgstat_progress_update_multi_param(1, progress_inds, progress_vals);
+ }
vacrel->dead_items = dead_items;
}
diff --git a/src/backend/storage/lmgr/lmgr.c b/src/backend/storage/lmgr/lmgr.c
index ee9b89a6726..a3acdce5ff5 100644
--- a/src/backend/storage/lmgr/lmgr.c
+++ b/src/backend/storage/lmgr/lmgr.c
@@ -912,6 +912,13 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
int total = 0;
int done = 0;
+ const int progress_index[] = {
+ PROGRESS_WAITFOR_TOTAL,
+ PROGRESS_WAITFOR_DONE,
+ PROGRESS_WAITFOR_CURRENT_PID
+ };
+ const int64 progress_values[] = {0, 0, 0};
+
/* Done if no locks to wait for */
if (locktags == NIL)
return;
@@ -930,7 +937,10 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
}
if (progress)
+ {
+ pgstat_progress_update_multi_param(3, progress_index, progress_values);
pgstat_progress_update_param(PROGRESS_WAITFOR_TOTAL, total);
+ }
/*
* Note: GetLockConflicts() never reports our own xid, hence we need not
@@ -960,19 +970,9 @@ WaitForLockersMultiple(List *locktags, LOCKMODE lockmode, bool progress)
pgstat_progress_update_param(PROGRESS_WAITFOR_DONE, ++done);
}
}
+
if (progress)
- {
- const int index[] = {
- PROGRESS_WAITFOR_TOTAL,
- PROGRESS_WAITFOR_DONE,
- PROGRESS_WAITFOR_CURRENT_PID
- };
- const int64 values[] = {
- 0, 0, 0
- };
-
- pgstat_progress_update_multi_param(3, index, values);
- }
+ pgstat_progress_update_multi_param(3, progress_index, progress_values);
list_free_deep(holders);
}
diff --git a/src/backend/utils/activity/backend_progress.c b/src/backend/utils/activity/backend_progress.c
index 63f9482b175..41ad6884521 100644
--- a/src/backend/utils/activity/backend_progress.c
+++ b/src/backend/utils/activity/backend_progress.c
@@ -117,6 +117,78 @@ pgstat_progress_asserts(void)
}
}
+/*
+ * Check that newval >= oldval, and that when total is not being set twice.
+ */
+static void
+pgstat_progress_assert_forward_progress(int command, int index,
+ int64 oldval, int64 newval)
+{
+ switch (command)
+ {
+ case PROGRESS_COMMAND_ANALYZE:
+ /*
+ * phase goes backwards for inheritance tables, which are sampled
+ * twice
+ */
+ if (index != PROGRESS_ANALYZE_CURRENT_CHILD_TABLE_RELID &&
+ index != PROGRESS_ANALYZE_PHASE)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_ANALYZE_BLOCKS_TOTAL ||
+ index == PROGRESS_ANALYZE_EXT_STATS_TOTAL ||
+ index == PROGRESS_ANALYZE_CHILD_TABLES_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CLUSTER:
+ if (index != PROGRESS_CLUSTER_INDEX_RELID)
+ Assert(newval >= oldval);
+ if (index == PROGRESS_CLUSTER_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_CREATE_INDEX:
+ if (index != PROGRESS_CREATEIDX_INDEX_OID &&
+ index != PROGRESS_CREATEIDX_SUBPHASE &&
+ index != PROGRESS_WAITFOR_CURRENT_PID &&
+ index != PROGRESS_CREATEIDX_ACCESS_METHOD_OID)
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_CREATEIDX_TUPLES_TOTAL ||
+ index == PROGRESS_CREATEIDX_PARTITIONS_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_BASEBACKUP:
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL &&
+ oldval == 0 && newval == -1)
+ return; /* Do nothing: this is the initial "null"
+ * state before the size is estimated */
+ Assert(newval >= oldval);
+
+ if (index == PROGRESS_BASEBACKUP_BACKUP_TOTAL ||
+ index == PROGRESS_BASEBACKUP_TBLSPC_TOTAL)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_COPY:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_COPY_BYTES_TOTAL)
+ Assert(oldval == 0);
+ break;
+ case PROGRESS_COMMAND_VACUUM:
+ Assert(newval >= oldval);
+ if (index == PROGRESS_VACUUM_TOTAL_HEAP_BLKS)
+ Assert(oldval == 0);
+ if (index == PROGRESS_VACUUM_MAX_DEAD_TUPLES)
+ Assert(oldval == 0);
+ break;
+
+ case PROGRESS_COMMAND_INVALID:
+ break;
+ }
+}
+
/*-----------
* pgstat_progress_update_param() -
*
@@ -133,6 +205,15 @@ pgstat_progress_update_param(int index, int64 val)
if (!beentry || !pgstat_track_activities)
return;
+ if (index != PROGRESS_SCAN_BLOCKS_DONE)
+ {
+ /* Check that progress does not go backwards */
+ int64 oldval = beentry->st_progress_param[index];
+
+ pgstat_progress_assert_forward_progress(beentry->st_progress_command,
+ index, oldval, val);
+ }
+
PGSTAT_BEGIN_WRITE_ACTIVITY(beentry);
beentry->st_progress_param[index] = val;
PGSTAT_END_WRITE_ACTIVITY(beentry);
--
2.34.1
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-25 15:55 ` Tom Lane <[email protected]>
2023-03-25 16:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-25 15:55 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> On Thu, Mar 23, 2023 at 04:35:46PM -0400, Tom Lane wrote:
>> Furthermore, as things stand it's not hard
>> for PARTITIONS_TOTAL to be zero --- there's at least one such case
>> in the regression tests --- and that seems just weird to me.
> I don't know why it'd seem weird. postgres doesn't create partitions
> automatically, so by default there are none. If we create a table but
> never load any data, it'll have no partitions.
My problem with it is that it's not clear how to tell "no partitioned
index creation in progress" from "partitioned index creation in progress,
but total = 0". Maybe there's some out-of-band way to tell that in the
stats reporting system, but still it's a weird corner case.
> Also, the TOTAL=0 case
> won't go away just because we start counting intermediate partitions.
That's why I wanted list_length() not list_length() - 1. We are
doing *something* at the top partitioned table, it just doesn't
involve a table scan, so I don't find this totally unreasonable.
If you agree we are doing work at intermediate partitioned tables,
how are we not doing work at the top one?
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 15:55 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-25 16:36 ` Justin Pryzby <[email protected]>
2023-03-25 17:11 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Justin Pryzby @ 2023-03-25 16:36 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Sat, Mar 25, 2023 at 11:55:13AM -0400, Tom Lane wrote:
> Justin Pryzby <[email protected]> writes:
> > On Thu, Mar 23, 2023 at 04:35:46PM -0400, Tom Lane wrote:
> >> Furthermore, as things stand it's not hard
> >> for PARTITIONS_TOTAL to be zero --- there's at least one such case
> >> in the regression tests --- and that seems just weird to me.
>
> > I don't know why it'd seem weird. postgres doesn't create partitions
> > automatically, so by default there are none. If we create a table but
> > never load any data, it'll have no partitions.
>
> My problem with it is that it's not clear how to tell "no partitioned
> index creation in progress" from "partitioned index creation in progress,
> but total = 0". Maybe there's some out-of-band way to tell that in the
> stats reporting system, but still it's a weird corner case.
>
> > Also, the TOTAL=0 case
> > won't go away just because we start counting intermediate partitions.
>
> That's why I wanted list_length() not list_length() - 1. We are
> doing *something* at the top partitioned table, it just doesn't
> involve a table scan, so I don't find this totally unreasonable.
> If you agree we are doing work at intermediate partitioned tables,
> how are we not doing work at the top one?
What you're proposing would redefine the meaning of
PARTITIONS_DONE/TOTAL, even in the absence of intermediate partitioned
tables. Which might be okay, but the scope of this thread/patch was to
fix the behavior involving intermediate partitioned tables.
It's somewhat weird to me that find_all_inheritors(rel) returns the rel
itself. But it's an internal function, and evidently that's what's
needed/desirable to do, so that's fine.
However, "PARTITIONS_TOTAL" has a certain user-facing definition, and
"Number of partitions" is easier to explain than "Number of partitions
plus the rel itself", and IMO an easier definition is a better one.
Your complaint seems similar to something I've said a few times before:
it's weird to expose macroscopic progress reporting of partitioned
tables in the same view and in the same *row* as microscopic progress of
its partitions. But changing that is a job for another patch. I won't
be opposed to it if someone were to propose a patch to remove
partitions_{done,total}. See also:
https://www.postgresql.org/message-id/flat/YCy5ZMt8xAyoOMmv%40paquier.xyz#b20d1be226a93dacd3fd40b402...
--
Justin
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 15:55 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 16:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
@ 2023-03-25 17:11 ` Tom Lane <[email protected]>
2023-03-25 19:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-25 17:11 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
Justin Pryzby <[email protected]> writes:
> On Sat, Mar 25, 2023 at 11:55:13AM -0400, Tom Lane wrote:
>> That's why I wanted list_length() not list_length() - 1. We are
>> doing *something* at the top partitioned table, it just doesn't
>> involve a table scan, so I don't find this totally unreasonable.
>> If you agree we are doing work at intermediate partitioned tables,
>> how are we not doing work at the top one?
> What you're proposing would redefine the meaning of
> PARTITIONS_DONE/TOTAL, even in the absence of intermediate partitioned
> tables. Which might be okay, but the scope of this thread/patch was to
> fix the behavior involving intermediate partitioned tables.
I'm a little skeptical of that argument, because this patch is already
redefining the meaning of PARTITIONS_TOTAL. The fact that the existing
documentation is vague enough to be read either way doesn't make it not
a change.
Still, in the interests of getting something done I'll drop the issue.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 15:55 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 16:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 17:11 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-25 19:43 ` Tom Lane <[email protected]>
2023-03-26 14:08 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
0 siblings, 1 reply; 44+ messages in thread
From: Tom Lane @ 2023-03-25 19:43 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
I pushed 0001 with some cosmetic changes (for instance, trying to
make the style of the doc entries for partitions_total/partitions_done
match the rest of their table).
I'm not touching 0002 or 0003, because I think they're fundamentally
a bad idea. Progress reporting is inherently inexact, because it's
so hard to predict the amount of work to be done in advance -- have
you ever seen a system anywhere whose progress bars reliably advance
at a uniform rate? I think adding assertions that the estimates are
error-free is just going to cause headaches. As an example, I added
a comment pointing out that the current fix won't crash and burn if
the caller failed to lock all the child tables in advance: the
find_all_inheritors call should be safe anyway, so the worst consequence
would be an imprecise partitions_total estimate. But that argument
falls down if we're going to add assertions that partitions_total
isn't in error.
regards, tom lane
^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: Progress report of CREATE INDEX for nested partitioned tables
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 15:55 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 16:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 17:11 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 19:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
@ 2023-03-26 14:08 ` Justin Pryzby <[email protected]>
0 siblings, 0 replies; 44+ messages in thread
From: Justin Pryzby @ 2023-03-26 14:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Ilya Gladyshev <[email protected]>; Matthias van de Meent <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; [email protected]
On Sat, Mar 25, 2023 at 03:43:32PM -0400, Tom Lane wrote:
> I pushed 0001 with some cosmetic changes (for instance, trying to
> make the style of the doc entries for partitions_total/partitions_done
> match the rest of their table).
Thanks.
> I'm not touching 0002 or 0003, because I think they're fundamentally
> a bad idea. Progress reporting is inherently inexact, because it's
Nobody could disagree that it's inexact. The assertions are for minimal
sanity tests and consistency. Like if "total" is set multiple times (as
in this patch), or if a progress value goes backwards. Anyway the
assertions exposed two other issues that would need to be fixed before
the assertions themselves could be proposed.
--
Justin
^ permalink raw reply [nested|flat] 44+ messages in thread
end of thread, other threads:[~2023-03-26 14:08 UTC | newest]
Thread overview: 44+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-04-19 05:42 Windows default locale vs initdb Thomas Munro <[email protected]>
2021-04-19 08:52 ` Pavel Stehule <[email protected]>
2021-04-19 10:52 ` Andrew Dunstan <[email protected]>
2021-04-19 10:57 ` Pavel Stehule <[email protected]>
2021-04-19 14:26 ` Dave Page <[email protected]>
2021-04-19 16:28 ` Andrew Dunstan <[email protected]>
2021-04-19 18:16 ` Peter Eisentraut <[email protected]>
2021-05-16 04:29 ` Noah Misch <[email protected]>
2021-12-15 10:32 ` Juan José Santamaría Flecha <[email protected]>
2022-07-18 22:58 ` Thomas Munro <[email protected]>
2022-07-19 02:46 ` Thomas Munro <[email protected]>
2022-07-20 10:26 ` Juan José Santamaría Flecha <[email protected]>
2022-07-20 11:44 ` Thomas Munro <[email protected]>
2022-07-22 11:58 ` Juan José Santamaría Flecha <[email protected]>
2022-07-29 03:33 ` Thomas Munro <[email protected]>
2022-12-23 04:36 ` Thomas Munro <[email protected]>
2022-07-20 08:34 ` Juan José Santamaría Flecha <[email protected]>
2022-07-31 00:25 [PATCH 4/8] cirrus/002_pg_upgrade: exercise --link and --clone Justin Pryzby <[email protected]>
2023-02-01 17:51 Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-02-01 18:24 ` Re: Progress report of CREATE INDEX for nested partitioned tables Matthias van de Meent <[email protected]>
2023-02-02 15:18 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-08 22:40 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-02-17 00:26 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-10 20:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 20:09 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 20:14 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-12 22:06 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-12 22:25 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 00:15 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-13 13:50 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-13 14:42 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:34 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-14 14:46 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-14 14:58 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-16 00:07 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-16 15:04 ` Re: Progress report of CREATE INDEX for nested partitioned tables Ilya Gladyshev <[email protected]>
2023-03-21 18:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-23 20:35 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 02:53 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 15:55 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 16:36 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
2023-03-25 17:11 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-25 19:43 ` Re: Progress report of CREATE INDEX for nested partitioned tables Tom Lane <[email protected]>
2023-03-26 14:08 ` Re: Progress report of CREATE INDEX for nested partitioned tables Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox