Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sVrlx-00C3zm-62 for pgsql-general@arkaria.postgresql.org; Mon, 22 Jul 2024 12:04:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sVrlv-000bnv-5Z for pgsql-general@arkaria.postgresql.org; Mon, 22 Jul 2024 12:04:07 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sVrk1-000X0r-BN for pgsql-general@lists.postgresql.org; Mon, 22 Jul 2024 12:02:10 +0000 Received: from mail-yb1-xb31.google.com ([2607:f8b0:4864:20::b31]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1sVrjy-000quD-Ed for pgsql-general@lists.postgresql.org; Mon, 22 Jul 2024 12:02:08 +0000 Received: by mail-yb1-xb31.google.com with SMTP id 3f1490d57ef6-e0875f1e9edso1705801276.1 for ; Mon, 22 Jul 2024 05:02:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; t=1721649725; x=1722254525; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=O9MmkJHsGO8DHqbnMwrhaBaXrI5AIzVzlUvknOKcb2g=; b=XBydPJqjRG0DrXWP3RduqDquPtil/61LLaQo3yrBSyv3sWOKe1YV68Fz2PDh6LHSI6 HIkXaKdQeCW4D34kQGh92B1xYbZ2LiNvwRXB9OHBHZfNYuU+kIBrwEimoVeRJJr5OkhY EQkG42PY+vHzQc0bveya7bQjacJjWBM6fY+Uch+Z5BcRcDlqCPbKyWxN7UQO1lVkQAJ9 Rxfl7utHZu3LNDDA/1WqTDbeLp3VNzKVyDkwmvbt19TEp2UWLSRtNar6HWrOcl1IL58b K939RQEcK8nrvD1MRpyNp2Nz7mMPcPkhDoOsuN6rBZ7omBM4Id5YIKwLFt/ilLPefn8h uC2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721649725; x=1722254525; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=O9MmkJHsGO8DHqbnMwrhaBaXrI5AIzVzlUvknOKcb2g=; b=HYbXhcoWKosJdusSwGX2XQXENtwsJcikmY9Lko4Tb0bqSTwYJTRe3Oy9PtBbZ5ntjJ nDyeU85Y02rBT4p0nI+ikDgY0CpZ1EPKeoWWDqtIdxCzvtH+DwvcAhcOIKXq2c4vyrg4 r13dBRGfoSZUJ3cHMstAY8quePbZgRJDo9i8FyHW4gzfze9APT4JmBZAhgr96s9E07PK jniIVPYTmnJ9VStSsUaL44koeZ8YBtpaq0dhqb+CWm7ziOlbp9y5id3K7AfeT0v5m1FW Wklop5lbpiB8ZIaNAzfzsK/VU9TsL5jm2jiy//Jgb2M16LoN+jIjbJn6zfFROcVf97Pq JrWg== X-Forwarded-Encrypted: i=1; AJvYcCVhCh4KNxETNjKN0lNq5+nmOJNEJrHZk/SEHrAWlq52yxDWd9xAVPQscM1MTld4piIEnWHEnxaTUQCCD0RYuh+feBljssNKyCee3Hka5Q+BYsa5 X-Gm-Message-State: AOJu0YyNYzeqMKaDAaB/ffyNIZbS9VdfzJoWR8OSpoENEnse1NJ82RHh uiCTSnMKinxm3bEAfx5uzaOmWB26eV1AjETJ/+o5eo6HpzDce0jeVZU5yGdL4qxP2dM2LWTeH6c KElloAcBJJ1Qs0PSwOP/qP3cLjeR8I2X91/7K X-Google-Smtp-Source: AGHT+IFJqSQTlOH+d7dlW35Ry4JB8NuCY6CQjDu7UizBsqF4MAb2JdpdaSDxZjQbbyRmgKwbTCHkeWQlZdcLyV1KmZw= X-Received: by 2002:a05:6902:f87:b0:e03:4e1d:77ed with SMTP id 3f1490d57ef6-e087029c418mr8876517276.24.1721649724811; Mon, 22 Jul 2024 05:02:04 -0700 (PDT) MIME-Version: 1.0 References: <80c9b0ea-c874-40ad-a006-fb1eb37464c2@aklaver.com> <44b44ece-dce6-4b4f-b751-8787a5a071e0@aklaver.com> In-Reply-To: From: Sandeep Thakkar Date: Mon, 22 Jul 2024 17:31:48 +0530 Message-ID: Subject: Re: Windows installation problem at post-install step To: Thomas Munro Cc: =?UTF-8?B?RXJ0YW4gS8O8w6fDvGtvZ2x1?= , Adrian Klaver , pgsql-general@lists.postgresql.org, Dave Page Content-Type: multipart/alternative; boundary="000000000000468fb4061dd4d0e1" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000468fb4061dd4d0e1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jul 22, 2024 at 5:21=E2=80=AFPM Sandeep Thakkar < sandeep.thakkar@enterprisedb.com> wrote: > Hi, > > EDB's windows installer gets the locales on the system using the > https://github.com/EnterpriseDB/edb-installers/blob/REL-16/server/scripts= /windows/getlocales/getlocales.cpp and > then substitute some patterns ( > https://github.com/EnterpriseDB/edb-installers/blob/REL-16/server/pgserve= r.xml.in#L2850) > I'm not sure why we do that but that is the old code and probably @Dave > Page may know but I'm not sure if that > piece of code is responsible for this change in encoding in this case. > > When I checked the installation log shared by Ertan, I do see that the > locale passed to initcluster script is the same as returned by the > getlocales executable. > > Executing C:\Windows\System32\cscript //NoLogo "C:\Program > Files\PostgreSQL\16/installer/server/initcluster.vbs" "NT > AUTHORITY\NetworkService" "postgres" "****" > "C:\Users\User1\AppData\Local\Temp/postgresql_installer_cd79fad8b7" > "C:\Program Files\PostgreSQL\16" "C:\DATA_PG16" 5432 "Turkish,T=C3=BCrkiy= e" 0 > > Apology about the top posting. Please ignore this thread. I've replied to another thread. > On Mon, Jul 22, 2024 at 6:43=E2=80=AFAM Thomas Munro > wrote: > >> On Mon, Jul 22, 2024 at 11:58=E2=80=AFAM Ertan K=C3=BC=C3=A7=C3=BCkoglu >> wrote: >> > Thomas Munro , 21 Tem 2024 Paz, 23:27 >> tarihinde =C5=9Funu yazd=C4=B1: >> >> 2. Some existing database clusters which had been installed with the >> >> name "Turkish_Turkey.1254" became unstartable when the OS upgrade >> >> renamed that locale to "Turkish_T=C3=BCrkiye.1254". I'm trying to pr= ovide >> >> a pathway[2] to fix such systems in core PostgreSQL in the next minor >> >> release. Everyone affected probably already found another way but at >> >> least next time a country is renamed this might help with the next >> >> point too. >> > >> > I was also hit by that OS update. >> > There is a Microsoft tool for creating a locale installer >> > https://www.microsoft.com/en-us/download/details.aspx?id=3D41158 >> > Using that tool and adding a second locale Turkish_Turkey.1254 (name >> before Microsoft update) in the OS can fix your broken PostgreSQL. >> > I believe most people simply choose this path. >> > There are also several blogs/articles written in Turkish about the >> problem. >> >> If that's easy and good enough then maybe I should abandon that >> on-the-fly renaming patch and we should just do a little documentation >> note... >> >> >> 3. I'd also like to teach initdb to use BCP47 names like "tr-TR" >> >> instead of those names by default (ie if you don't specify a locale >> >> name explicitly), and have proposed that before[3] but it hasn't gone >> >> in due to lack of testing/reviews from Windows users. It seems like >> >> that doesn't matter much in practice to all the people using the >> >> popular EDB installer, since it apparently takes control of picking >> >> the locale and explicitly passes it in (and screws up the encoding as >> >> we have now learned). >> > >> > If I am not mistaken BCP47 names are already used in Linux systems. >> > Using them would make PostgreSQL use the same locale names across Linu= x >> and Windows systems. >> >> Not exactly. POSIX systems use >> [language[_territory][.codeset][@modifier]], but POSIX doesn't say >> what any of those components are[1] (are they ISO country codes? >> English words? Hieroglyphs?), so, curiously, those Windows names like >> "English_United States.1252" are probably POSIX-conforming. Every >> real POSIX system of course uses ISO language and country codes these >> days (though I still recall other names being used years ago), so they >> look similar to the simpler kinds of BCP47 tags, which are just >> language-country with the same ISO codes but a different separator. >> They diverge further once you get into the finer points with more >> components. Incidentally that lack of standardisation is the reason >> you can't say that the glibc ".utf8" ending is "wrong", even though it >> is obviously stupid :-p (all systems I know accept .UTF-8, 'cause >> that's what Ken Thompson, Rob Pike and the Unicode standard called >> it). I suspect that Windows accepts the POSIX style en_US too, but >> it's not what the manual tells you to use. >> >> But really we shouldn't have to know or care how locales are named; we >> should get the names from the OS in the first place, and then we >> should remember them and give them back to the OS at the right times. >> The two problems here is that Windows has two kinds, one unstable over >> time and with illegal (for us) characters in the name, and one stable; >> we need to find all the places where the old unstable ones can get >> into our system, and block them off. I'm aware of two places now: the >> EDB installer, and initdb's default for people who run it on the >> command line with giving an explicit name. >> >> > I can help with the testing part. Let me know the details, please. >> >> Thanks! I will rebase that patch, and CC you on the thread. >> >> [1] >> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html >> > > > -- > Sandeep Thakkar > > > --=20 Sandeep Thakkar --000000000000468fb4061dd4d0e1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Mon, Jul 22, 2024 at 5:21=E2= =80=AFPM Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> wrote:
Hi,

EDB's windows installer gets the locales on the sys= tem using the=C2=A0https://github.com/EnterpriseDB/edb-installers/blob/REL-16/server/sc= ripts/windows/getlocales/getlocales.cpp=C2=A0and then substitute=C2=A0s= ome patterns (https://git= hub.com/EnterpriseDB/edb-installers/blob/REL-16/server/pgserver.xml.in#L285= 0) I'm not sure why we do that but that is the old code and probabl= y=C2=A0@Dave Pa= ge=C2=A0 may know but I'm not sure if that piece of code is respons= ible for this change in encoding=C2=A0in this case.

When I checked t= he installation log shared by Ertan, I do see that the locale passed=C2=A0t= o initcluster script is the same as returned by the getlocales executable.<= br>
Executing C:\Windows\System32\cscript //NoLogo "C:\Program File= s\PostgreSQL\16/installer/server/initcluster.vbs" "NT AUTHORITY\N= etworkService" "postgres" "****" "C:\Users\Us= er1\AppData\Local\Temp/postgresql_installer_cd79fad8b7" "C:\Progr= am Files\PostgreSQL\16" "C:\DATA_PG16" 5432 "Turkish,T= =C3=BCrkiye" 0

Apology about the= top posting. Please ignore this thread. I've replied to another thread= .
=C2=A0
<= div class=3D"gmail_quote">
On Mon, Jul= 22, 2024 at 6:43=E2=80=AFAM Thomas Munro <thomas.munro@gmail.com> wrote:
On Mon, Jul 22, 2024 a= t 11:58=E2=80=AFAM Ertan K=C3=BC=C3=A7=C3=BCkoglu
<ertan.ku= cukoglu@gmail.com> wrote:
> Thomas Munro <thomas.munro@gmail.com>, 21 Tem 2024 Paz, 23:27 tarihinde =C5= =9Funu yazd=C4=B1:
>> 2.=C2=A0 Some existing database clusters which had been installed = with the
>> name "Turkish_Turkey.1254" became unstartable when the O= S upgrade
>> renamed that locale to "Turkish_T=C3=BCrkiye.1254".=C2= =A0 I'm trying to provide
>> a pathway[2] to fix such systems in core PostgreSQL in the next mi= nor
>> release.=C2=A0 Everyone affected probably already found another wa= y but at
>> least next time a country is renamed this might help with the next=
>> point too.
>
> I was also hit by that OS update.
> There is a Microsoft tool for creating a locale installer
> https://www.microsoft.com/en-us= /download/details.aspx?id=3D41158
> Using that tool and adding a second locale Turkish_Turkey.1254 (name b= efore Microsoft update) in the OS can fix your broken PostgreSQL.
> I believe most people simply choose this path.
> There are also several blogs/articles written in Turkish about the pro= blem.

If that's easy and good enough then maybe I should abandon that
on-the-fly renaming patch and we should just do a little documentation
note...

>> 3.=C2=A0 I'd also like to teach initdb to use BCP47 names like= "tr-TR"
>> instead of those names by default (ie if you don't specify a l= ocale
>> name explicitly), and have proposed that before[3] but it hasn'= ;t gone
>> in due to lack of testing/reviews from Windows users.=C2=A0 It see= ms like
>> that doesn't matter much in practice to all the people using t= he
>> popular EDB installer, since it apparently takes control of pickin= g
>> the locale and explicitly passes it in (and screws up the encoding= as
>> we have now learned).
>
> If I am not mistaken BCP47 names are already used in Linux systems. > Using them would make PostgreSQL use the same locale names across Linu= x and Windows systems.

Not exactly.=C2=A0 POSIX systems use
[language[_territory][.codeset][@modifier]], but POSIX doesn't say
what any of those components are[1] (are they ISO country codes?
English words?=C2=A0 Hieroglyphs?), so, curiously, those Windows names like=
"English_United States.1252" are probably POSIX-conforming.=C2=A0= Every
real POSIX system of course uses ISO language and country codes these
days (though I still recall other names being used years ago), so they
look similar to the simpler kinds of BCP47 tags, which are just
language-country with the same ISO codes but a different separator.
They diverge further once you get into the finer points with more
components.=C2=A0 Incidentally that lack of standardisation is the reason you can't say that the glibc ".utf8" ending is "wrong&qu= ot;, even though it
is obviously stupid :-p (all systems I know accept .UTF-8, 'cause
that's what Ken Thompson, Rob Pike and the Unicode standard called
it).=C2=A0 I suspect that Windows accepts the POSIX style en_US too, but it's not what the manual tells you to use.

But really we shouldn't have to know or care how locales are named; we<= br> should get the names from the OS in the first place, and then we
should remember them and give them back to the OS at the right times.
The two problems here is that Windows has two kinds, one unstable over
time and with illegal (for us) characters in the name, and one stable;
we need to find all the places where the old unstable ones can get
into our system, and block them off.=C2=A0 I'm aware of two places now:= the
EDB installer, and initdb's default for people who run it on the
command line with giving an explicit name.

> I can help with the testing part. Let me know the details, please.

Thanks!=C2=A0 I will rebase that patch, and CC you on the thread.

[1] https://pubs.opengroup.o= rg/onlinepubs/9699919799/basedefs/V1_chap08.html


--
Sandeep Thakkar




--
Sandeep Thakkar


--000000000000468fb4061dd4d0e1--