pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPG v12.2 - Setting jit_above_cost is causing the server to crash
9+ messages / 6 participants
[nested] [flat]
* PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-24 06:46 Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
0 siblings, 2 replies; 9+ messages in thread
From: Aditya Toshniwal @ 2020-02-24 06:46 UTC (permalink / raw)
To: pgsql-admin
Hi Team,
The PG 12.2 server is crashing on setting the jit_above_cost param. Below
is the output.
postgres=# select version();
version
----------------------------------------------------------------------------------------------------------------------------
PostgreSQL 12.2 on x86_64-apple-darwin, compiled by Apple LLVM version 6.0
(clang-600.0.54) (based on LLVM 3.5svn), 64-bit
(1 row)
postgres=# SET jit_above_cost=10;
SET
postgres=# SELECT count(*) FROM pg_class;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>
--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-24 06:59 Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
parent: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
1 sibling, 1 reply; 9+ messages in thread
From: Aditya Toshniwal @ 2020-02-24 06:59 UTC (permalink / raw)
To: pgsql-admin; pgsql-hackers@postgresql.org
++hackers.
On Mon, Feb 24, 2020 at 12:16 PM Aditya Toshniwal <
aditya.toshniwal@enterprisedb.com> wrote:
> Hi Team,
>
> The PG 12.2 server is crashing on setting the jit_above_cost param. Below
> is the output.
>
> postgres=# select version();
>
> version
>
>
>
> ----------------------------------------------------------------------------------------------------------------------------
>
> PostgreSQL 12.2 on x86_64-apple-darwin, compiled by Apple LLVM version
> 6.0 (clang-600.0.54) (based on LLVM 3.5svn), 64-bit
>
> (1 row)
>
>
> postgres=# SET jit_above_cost=10;
>
> SET
>
> postgres=# SELECT count(*) FROM pg_class;
>
> server closed the connection unexpectedly
>
> This probably means the server terminated abnormally
>
> before or while processing the request.
>
> The connection to the server was lost. Attempting reset: Failed.
>
> !>
>
> --
> Thanks and Regards,
> Aditya Toshniwal
> pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
> "Don't Complain about Heat, Plant a TREE"
>
--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-24 07:16 Andres Freund <andres@anarazel.de>
parent: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
0 siblings, 1 reply; 9+ messages in thread
From: Andres Freund @ 2020-02-24 07:16 UTC (permalink / raw)
To: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>; +Cc: pgsql-admin; pgsql-hackers@postgresql.org
Hi,
On 2020-02-24 12:16:08 +0530, Aditya Toshniwal wrote:
> The PG 12.2 server is crashing on setting the jit_above_cost param. Below
> is the output.
> postgres=# SELECT count(*) FROM pg_class;
>
> server closed the connection unexpectedly
>
> This probably means the server terminated abnormally
>
> before or while processing the request.
>
> The connection to the server was lost. Attempting reset: Failed.
This isn't reproducible here. Are you sure that you're running on a
clean installation?
If not, we'd at least need a backtrace to figure out what's going on.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-24 07:35 Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 9+ messages in thread
From: Aditya Toshniwal @ 2020-02-24 07:35 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-admin; pgsql-hackers@postgresql.org
Hi Andres,
On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2020-02-24 12:16:08 +0530, Aditya Toshniwal wrote:
> > The PG 12.2 server is crashing on setting the jit_above_cost param. Below
> > is the output.
>
> > postgres=# SELECT count(*) FROM pg_class;
> >
> > server closed the connection unexpectedly
> >
> > This probably means the server terminated abnormally
> >
> > before or while processing the request.
> >
> > The connection to the server was lost. Attempting reset: Failed.
>
> This isn't reproducible here. Are you sure that you're running on a
> clean installation?
>
Yes I did a fresh installation using installer provided here -
https://www.enterprisedb.com/downloads/postgresql
>
> If not, we'd at least need a backtrace to figure out what's going on.
>
Please let me know how can I provide required info.
>
> Greetings,
>
> Andres Freund
>
--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-24 07:44 Imre Samu <pella.samu@gmail.com>
parent: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
1 sibling, 0 replies; 9+ messages in thread
From: Imre Samu @ 2020-02-24 07:44 UTC (permalink / raw)
To: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>; +Cc: pgsql-admin
>PostgreSQL 12.2 ...
>compiled by Apple LLVM version 6.0(clang-600.0.54)* (based on LLVM 3.5svn)*
this LLVM is >= 3.9 ?
according to the docs : "Build with support for LLVM based JIT compilation
.... *The minimum required version of LLVM is currently 3.9.*"
https://www.postgresql.org/docs/12/install-procedure.html
Regards,
Imre
Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> ezt írta (időpont:
2020. febr. 24., H, 7:46):
> Hi Team,
>
> The PG 12.2 server is crashing on setting the jit_above_cost param. Below
> is the output.
>
> postgres=# select version();
>
> version
>
>
>
> ----------------------------------------------------------------------------------------------------------------------------
>
> PostgreSQL 12.2 on x86_64-apple-darwin, compiled by Apple LLVM version
> 6.0 (clang-600.0.54) (based on LLVM 3.5svn), 64-bit
>
> (1 row)
>
>
> postgres=# SET jit_above_cost=10;
>
> SET
>
> postgres=# SELECT count(*) FROM pg_class;
>
> server closed the connection unexpectedly
>
> This probably means the server terminated abnormally
>
> before or while processing the request.
>
> The connection to the server was lost. Attempting reset: Failed.
>
> !>
>
> --
> Thanks and Regards,
> Aditya Toshniwal
> pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
> "Don't Complain about Heat, Plant a TREE"
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-24 14:50 Tom Lane <tgl@sss.pgh.pa.us>
parent: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
0 siblings, 2 replies; 9+ messages in thread
From: Tom Lane @ 2020-02-24 14:50 UTC (permalink / raw)
To: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:
> On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de> wrote:
>> This isn't reproducible here. Are you sure that you're running on a
>> clean installation?
> Yes I did a fresh installation using installer provided here -
> https://www.enterprisedb.com/downloads/postgresql
There is apparently something wrong with the JIT stuff in EDB's 12.2
build for macOS. At least, that's the conclusion I came to after
off-list discussion with the submitter of bug #16264, which has pretty
much exactly this symptom (especially if you're seeing "signal 9"
reports in the postmaster log). For him, either disabling JIT or
reverting to 12.1 made it go away.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-25 05:23 Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 9+ messages in thread
From: Aditya Toshniwal @ 2020-02-25 05:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andres Freund <andres@anarazel.de>; pgsql-hackers@postgresql.org
On Mon, Feb 24, 2020 at 8:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:
> > On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de>
> wrote:
> >> This isn't reproducible here. Are you sure that you're running on a
> >> clean installation?
>
> > Yes I did a fresh installation using installer provided here -
> > https://www.enterprisedb.com/downloads/postgresql
>
> There is apparently something wrong with the JIT stuff in EDB's 12.2
> build for macOS. At least, that's the conclusion I came to after
> off-list discussion with the submitter of bug #16264, which has pretty
> much exactly this symptom (especially if you're seeing "signal 9"
> reports in the postmaster log). For him, either disabling JIT or
> reverting to 12.1 made it go away.
>
Yes it seems like issue with EDB build. It works fine on macOS < Catalina.
>
> regards, tom lane
>
--
Thanks and Regards,
Aditya Toshniwal
pgAdmin Hacker | Sr. Software Engineer | EnterpriseDB India | Pune
"Don't Complain about Heat, Plant a TREE"
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-02-27 12:53 Dave Page <dpage@pgadmin.org>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 9+ messages in thread
From: Dave Page @ 2020-02-27 12:53 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers <pgsql-hackers@postgresql.org>
Hi
On Thu, Feb 27, 2020 at 12:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:
> > On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de>
> wrote:
> >> This isn't reproducible here. Are you sure that you're running on a
> >> clean installation?
>
> > Yes I did a fresh installation using installer provided here -
> > https://www.enterprisedb.com/downloads/postgresql
>
> There is apparently something wrong with the JIT stuff in EDB's 12.2
> build for macOS. At least, that's the conclusion I came to after
> off-list discussion with the submitter of bug #16264, which has pretty
> much exactly this symptom (especially if you're seeing "signal 9"
> reports in the postmaster log). For him, either disabling JIT or
> reverting to 12.1 made it go away.
>
We've been looking into this;
Apple started a notarisation process some time ago, designed to mark their
applications as conforming to various security requirements, but prior to
Catalina it was essentially optional. When Catalina was released, they made
notarisation for distributed software a requirement, but had the process
issue warnings for non-compliance. As-of the end of January, those warnings
became hard errors, so now our packages must be notarised, and for that to
happen, must be hardened by linking with a special runtime and having
securely time stamped signatures on every binary before being checked and
notarised as such by Apple. Without that, users would have to disable
security features on their systems before they could run our software.
Our packages are being successfully notarised at the moment, because that's
essentially done through a static analysis. We can (and have) added what
Apple call an entitlement in test builds which essentially puts a flag in
the notarisation for the product that declares that it will do JIT
operations, however, it seems that this alone is not enough and that in
addition to the entitlement, we also need to include the MAP_JIT flag in
mmap() calls. See
https://developer.apple.com/documentation/security/hardened_runtime and
https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-j...
We're working on trying to test a patch for that at the moment.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: PG v12.2 - Setting jit_above_cost is causing the server to crash
@ 2020-03-19 11:16 Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
parent: Dave Page <dpage@pgadmin.org>
0 siblings, 0 replies; 9+ messages in thread
From: Sandeep Thakkar @ 2020-03-19 11:16 UTC (permalink / raw)
To: Dave Page <dpage@pgadmin.org>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>; Andres Freund <andres@anarazel.de>; pgsql-hackers <pgsql-hackers@postgresql.org>
Hi,
On Thu, Feb 27, 2020 at 6:23 PM Dave Page <dpage@pgadmin.org> wrote:
> Hi
>
> On Thu, Feb 27, 2020 at 12:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>> Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> writes:
>> > On Mon, Feb 24, 2020 at 12:46 PM Andres Freund <andres@anarazel.de>
>> wrote:
>> >> This isn't reproducible here. Are you sure that you're running on a
>> >> clean installation?
>>
>> > Yes I did a fresh installation using installer provided here -
>> > https://www.enterprisedb.com/downloads/postgresql
>>
>> There is apparently something wrong with the JIT stuff in EDB's 12.2
>> build for macOS. At least, that's the conclusion I came to after
>> off-list discussion with the submitter of bug #16264, which has pretty
>> much exactly this symptom (especially if you're seeing "signal 9"
>> reports in the postmaster log). For him, either disabling JIT or
>> reverting to 12.1 made it go away.
>>
>
> We've been looking into this;
>
> Apple started a notarisation process some time ago, designed to mark their
> applications as conforming to various security requirements, but prior to
> Catalina it was essentially optional. When Catalina was released, they made
> notarisation for distributed software a requirement, but had the process
> issue warnings for non-compliance. As-of the end of January, those warnings
> became hard errors, so now our packages must be notarised, and for that to
> happen, must be hardened by linking with a special runtime and having
> securely time stamped signatures on every binary before being checked and
> notarised as such by Apple. Without that, users would have to disable
> security features on their systems before they could run our software.
>
> Our packages are being successfully notarised at the moment, because
> that's essentially done through a static analysis. We can (and have) added
> what Apple call an entitlement in test builds which essentially puts a flag
> in the notarisation for the product that declares that it will do JIT
> operations, however, it seems that this alone is not enough and that in
> addition to the entitlement, we also need to include the MAP_JIT flag in
> mmap() calls. See
> https://developer.apple.com/documentation/security/hardened_runtime and
> https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-j...
>
> We're working on trying to test a patch for that at the moment.
>
>
We have fixed the issue. To explain in brief, It was related to the
hardened runtime. Hardening the runtime can produce such issues, and
therefore Apple provides the runtime exceptions. We were previously using
an entitlement "com.apple.security.cs.disable-library-validation" for the
app bundle and then tried adding
"com.apple.security.cs.allow-unsigned-executable-memory" but still the
query would crash the server process when JIT is enabled. Later we applied
these entitlements to the PG binaries (postgres, pg_ctl and others) and the
bundles (llvmjit.so and others) which actually resolved the problem.
The updates will be released in a couple of days.
--
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
--
Sandeep Thakkar
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2020-03-19 11:16 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-02-24 06:46 PG v12.2 - Setting jit_above_cost is causing the server to crash Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
2020-02-24 06:59 ` Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
2020-02-24 07:16 ` Andres Freund <andres@anarazel.de>
2020-02-24 07:35 ` Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
2020-02-24 14:50 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-02-25 05:23 ` Aditya Toshniwal <aditya.toshniwal@enterprisedb.com>
2020-02-27 12:53 ` Dave Page <dpage@pgadmin.org>
2020-03-19 11:16 ` Sandeep Thakkar <sandeep.thakkar@enterprisedb.com>
2020-02-24 07:44 ` Imre Samu <pella.samu@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox