pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPG 11.8 > PG 12.3 : Unable to allocate section memory
6+ messages / 3 participants
[nested] [flat]
* PG 11.8 > PG 12.3 : Unable to allocate section memory
@ 2020-08-12 14:09 JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 14:26 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-12 14:41 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 6+ messages in thread
From: JOIGNY Michael @Neteven @ 2020-08-12 14:09 UTC (permalink / raw)
To: pgsql-admin
Hi Community,
We have migrated from postgresql 11.8 to 12.3,unfortunately we have
sometimes the following errors in the logs, causing postgres process to
be killed.
2020-08-12 14:20:29.043 CEST [134644]: [71-1]
user=xx,db=xx,app=[unknown],client=localhost FATAL: out of memory
2020-08-12 14:20:29.049 CEST [30022]: [6126-1]
user=xx,db=xx,app=[unknown],client=localhost FATAL: fatal llvm error:
Unable to allocate section memory!
2020-08-12 14:20:29.197 CEST [78792]: [1-1] user=,db=,app=,client=
FATAL: terminating connection due to administrator command
2020-08-12 14:20:29.197 CEST [78791]: [1-1] user=,db=,app=,client=
FATAL: terminating connection due to administrator command
2020-08-12 14:20:29.197 CEST [78793]: [1-1] user=,db=,app=,client=
FATAL: terminating connection due to administrator command
2020-08-12 14:20:29.197 CEST [78790]: [1-1] user=,db=,app=,client=
FATAL: terminating connection due to administrator command
2020-08-12 14:21:02.621 CEST [86289]: [2-1]
user=postgres,db=template1,app=[unknown],client=[local] FATAL: out of memory
2020-08-12 14:21:04.632 CEST [29597]: [26704-1]
user=xx,db=xx,app=[unknown],client=localhost FATAL: fatal llvm error:
Unable to allocate section memory!
Here are our settings :
_PG :_
max_connections = 3600
shared_buffers = 32GB
temp_buffers = 16MB
work_mem = 96MB
maintenance_work_mem = 8GB
max_stack_depth = 6MB
dynamic_shared_memory_type = posix
We use postgresql with pgbouncer as a pooler.
_PGbouncer :
_
pooler mode : session
max_client_conn = 6000
default_pool_size = 3000
We have ~ 2000 connections at the same time with ~ 20/30 are active.
Note that we had the same configuration under PG11 without any problem.
Do you have an idea ?
Regards.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PG 11.8 > PG 12.3 : Unable to allocate section memory
2020-08-12 14:09 PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
@ 2020-08-12 14:26 ` MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-12 14:54 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
1 sibling, 1 reply; 6+ messages in thread
From: MichaelDBA @ 2020-08-12 14:26 UTC (permalink / raw)
To: mjoigny@neteven.com; +Cc: pgsql-admin
Please specify some other stuff:
1. PG onprem or cloud? If cloud, which one and what instance type.
2. Is it a dedicated PG host? How many hyper threads and memory do you
have?
3. Does PGBouncer run on another host or on the same host as PG?
JOIGNY Michael @Neteven wrote on 8/12/2020 10:09 AM:
>
> Hi Community,
>
> We have migrated from postgresql 11.8 to 12.3,unfortunately we have
> sometimes the following errors in the logs, causing postgres process
> to be killed.
>
> 2020-08-12 14:20:29.043 CEST [134644]: [71-1]
> user=xx,db=xx,app=[unknown],client=localhost FATAL: out of memory
> 2020-08-12 14:20:29.049 CEST [30022]: [6126-1]
> user=xx,db=xx,app=[unknown],client=localhost FATAL: fatal llvm error:
> Unable to allocate section memory!
> 2020-08-12 14:20:29.197 CEST [78792]: [1-1] user=,db=,app=,client=
> FATAL: terminating connection due to administrator command
> 2020-08-12 14:20:29.197 CEST [78791]: [1-1] user=,db=,app=,client=
> FATAL: terminating connection due to administrator command
> 2020-08-12 14:20:29.197 CEST [78793]: [1-1] user=,db=,app=,client=
> FATAL: terminating connection due to administrator command
> 2020-08-12 14:20:29.197 CEST [78790]: [1-1] user=,db=,app=,client=
> FATAL: terminating connection due to administrator command
> 2020-08-12 14:21:02.621 CEST [86289]: [2-1]
> user=postgres,db=template1,app=[unknown],client=[local] FATAL: out of
> memory
> 2020-08-12 14:21:04.632 CEST [29597]: [26704-1]
> user=xx,db=xx,app=[unknown],client=localhost FATAL: fatal llvm error:
> Unable to allocate section memory!
>
> Here are our settings :
>
> _PG :_
>
> max_connections = 3600
>
> shared_buffers = 32GB
> temp_buffers = 16MB
> work_mem = 96MB
> maintenance_work_mem = 8GB
> max_stack_depth = 6MB
> dynamic_shared_memory_type = posix
>
> We use postgresql with pgbouncer as a pooler.
>
>
> _PGbouncer :
> _
>
> pooler mode : session
> max_client_conn = 6000
> default_pool_size = 3000
>
> We have ~ 2000 connections at the same time with ~ 20/30 are active.
>
> Note that we had the same configuration under PG11 without any problem.
>
> Do you have an idea ?
>
> Regards.
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PG 11.8 > PG 12.3 : Unable to allocate section memory
2020-08-12 14:09 PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 14:26 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory MichaelDBA <MichaelDBA@sqlexec.com>
@ 2020-08-12 14:54 ` JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 15:56 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory MichaelDBA <MichaelDBA@sqlexec.com>
0 siblings, 1 reply; 6+ messages in thread
From: JOIGNY Michael @Neteven @ 2020-08-12 14:54 UTC (permalink / raw)
To: MichaelDBA <MichaelDBA@sqlexec.com>; tgl@sss.pgh.pa.us; +Cc: pgsql-admin
Hi Michael, Tom,
Thanks for your help.
Le 12/08/2020 à 16:26, MichaelDBA a écrit :
> Please specify some other stuff:
>
> 1. PG onprem or cloud? If cloud, which one and what instance type.
Onprem
> 2. Is it a dedicated PG host? How many hyper threads and memory do
> you have?
Yes dedicated PG host, 56 hyper threads and 380Go of memory.
> 3. Does PGBouncer run on another host or on the same host as PG?
I have a primary / stanby configuration using repmgr for replication.
PGbouncer is running on both of them.
@Tom, thanks i will have a look at this parameter.
"One thing that you might not have noticed is that "jit" defaults to on
as of v12. Turning it off would prevent this specific crash case. (The
fact that it's a crash comes from llvm's crummy error handling :-() You
might still have OOM problems, but with luck they'd not take down the
whole database."
Regards.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PG 11.8 > PG 12.3 : Unable to allocate section memory
2020-08-12 14:09 PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 14:26 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-12 14:54 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
@ 2020-08-12 15:56 ` MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-13 06:34 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
0 siblings, 1 reply; 6+ messages in thread
From: MichaelDBA @ 2020-08-12 15:56 UTC (permalink / raw)
To: mjoigny@neteven.com; +Cc: tgl@sss.pgh.pa.us; pgsql-admin
Just to see if work_mem is related to these memory problems, please
lower it from 96MB to 16MB and do a reload:
pg_ctl -D <your pg data directory> reload
If you still see memory problems, the problem is elsewhere. Please
report back. Thanks.
Don't forget to reset it back to 96MB if it's not the cause of the problem.
Regards,
Michael Vitale
JOIGNY Michael @Neteven wrote on 8/12/2020 10:54 AM:
> Hi Michael, Tom,
>
> Thanks for your help.
>
> Le 12/08/2020 à 16:26, MichaelDBA a écrit :
>> Please specify some other stuff:
>>
>> 1. PG onprem or cloud? If cloud, which one and what instance type.
> Onprem
>> 2. Is it a dedicated PG host? How many hyper threads and memory do
>> you have?
> Yes dedicated PG host, 56 hyper threads and 380Go of memory.
>> 3. Does PGBouncer run on another host or on the same host as PG?
>
> I have a primary / stanby configuration using repmgr for replication.
>
> PGbouncer is running on both of them.
>
>
> @Tom, thanks i will have a look at this parameter.
>
>
> "One thing that you might not have noticed is that "jit" defaults to
> on as of v12. Turning it off would prevent this specific crash case.
> (The fact that it's a crash comes from llvm's crummy error handling
> :-() You might still have OOM problems, but with luck they'd not take
> down the whole database."
>
> Regards.
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PG 11.8 > PG 12.3 : Unable to allocate section memory
2020-08-12 14:09 PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 14:26 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-12 14:54 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 15:56 ` Re: PG 11.8 > PG 12.3 : Unable to allocate section memory MichaelDBA <MichaelDBA@sqlexec.com>
@ 2020-08-13 06:34 ` JOIGNY Michael @Neteven <mjoigny@neteven.com>
0 siblings, 0 replies; 6+ messages in thread
From: JOIGNY Michael @Neteven @ 2020-08-13 06:34 UTC (permalink / raw)
To: MichaelDBA <MichaelDBA@sqlexec.com>; +Cc: tgl@sss.pgh.pa.us; pgsql-admin
Since the deactivation of the JIT param, no more error message.
Thanks for your help.
Have a good day.
Le 12/08/2020 à 17:56, MichaelDBA a écrit :
> Just to see if work_mem is related to these memory problems, please
> lower it from 96MB to 16MB and do a reload:
> pg_ctl -D <your pg data directory> reload
>
> If you still see memory problems, the problem is elsewhere. Please
> report back. Thanks.
>
> Don't forget to reset it back to 96MB if it's not the cause of the
> problem.
>
> Regards,
> Michael Vitale
>
>
> JOIGNY Michael @Neteven wrote on 8/12/2020 10:54 AM:
>> Hi Michael, Tom,
>>
>> Thanks for your help.
>>
>> Le 12/08/2020 à 16:26, MichaelDBA a écrit :
>>> Please specify some other stuff:
>>>
>>> 1. PG onprem or cloud? If cloud, which one and what instance type.
>> Onprem
>>> 2. Is it a dedicated PG host? How many hyper threads and memory do
>>> you have?
>> Yes dedicated PG host, 56 hyper threads and 380Go of memory.
>>> 3. Does PGBouncer run on another host or on the same host as PG?
>>
>> I have a primary / stanby configuration using repmgr for replication.
>>
>> PGbouncer is running on both of them.
>>
>>
>> @Tom, thanks i will have a look at this parameter.
>>
>>
>> "One thing that you might not have noticed is that "jit" defaults to
>> on as of v12. Turning it off would prevent this specific crash case.
>> (The fact that it's a crash comes from llvm's crummy error handling
>> :-() You might still have OOM problems, but with luck they'd not take
>> down the whole database."
>>
>> Regards.
>>
>
--
Cordialement.
--------------------
NETEVEN
Michael JOIGNY
Service Exploitation - Ingénieur Systèmes
190 Avenue Jean Jaures - 75019 Paris
mjoigny@neteven.com / adminsys@neteven.com
www.neteven.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: PG 11.8 > PG 12.3 : Unable to allocate section memory
2020-08-12 14:09 PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
@ 2020-08-12 14:41 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 6+ messages in thread
From: Tom Lane @ 2020-08-12 14:41 UTC (permalink / raw)
To: mjoigny@neteven.com; +Cc: pgsql-admin
"JOIGNY Michael @Neteven" <mjoigny@neteven.com> writes:
> We have migrated from postgresql 11.8 to 12.3,unfortunately we have
> sometimes the following errors in the logs, causing postgres process to
> be killed.
> 2020-08-12 14:20:29.043 CEST [134644]: [71-1]
> user=xx,db=xx,app=[unknown],client=localhost FATAL: out of memory
> 2020-08-12 14:20:29.049 CEST [30022]: [6126-1]
> user=xx,db=xx,app=[unknown],client=localhost FATAL: fatal llvm error:
> Unable to allocate section memory!
Hmm, seems like your system is under undue memory pressure ...
> max_connections = 3600
This seems slightly insane. Get a pooler ... or, since you say you
already have one, let *it* do the multiplexing.
> Note that we had the same configuration under PG11 without any problem.
One thing that you might not have noticed is that "jit" defaults to on
as of v12. Turning it off would prevent this specific crash case.
(The fact that it's a crash comes from llvm's crummy error handling :-()
You might still have OOM problems, but with luck they'd not take down
the whole database.
regards, tom lane
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2020-08-13 06:34 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-08-12 14:09 PG 11.8 > PG 12.3 : Unable to allocate section memory JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 14:26 ` MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-12 14:54 ` JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 15:56 ` MichaelDBA <MichaelDBA@sqlexec.com>
2020-08-13 06:34 ` JOIGNY Michael @Neteven <mjoigny@neteven.com>
2020-08-12 14:41 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox