pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPG10 - Out of shared memory / invalid memory alloc request size
8+ messages / 4 participants
[nested] [flat]
* PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 19:44 Arjun Ranade <ranade@nodalexchange.com>
0 siblings, 1 reply; 8+ messages in thread
From: Arjun Ranade @ 2017-11-14 19:44 UTC (permalink / raw)
To: pgsql-admin
We recently upgraded a database that a number of developers use from
Postgres 9.4 to 10. Our memory parameters are unchanged.
We are seeing a lot of errors that look like this in the postgres log:
< 2017-11-14 13:13:59 EST [144172] : [1-1]
user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no PostgreSQL
user name specified in startup packet
< 2017-11-14 13:52:22 EST [6273] : [1-1]
user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
shared memory
< 2017-11-14 14:17:53 EST [6200] : [3-1]
user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
invalid memory alloc request size 34359738368
Anecdotally it seems that these errors are happening when
powa_take_snapshot() (Postgres workload analyzer) is running as well but I
can't confirm that.
Does anyone know how to debug these types of errors? And once the db is in
this state, how do we resolve sessions reporting "Error: invalid memory
alloc request size" - killing the sessions and having them reconnect does
not fix the issue.
Thanks,
Arjun
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 19:53 Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>
parent: Arjun Ranade <ranade@nodalexchange.com>
0 siblings, 1 reply; 8+ messages in thread
From: Alvaro Aguayo Garcia-Rada @ 2017-11-14 19:53 UTC (permalink / raw)
To: Arjun Ranade <ranade@nodalexchange.com>; +Cc: pgsql-admin
I remember an old issue I had with postgres and shared memory being set too low by the distro I was using.
What's the value of "/proc/sys/kernel/shmmax"? Try setting to 18446744073709551615
Regards,
Alvaro Aguayo
Operations Manager
Open Comb Systems E.I.R.L.
Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
Web: www.ocs.pe
----- Original Message -----
From: "Arjun Ranade" <ranade@nodalexchange.com>
To: "pgsql-admin" <pgsql-admin@postgresql.org>
Sent: Tuesday, 14 November, 2017 14:44:08
Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size
We recently upgraded a database that a number of developers use from
Postgres 9.4 to 10. Our memory parameters are unchanged.
We are seeing a lot of errors that look like this in the postgres log:
< 2017-11-14 13:13:59 EST [144172] : [1-1]
user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no PostgreSQL
user name specified in startup packet
< 2017-11-14 13:52:22 EST [6273] : [1-1]
user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
shared memory
< 2017-11-14 14:17:53 EST [6200] : [3-1]
user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
invalid memory alloc request size 34359738368
Anecdotally it seems that these errors are happening when
powa_take_snapshot() (Postgres workload analyzer) is running as well but I
can't confirm that.
Does anyone know how to debug these types of errors? And once the db is in
this state, how do we resolve sessions reporting "Error: invalid memory
alloc request size" - killing the sessions and having them reconnect does
not fix the issue.
Thanks,
Arjun
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 20:00 Arjun Ranade <ranade@nodalexchange.com>
parent: Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>
0 siblings, 1 reply; 8+ messages in thread
From: Arjun Ranade @ 2017-11-14 20:00 UTC (permalink / raw)
To: Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>; +Cc: pgsql-admin
/proc/sys/kernel/shmax = 68719476736
On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
aaguayo@opensysperu.com> wrote:
> I remember an old issue I had with postgres and shared memory being set
> too low by the distro I was using.
>
> What's the value of "/proc/sys/kernel/shmmax"? Try setting to
> 18446744073709551615
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> Web: www.ocs.pe
>
> ----- Original Message -----
> From: "Arjun Ranade" <ranade@nodalexchange.com>
> To: "pgsql-admin" <pgsql-admin@postgresql.org>
> Sent: Tuesday, 14 November, 2017 14:44:08
> Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> request size
>
> We recently upgraded a database that a number of developers use from
> Postgres 9.4 to 10. Our memory parameters are unchanged.
>
> We are seeing a lot of errors that look like this in the postgres log:
>
> < 2017-11-14 13:13:59 EST [144172] : [1-1]
> user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no
> PostgreSQL
> user name specified in startup packet
> < 2017-11-14 13:52:22 EST [6273] : [1-1]
> user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
> shared memory
> < 2017-11-14 14:17:53 EST [6200] : [3-1]
> user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
> invalid memory alloc request size 34359738368
>
> Anecdotally it seems that these errors are happening when
> powa_take_snapshot() (Postgres workload analyzer) is running as well but I
> can't confirm that.
>
> Does anyone know how to debug these types of errors? And once the db is in
> this state, how do we resolve sessions reporting "Error: invalid memory
> alloc request size" - killing the sessions and having them reconnect does
> not fix the issue.
>
> Thanks,
> Arjun
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 20:05 Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>
parent: Arjun Ranade <ranade@nodalexchange.com>
0 siblings, 1 reply; 8+ messages in thread
From: Alvaro Aguayo Garcia-Rada @ 2017-11-14 20:05 UTC (permalink / raw)
To: Arjun Ranade <ranade@nodalexchange.com>; +Cc: pgsql-admin
Try with:
echo "18446744073709551615" > /proc/sys/kernel/shmax
Then restart postgres
Regards,
Alvaro Aguayo
Operations Manager
Open Comb Systems E.I.R.L.
Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
Web: www.ocs.pe
----- Original Message -----
From: "Arjun Ranade" <ranade@nodalexchange.com>
To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com>
Cc: "pgsql-admin" <pgsql-admin@postgresql.org>
Sent: Tuesday, 14 November, 2017 15:00:03
Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory alloc request size
/proc/sys/kernel/shmax = 68719476736
On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
aaguayo@opensysperu.com> wrote:
> I remember an old issue I had with postgres and shared memory being set
> too low by the distro I was using.
>
> What's the value of "/proc/sys/kernel/shmmax"? Try setting to
> 18446744073709551615
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> Web: www.ocs.pe
>
> ----- Original Message -----
> From: "Arjun Ranade" <ranade@nodalexchange.com>
> To: "pgsql-admin" <pgsql-admin@postgresql.org>
> Sent: Tuesday, 14 November, 2017 14:44:08
> Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> request size
>
> We recently upgraded a database that a number of developers use from
> Postgres 9.4 to 10. Our memory parameters are unchanged.
>
> We are seeing a lot of errors that look like this in the postgres log:
>
> < 2017-11-14 13:13:59 EST [144172] : [1-1]
> user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no
> PostgreSQL
> user name specified in startup packet
> < 2017-11-14 13:52:22 EST [6273] : [1-1]
> user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
> shared memory
> < 2017-11-14 14:17:53 EST [6200] : [3-1]
> user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
> invalid memory alloc request size 34359738368
>
> Anecdotally it seems that these errors are happening when
> powa_take_snapshot() (Postgres workload analyzer) is running as well but I
> can't confirm that.
>
> Does anyone know how to debug these types of errors? And once the db is in
> this state, how do we resolve sessions reporting "Error: invalid memory
> alloc request size" - killing the sessions and having them reconnect does
> not fix the issue.
>
> Thanks,
> Arjun
>
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 20:07 Arjun Ranade <ranade@nodalexchange.com>
parent: Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>
0 siblings, 2 replies; 8+ messages in thread
From: Arjun Ranade @ 2017-11-14 20:07 UTC (permalink / raw)
To: Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>; +Cc: pgsql-admin
We will try that. I am wondering why we need to worry about this in pg10
whereas we had no issues with shared memory in 9.4
Thanks,
Arjun
On Tue, Nov 14, 2017 at 3:05 PM, Alvaro Aguayo Garcia-Rada <
aaguayo@opensysperu.com> wrote:
> Try with:
>
> echo "18446744073709551615" > /proc/sys/kernel/shmax
>
> Then restart postgres
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> Web: www.ocs.pe
>
> ----- Original Message -----
> From: "Arjun Ranade" <ranade@nodalexchange.com>
> To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com>
> Cc: "pgsql-admin" <pgsql-admin@postgresql.org>
> Sent: Tuesday, 14 November, 2017 15:00:03
> Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> request size
>
> /proc/sys/kernel/shmax = 68719476736
>
>
> On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
> aaguayo@opensysperu.com> wrote:
>
> > I remember an old issue I had with postgres and shared memory being set
> > too low by the distro I was using.
> >
> > What's the value of "/proc/sys/kernel/shmmax"? Try setting to
> > 18446744073709551615
> >
> > Regards,
> >
> > Alvaro Aguayo
> > Operations Manager
> > Open Comb Systems E.I.R.L.
> >
> > Office: (+51-1) 3377813 | Mobile: (+51) 995540103 | (+51) 954183248
> > Web: www.ocs.pe
> >
> > ----- Original Message -----
> > From: "Arjun Ranade" <ranade@nodalexchange.com>
> > To: "pgsql-admin" <pgsql-admin@postgresql.org>
> > Sent: Tuesday, 14 November, 2017 14:44:08
> > Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> > request size
> >
> > We recently upgraded a database that a number of developers use from
> > Postgres 9.4 to 10. Our memory parameters are unchanged.
> >
> > We are seeing a lot of errors that look like this in the postgres log:
> >
> > < 2017-11-14 13:13:59 EST [144172] : [1-1]
> > user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no
> > PostgreSQL
> > user name specified in startup packet
> > < 2017-11-14 13:52:22 EST [6273] : [1-1]
> > user=deployment_app,db=qa5,remote=192.168.104.83(45384) > ERROR: out of
> > shared memory
> > < 2017-11-14 14:17:53 EST [6200] : [3-1]
> > user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) > ERROR:
> > invalid memory alloc request size 34359738368
> >
> > Anecdotally it seems that these errors are happening when
> > powa_take_snapshot() (Postgres workload analyzer) is running as well but
> I
> > can't confirm that.
> >
> > Does anyone know how to debug these types of errors? And once the db is
> in
> > this state, how do we resolve sessions reporting "Error: invalid memory
> > alloc request size" - killing the sessions and having them reconnect does
> > not fix the issue.
> >
> > Thanks,
> > Arjun
> >
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 20:15 Tom Lane <tgl@sss.pgh.pa.us>
parent: Arjun Ranade <ranade@nodalexchange.com>
1 sibling, 0 replies; 8+ messages in thread
From: Tom Lane @ 2017-11-14 20:15 UTC (permalink / raw)
To: Arjun Ranade <ranade@nodalexchange.com>; +Cc: Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>; pgsql-admin
Arjun Ranade <ranade@nodalexchange.com> writes:
> We will try that. I am wondering why we need to worry about this in pg10
> whereas we had no issues with shared memory in 9.4
What it looks like to me is something is miscomputing a memory request
size and asking for some silly amount of memory. I think you're dealing
with a software bug, not inadequate configuration.
> Anecdotally it seems that these errors are happening when
> powa_take_snapshot() (Postgres workload analyzer) is running as well but
> I can't confirm that.
The first theory that comes to mind is that powa_take_snapshot() requires
some updates for PG10. I do not know what that function is or where you
got it from, but if there's any C code involved, you need to discuss this
with its developer(s).
regards, tom lane
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-14 23:24 Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
parent: Arjun Ranade <ranade@nodalexchange.com>
1 sibling, 1 reply; 8+ messages in thread
From: Mark Kirkwood @ 2017-11-14 23:24 UTC (permalink / raw)
To: Arjun Ranade <ranade@nodalexchange.com>; Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>; +Cc: pgsql-admin
I don't think this is the issue (Tom's theory sounds more likely). Also
changing the IPC shared memory limit is not going to make any difference
for these later versions of Postgres (>= 9.3) as it uses POSIX shared
memory (I believe the whole point of that change was to avoid the need
to tweak these IPC parameters)!
regards
Mark
On 15/11/17 09:07, Arjun Ranade wrote:
> We will try that. I am wondering why we need to worry about this in
> pg10 whereas we had no issues with shared memory in 9.4
>
> Thanks,
> Arjun
>
> On Tue, Nov 14, 2017 at 3:05 PM, Alvaro Aguayo Garcia-Rada
> <aaguayo@opensysperu.com <mailto:aaguayo@opensysperu.com>> wrote:
>
> Try with:
>
> echo "18446744073709551615" > /proc/sys/kernel/shmax
>
> Then restart postgres
>
> Regards,
>
> Alvaro Aguayo
> Operations Manager
> Open Comb Systems E.I.R.L.
>
> Office: (+51-1) 3377813 <tel:%28%2B51-1%29%203377813> | Mobile:
> (+51) 995540103 <tel:%28%2B51%29%20995540103> | (+51) 954183248
> <tel:%28%2B51%29%20954183248>
> Web: www.ocs.pe <http://www.ocs.pe;
>
> ----- Original Message -----
> From: "Arjun Ranade" <ranade@nodalexchange.com
> <mailto:ranade@nodalexchange.com>>
> To: "Alvaro Aguayo Garcia-Rada" <aaguayo@opensysperu.com
> <mailto:aaguayo@opensysperu.com>>
> Cc: "pgsql-admin" <pgsql-admin@postgresql.org
> <mailto:pgsql-admin@postgresql.org>>
> Sent: Tuesday, 14 November, 2017 15:00:03
> Subject: Re: [ADMIN] PG10 - Out of shared memory / invalid memory
> alloc request size
>
> /proc/sys/kernel/shmax = 68719476736
>
>
> On Tue, Nov 14, 2017 at 2:53 PM, Alvaro Aguayo Garcia-Rada <
> aaguayo@opensysperu.com <mailto:aaguayo@opensysperu.com>> wrote:
>
> > I remember an old issue I had with postgres and shared memory
> being set
> > too low by the distro I was using.
> >
> > What's the value of "/proc/sys/kernel/shmmax"? Try setting to
> > 18446744073709551615
> >
> > Regards,
> >
> > Alvaro Aguayo
> > Operations Manager
> > Open Comb Systems E.I.R.L.
> >
> > Office: (+51-1) 3377813 <tel:%28%2B51-1%29%203377813> | Mobile:
> (+51) 995540103 <tel:%28%2B51%29%20995540103> | (+51) 954183248
> <tel:%28%2B51%29%20954183248>
> > Web: www.ocs.pe <http://www.ocs.pe;
> >
> > ----- Original Message -----
> > From: "Arjun Ranade" <ranade@nodalexchange.com
> <mailto:ranade@nodalexchange.com>>
> > To: "pgsql-admin" <pgsql-admin@postgresql.org
> <mailto:pgsql-admin@postgresql.org>>
> > Sent: Tuesday, 14 November, 2017 14:44:08
> > Subject: [ADMIN] PG10 - Out of shared memory / invalid memory alloc
> > request size
> >
> > We recently upgraded a database that a number of developers use from
> > Postgres 9.4 to 10. Our memory parameters are unchanged.
> >
> > We are seeing a lot of errors that look like this in the
> postgres log:
> >
> > < 2017-11-14 13:13:59 EST [144172] : [1-1]
> > user=[unknown],db=qa16,remote=192.168.104.22(35238) > FATAL: no
> > PostgreSQL
> > user name specified in startup packet
> > < 2017-11-14 13:52:22 EST [6273] : [1-1]
> > user=deployment_app,db=qa5,remote=192.168.104.83(45384) >
> ERROR: out of
> > shared memory
> > < 2017-11-14 14:17:53 EST [6200] : [3-1]
> > user=service_tracker_app,db=qa16,remote=192.168.104.22(46996) >
> ERROR:
> > invalid memory alloc request size 34359738368
> >
> > Anecdotally it seems that these errors are happening when
> > powa_take_snapshot() (Postgres workload analyzer) is running as
> well but I
> > can't confirm that.
> >
> > Does anyone know how to debug these types of errors? And once
> the db is in
> > this state, how do we resolve sessions reporting "Error: invalid
> memory
> > alloc request size" - killing the sessions and having them
> reconnect does
> > not fix the issue.
> >
> > Thanks,
> > Arjun
> >
>
>
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: PG10 - Out of shared memory / invalid memory alloc request size
@ 2017-11-15 01:34 Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
parent: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
0 siblings, 0 replies; 8+ messages in thread
From: Mark Kirkwood @ 2017-11-15 01:34 UTC (permalink / raw)
To: Arjun Ranade <ranade@nodalexchange.com>; Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>; +Cc: pgsql-admin
Sorry: I've blithely typed IPC when I should have said SYS V IPC...
On 15/11/17 12:24, Mark Kirkwood wrote:
> I don't think this is the issue (Tom's theory sounds more likely).
> Also changing the IPC shared memory limit is not going to make any
> difference for these later versions of Postgres (>= 9.3) as it uses
> POSIX shared memory (I believe the whole point of that change was to
> avoid the need to tweak these IPC parameters)!
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2017-11-15 01:34 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-11-14 19:44 PG10 - Out of shared memory / invalid memory alloc request size Arjun Ranade <ranade@nodalexchange.com>
2017-11-14 19:53 ` Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>
2017-11-14 20:00 ` Arjun Ranade <ranade@nodalexchange.com>
2017-11-14 20:05 ` Alvaro Aguayo Garcia-Rada <aaguayo@opensysperu.com>
2017-11-14 20:07 ` Arjun Ranade <ranade@nodalexchange.com>
2017-11-14 20:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2017-11-14 23:24 ` Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
2017-11-15 01:34 ` Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox