pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_dump being killed by oom killer
7+ messages / 3 participants
[nested] [flat]
* pg_dump being killed by oom killer
@ 2013-10-29 12:17 Paul Warren <pw@researchresearch.com>
0 siblings, 1 reply; 7+ messages in thread
From: Paul Warren @ 2013-10-29 12:17 UTC (permalink / raw)
To: pgsql-admin
Hi,
I'm to the lists so bare with me please.
I have a vm running debian 6 with linux kernel 2.6.32-11-pve, when i run
pg_dump -Fc -U postgres db name > db name.dump
after so long the process is killed, when I look in the /var/log/kern.log i
see the following entry
Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump) vm:5269904kB,
rss:5210224kB, swap:0kB
The vm has 16 cpu's and is using around 10%, when I run pg_dump it goes up
by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram and 1gb of
swap, the system uses around 2mb of swap and 15gb of ram, when pg_dump runs
the swap doesn't move but the ram goes up by 3 / 4 gb before the process is
killed.
The machine had only 20gb of ram so I've increased the ram usage and still
get the same issue. if I run pg_dump on any other db it works fine, if we
run a pg_dump script by table on the failing database it works fine.
The backup is going directly to a nas device which has 1.2 tb free. The vm
has aroudn 40gb free as well.
Any ideas what this could be? it seems to be a memory issue but I can't
seem to see what / why it's happening.
oh the DB is around 1.5 gb / 2gb in size
Thanks
Paul
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_dump being killed by oom killer
@ 2013-10-29 13:01 Sergey Klochkov <klochkov@iqbuzz.ru>
parent: Paul Warren <pw@researchresearch.com>
0 siblings, 1 reply; 7+ messages in thread
From: Sergey Klochkov @ 2013-10-29 13:01 UTC (permalink / raw)
To: pgsql-admin; pw@researchresearch.com
Hello Paul,
Recently I've run into similar issue, see my message to this mailing
list sent on October 1st, 2013. It happens if there is a large enough
amount of rows in pg_largeobject of the relevant database. pg_dump loads
metadata (owner, access rights, etc) of each large object deparately
until it is out of memory. How many large objects are in your DB?
Hope this helps.
29.10.2013 16:17, Paul Warren пишет:
> Hi,
>
> I'm to the lists so bare with me please.
>
> I have a vm running debian 6 with linux kernel 2.6.32-11-pve, when i run
> pg_dump -Fc -U postgres db name > db name.dump
>
> after so long the process is killed, when I look in the
> /var/log/kern.log i see the following entry
> Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump)
> vm:5269904kB, rss:5210224kB, swap:0kB
>
>
> The vm has 16 cpu's and is using around 10%, when I run pg_dump it goes
> up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram and 1gb
> of swap, the system uses around 2mb of swap and 15gb of ram, when
> pg_dump runs the swap doesn't move but the ram goes up by 3 / 4 gb
> before the process is killed.
>
> The machine had only 20gb of ram so I've increased the ram usage and
> still get the same issue. if I run pg_dump on any other db it works
> fine, if we run a pg_dump script by table on the failing database it
> works fine.
>
> The backup is going directly to a nas device which has 1.2 tb free. The
> vm has aroudn 40gb free as well.
>
> Any ideas what this could be? it seems to be a memory issue but I can't
> seem to see what / why it's happening.
>
> oh the DB is around 1.5 gb / 2gb in size
>
>
> Thanks
> Paul
--
Best regards, Klochkov Sergey
www.iqbuzz.ru
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_dump being killed by oom killer
@ 2013-10-29 13:04 Paul Warren <pw@researchresearch.com>
parent: Sergey Klochkov <klochkov@iqbuzz.ru>
0 siblings, 1 reply; 7+ messages in thread
From: Paul Warren @ 2013-10-29 13:04 UTC (permalink / raw)
To: Sergey Klochkov <klochkov@iqbuzz.ru>; +Cc: pgsql-admin
Hey,
I think it might not be a issue with pg_dump / db etc but more to do with
proxmox and it's container limits, this is a VM running on proxmox and
could be the cause of the issue. Will look into that first and come back if
it's not.
thanks
Paul
On Tue, Oct 29, 2013 at 1:01 PM, Sergey Klochkov <klochkov@iqbuzz.ru> wrote:
> Hello Paul,
>
> Recently I've run into similar issue, see my message to this mailing list
> sent on October 1st, 2013. It happens if there is a large enough amount of
> rows in pg_largeobject of the relevant database. pg_dump loads metadata
> (owner, access rights, etc) of each large object deparately until it is out
> of memory. How many large objects are in your DB?
>
> Hope this helps.
>
> 29.10.2013 16:17, Paul Warren пишет:
>
> Hi,
>>
>> I'm to the lists so bare with me please.
>>
>> I have a vm running debian 6 with linux kernel 2.6.32-11-pve, when i run
>> pg_dump -Fc -U postgres db name > db name.dump
>>
>> after so long the process is killed, when I look in the
>> /var/log/kern.log i see the following entry
>> Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump)
>> vm:5269904kB, rss:5210224kB, swap:0kB
>>
>>
>> The vm has 16 cpu's and is using around 10%, when I run pg_dump it goes
>> up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram and 1gb
>> of swap, the system uses around 2mb of swap and 15gb of ram, when
>> pg_dump runs the swap doesn't move but the ram goes up by 3 / 4 gb
>> before the process is killed.
>>
>> The machine had only 20gb of ram so I've increased the ram usage and
>> still get the same issue. if I run pg_dump on any other db it works
>> fine, if we run a pg_dump script by table on the failing database it
>> works fine.
>>
>> The backup is going directly to a nas device which has 1.2 tb free. The
>> vm has aroudn 40gb free as well.
>>
>> Any ideas what this could be? it seems to be a memory issue but I can't
>> seem to see what / why it's happening.
>>
>> oh the DB is around 1.5 gb / 2gb in size
>>
>>
>> Thanks
>> Paul
>>
>
>
> --
> Best regards, Klochkov Sergey
> www.iqbuzz.ru
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_dump being killed by oom killer
@ 2013-10-29 13:06 Sergey Klochkov <klochkov@iqbuzz.ru>
parent: Paul Warren <pw@researchresearch.com>
0 siblings, 1 reply; 7+ messages in thread
From: Sergey Klochkov @ 2013-10-29 13:06 UTC (permalink / raw)
To: Paul Warren <pw@researchresearch.com>; +Cc: pgsql-admin
Hello Paul,
If so, what does ulimit -m show?
29.10.2013 17:04, Paul Warren пишет:
> Hey,
>
> I think it might not be a issue with pg_dump / db etc but more to do
> with proxmox and it's container limits, this is a VM running on proxmox
> and could be the cause of the issue. Will look into that first and come
> back if it's not.
>
>
> thanks
> Paul
>
>
> On Tue, Oct 29, 2013 at 1:01 PM, Sergey Klochkov <klochkov@iqbuzz.ru
> <mailto:klochkov@iqbuzz.ru>> wrote:
>
> Hello Paul,
>
> Recently I've run into similar issue, see my message to this mailing
> list sent on October 1st, 2013. It happens if there is a large
> enough amount of rows in pg_largeobject of the relevant database.
> pg_dump loads metadata (owner, access rights, etc) of each large
> object deparately until it is out of memory. How many large objects
> are in your DB?
>
> Hope this helps.
>
> 29.10.2013 16:17, Paul Warren пишет:
>
> Hi,
>
> I'm to the lists so bare with me please.
>
> I have a vm running debian 6 with linux kernel 2.6.32-11-pve,
> when i run
> pg_dump -Fc -U postgres db name > db name.dump
>
> after so long the process is killed, when I look in the
> /var/log/kern.log i see the following entry
> Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump)
> vm:5269904kB, rss:5210224kB, swap:0kB
>
>
> The vm has 16 cpu's and is using around 10%, when I run pg_dump
> it goes
> up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram
> and 1gb
> of swap, the system uses around 2mb of swap and 15gb of ram, when
> pg_dump runs the swap doesn't move but the ram goes up by 3 / 4 gb
> before the process is killed.
>
> The machine had only 20gb of ram so I've increased the ram usage and
> still get the same issue. if I run pg_dump on any other db it works
> fine, if we run a pg_dump script by table on the failing database it
> works fine.
>
> The backup is going directly to a nas device which has 1.2 tb
> free. The
> vm has aroudn 40gb free as well.
>
> Any ideas what this could be? it seems to be a memory issue but
> I can't
> seem to see what / why it's happening.
>
> oh the DB is around 1.5 gb / 2gb in size
>
>
> Thanks
> Paul
>
>
>
> --
> Best regards, Klochkov Sergey
> www.iqbuzz.ru <http://www.iqbuzz.ru;
>
>
--
С уважением, Клочков Сергей
www.iqbuzz.ru
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_dump being killed by oom killer
@ 2013-10-29 13:08 Paul Warren <pw@researchresearch.com>
parent: Sergey Klochkov <klochkov@iqbuzz.ru>
0 siblings, 1 reply; 7+ messages in thread
From: Paul Warren @ 2013-10-29 13:08 UTC (permalink / raw)
To: Sergey Klochkov <klochkov@iqbuzz.ru>; +Cc: pgsql-admin
root@www1:/# ulimit -m
unlimited
On Tue, Oct 29, 2013 at 1:06 PM, Sergey Klochkov <klochkov@iqbuzz.ru> wrote:
> Hello Paul,
>
> If so, what does ulimit -m show?
>
>
> 29.10.2013 17:04, Paul Warren пишет:
>
>> Hey,
>>
>> I think it might not be a issue with pg_dump / db etc but more to do
>> with proxmox and it's container limits, this is a VM running on proxmox
>> and could be the cause of the issue. Will look into that first and come
>> back if it's not.
>>
>>
>> thanks
>> Paul
>>
>>
>> On Tue, Oct 29, 2013 at 1:01 PM, Sergey Klochkov <klochkov@iqbuzz.ru
>> <mailto:klochkov@iqbuzz.ru>> wrote:
>>
>> Hello Paul,
>>
>> Recently I've run into similar issue, see my message to this mailing
>> list sent on October 1st, 2013. It happens if there is a large
>> enough amount of rows in pg_largeobject of the relevant database.
>> pg_dump loads metadata (owner, access rights, etc) of each large
>> object deparately until it is out of memory. How many large objects
>> are in your DB?
>>
>> Hope this helps.
>>
>> 29.10.2013 16:17, Paul Warren пишет:
>>
>> Hi,
>>
>> I'm to the lists so bare with me please.
>>
>> I have a vm running debian 6 with linux kernel 2.6.32-11-pve,
>> when i run
>> pg_dump -Fc -U postgres db name > db name.dump
>>
>> after so long the process is killed, when I look in the
>> /var/log/kern.log i see the following entry
>> Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump)
>> vm:5269904kB, rss:5210224kB, swap:0kB
>>
>>
>> The vm has 16 cpu's and is using around 10%, when I run pg_dump
>> it goes
>> up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram
>> and 1gb
>> of swap, the system uses around 2mb of swap and 15gb of ram, when
>> pg_dump runs the swap doesn't move but the ram goes up by 3 / 4 gb
>> before the process is killed.
>>
>> The machine had only 20gb of ram so I've increased the ram usage
>> and
>> still get the same issue. if I run pg_dump on any other db it
>> works
>> fine, if we run a pg_dump script by table on the failing database
>> it
>> works fine.
>>
>> The backup is going directly to a nas device which has 1.2 tb
>> free. The
>> vm has aroudn 40gb free as well.
>>
>> Any ideas what this could be? it seems to be a memory issue but
>> I can't
>> seem to see what / why it's happening.
>>
>> oh the DB is around 1.5 gb / 2gb in size
>>
>>
>> Thanks
>> Paul
>>
>>
>>
>> --
>> Best regards, Klochkov Sergey
>> www.iqbuzz.ru <http://www.iqbuzz.ru;
>>
>>
>>
>
> --
> С уважением, Клочков Сергей
> www.iqbuzz.ru
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_dump being killed by oom killer
@ 2013-10-30 12:31 JotaComm <jota.comm@gmail.com>
parent: Paul Warren <pw@researchresearch.com>
0 siblings, 1 reply; 7+ messages in thread
From: JotaComm @ 2013-10-30 12:31 UTC (permalink / raw)
To: Paul Warren <pw@researchresearch.com>; +Cc: Sergey Klochkov <klochkov@iqbuzz.ru>; pgsql-admin
Hello,
2013/10/29 Paul Warren <pw@researchresearch.com>
> root@www1:/# ulimit -m
> unlimited
>
> And parameters: overcommit_memory and overcommit_ratio.
>
>
> On Tue, Oct 29, 2013 at 1:06 PM, Sergey Klochkov <klochkov@iqbuzz.ru>wrote:
>
>> Hello Paul,
>>
>> If so, what does ulimit -m show?
>>
>>
>> 29.10.2013 17:04, Paul Warren пишет:
>>
>>> Hey,
>>>
>>> I think it might not be a issue with pg_dump / db etc but more to do
>>> with proxmox and it's container limits, this is a VM running on proxmox
>>> and could be the cause of the issue. Will look into that first and come
>>> back if it's not.
>>>
>>>
>>> thanks
>>> Paul
>>>
>>>
>>> On Tue, Oct 29, 2013 at 1:01 PM, Sergey Klochkov <klochkov@iqbuzz.ru
>>> <mailto:klochkov@iqbuzz.ru>> wrote:
>>>
>>> Hello Paul,
>>>
>>> Recently I've run into similar issue, see my message to this mailing
>>> list sent on October 1st, 2013. It happens if there is a large
>>> enough amount of rows in pg_largeobject of the relevant database.
>>> pg_dump loads metadata (owner, access rights, etc) of each large
>>> object deparately until it is out of memory. How many large objects
>>> are in your DB?
>>>
>>> Hope this helps.
>>>
>>> 29.10.2013 16:17, Paul Warren пишет:
>>>
>>> Hi,
>>>
>>> I'm to the lists so bare with me please.
>>>
>>> I have a vm running debian 6 with linux kernel 2.6.32-11-pve,
>>> when i run
>>> pg_dump -Fc -U postgres db name > db name.dump
>>>
>>> after so long the process is killed, when I look in the
>>> /var/log/kern.log i see the following entry
>>> Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump)
>>> vm:5269904kB, rss:5210224kB, swap:0kB
>>>
>>>
>>> The vm has 16 cpu's and is using around 10%, when I run pg_dump
>>> it goes
>>> up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram
>>> and 1gb
>>> of swap, the system uses around 2mb of swap and 15gb of ram, when
>>> pg_dump runs the swap doesn't move but the ram goes up by 3 / 4
>>> gb
>>> before the process is killed.
>>>
>>> The machine had only 20gb of ram so I've increased the ram usage
>>> and
>>> still get the same issue. if I run pg_dump on any other db it
>>> works
>>> fine, if we run a pg_dump script by table on the failing
>>> database it
>>> works fine.
>>>
>>> The backup is going directly to a nas device which has 1.2 tb
>>> free. The
>>> vm has aroudn 40gb free as well.
>>>
>>> Any ideas what this could be? it seems to be a memory issue but
>>> I can't
>>> seem to see what / why it's happening.
>>>
>>> oh the DB is around 1.5 gb / 2gb in size
>>>
>>>
>>> Thanks
>>> Paul
>>>
>>>
>>>
>>> --
>>> Best regards, Klochkov Sergey
>>> www.iqbuzz.ru <http://www.iqbuzz.ru;
>>>
>>>
>>>
>>
>> --
>> С уважением, Клочков Сергей
>> www.iqbuzz.ru
>>
>
>
Best regards
--
JotaComm
http://jotacomm.wordpress.com
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_dump being killed by oom killer
@ 2013-11-01 10:46 Paul Warren <pw@researchresearch.com>
parent: JotaComm <jota.comm@gmail.com>
0 siblings, 0 replies; 7+ messages in thread
From: Paul Warren @ 2013-11-01 10:46 UTC (permalink / raw)
To: JotaComm <jota.comm@gmail.com>; +Cc: Sergey Klochkov <klochkov@iqbuzz.ru>; pgsql-admin
Yesterday I turned off the oom killer on the psychical server rather then
on the VM the vm is running on promox which is a container so oom killer
wasn't a option to turn off on the vm. When I did this it caused the server
to run out of memory, I rebooted server and pg dump now doesn't get killed
by oom killer but it now runs then uses most fo the ram up and then hangs
on terminal and i ended up killing it with loads of pg_dump command not
found errors
On Wed, Oct 30, 2013 at 12:31 PM, JotaComm <jota.comm@gmail.com> wrote:
> Hello,
>
>
> 2013/10/29 Paul Warren <pw@researchresearch.com>
>
>> root@www1:/# ulimit -m
>> unlimited
>>
>> And parameters: overcommit_memory and overcommit_ratio.
>
>>
>>
>> On Tue, Oct 29, 2013 at 1:06 PM, Sergey Klochkov <klochkov@iqbuzz.ru>wrote:
>>
>>> Hello Paul,
>>>
>>> If so, what does ulimit -m show?
>>>
>>>
>>> 29.10.2013 17:04, Paul Warren пишет:
>>>
>>>> Hey,
>>>>
>>>> I think it might not be a issue with pg_dump / db etc but more to do
>>>> with proxmox and it's container limits, this is a VM running on proxmox
>>>> and could be the cause of the issue. Will look into that first and come
>>>> back if it's not.
>>>>
>>>>
>>>> thanks
>>>> Paul
>>>>
>>>>
>>>> On Tue, Oct 29, 2013 at 1:01 PM, Sergey Klochkov <klochkov@iqbuzz.ru
>>>> <mailto:klochkov@iqbuzz.ru>> wrote:
>>>>
>>>> Hello Paul,
>>>>
>>>> Recently I've run into similar issue, see my message to this mailing
>>>> list sent on October 1st, 2013. It happens if there is a large
>>>> enough amount of rows in pg_largeobject of the relevant database.
>>>> pg_dump loads metadata (owner, access rights, etc) of each large
>>>> object deparately until it is out of memory. How many large objects
>>>> are in your DB?
>>>>
>>>> Hope this helps.
>>>>
>>>> 29.10.2013 16:17, Paul Warren пишет:
>>>>
>>>> Hi,
>>>>
>>>> I'm to the lists so bare with me please.
>>>>
>>>> I have a vm running debian 6 with linux kernel 2.6.32-11-pve,
>>>> when i run
>>>> pg_dump -Fc -U postgres db name > db name.dump
>>>>
>>>> after so long the process is killed, when I look in the
>>>> /var/log/kern.log i see the following entry
>>>> Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump)
>>>> vm:5269904kB, rss:5210224kB, swap:0kB
>>>>
>>>>
>>>> The vm has 16 cpu's and is using around 10%, when I run pg_dump
>>>> it goes
>>>> up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram
>>>> and 1gb
>>>> of swap, the system uses around 2mb of swap and 15gb of ram,
>>>> when
>>>> pg_dump runs the swap doesn't move but the ram goes up by 3 / 4
>>>> gb
>>>> before the process is killed.
>>>>
>>>> The machine had only 20gb of ram so I've increased the ram
>>>> usage and
>>>> still get the same issue. if I run pg_dump on any other db it
>>>> works
>>>> fine, if we run a pg_dump script by table on the failing
>>>> database it
>>>> works fine.
>>>>
>>>> The backup is going directly to a nas device which has 1.2 tb
>>>> free. The
>>>> vm has aroudn 40gb free as well.
>>>>
>>>> Any ideas what this could be? it seems to be a memory issue but
>>>> I can't
>>>> seem to see what / why it's happening.
>>>>
>>>> oh the DB is around 1.5 gb / 2gb in size
>>>>
>>>>
>>>> Thanks
>>>> Paul
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards, Klochkov Sergey
>>>> www.iqbuzz.ru <http://www.iqbuzz.ru;
>>>>
>>>>
>>>>
>>>
>>> --
>>> С уважением, Клочков Сергей
>>> www.iqbuzz.ru
>>>
>>
>>
>
> Best regards
> --
> JotaComm
> http://jotacomm.wordpress.com
>
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2013-11-01 10:46 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2013-10-29 12:17 pg_dump being killed by oom killer Paul Warren <pw@researchresearch.com>
2013-10-29 13:01 ` Sergey Klochkov <klochkov@iqbuzz.ru>
2013-10-29 13:04 ` Paul Warren <pw@researchresearch.com>
2013-10-29 13:06 ` Sergey Klochkov <klochkov@iqbuzz.ru>
2013-10-29 13:08 ` Paul Warren <pw@researchresearch.com>
2013-10-30 12:31 ` JotaComm <jota.comm@gmail.com>
2013-11-01 10:46 ` Paul Warren <pw@researchresearch.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox