User-Agent: Microsoft-MacOutlook/14.2.3.120616
Date: Mon, 19 Aug 2013 11:45:08 +0300
Subject: Re: postgres crashes on insert in 40 different threads
From: Dzmitry <dzmitry.nikitsin@gmail.com>
To: =?ISO-8859-1?B?U3TpcGhhbmU=?= Schildknecht
 <stephane.schildknecht@postgresql.fr>, <pgsql-admin@postgresql.org>
Message-ID: <CE37B4F2.116D1%dzmitry.nikitsin@gmail.com>
Thread-Topic: [ADMIN] postgres crashes on insert in 40 different threads
In-Reply-To: <5211D917.5060805@postgresql.fr>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

I don't think it's the case. I am using newrelic for monitoring my DB
servers(I have one master and 2 slaves - all use the same configuration) -
memory is not going above 12.5GB, so I have a good reserve, also I don't
see any swapping there :(

Thanks,
  Dzmitry


On 8/19/13 11:36 AM, "St=E9phane Schildknecht"
<stephane.schildknecht@postgresql.fr> wrote:

>Le 19/08/2013 10:07, Dzmitry a =E9crit :
>> Hey folks,
>>  I have postgres server running on ubuntu 12,Intel Xeon 8 CPUs 29 GB
>>RAM.
>> With following settings:
>> max_connections =3D 550
>> shared_buffers =3D 12GB
>> temp_buffers =3D 8MB
>> max_prepared_transactions =3D 0
>> work_mem =3D 50MB
>> maintenance_work_mem =3D 1GB
>> fsync =3D on
>> wal_buffers =3D 16MB
>> commit_delay =3D 50
>> commit_siblings =3D 7
>> checkpoint_segments =3D 32
>> checkpoint_completion_target =3D 0.9
>> effective_cache_size =3D 22GB
>> autovacuum =3D on
>> autovacuum_vacuum_threshold =3D 1800
>> autovacuum_analyze_threshold =3D 900
>>
>> I am doing a lot of writes to DB in 40 different threads =AD so every
>>thread=20
>> check if record exists =AD if not =3D> insert record, if exists =3D> upd=
ate
>>record.
>> During this update, my disk IO almost always =AD 100% and sometimes it
>>crash my=20
>> DB with following message:
>>
>> 2013-08-19 03:18:00 UTC LOG:  checkpointer process (PID 28354) was
>>terminated=20
>> by signal 9: Killed
>> 2013-08-19 03:18:00 UTC LOG:  terminating any other active server
>>processes
>> 2013-08-19 03:18:00 UTC WARNING:  terminating connection because of
>>crash of=20
>> another server process
>> 2013-08-19 03:18:00 UTC DETAIL:  The postmaster has commanded this
>>server=20
>> process to roll back the current transaction and exit, because another
>>server=20
>> process exited abnormally and possibly corrupted shared memory.
>> 2013-08-19 03:18:00 UTC HINT:  In a moment you should be able to
>>reconnect to=20
>> the database and repeat your command.
>> 2013-08-19 03:18:00 UTC WARNING:  terminating connection because of
>>crash of=20
>> another server process
>> 2013-08-19 03:18:00 UTC DETAIL:  The postmaster has commanded this
>>server=20
>> process to roll back the current transaction and exit, because another
>>server=20
>> process exited abnormally and possibly corrupted shared memory.
>> 2013-08-19 03:18:00 UTC HINT:  In a moment you should be able to
>>reconnect to=20
>> the database and repeat your command.
>> 2013-08-19 03:18:00 UTC WARNING:  terminating connection because of
>>crash of=20
>> another server process
>> 2013-08-19 03:18:00 UTC DETAIL:  The postmaster has commanded this
>>server=20
>> process to roll back the current transaction and exit, because another
>>server=20
>> process exited abnormally and possibly corrupted shared memory.
>>
>> My DB size is not very big =AD 169GB.
>>
>> Anyone know how can I get rid of DB crash  ?
>>
>>
>> Thanks,
>>   Dzmitry
>>
>
>The fact that the checkpointer was killed -9 let me think the OOMKiller
>has=20
>detected you were out of memory.
>
>Could that be the case?
>
>12GB of shared_buffers on a 29Gb box is too high. You should try to lower
>that=20
>value to 6GB, for instance.
>550*50MB, that is 27GB of RAM that PostgreSQL could try to adress.
>
>I can imagine your system is swapping a lot, and you exhaust swap memory
>before=20
>crash.
>
>Regards,
>
>--=20
>St=E9phane Schildknecht
>Loxodata - Conseil, expertise et formations
>
>
>
>--=20
>Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-admin


--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin