Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id CF2A8632EA2 for ; Fri, 7 May 2010 12:42:50 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 51359-09 for ; Fri, 7 May 2010 15:42:42 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail.sdb.it (mail.sdb.it [212.141.11.2]) by mail.postgresql.org (Postfix) with ESMTP id B2F66632337 for ; Fri, 7 May 2010 12:42:41 -0300 (ADT) Received: by mail.sdb.it (Postfix, from userid 500) id ADF448100; Fri, 7 May 2010 17:42:40 +0200 (CEST) Received: from geronimo.sdb.it (geronimo.sdb.it [10.101.1.20]) by mail.sdb.it (Postfix) with ESMTP id A7DF68100; Fri, 7 May 2010 17:42:40 +0200 (CEST) Received: from mailtech.sdb.it (sdbit-srv01.sdb.it [10.118.122.250]) by geronimo.sdb.it (Postfix) with ESMTP id 767B53808A1; Fri, 7 May 2010 17:42:29 +0200 (CEST) Received: from [10.118.101.88] (silviob.sdb.it [10.118.101.88]) by mailtech.sdb.it (Postfix) with ESMTP id 8AE1B58ABA; Fri, 7 May 2010 17:42:40 +0200 (CEST) Message-ID: <4BE434EE.2050208@tech.sdb.it> Date: Fri, 07 May 2010 17:42:38 +0200 From: Silvio Brandani User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Lacey Powers Cc: pgsql-admin@postgresql.org Subject: Re: postgres invoked oom-killer References: <4BE4232B.3060401@tech.sdb.it> <4BE42E82.3070801@commandprompt.com> In-Reply-To: <4BE42E82.3070801@commandprompt.com> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Private-Material: Please read notice X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.81 tagged_above=-10 required=5 tests=BAYES_50=0.8, T_FRT_FOLLOW1=0.01, T_FRT_FOLLOW2=0.01, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201005/100 X-Sequence-Number: 34573 Lacey Powers ha scritto: > Silvio Brandani wrote: >> We have a postgres 8.3.8 on linux >> >> We get following messages int /var/log/messages: >> >> May 6 22:31:01 pgblade02 kernel: postgres invoked oom-killer: >> gfp_mask=0x201d2, order=0, oomkilladj=0 >> May 6 22:31:01 pgblade02 kernel: >> May 6 22:31:01 pgblade02 kernel: Call Trace: >> May 6 22:31:19 pgblade02 kernel: [] >> out_of_memory+0x8e/0x2f5 >> May 6 22:31:19 pgblade02 kernel: [] >> __alloc_pages+0x22b/0x2b4 >> May 6 22:31:19 pgblade02 kernel: [] >> __do_page_cache_readahead+0x95/0x1d9 >> May 6 22:31:19 pgblade02 kernel: [] >> __wait_on_bit_lock+0x5b/0x66 >> May 6 22:31:19 pgblade02 kernel: [] >> :dm_mod:dm_any_congested+0x38/0x3f >> May 6 22:31:19 pgblade02 kernel: [] >> filemap_nopage+0x148/0x322 >> May 6 22:31:19 pgblade02 kernel: [] >> __handle_mm_fault+0x1f8/0xdf4 >> May 6 22:31:19 pgblade02 kernel: [] >> do_page_fault+0x4b8/0x81d >> May 6 22:31:19 pgblade02 kernel: [] >> thread_return+0x0/0xeb >> May 6 22:31:19 pgblade02 kernel: [] >> error_exit+0x0/0x84 >> May 6 22:31:27 pgblade02 kernel: >> May 6 22:31:28 pgblade02 kernel: Mem-info: >> May 6 22:31:28 pgblade02 kernel: Node 0 DMA per-cpu: >> May 6 22:31:28 pgblade02 kernel: cpu 0 hot: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 0 cold: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 1 hot: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 1 cold: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 2 hot: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 2 cold: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 3 hot: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: cpu 3 cold: high 0, batch 1 used:0 >> May 6 22:31:28 pgblade02 kernel: Node 0 DMA32 per-cpu: >> May 6 22:31:28 pgblade02 kernel: cpu 0 hot: high 186, batch 31 used:27 >> May 6 22:31:29 pgblade02 kernel: cpu 0 cold: high 62, batch 15 used:54 >> May 6 22:31:29 pgblade02 kernel: cpu 1 hot: high 186, batch 31 used:23 >> May 6 22:31:29 pgblade02 kernel: cpu 1 cold: high 62, batch 15 used:49 >> May 6 22:31:29 pgblade02 kernel: cpu 2 hot: high 186, batch 31 used:12 >> May 6 22:31:29 pgblade02 kernel: cpu 2 cold: high 62, batch 15 used:14 >> May 6 22:31:29 pgblade02 kernel: cpu 3 hot: high 186, batch 31 used:50 >> May 6 22:31:29 pgblade02 kernel: cpu 3 cold: high 62, batch 15 used:60 >> May 6 22:31:29 pgblade02 kernel: Node 0 Normal per-cpu: >> May 6 22:31:29 pgblade02 kernel: cpu 0 hot: high 186, batch 31 used:5 >> May 6 22:31:29 pgblade02 kernel: cpu 0 cold: high 62, batch 15 used:48 >> May 6 22:31:29 pgblade02 kernel: cpu 1 hot: high 186, batch 31 used:11 >> May 6 22:31:29 pgblade02 kernel: cpu 1 cold: high 62, batch 15 used:39 >> May 6 22:31:29 pgblade02 kernel: cpu 2 hot: high 186, batch 31 used:14 >> May 6 22:31:29 pgblade02 kernel: cpu 2 cold: high 62, batch 15 used:57 >> May 6 22:31:29 pgblade02 kernel: cpu 3 hot: high 186, batch 31 used:94 >> May 6 22:31:29 pgblade02 kernel: cpu 3 cold: high 62, batch 15 used:36 >> May 6 22:31:29 pgblade02 kernel: Node 0 HighMem per-cpu: empty >> May 6 22:31:29 pgblade02 kernel: Free pages: 41788kB (0kB >> HighMem) >> May 6 22:31:29 pgblade02 kernel: Active:974250 inactive:920579 >> dirty:0 writeback:0 unstable:0 free:10447 slab:11470 mapped-file:985 >> mapped-anon:1848625 pagetables:111027 >> May 6 22:31:29 pgblade02 kernel: Node 0 DMA free:11172kB min:12kB >> low:12kB high:16kB active:0kB inactive:0kB present:10816kB >> pages_scanned:0 all_unreclaimable? yes >> May 6 22:31:29 pgblade02 kernel: lowmem_reserve[]: 0 3254 8052 8052 >> May 6 22:31:29 pgblade02 kernel: Node 0 DMA32 free:23804kB >> min:4636kB low:5792kB high:6952kB active:1555260kB inactive:1566144kB >> present:3332668kB pages_scanned:35703257 all_unreclaimable? yes >> May 6 22:31:29 pgblade02 kernel: lowmem_reserve[]: 0 0 4797 4797 >> May 6 22:31:29 pgblade02 kernel: Node 0 Normal free:6812kB >> min:6836kB low:8544kB high:10252kB active:2342332kB >> inactive:2115836kB present:4912640kB pages_scanned:10165709 >> all_unreclaimable? yes >> May 6 22:31:29 pgblade02 kernel: lowmem_reserve[]: 0 0 0 0 >> May 6 22:31:29 pgblade02 kernel: Node 0 HighMem free:0kB min:128kB >> low:128kB high:128kB active:0kB inactive:0kB present:0kB >> pages_scanned:0 all_unreclaimable? no >> May 6 22:31:29 pgblade02 kernel: lowmem_reserve[]: 0 0 0 0 >> May 6 22:31:29 pgblade02 kernel: Node 0 DMA: 3*4kB 5*8kB 3*16kB >> 6*32kB 4*64kB 3*128kB 0*256kB 0*512kB 2*1024kB 0*2048kB 2*4096kB = >> 11172kB >> May 6 22:31:29 pgblade02 kernel: Node 0 DMA32: 27*4kB 0*8kB 1*16kB >> 0*32kB 2*64kB 4*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 5*4096kB = >> 23804kB >> May 6 22:31:29 pgblade02 ker >> if it asks for more memory than is actually available. >> nel: Node 0 Normal: 21*4kB 9*8kB 26*16kB 3*32kB 6*64kB 5*128kB >> 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 6812kB >> May 6 22:31:29 pgblade02 kernel: Node 0 HighMem: empty >> May 6 22:31:29 pgblade02 kernel: Swap cache: add 71286821, delete >> 71287152, find 207780333/216904318, race 1387+10506 >> May 6 22:31:29 pgblade02 kernel: Free swap = 0kB >> May 6 22:31:30 pgblade02 kernel: Total swap = 8388600kB >> May 6 22:31:30 pgblade02 kernel: Free swap: 0kB >> May 6 22:31:30 pgblade02 kernel: 2293759 pages of RAM >> May 6 22:31:30 pgblade02 kernel: 249523 reserved pages >> May 6 22:31:30 pgblade02 kernel: 56111 pages shared >> May 6 22:31:30 pgblade02 kernel: 260 pages swap cached >> May 6 22:31:30 pgblade02 kernel: Out of memory: Killed process 29076 >> (postgres). >> >> >> We get folloowing errors in the postgres log: >> >> A couple of time: >> 2010-05-06 22:26:28 CEST [23001]: [2-1] WARNING: worker took too >> long to start; cancelled >> Then: >> 2010-05-06 22:31:21 CEST [29059]: [27-1] LOG: system logger process >> (PID 29076) was terminated by signal 9: Killed >> Finally: >> 2010-05-06 22:50:20 CEST [29059]: [28-1] LOG: background writer >> process (PID 22999) was terminated by signal 9: Killed >> 2010-05-06 22:50:20 CEST [29059]: [29-1] LOG: terminating any other >> active server processes >> >> Any help higly apprecaited, >> >> --- >> > > Hello Silvio, > > Is this machine dedicated to PostgreSQL? > > If so, I'd recommend adding these two parameters to your sysctl.conf > > vm.overcommit_memory = 2 > vm.overcommit_ratio = 0 > > So that OOMKiller is turned off. > > PostgreSQL should gracefully degrade if a malloc() fails because it > asks for too much memory. > > Hope that helps. =) > > Regards, > Lacey > > Thanks a lot, yes the server is dedicated to PostgreSQL. Could be a bug of PostgreSQL the fact that the system went Out of Memory?? Wath can be the cause of it? Regards, Silvio -- Silvio Brandani Infrastructure Administrator SDB Information Technology Phone: +39.055.3811222 Fax: +39.055.5201119 --- Utilizziamo i dati personali che la riguardano esclusivamente per nostre finalitą amministrative e contabili, anche quando li comunichiamo a terzi. Informazioni dettagliate, anche in ordine al Suo diritto di accesso e agli altri Suoi diritti, sono riportate alla pagina http://www.savinodelbene.com/news/privacy.html Se avete ricevuto questo messaggio per errore Vi preghiamo di ritornarlo al mittente eliminandolo assieme agli eventuali allegati, ai sensi art. 616 codice penale http://www.savinodelbene.com/codice_penale_616.html L'Azienda non si assume alcuna responsabilitą giuridica qualora pervengano da questo indirizzo messaggi estranei all'attivitą lavorativa o contrari a norme. --