Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1t0QRH-00BMiY-Az for pgsql-general@arkaria.postgresql.org; Mon, 14 Oct 2024 19:09:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1t0QRE-00GGdg-QM for pgsql-general@arkaria.postgresql.org; Mon, 14 Oct 2024 19:09:05 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1t0QRE-00GGdY-4d for pgsql-general@lists.postgresql.org; Mon, 14 Oct 2024 19:09:04 +0000 Received: from mail-yb1-xb32.google.com ([2607:f8b0:4864:20::b32]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1t0QRA-0013P4-FG for pgsql-general@lists.postgresql.org; Mon, 14 Oct 2024 19:09:03 +0000 Received: by mail-yb1-xb32.google.com with SMTP id 3f1490d57ef6-e28fc40fdccso4190302276.1 for ; Mon, 14 Oct 2024 12:09:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joeconway.com; s=google; t=1728932939; x=1729537739; darn=lists.postgresql.org; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:to:subject:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=NSZCeqXjdNxkUXmlw1VOs6mdGZ+hx/BSG3TY+OrgrAU=; b=MhEPM2a6vfDQAKK+I+nqp273FjJyTiO61c4YRmNy20U6W4fqkXShH6uyVm2vIpUvXQ cF/WtrfxaNpHkOYIqdazkKc+fsN1vqMpV9c/7cqHtTvLbU6sXa5m3vR9NjI9Ce8c5nsW 7/rJDrGTft3BiVa3j4zO3YMdFSuk1LzbX/6btkIKn4PkG6SoD5p5WSACtveMme29qqTy FrB0xFx6FlYO9WjCmCAvNrrMC2e6HBQ6HHwHXOK6X84gj96UAzGfXaekvXNkNa7TaSge ZCR/w90na2oUpmSvcNUXgeILPx2c1U+WmWzDyRC3RNrnYKb6nTrE8AV+A1iSUYMB7wEM gEUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728932939; x=1729537739; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:to:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NSZCeqXjdNxkUXmlw1VOs6mdGZ+hx/BSG3TY+OrgrAU=; b=DYMNv7PuV1j0vLaP8ZlYfn03VDVv0fRjh4n6etJtjUy6kQDJm0y2z9EHjLyGBnUqdf sb1tMf/qyOG8dE9J9/QEdSYTRZVbEvgkER388jObiZhT5MybQ1x0HNwmkoT4kAkDnDj2 M6XdKy+d/q4VtsVtG9Z9uznC75fN3nOpjmTk8o8Szr6eHACWuq3BHlYRRQJLONzrHHI1 LKF9KKJQKdvUWB3xOy4SSyxa0Itk2fsW+hmADjucBCPqTISJTaLs/BXiBAWtiZR20QFy 3hQHuWvPZCyOjD1VAf/HA3PBM7t9ICsUXmgWjuNYS3c4OArBwAVpvbIYAfXqHkXhoPr9 b3kA== X-Forwarded-Encrypted: i=1; AJvYcCV6aXaPg0OoUe5JWIjrq5yaSw1TivBMqhWUs73vGLJniiz6hy31AyqAYRh3OB5ARO4FFneIBJLBuG13bqBi@lists.postgresql.org X-Gm-Message-State: AOJu0YyFB7LAg6MtwJUp4TpcT6935KBeYBx8boM++FPbigEPoAZT54+A oFCxf35PZijf8Q6smE5pk+gElGivfJrgk1AE5xvZlMOS7iGVRKG7a7zXOcE38xY= X-Google-Smtp-Source: AGHT+IE5gt9ImivYcqK0O9vjuE4mmsuNBzBzVZ1DFun7NhTk4OwIyfYKq/vDHVh4aX0z626QAlHIOw== X-Received: by 2002:a05:6902:1b81:b0:e20:2789:80ae with SMTP id 3f1490d57ef6-e2931b157c2mr5639923276.10.1728932939436; Mon, 14 Oct 2024 12:08:59 -0700 (PDT) Received: from [192.168.4.41] (162-239-31-113.lightspeed.dybhfl.sbcglobal.net. [162.239.31.113]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e290ee000d4sm2557937276.21.2024.10.14.12.08.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 14 Oct 2024 12:08:58 -0700 (PDT) Message-ID: <37308daa-3860-455f-b9f0-567a982c4554@joeconway.com> Date: Mon, 14 Oct 2024 15:08:58 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Help in dealing with OOM To: Siraj G , pgsql-general@lists.postgresql.org References: Content-Language: en-US From: Joe Conway Autocrypt: addr=mail@joeconway.com; keydata= xsFNBEpXMCsBEADDnXUQzjlyi/cX02Gtdy2CLcroE5CsC7DJKdOBDbfgn0kfiIYoV5JniG4l VyzZUodY8yUAagqLYolh0UkBzs9N+qkm7erde4ypw3jzVQ37BuzIvk3nMUbuDZDgxWqX+nVS sKc+BQ5BpzgCHg48leoRO2ohjvYnUhgH3j2rFZCzaj6qQ7mv+XoxOJmUlVQtG06Jwkk7Vu14 7U9nMMM6hyUKzVnmCphnlcMNo26UyVU70MwFfFJgcI0c5fpp8byN56eD6VJVnufO5WAuEhzE qcrSJR2FAlmM90GBY+6vP29twLDCHuSFvrnujNCx/BvCC/a3/gPvyAFp4JtMm9eXAmq3m/Kw 94nTJXVdcbQeQQDp3KIG7MmWS4lnGvPn8v0CjgNaLvZXFLo1FgmUVsyEq1Lww4iRLa6sbpXJ ESx15UEue1k1YZM9C+4F/o3aeKNsAienjw2EXFzcaxIg/C4P493VMi3Qa8ycVxR5iYhUbYdo DFIUQhbFNsYfrtW/qZAELT3FCYFpZYG01e9Hj+cBrXXgyDDkQ5Lq4mlvmkRvuxn61V6Au4HA 0sJiCox5pM1FvzT+aI8HY1BYaiB9Pl4fhpKgmhhlSuglk9v39S4jmlUIb45iLAUVpeNM6Qjm 69pf5da9sm4aGFa7YlDSKf/WcU7z9ITZxsilOi2n7YJiwG7kTQARAQABzSRKb3NlcGggRSBD b253YXkgPG1haWxAam9lY29ud2F5LmNvbT7CwXoEEwEIACQCGwMCHgECF4AFCwkIBwMFFQoJ CAsFFgIDAQAFAlWTVvUCGQEACgkQMyt+aLaZQ0oPCQ/9HyRewMyvAIJRmoXoLAr8AoFLId6R qBJnNX0Lll0RLZui65aQ0+exwX7aH7TxWR16B2gWX3OmLfGT8XITOoG+zt9zsEpLvNkHchkF T/jyAcbuRj5WX9hamZgMbjXAJeCdlhW+fRA9Upb0w4dgBjqK5OgsqMikASL7t2vogHl9H08j vSoQLW+8wTnSBXBeBTBwB7xLIin5WVivzFHUCrnD2UsjeBIW3fmGdpTAjSxRzG+UPYVwXQ8F FLt7DpEytvLWapmZWMRdj0WZ/Q3SOO/Ed0yFqbzuwKaWcFrQBNeS2Sig+FefBNS98f9Hx7ku H3DW34qX/zSSdDh0jLs7X3PkIgF6BZR2TxaCwHPP9ERDiDaUInC9U7We1iZE1DjW8rLMEVJB hY0ClrrF67pnUKTbcU+uajpPn+2Jl74T0Set/XxpHZ4cezcJuqg31R8vHZgd5cf1WKP0D0pc qiuS02BBFkNCs1jQ+raTWcDuE6F1mUO2nvjUBN9r4y5DUbCNSqLKeAe/aA6JaSDkBpoXKdNS +c4rbzbktWkfUW8EhVlCGzNpy4ezEoVsqV2Ex7fNoxsE2vnSylLT9hycAmYf8ryMvniRZqnD T4JgLenIcQlkhB896T7wApOXfD8OJj1/XFxAfPi6vdlsr81uoxuB4euLp8IyduwLORRUogO9 zmAXG5jOwU0ESlcyJwEQAOkTBb9yDhJbMUgvhM11rZwT5tm4Y9TqtEHn0Zy3t9g7bdFFpMva v/KENd3oAtLFpMDf+H3AggFk4ftUwJwiVgJ88ilvCynJUGXiuYIaexY4DLgn4xpnuiEpYEFV dWnlw7dWVTc62exfqIz9bSWRzwfBCY9ruYGEb4RDPDSNSAVyI7sxHzef2asiYxIcxrTrw5Vu gWNlPZcV5/EJ6PUvATjBF2TBkXV7KOciQng2tsQGrGMkY5mduNqwpuh6zfPcVF8LeObe96wv 5ZhPRpO79nef7hnK2lJogp3JIo558Jlbz9WHtQEMZR85+bUhtI825QyNAFz3Jrn7NMgvDikc 2OrWo7YMgMC5hDSWVFqA6/EQCNnDWGABWgeYHZFpnPwsvUWIYdhSilUuj/Tuzvz9ZmucFNbQ bauDQw6VQ38ofGnoYDZFJsGncprB8dBi4tDrIQ+1RlIh6C2Z/eMipqJOT26+spluTjouvnKT 0S5yOgyX0PjbsysgwQdCGNJLHOjhHbSpSmOLaduV3CQo/0+DHT/TBjYfIXjTWouY9TkGxG4e NrxU0u2xAy5bMqOPmsFdjLTWlQUlF/fTMhB54XwI3FHWgnSnXZzStDTmTebLNdT/ftgliAzA 81uMj49j0exv731/v+7udLA1bV8gnZ01zQCASDpWiRQR3fgwcugSUqgRABEBAAHCwV8EGAEI AAkFAkpXMicCGwwACgkQMyt+aLaZQ0pwAQ//bjcWnZg/jjRQ9gbZUGMqniItZYRglBMKIqt4 Fia379JmHwTvavnFkJ8XMZ56UB0FIrgS+sUkRH6cPRQR+7Qi392LD021DXgSsz9CwFHjFyBG HwLEOTRcfYQbtJy0shHDJB4aQTOX3ERDH1PsvJNuevmQMzS0DWFav9+xMz9rKP4N+HffoBIZ E0C1xIE43nD4eLsbycte9sVIrmlNuUti3qUxJAQw8HwfJ6ZbBInHxquApR16uD1u99o6Xlnd FrDlY22tRmHCM0bR81GfGNdcU3Uo+rG/R/k4qa7s9/dgKvMbyH3fHhp/ceKag80Xo8IFurRl 0ZJP3sHJ2QDHCVLat7jRZ+43hi1WlIhFbrgn6IyI0i7XR/W8JjrC5MsKq4TUwGH077sU/kcH YebVJZRbUUst2hAGHDFVBcG12qoKf+ltL9qXJc1y7BGeCoUW6QjOpljpq6ZL4FQUsM0RSRjs 5egE3szPcIf5SyPK6WDOApoAq6M7BBFMGDZwEylYMtr0YekA1u86UA9D2xwLHEbBBp/uiby1 c9JbPJ1Pn8zJP8WZNeRw4Q9TtqVK09+oLirMUSpIDd6KdZ1VgRxOK2re7tjDvkVuYsSrsiJ+ 1iJNEnp9iK0ok0DlJpSCe6KhkxpaTdeoWMXdKuJWec0NIqoAd54ZgBPnr+UPxTixgPq/p6Q= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 10/14/24 14:37, Siraj G wrote: > This is from the OS log (/var/log/kern.log): > > oom- > kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.sli      ce/system-postgresql.slice/postgresql@12-main.service,task=postgres,pid=2334587,uid=114 >   494 Oct 14 09:58:10 gce-k12-prod-as1-erp-pg-secondary kernel: > [6905020.514569] Out of memory: Killed process 2334587 (postgres) total- > vm:26349584kB, anon-rss:3464kB, file-rss:0kB, shmem-rs >  s:21813032kB, UID:114 pgtables:49024kB oom_score_adj:0 1. Do you happen to have swap disabled? If so, don't do that. 2. Does the postgres cgroup have memory.limit (cgroup v1) or memory.max (cgroup v2) set? 3. If #2 answer is no, have you followed the documented guidance here (in particular vm.overcommit_memory=2): https://www.postgresql.org/docs/12/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com