Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1j3S8U-00025v-4O for pgsql-bugs@arkaria.postgresql.org; Sun, 16 Feb 2020 22:11:34 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1j3S8S-0002gt-0z for pgsql-bugs@arkaria.postgresql.org; Sun, 16 Feb 2020 22:11:32 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1j3S8R-0002dG-Oy for pgsql-bugs@lists.postgresql.org; Sun, 16 Feb 2020 22:11:31 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j3S8J-000559-Nh for pgsql-bugs@lists.postgresql.org; Sun, 16 Feb 2020 22:11:30 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id 01GMBJ7x015693; Sun, 16 Feb 2020 17:11:19 -0500 From: Tom Lane To: mingjuwu0505@gmail.com cc: pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16260: Repetitive accessing to partitioned table inside transaction causes server process crash In-reply-to: <16260-ceaef56a5acbf449@postgresql.org> References: <16260-ceaef56a5acbf449@postgresql.org> Comments: In-reply-to PG Bug reporting form message dated "Sun, 16 Feb 2020 09:15:40 +0000" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <15691.1581891079.1@sss.pgh.pa.us> Content-Transfer-Encoding: quoted-printable Date: Sun, 16 Feb 2020 17:11:19 -0500 Message-ID: <15692.1581891079@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk PG Bug reporting form writes: > Accessing to a partitioned table with large partitions (20K) inside a > transaction can cause postgres to use large amount of memory (>30GB) and > ultimately leading to server process crash. This isn't particularly surprising. > 2020-02-16 17:01:35.449 CST [22652] LOG: server process (PID 23149) was > terminated by signal 9: Killed Notice that this is an external SIGKILL, not an internal-to-postgres crash. Presumably, the process got killed by Linux's infamous OOM killer. We typically recommend that Postgres servers be set up to avoid OOM kills by disabling kernel memory overcommit [1]. However, the fundamental problem here is that lots of partitions translates to lots of memory usage. Our current recommendation is to not exceed "a few thousand" partitions [2]. There has been considerable work already on reducing the memory consumption for heavily-partitioned queries, and we'll continue to work on it, but don't hold your breath --- and definitely don't expect that there will be any back-patched bug fixes for it. Significant forward progress will probably require major rewrites, on the scale of what's discussed in [3]. In short: use fewer partitions. regards, tom lane [1] https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-ME= MORY-OVERCOMMIT [2] https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PART= ITIONING-DECLARATIVE-BEST-PRACTICES [3] https://www.postgresql.org/message-id/flat/357.1550612935%40sss.pgh.pa= .us