From: Tom Lane <tgl@sss.pgh.pa.us>
To: mingjuwu0505@gmail.com
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16260: Repetitive accessing to partitioned table inside transaction causes server process crash
Date: Sun, 16 Feb 2020 17:11:19 -0500
Message-ID: <15692.1581891079@sss.pgh.pa.us> (raw)
In-Reply-To: <16260-ceaef56a5acbf449@postgresql.org>
References: <16260-ceaef56a5acbf449@postgresql.org>
PG Bug reporting form <noreply@postgresql.org> writes:
> Accessing to a partitioned table with large partitions (20K) inside a
> transaction can cause postgres to use large amount of memory (>30GB) and
> ultimately leading to server process crash.
This isn't particularly surprising.
> 2020-02-16 17:01:35.449 CST [22652] LOG: server process (PID 23149) was
> terminated by signal 9: Killed
Notice that this is an external SIGKILL, not an internal-to-postgres
crash. Presumably, the process got killed by Linux's infamous
OOM killer. We typically recommend that Postgres servers be set up
to avoid OOM kills by disabling kernel memory overcommit [1]. However,
the fundamental problem here is that lots of partitions translates to
lots of memory usage. Our current recommendation is to not exceed
"a few thousand" partitions [2].
There has been considerable work already on reducing the memory
consumption for heavily-partitioned queries, and we'll continue to work
on it, but don't hold your breath --- and definitely don't expect that
there will be any back-patched bug fixes for it. Significant forward
progress will probably require major rewrites, on the scale of what's
discussed in [3].
In short: use fewer partitions.
regards, tom lane
[1] https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
[2] https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-BEST-PRAC...
[3] https://www.postgresql.org/message-id/flat/357.1550612935%40sss.pgh.pa.us
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, mingjuwu0505@gmail.com
Subject: Re: BUG #16260: Repetitive accessing to partitioned table inside transaction causes server process crash
In-Reply-To: <15692.1581891079@sss.pgh.pa.us>
References: <16260-ceaef56a5acbf449@postgresql.org> <15692.1581891079@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox