pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tom Lane <tgl@sss.pgh.pa.us>
To: mingjuwu0505@gmail.com
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16260: Repetitive accessing to partitioned table inside transaction causes server process crash
Date: Sun, 16 Feb 2020 17:11:19 -0500
Message-ID: <15692.1581891079@sss.pgh.pa.us> (raw)
In-Reply-To: <16260-ceaef56a5acbf449@postgresql.org>
References: <16260-ceaef56a5acbf449@postgresql.org>

PG Bug reporting form <noreply@postgresql.org> writes:
> Accessing to a partitioned table with large partitions (20K) inside a
> transaction can cause postgres to use large amount of memory (>30GB) and
> ultimately leading to server process crash.

This isn't particularly surprising.

> 2020-02-16 17:01:35.449 CST [22652] LOG:  server process (PID 23149) was
> terminated by signal 9: Killed

Notice that this is an external SIGKILL, not an internal-to-postgres
crash.  Presumably, the process got killed by Linux's infamous
OOM killer.  We typically recommend that Postgres servers be set up
to avoid OOM kills by disabling kernel memory overcommit [1].  However,
the fundamental problem here is that lots of partitions translates to
lots of memory usage.  Our current recommendation is to not exceed
"a few thousand" partitions [2].

There has been considerable work already on reducing the memory
consumption for heavily-partitioned queries, and we'll continue to work
on it, but don't hold your breath --- and definitely don't expect that
there will be any back-patched bug fixes for it.  Significant forward
progress will probably require major rewrites, on the scale of what's
discussed in [3].

In short: use fewer partitions.

			regards, tom lane

[1] https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT
[2] https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-BEST-PRAC...
[3] https://www.postgresql.org/message-id/flat/357.1550612935%40sss.pgh.pa.us





view thread (2+ messages)

Message-ID: <15692.1581891079@sss.pgh.pa.us>
Permalink:  ../15692.1581891079@sss.pgh.pa.us/
Also on:    postgresql.org/message-id/15692.1581891079@sss.pgh.pa.us

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: tgl@sss.pgh.pa.us, mingjuwu0505@gmail.com
  Subject: Re: BUG #16260: Repetitive accessing to partitioned table inside transaction causes server process crash
  In-Reply-To: <15692.1581891079@sss.pgh.pa.us>
  References: <16260-ceaef56a5acbf449@postgresql.org> <15692.1581891079@sss.pgh.pa.us>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox