pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Ron <ronljohnsonjr@gmail.com>
To: pgsql-admin@lists.postgresql.org
Subject: Re: Large number of partitions of a table
Date: Tue, 18 Jan 2022 08:47:20 -0600
Message-ID: <d45899c8-187d-4a1d-563b-f5842243e770@gmail.com> (raw)
In-Reply-To: <YeZ3/4TSd+xHAlDl@admin.sibptus.ru>
References: <YeTYA63gEXNePfkv@admin.sibptus.ru>
	<3919502.1642393668@sss.pgh.pa.us>
	<YeUAx56Juri43xGY@admin.sibptus.ru>
	<671340a5-1960-e5dc-1d09-4b7f36cda2c8@gmail.com>
	<YeYr88f3jXHlijU9@admin.sibptus.ru>
	<254601.1642480213@sss.pgh.pa.us>
	<YeZV0QSOly8Pnx5d@admin.sibptus.ru>
	<268771.1642489687@sss.pgh.pa.us>
	<YeZ3/4TSd+xHAlDl@admin.sibptus.ru>

On 1/18/22 2:19 AM, Victor Sudakov wrote:
> Tom Lane wrote:
>> Victor Sudakov <vas@sibptus.ru> writes:
>>> Tom Lane wrote:
>>>> Well, yeah, actually.  An ill-advised query will blow out your backend's
>>>> memory consumption, potentially leading to a SIGKILL from the dreaded OOM
>>>> killer[1] (if you're on Linux), resulting in a backend crash and cluster
>>>> restart.
>>> Why should "SELECT COUNT(*) FROM t" ever consume more than work_mem
>>> even if t has 10000 partitions?
>> Sure, COUNT(*)'s runtime memory consumption is negligible.
>> But you're not thinking about overhead --- specifically,
>>
>> 1. 10000 relcache entries for the base tables.
>>
>> 2. If you have N indexes per table, N*10000 relcache entries for
>> the indexes.  (The planner will probably have looked at all those
>> indexes, even if it didn't find any use for them.)
>>
>> 3. 10000 SeqScan plan nodes and associated rangetable entries,
>>
>> 4. Likewise, 10000 instances of executor per-node state.
>>
>> 5. 10000 lock table entries (both shared and local lock tables).
>>
>> 6. Probably a few per-relation things I didn't think of.
> I see your point about all that query-related stuff. I hope the
> testing of queries in a staging environment should help to detect such
> situations.
>
> What about the system catalogs however? Will the extra 10000
> tables and 500000 indexes negatively impact the performance of the
> system catalogs? Are there any caveats you could think of?

EXPLAIN plans are going to be hilariously gigantic, which means that query 
planning would take a loooong time,  And the query planner (in v12, at 
least) can generate some pretty bad plans in partitioned tables; I bet there 
are edge cases in the QP code that don't work well with 10000 partitions and 
50000 indices.

-- 
Angular momentum makes the world go 'round.





view thread (22+ messages)  latest in thread

Message-ID: <d45899c8-187d-4a1d-563b-f5842243e770@gmail.com>
Permalink:  ../d45899c8-187d-4a1d-563b-f5842243e770@gmail.com/
Also on:    postgresql.org/message-id/d45899c8-187d-4a1d-563b-f5842243e770@gmail.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-admin@postgresql.org
  Cc: ronljohnsonjr@gmail.com, pgsql-admin@lists.postgresql.org
  Subject: Re: Large number of partitions of a table
  In-Reply-To: <d45899c8-187d-4a1d-563b-f5842243e770@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox