Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hxGXt-0007mW-Be for pgsql-performance@arkaria.postgresql.org; Mon, 12 Aug 2019 20:03:57 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1hxGXq-0004sf-L3 for pgsql-performance@arkaria.postgresql.org; Mon, 12 Aug 2019 20:03:54 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hxGXq-0004p1-13 for pgsql-performance@lists.postgresql.org; Mon, 12 Aug 2019 20:03:54 +0000 Received: from mail-lj1-x244.google.com ([2a00:1450:4864:20::244]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hxGXn-000587-An for pgsql-performance@lists.postgresql.org; Mon, 12 Aug 2019 20:03:53 +0000 Received: by mail-lj1-x244.google.com with SMTP id e27so4293222ljb.7 for ; Mon, 12 Aug 2019 13:03:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:cc:from:subject:date:importance :in-reply-to:references; bh=5tNCfLIR1a2Geg2DF3IsGzJwY1RELRo8wDL8tERS7mg=; b=YHO2AHMNtBdQ2cALg3+0EBcLwwbAZT1CmJWr4PjpOXDxBAnFdDYoU02rHWykk0Nbp7 rcg3kXgMKbK+QOmZ76chN4HYjvMDtMQvfG0enkMOBDNFCCUVPNe3N0UrxhRJKr8rX7Lh 4vOkioFZ6v05+KPh6DdyUqrKuwygvrhStCMXV4AiV/U1JzXA1DaOekbWJtj3jSlUDSkJ 7rVfhgIOXUxaC4Sn3ETWF2HzDiuff1iz5f7f2Vf13YnHQM895+IR7hjKym9/rO6Rht9O 6HHUDxdlFWS73yUgZVGUpnMCgUpPETHvYwPPsabOgIj5ZaMx830RsQIZwscrEzxCDDct UB0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:cc:from:subject:date :importance:in-reply-to:references; bh=5tNCfLIR1a2Geg2DF3IsGzJwY1RELRo8wDL8tERS7mg=; b=a+YWiaOEVX/jTBWQQy1ECsqSutQbS7JhQ8jLy6wdR1SgiVw4aH7UayDob5zJ8fcEiz jzULZBNcONOlL7ixdjIyx1WUAr7/TLiN+/GBKw8bkumb1SbGQHEKxF1+Q0uGrMugwQfn 2K2v5iqqCRVKVoX2trCI5hx92k8XIk52miGTdmVZY9nBgluADAOLc7ZKzPs10ojAU5uI xXTNVDJM/VP5WmRmglOpyFV7emazzbUjtKZhGlMz6oOhKcw2LpkJljGDyphtVhLfAgyP NrIu/tUXsFL2xgJ/ulUSw0HCf2I1eWGNnp8TgP5ehLnYKMgkjajkJtbtnlRTR/RtKOIr 8dZg== X-Gm-Message-State: APjAAAWV21j+uzDt5J5bPXM+oq0dncJQg2wyoUM9solp+lOLGADkSx/c w/lw5UKz9TUPMjxBM0OUd1U= X-Google-Smtp-Source: APXvYqwnr8vX9xKQV/fBFgmPd4knFmKZp5q/Qww+qbJHkqMl+5JhHtOl1hFc3R60s5XfCYNUK67KTQ== X-Received: by 2002:a2e:8183:: with SMTP id e3mr19223888ljg.97.1565640227575; Mon, 12 Aug 2019 13:03:47 -0700 (PDT) Received: from ?IPv6:::ffff:192.168.1.2? ([212.180.219.12]) by smtp.gmail.com with ESMTPSA id d21sm19257640lfc.73.2019.08.12.13.03.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Aug 2019 13:03:46 -0700 (PDT) Message-ID: <5d51c622.1c69fb81.4032d.34d9@mx.google.com> MIME-Version: 1.0 To: MichaelDBA , Michael Lewis Cc: =?utf-8?Q?Piotr_W=C5=82odarczyk?= , "pgsql-performance@lists.postgresql.org" From: =?utf-8?Q?Piotr_W=C5=82odarczyk?= Subject: ODP: Planner performance in partitions Date: Mon, 12 Aug 2019 22:03:48 +0200 Importance: normal X-Priority: 3 In-Reply-To: <458dfc43-b097-29f3-4e0e-857b9c30b427@sqlexec.com> References: <5675f9f771f04d9f8b5cffe16f4a4175@EX13MBX02.gb.ad.corp> <458dfc43-b097-29f3-4e0e-857b9c30b427@sqlexec.com> Content-Type: multipart/alternative; boundary="_2FF31A7E-56B7-4227-9540-C3028176FD63_" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --_2FF31A7E-56B7-4227-9540-C3028176FD63_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" @Michael Lewis: I know documentation. I'm just considerations about possibl= e performance tricks in current production version. I've tested this on V12= on another computer and I can say that I'm impressed. I've checked on 1200= partitions and times are: PostgreSQL11.5: =E2=80=A2 select on main partition (public.book): planner: 60ms, execution:= 5ms =E2=80=A2 select on=C2=A0partitioned table (public.book_1-1000): planner: 2= .7 ms, execution: 2,4 ms PostgreSQL 12B3: =E2=80=A2 select on main partition (public.book): planner: 2,5ms , executio= n: 1,2ms =E2=80=A2 select on=C2=A0partitioned table (public.book_1-1000): planner: 2= .5 ms, execution: 1,2 ms So looking at above results we have two options: =E2=80=A2 Wait for 12.0 stable version =F0=9F=98=89 =E2=80=A2 Wait for patches to 11 =E2=80=93 PostgreSQL Team: can You do this= ? =F0=9F=98=8A Pozdrawiam, Piotr W=C5=82odarczyk Od: MichaelDBA Wys=C5=82ano: poniedzia=C5=82ek, 12 sierpnia 2019 21:25 Do: Michael Lewis DW: Piotr W=C5=82odarczyk; pgsql-performance@lists.postgresql.org Temat: Re: Planner performance in partitions Queries against tables with a lot of partitions (> 1000) start to incur an = increasing planning time duration even with the current version, V11.=C2=A0= V12 purportedly has fixed this problem, allowing thousands of partitioned = tables without a heavy planning cost.=C2=A0 Can't seem to find the threads = on this topic, but there are out there.=C2=A0 I personally noted a gigantic= increase in planning time once I got past 1500 partitioned tables in V11. On another note, hopefully they have fixed runtime partition pruning in V12= since V11 introduced it but some query plans don't use it, so you have to = reconstruct some queries to sub queries to make it work correctly. Regards, Michael Vitale Michael Lewis wrote on 8/12/2019 3:05 PM: "It is also important to consider the overhead of partitioning during query= planning and execution. The query planner is generally able to handle part= ition hierarchies with up to a few hundred partitions fairly well, provided= that typical queries allow the query planner to prune all but a small numb= er of partitions. Planning times become longer and memory consumption becom= es higher as more partitions are added." (emphasis added) --https://www.postgresql.org/docs/current/ddl-partitioning.html --_2FF31A7E-56B7-4227-9540-C3028176FD63_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

@Michael Lewis: I know documentation. I'm just considerations about = possible performance tricks in current production version. I've tested this= on V12 on another computer and I can say that I'm impressed. I've checked = on 1200 partitions and times are:

 

PostgreSQL11.5:

  • select on main partition (public.book): plan= ner: 60ms, execution: 5ms
  • select on partitioned table (public.= book_1-1000): planner: 2.7 ms, execution: 2,4 ms

PostgreSQL 12B3:

  • select on main partition (public.book): planner: 2,5ms , = execution: 1,2ms
  • select on partitioned table (publi= c.book_1-1000): planner: 2.5 ms, execution: 1,2 ms

 

So looking at ab= ove results we have two options:

 

Queries against tables with a lot of partitions (> 1000) st= art to incur an increasing planning time duration even with the current ver= sion, V11.  V12 purportedly has fixed this problem, allowing thousands= of partitioned tables without a heavy planning cost.  Can't seem to f= ind the threads on this topic, but there are out there.  I personally = noted a gigantic increase in planning time once I got past 1500 partitioned= tables in V11.

On another note, hopefully they have fixed runtime p= artition pruning in V12 since V11 introduced it but some query plans don't = use it, so you have to reconstruct some queries to sub queries to make it w= ork correctly.

Regards,
Michael Vitale


Michael Lewis w= rote on 8/12/2019 3:05 PM:

"It is= also important to consider the overhead of partitioning during query plann= ing and execution. The query planner is generally able to handle partition = hierarchies with up to a few hundred partitions fairly well, = provided that typical queries allow the query planner to prune all but a sm= all number of partitions. Planning times become longer and memory consumpti= on becomes higher as more partitions are added." (emphasis added)=

 

&nb= sp;

= --_2FF31A7E-56B7-4227-9540-C3028176FD63_--