Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1smI3F-004cZW-Vh for pgsql-advocacy@arkaria.postgresql.org; Thu, 05 Sep 2024 19:21:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1smI3D-00GDkx-Jz for pgsql-advocacy@arkaria.postgresql.org; Thu, 05 Sep 2024 19:21:52 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1smI3D-00GDkn-8b for pgsql-advocacy@lists.postgresql.org; Thu, 05 Sep 2024 19:21:51 +0000 Received: from mail-oo1-xc30.google.com ([2607:f8b0:4864:20::c30]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1smI39-000Kej-Kz for pgsql-advocacy@lists.postgresql.org; Thu, 05 Sep 2024 19:21:51 +0000 Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-5e1972a9622so747048eaf.0 for ; Thu, 05 Sep 2024 12:21:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725564106; x=1726168906; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=BA/bbzowJduuC8SBRbal3HDdiFTz5wTQPHdI+ejbPA0=; b=XvAAb2I/Z4Oc6zC4kqF42MGLhPPgoWi7WKmOX3by4/vzvaN4/8JLLVs/bCR+rMYzM1 mkC8/oYptBHCwP28x4lh0V+edLOT9NsLyj+aZEX+ealIM8D0xtVzY8yqDmr9HfT3ohtJ y41rQ34L2llKObj+2hjg7oYocuUbRemQEo6B3tHDOwC0RRwpvED4RF5jcsR1Qg8mk8Vq 3Eh79TQuPDNf/DiFT10Z7cZXk/2RHW3LQ5zO11ov99bGBGP9PnHtgTMamF06+Da1kMfu j/AYDtgbHk5So/G4uZRpu49IsobXFtb0y0G7nIc6/2/OeJk6aqJEGtRbRXpjV6NSw1Ki PZcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725564106; x=1726168906; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=BA/bbzowJduuC8SBRbal3HDdiFTz5wTQPHdI+ejbPA0=; b=kruE8EZWtLEI4tfJ5fMk2hGluyHm4u9tUF0aGUndy2UpMIdPquHVl/FKfgOd1iEs71 0fwbocYMRhWBBU7rfiUIvxKccXwByKVM5BDBP48h3NS+afAov1VBLREnvMyJ9dXXVIk2 WBqf6eFmtBUJZUlXmH+T9Ixk+I593vIwz7pFJF2+ZfyjQU1rP7nxLlIAMcr4d5I8EsCm 22iMNvw0axgdLMdukRdEBfXh3iNN+5A9AidqOsOkxYeEeaiyWIpl1FraUneGsf0fM0F2 gRcNktcG8wPVuDbLrcqnjK+2+ymwDTsfxTBRVkAXTVyg4cNyLveXMSbY3ApHwmebVd7x mxeg== X-Forwarded-Encrypted: i=1; AJvYcCXdZLC2DOY/tckyEg6j6ZwHC+Gtpl5sLtAcGPSq8Sc/foJ0lZZX3eI0IYq6sGnT6PsXDK+ow8rEj1SnjgAlxQ==@lists.postgresql.org X-Gm-Message-State: AOJu0YxbCMr4f1wJqP+cU87MjcTDePf6b0LyG1yj+Tl/giGq/t6JE0EL 617nX0VDdh4NwpHR+9nWbE75yFBJUY9sjK8TfEDTK+fbveuZfVspdNrkYiZ9QLawDoeBk8u4/61 zKXySEdx1IGa1e6tFIiulGm1y8l4= X-Google-Smtp-Source: AGHT+IHWHSqDBXX6SSeVLeF7eEVYvXM4XbeqE4LgL0gS61bqUQEbtdnG9R3wxPEOFyZQZYFEUK2NfJ6exERe2rieRJQ= X-Received: by 2002:a05:6808:1385:b0:3d9:3f72:715e with SMTP id 5614622812f47-3df05c461b9mr31166051b6e.10.1725564105766; Thu, 05 Sep 2024 12:21:45 -0700 (PDT) MIME-Version: 1.0 References: <81af1d33-1abb-4ddf-a7e9-859db718281c@postgresql.org> <906be4b7-9066-43db-8047-d34d85dff3ad@eisentraut.org> In-Reply-To: From: Andrew Atkinson Date: Thu, 5 Sep 2024 14:21:09 -0500 Message-ID: Subject: Re: PostgreSQL 17 release announcement draft To: Robert Haas Cc: Peter Eisentraut , "Jonathan S. Katz" , PostgreSQL Advocacy Content-Type: multipart/alternative; boundary="0000000000008fbf900621643390" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000008fbf900621643390 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'll send one more bigger email. I also have made these as comments in a google doc I'm happy to add anyone to if they're interested. Let me know. > PostgreSQL 17 extends its performance gains to query execution. Planner statistics help PostgreSQL to determine the best way to search for data, and PostgreSQL 17 can now use planner statistics and the sort order [common table expressions](https://www.postgresql.org/docs/17/queries-with.html ) ([`WITH` queries](https://www.postgresql.org/docs/17/queries-with.html )) to further speed up queries. Was that meant to be the sort order "from" CTEs? Maybe a missing "From"? > to support more expressive JSON processing capabilities and implementation of the SQL/JSON standard. Was that meant to be something like "and to expand the implementation" of or "an expanded implementation"? > "with an emphasis of converting JSON data to a native PostgreSQL data type" Was that meant to be an emphasis "*on" and not "of"* ? > For the merge section, here's a proposed reworked version: PostgreSQL 17 provides expanded control over `MERGE` behavior, including a new `merge_action` function that describes how a row was modified. PostgreSQL 17 provides new capabilities and performance improvements for bulk data loading and exporting. This includes improving exports of large rows by up to 2x using the [`COPY`]( https://www.postgresql.org/docs/17/sql-copy.html ) command, and performance improvements for cases when the source and destination encodings match. A new option `ON_ERROR` was added which can be used to allow imports to continue even when errors happen for inserts. > For the predefined role, something to consider would be using action verbs. I don't know if this seems too inconsistent though. It makes it easier to write. For example: Use the new `pg_maintain` [predefined role]( https://www.postgresql.org/docs/17/predefined-roles.html ) to grant permissions for maintenance operations. > For the EXPLAIN enhancements: PostgreSQL 17 [`EXPLAIN`]( https://www.postgresql.org/docs/17/sql-explain.html ) now shows the time spent for I/O block reads and writes, and includes two new options: `SERIALIZE` and `MEMORY`, useful for seeing the time spent in data conversion for network transmission, and how much memory was used. That's all I've got. This is an exciting release! Nice work! On Thu, Sep 5, 2024 at 1:44=E2=80=AFPM Andrew Atkinson wrote: > Additional chunk: > > Proposed: > PostgreSQL 17 improves the performance of queries with `IN` clauses that > use [B-tree]( > https://www.postgresql.org/docs/17/indexes-types.html#INDEXES-TYPES-BTREE= ) > indexes, the default index method in PostgreSQL). Support for parallel > index builds were added for [BRIN]( > https://www.postgresql.org/docs/17/brin.html) indexing. > > PostgreSQL 17 leverages the constraints you've defined. The query planner > removes redundant `IS NOT NULL` statements when a column has a `NOT NULL` > constraint, and skips over `IS NULL` clauses for columns with an `IS NOT > NULL` constraint. > > > On Thu, Sep 5, 2024 at 11:58=E2=80=AFAM Andrew Atkinson > wrote: > >> The write operations seem to be under-sold here! These seem very excitin= g >> and would be beneficial to all Postgres databases I've worked on. >> >> Is this a fair way to reduce the three items mentioned, as much as >> possible? I'm not proposing they change, but just sort of pushing on >> building down the items tied to their benefit to the end user as much as >> possible. Possibly shortening them and making them punchier will help. >> >> Faster Write throughput: >> - "Fewer WAL locks shorter lock lengths" (check this, I'm making it up, >> but I'm wondering if it's fewer WAL locks, or the locks are held for les= s >> time), thus better throughput >> - "Faster Sequential scans" - is the benefit that they're faster? >> - "Faster ANALYZE" - is it that ANALYZE runs faster on 17 vs. the same >> operation on 16? >> >> The last point "Allowing extensions to be integrated further." I didn't >> grab on to as much. I'm wondering if it's something like "new write >> operation APIs are now available for extension creators" or something li= ke >> that? >> >> On Thu, Sep 5, 2024 at 11:49=E2=80=AFAM Andrew Atkinson >> wrote: >> >>> Hi Jonathan. Do you want change proposals here as text snippets in >>> emails? It seems the patch process isn't used here. >>> >>> If so, here's an attempted reduction that echoes what Robert said. I >>> also thought explaining Vacuum wouldn't be necessary for this audience,= so >>> less lead-in could work. Is the benefit to end users that there is less >>> memory and CPU needed by vacuum, thus more CPU and memory is available = to >>> their foreground workload? >>> >>> Original: >>> > A foundational feature of PostgreSQL is [vacuum]( >>> https://www.postgresql.org/docs/17/routine-vacuuming.html), which is >>> used to reclaim storage from data that was marked as removed. Reducing >>> resources required for vacuuming directly helps other areas of PostgreS= QL, >>> particularly on very busy systems. PostgreSQL 17 introduces a new inter= nal >>> memory structure for vacuum that's shown up to a 20x reduction in memor= y >>> and improvements in overall vacuuming speed. >>> >>> Proposed: >>> The PostgreSQL [vacuum]( >>> https://www.postgresql.org/docs/17/routine-vacuuming.html) process is >>> critical for healthy operations, requiring server instance resources to >>> operate. With PostgreSQL 17, a new internal memory structure for vacuum= was >>> used that consumes up to 20x less memory. This improves vacuum speed an= d >>> also reduces the use of shared resources, making more available for you= r >>> workload. >>> >>> >>> Something along those lines, where the benefit to the user is they coul= d >>> expect more CPU/mem etc. available for their SQL operations, right? Th= is >>> could be something folks want to benchmark as well as a reason to upgra= de, >>> at least for Vacuum-intensive workloads, high UPDATE and DELETE operati= ons >>> etc. >>> >>> >>> >>> >>> >>> On Thu, Sep 5, 2024 at 11:04=E2=80=AFAM Robert Haas >>> wrote: >>> >>>> On Thu, Sep 5, 2024 at 5:22=E2=80=AFAM Peter Eisentraut >>>> wrote: >>>> > On 04.09.24 23:05, Jonathan S. Katz wrote: >>>> > > Attached is the draft of the PostgreSQL 17 release announcement. >>>> This is >>>> > > a draft of the text that will go into the press kit, with the key >>>> > > portions to review starting from the top of the document, up until >>>> the >>>> > > "About PostgreSQL" section. >>>> > >>>> > I noticed that we don't yet have a list of major features in the PG1= 7 >>>> > release notes. We should probably put that in soon, so that what we >>>> > list there and what is in the announcement are consistent. >>>> >>>> +1. >>>> >>>> > On the actual list, there will be lots of opinions to be had, but I'= ll >>>> > just offer one: I don't think the MERGE RETURNING clause deserves >>>> twice >>>> > as much space as incremental backup. >>>> >>>> I agree with that, although obviously I'm biased. >>>> >>>> I also feel like this whole thing could just be shorter. If it were >>>> half as long and mentioned fewer things and those more briefly, would >>>> we be worse off? I think we might be better off, because it just feels >>>> wordy to me right now. For example: >>>> >>>> A foundational feature of PostgreSQL is >>>> [vacuum](https://www.postgresql.org/docs/17/routine-vacuuming.html), >>>> which is used to reclaim storage from data that was marked as removed. >>>> Reducing resources required for vacuuming directly helps other areas >>>> of PostgreSQL, particularly on very busy systems. PostgreSQL 17 >>>> introduces a new internal memory structure for vacuum that's shown up >>>> to a 20x reduction in memory and improvements in overall vacuuming >>>> speed. This release also removes the `1GB` limit on the memory it can >>>> use (controlled by >>>> [`maintenance_work_mem`]( >>>> https://www.postgresql.org/docs/17/runtime-config-resource.html#GUC-MA= INTENANCE-WORK-MEM) >>>> ), >>>> letting users apply more resources to vacuuming, which is beneficial >>>> for systems with lots of changes. >>>> >>>> It seems to me that the first two sentences could just be completely >>>> nuked, and everything from "letting users" to the end could also be >>>> nuked. At least to me, all of that stuff reads as unnecessarily >>>> filler. I'm not at all sure that removing the 1GB limit on >>>> maintenance_work_mem is important enough that it needs to be in the >>>> release announcement -- I agree it's a good improvement, but to have >>>> it be one of the first things in the press release seems like an odd >>>> choice from my perspective. Nobody's going to look back on this >>>> release years from now and say "oh, that was the release where could >>>> finally set maintenance_work_mem=3D4GB, that was so much better". If >>>> they think about VACUUM, they'll think about the 20x memory reduction >>>> stuff which made the ability to configure values larger than 1GB >>>> irrelevant in the first place. So I'd probably delete the part about >>>> lifting the 1GB cap entirely. But even if you don't do that, the >>>> paragraph could be half as long without losing anything, from my >>>> perspective. >>>> >>>> -- >>>> Robert Haas >>>> EDB: http://www.enterprisedb.com >>>> >>>> >>>> --0000000000008fbf900621643390 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I'll send one more bigger email. I also have made thes= e as comments in a google doc I'm happy to add anyone to if they're= interested. Let me know.

= >=C2=A0PostgreSQL 17 extends its performance gains to query execution. P= lanner statistics help PostgreSQL to determine the best way to search for d= ata, and PostgreSQL 17 can now use planner statistics and the sort order [c= ommon table expressions](https://ww= w.postgresql.org/docs/17/queries-with.html) ([`WITH` queries](https://www.post= gresql.org/docs/17/queries-with.html)) to= further speed up queries.

Was that meant to be the sort order &quo= t;from" CTEs? Maybe a missing "From"?

>=C2=A0to support more expressive JSON processing capab= ilities and implementation of the SQL/JSON standard.=C2=A0

Was that meant to be = something like "and to expand the implementation" of or "an = expanded implementation"?

<= /font>

> "with an emphasis of converting JSON data= to a native PostgreSQL data type"
<= br style=3D"color:rgb(68,71,70);font-family:"Google Sans",Roboto,= sans-serif;font-size:14px">Was that meant to = be an emphasis=C2=A0"on" and not "of"=C2= =A0?

<= font color=3D"#444746" face=3D"Google Sans, Roboto, sans-serif">> For the merge section, here's a proposed rewor= ked version:

PostgreSQL 17 provides expanded c= ontrol over `MERGE` behavior, including a new `merge_action` function that = describes how a row was modified.
PostgreSQL 17 provi= des new capabilities and performance improvements for bulk data loading and= exporting. This includes improving exports of large rows by up to 2x using= the [`COPY`](https://www.postgresql.or= g/docs/17/sql-copy.html) command, and per= formance improvements for cases when the source and destination encodings m= atch. A new option `ON_ERROR` was added which can be used to allow imports = to continue even when errors happen for inserts.


Use the new `pg_main= tain` [predefined role](https://w= ww.postgresql.org/docs/17/predefined-roles.html) to grant permissions for maintenance operations.


> For the EXPLAIN enhancements:

PostgreSQL 17 [`EXPLAIN`](https://www.postgresql.org/docs/17/sql-explain.html) now shows the time spent for I/O block reads and writes, a= nd includes=C2=A0two new options: `SERIALIZE` and `MEMORY`, useful for seei= ng the time spent in data conversion for network transmission, and how much= memory was used.



That's all I've got.

This is an ex= citing release! Nice work!





On Thu, Sep 5, 2024 at 1:44=E2=80=AFPM Andrew= Atkinson <andyatkinson@gmail.= com> wrote:
Additional chunk:

Proposed:
https://www.postgresql.org/docs/1= 7/indexes-types.html#INDEXES-TYPES-BTREE) indexes, the default index me= thod in PostgreSQL). Support for parallel index builds were added for [BRIN= ](https://www.postgresql.org/docs/17/brin.html) indexing.

PostgreSQL 17 = leverages the constraints you've defined. The query planner removes red= undant `IS NOT NULL` statements when a column has a `NOT NULL` constraint, = and skips over `IS NULL` clauses for columns with an `IS NOT NULL` constrai= nt.


On Thu, Sep 5, 2024 at 11:5= 8=E2=80=AFAM Andrew Atkinson <andyatkinson@gmail.com> wrote:
The write operations seem to be under-sold here! These seem very excitin= g and would be beneficial to all Postgres databases I've worked on.
Is this a fair way to reduce the three items mentioned, as = much as possible? I'm not proposing they change, but just sort of pushi= ng on building=C2=A0down the items tied to their benefit to the end user as= much as possible. Possibly shortening them and making them punchier will h= elp.

Faster Write throughput:
- "Fe= wer WAL locks shorter lock lengths" (check this, I'm making it up,= but I'm wondering if it's fewer WAL locks, or the locks are held f= or less time), thus better throughput
- "Faster Sequential s= cans" =C2=A0- is the benefit that they're faster?
- &quo= t;Faster ANALYZE" =C2=A0- is it that ANALYZE runs faster on 17 vs. the= same operation on 16?

The last point "Allowi= ng extensions to be integrated further." I didn't grab on to as mu= ch. I'm wondering if it's something like "new write operation = APIs are now available for extension creators" or something like that?=

On Thu, Sep 5, 2024 at 11:49=E2=80=AFAM Andrew Atkinson <andyatkinson@gmail= .com> wrote:
Hi Jonathan. Do you want change proposals here as text snippets in ema= ils? It seems the=C2=A0patch process isn't used here.

If so, here's an attempted reduction that echoes what Robert said. I = also thought explaining Vacuum wouldn't be necessary for this audience,= so less lead-in could work. Is the=C2=A0benefit to end users that there is= less memory and CPU needed by vacuum, thus more CPU and memory is availabl= e=C2=A0to their foreground=C2=A0workload?

Original= :
> A foundational feature of Po= stgreSQL is [vacuum](https://www.postgresql.org/docs/17/routin= e-vacuuming.html), which is used to reclaim storage from data that was = marked as removed. Reducing resources required for vacuuming directly helps= other areas of PostgreSQL, particularly on very busy systems. PostgreSQL 1= 7 introduces a new internal memory structure for vacuum that's shown up= to a 20x reduction in memory and improvements in overall vacuuming speed.<= /span>

P= roposed:
The PostgreSQL [vacuum](https://www.postgresql.org/docs/17/routine-vacuuming.html) proces= s is critical for healthy operations, requiring server instance resources t= o operate. With PostgreSQL 17, a new internal memory structure for vacuum w= as used that consumes up to 20x less memory. This improves vacuum speed and= also=C2=A0reduces the use of shared resources,=C2=A0making more available = for your workload.


Something alo= ng those lines, where the benefit to the user is they could expect more CPU= /mem etc. available for their SQL operations, right?=C2=A0 This could be so= mething folks want to benchmark as well as a reason to upgrade, at least fo= r Vacuum-intensive workloads,=C2=A0high UPDATE and DELETE operations etc.





<= div class=3D"gmail_quote">
On Thu, Sep= 5, 2024 at 11:04=E2=80=AFAM Robert Haas <robertmhaas@gmail.com> wrote:
=
On Thu, Sep 5, 2024 at 5:22=E2=80=AFAM Peter Eisentraut &l= t;peter@eisentrau= t.org> wrote:
> On 04.09.24 23:05, Jonathan S. Katz wrote:
> > Attached is the draft of the PostgreSQL 17 release announcement. = This is
> > a draft of the text that will go into the press kit, with the key=
> > portions to review starting from the top of the document, up unti= l the
> > "About PostgreSQL" section.
>
> I noticed that we don't yet have a list of major features in the P= G17
> release notes.=C2=A0 We should probably put that in soon, so that what= we
> list there and what is in the announcement are consistent.

+1.

> On the actual list, there will be lots of opinions to be had, but I= 9;ll
> just offer one:=C2=A0 I don't think the MERGE RETURNING clause des= erves twice
> as much space as incremental backup.

I agree with that, although obviously I'm biased.

I also feel like this whole thing could just be shorter. If it were
half as long and mentioned fewer things and those more briefly, would
we be worse off? I think we might be better off, because it just feels
wordy to me right now. For example:

A foundational feature of PostgreSQL is
[vacuum](https://www.postgresql.org/docs/17= /routine-vacuuming.html),
which is used to reclaim storage from data that was marked as removed.
Reducing resources required for vacuuming directly helps other areas
of PostgreSQL, particularly on very busy systems. PostgreSQL 17
introduces a new internal memory structure for vacuum that's shown up to a 20x reduction in memory and improvements in overall vacuuming
speed. This release also removes the=C2=A0 `1GB` limit on the memory it can=
use (controlled by
[`maintenance_work_mem`](https://www.postgresql.org/docs/17/runtime-config-resource.ht= ml#GUC-MAINTENANCE-WORK-MEM)),
letting users apply more resources to vacuuming, which is beneficial
for systems with lots of changes.

It seems to me that the first two sentences could just be completely
nuked, and everything from "letting users" to the end could also = be
nuked. At least to me, all of that stuff reads as unnecessarily
filler. I'm not at all sure that removing the 1GB limit on
maintenance_work_mem is important enough that it needs to be in the
release announcement -- I agree it's a good improvement, but to have it be one of the first things in the press release seems like an odd
choice from my perspective. Nobody's going to look back on this
release years from now and say "oh, that was the release where could finally set maintenance_work_mem=3D4GB, that was so much better". If they think about VACUUM, they'll think about the 20x memory reduction stuff which made the ability to configure values larger than 1GB
irrelevant in the first place. So I'd probably delete the part about lifting the 1GB cap entirely. But even if you don't do that, the
paragraph could be half as long without losing anything, from my
perspective.

--
Robert Haas
EDB: http://www.enterprisedb.com


--0000000000008fbf900621643390--