public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 03/10] Make sure published XIDs are persistent
4+ messages / 3 participants
[nested] [flat]
* [PATCH 03/10] Make sure published XIDs are persistent
@ 2021-03-08 06:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Kyotaro Horiguchi @ 2021-03-08 06:43 UTC (permalink / raw)
pg_xact_status() premises that XIDs obtained by
pg_current_xact_id(_if_assigned)() are persistent beyond a crash. But
XIDs are not guaranteed to go beyond WAL buffers before commit and
thus XIDs may vanish if server crashes before commit. This patch
guarantees the XID shown by the functions to be flushed out to disk.
Copied from: https://www.postgresql.org/message-id/20210308.173242.463790587797836129.horikyota.ntt%40gmail.com
---
src/backend/access/transam/xact.c | 55 +++++++++++++++++++++++++------
src/backend/access/transam/xlog.c | 2 +-
src/backend/utils/adt/xid8funcs.c | 12 ++++++-
src/include/access/xact.h | 3 +-
4 files changed, 59 insertions(+), 13 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6395a9b240..38e978d238 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -201,7 +201,7 @@ typedef struct TransactionStateData
int prevSecContext; /* previous SecurityRestrictionContext */
bool prevXactReadOnly; /* entry-time xact r/o state */
bool startedInRecovery; /* did we start in recovery? */
- bool didLogXid; /* has xid been included in WAL record? */
+ XLogRecPtr minLSN; /* LSN needed to reach to record the xid */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
bool assigned; /* assigned to top-level XID */
@@ -520,14 +520,46 @@ GetCurrentFullTransactionIdIfAny(void)
* MarkCurrentTransactionIdLoggedIfAny
*
* Remember that the current xid - if it is assigned - now has been wal logged.
+ *
+ * upto is the LSN up to which we need to flush WAL to ensure the current xid
+ * to be persistent. See EnsureCurrentTransactionIdLogged().
*/
void
-MarkCurrentTransactionIdLoggedIfAny(void)
+MarkCurrentTransactionIdLoggedIfAny(XLogRecPtr upto)
{
- if (FullTransactionIdIsValid(CurrentTransactionState->fullTransactionId))
- CurrentTransactionState->didLogXid = true;
+ if (FullTransactionIdIsValid(CurrentTransactionState->fullTransactionId) &&
+ XLogRecPtrIsInvalid(CurrentTransactionState->minLSN))
+ CurrentTransactionState->minLSN = upto;
}
+/*
+ * EnsureCurrentTransactionIdLogged
+ *
+ * Make sure that the current top XID is WAL-logged.
+ */
+void
+EnsureTopTransactionIdLogged(void)
+{
+ /*
+ * We need at least one WAL record for the current top transaction to be
+ * flushed out. Write one if we don't have one yet.
+ */
+ if (XLogRecPtrIsInvalid(TopTransactionStateData.minLSN))
+ {
+ xl_xact_assignment xlrec;
+
+ xlrec.xtop = XidFromFullTransactionId(XactTopFullTransactionId);
+ Assert(TransactionIdIsValid(xlrec.xtop));
+ xlrec.nsubxacts = 0;
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, MinSizeOfXactAssignment);
+ TopTransactionStateData.minLSN =
+ XLogInsert(RM_XACT_ID, XLOG_XACT_ASSIGNMENT);
+ }
+
+ XLogFlush(TopTransactionStateData.minLSN);
+}
/*
* GetStableLatestTransactionId
@@ -616,14 +648,14 @@ AssignTransactionId(TransactionState s)
* When wal_level=logical, guarantee that a subtransaction's xid can only
* be seen in the WAL stream if its toplevel xid has been logged before.
* If necessary we log an xact_assignment record with fewer than
- * PGPROC_MAX_CACHED_SUBXIDS. Note that it is fine if didLogXid isn't set
+ * PGPROC_MAX_CACHED_SUBXIDS. Note that it is fine if minLSN isn't set
* for a transaction even though it appears in a WAL record, we just might
* superfluously log something. That can happen when an xid is included
* somewhere inside a wal record, but not in XLogRecord->xl_xid, like in
* xl_standby_locks.
*/
if (isSubXact && XLogLogicalInfoActive() &&
- !TopTransactionStateData.didLogXid)
+ XLogRecPtrIsInvalid(TopTransactionStateData.minLSN))
log_unknown_top = true;
/*
@@ -693,6 +725,7 @@ AssignTransactionId(TransactionState s)
log_unknown_top)
{
xl_xact_assignment xlrec;
+ XLogRecPtr endptr;
/*
* xtop is always set by now because we recurse up transaction
@@ -707,11 +740,13 @@ AssignTransactionId(TransactionState s)
XLogRegisterData((char *) unreportedXids,
nUnreportedXids * sizeof(TransactionId));
- (void) XLogInsert(RM_XACT_ID, XLOG_XACT_ASSIGNMENT);
+ endptr = XLogInsert(RM_XACT_ID, XLOG_XACT_ASSIGNMENT);
nUnreportedXids = 0;
- /* mark top, not current xact as having been logged */
- TopTransactionStateData.didLogXid = true;
+
+ /* set minLSN of top, not of current xact if not yet */
+ if (XLogRecPtrIsInvalid(TopTransactionStateData.minLSN))
+ TopTransactionStateData.minLSN = endptr;
}
}
}
@@ -2022,7 +2057,7 @@ StartTransaction(void)
* initialize reported xid accounting
*/
nUnreportedXids = 0;
- s->didLogXid = false;
+ s->minLSN = InvalidXLogRecPtr;
/*
* must initialize resource-management stuff first
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 15da91a8dd..6eb46ea8a7 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1172,7 +1172,7 @@ XLogInsertRecord(XLogRecData *rdata,
*/
WALInsertLockRelease();
- MarkCurrentTransactionIdLoggedIfAny();
+ MarkCurrentTransactionIdLoggedIfAny(EndPos);
END_CRIT_SECTION();
diff --git a/src/backend/utils/adt/xid8funcs.c b/src/backend/utils/adt/xid8funcs.c
index cc2b4ac797..992482f8c8 100644
--- a/src/backend/utils/adt/xid8funcs.c
+++ b/src/backend/utils/adt/xid8funcs.c
@@ -357,6 +357,8 @@ bad_format:
Datum
pg_current_xact_id(PG_FUNCTION_ARGS)
{
+ FullTransactionId xid;
+
/*
* Must prevent during recovery because if an xid is not assigned we try
* to assign one, which would fail. Programs already rely on this function
@@ -365,7 +367,12 @@ pg_current_xact_id(PG_FUNCTION_ARGS)
*/
PreventCommandDuringRecovery("pg_current_xact_id()");
- PG_RETURN_FULLTRANSACTIONID(GetTopFullTransactionId());
+ xid = GetTopFullTransactionId();
+
+ /* the XID is going to be published, make sure it is psersistent */
+ EnsureTopTransactionIdLogged();
+
+ PG_RETURN_FULLTRANSACTIONID(xid);
}
/*
@@ -380,6 +387,9 @@ pg_current_xact_id_if_assigned(PG_FUNCTION_ARGS)
if (!FullTransactionIdIsValid(topfxid))
PG_RETURN_NULL();
+ /* the XID is going to be published, make sure it is psersistent */
+ EnsureTopTransactionIdLogged();
+
PG_RETURN_FULLTRANSACTIONID(topfxid);
}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 34cfaf542c..a61e4d6da5 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -386,7 +386,8 @@ extern FullTransactionId GetTopFullTransactionId(void);
extern FullTransactionId GetTopFullTransactionIdIfAny(void);
extern FullTransactionId GetCurrentFullTransactionId(void);
extern FullTransactionId GetCurrentFullTransactionIdIfAny(void);
-extern void MarkCurrentTransactionIdLoggedIfAny(void);
+extern void MarkCurrentTransactionIdLoggedIfAny(XLogRecPtr upto);
+extern void EnsureTopTransactionIdLogged(void);
extern bool SubTransactionIsActive(SubTransactionId subxid);
extern CommandId GetCurrentCommandId(bool used);
extern void SetParallelStartTimestamps(TimestampTz xact_ts, TimestampTz stmt_ts);
--
2.17.0
--XsQoSWH+UP9D9v3l
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-wal_compression_method-default-to-zlib.patch"
^ permalink raw reply [nested|flat] 4+ messages in thread
* PostgreSQL 17 Beta 1 release announcement draft
@ 2024-05-16 01:45 Jonathan S. Katz <[email protected]>
2024-05-16 02:42 ` Re: PostgreSQL 17 Beta 1 release announcement draft David Rowley <[email protected]>
2024-05-19 21:34 ` Re: PostgreSQL 17 Beta 1 release announcement draft Jonathan S. Katz <[email protected]>
0 siblings, 2 replies; 4+ messages in thread
From: Jonathan S. Katz @ 2024-05-16 01:45 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Hi,
Attached is a copy of the PostgreSQL 17 Beta 1 release announcement
draft. This contains a user-facing summary of some of the features that
will be available in the Beta, as well as a call to test. I've made an
effort to group them logically around the different workflows they affect.
A few notes:
* The section with the features is not 80-char delimited. I will do that
before the final copy
* There is an explicit callout that we've added in the SQL/JSON features
that were previously reverted in PG15. I want to ensure we're
transparent about that, but also use it as a hook to get people testing.
When reviewing:
* Please check for correctness of feature descriptions, keeping in mind
this is targeted for a general audience
* Please indicate if you believe there's a notable omission, or if we
should omit any of these callouts
* Please indicate if a description is confusing - I'm happy to rewrite
to ensure it's clearer.
Please provide feedback no later than Wed 2024-05-22 18:00 UTC. As the
beta release takes some extra effort, I want to ensure all changes are
in with time to spare before release day.
Thanks,
Jonathan
The PostgreSQL Global Development Group announces that the first beta release of
PostgreSQL 17 is now [available for download](https://www.postgresql.org/download/).
This release contains previews of all features that will be available when
PostgreSQL 17 is made generally available, though some details of the release
can change during the beta period.
You can find information about all of the features and changes found in
PostgreSQL 17 in the [release notes](https://www.postgresql.org/docs/17/release-17.html):
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
In the spirit of the open source PostgreSQL community, we strongly encourage you
to test the new features of PostgreSQL 17 on your systems to help us eliminate
bugs or other issues that may exist. While we do not advise you to run
PostgreSQL 17 Beta 1 in production environments, we encourage you to find ways
to run your typical application workloads against this beta release.
Your testing and feedback will help the community ensure that the PostgreSQL 17
release upholds our standards of delivering a stable, reliable release of the
world's most advanced open source relational database. Please read more about
our [beta testing process](https://www.postgresql.org/developer/beta/) and how
you can contribute:
[https://www.postgresql.org/developer/beta/](https://www.postgresql.org/developer/beta/)
PostgreSQL 17 Feature Highlights
--------------------------------
### Query and Operational Performance Improvements
PostgreSQL 17 builds on recent releases and continues to improve performance across the entire system. [Vacuum](https://www.postgresql.org/docs/17/routine-vacuuming.html), the PostgreSQL process responsible for reclaiming storage, has a new internal data structure that reduces memory usage and has shown up to a 6x improvement in overall time to complete its work. Additionally, the vacuum process no longer has a `1GB` limit on the memory it can use (controlled by [`maintenance_work_mem`](https://www.postgresql.org/docs/17/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM)), giving you the option to apply more resources to vacuuming.
PostgreSQL 17 can now use both planner statistics and the sort order of [common table expressions](https://www.postgresql.org/docs/17/queries-with.html) (aka [`WITH` queries](https://www.postgresql.org/docs/17/queries-with.html)) to further optimize these queries and help them to execute more quickly. Additionally, this release significantly improves execution time of queries that use the `IN` clause with a [B-tree index](https://www.postgresql.org/docs/17/indexes-types.html#INDEXES-TYPES-BTREE). Starting with this release, PostgreSQL can remove redundant `IS NOT NULL` statements from execution on columns that have a `NOT NULL` constraint, and no longer needs to do work on queries that contain an `IS NULL` clause on an `IS NOT NULL` column. Now as of PostgreSQL 17, you can now use parallel index builds for [BRIN](https://www.postgresql.org/docs/17/brin.html) indexes.
Workloads with highly concurrent changes can benefit from PostgreSQL 17 with an improvement to how [write-ahead log](https://www.postgresql.org/docs/17/wal-intro.html) ([WAL](https://www.postgresql.org/docs/17/wal-intro.html)) locks are managed, with some tests showing up to a 2x performance improvement.
Finally, PostgreSQL 17 adds more explicitly SIMD instructions, including AVX-512 support for the [`bit_count](https://www.postgresql.org/docs/17/functions-bitstring.html) function.
### Partitioned and distributed workloads enhancements
PostgreSQL 17 brings more flexibility to partition management, adding the ability to both split and merge partitions, and adds support to partitioned tables for identity columns and exclusion constraints. Additionally, the [PostgreSQL foreign data wrapper](https://www.postgresql.org/docs/17/postgres-fdw.html) ([`postgres_fdw`](https://www.postgresql.org/docs/17/postgres-fdw.html)) can realize performance benefits from queries with `EXISTS` and `IN` subqueries as these can now be pushed down to the remote server.
PostgreSQL 17 adds new capabilities to logical replication that make it simpler to use it in high availability workloads and on upgrades. Starting with PostgreSQL 17, you no longer need to drop [logical replication slots](https://www.postgresql.org/docs/17/logical-replication-subscription.html#LOGICAL-REPLICATION-SUBSCRI...) when using [`pg_upgrade`](https://www.postgresql.org/docs/17/pgupgrade.html), allowing you to avoid resynchronizing data after an upgrade. Additionally, this release introduces failover control for logical replication, providing more control for managing PostgreSQL databases in high availability environments. PostgreSQL 17 also lets logical replication subscribers use `hash` indexes for lookups, and introduces the `pg_createsubscriber` command-line tool for adding logical replication on a replica using physical replication.
### Developer Experience
PostgreSQL 17 continues to build on the SQL/JSON standard, adding support for the `JSON_TABLE` features that can convert JSON to a standard PostgreSQL table, and SQL/JSON constructor (`JSON`, `JSON_SCALAR`, `JSON_SERIALIZE`) and query functions (`JSON_EXISTS`, `JSON_QUERY`, `JSON_VALUE`). Notably, these features were originally planned for the PostgreSQL 15 release but were reverted during the beta period due to design considerations, which is one reason we ask for you to help us test features during beta! Additionally, PostgreSQL 17 adds more functionality to its `jsonpath` implementation, including the ability to convert JSON values to different data types.
The [`MERGE`](https://www.postgresql.org/docs/17/sql-merge.html) command now supports the `RETURNING` clause, letting you further work with modified rows. You can also see what part of the `MERGE` command modified the row using the new `merge_action` function. PostgreSQL 17 also lets you update views using the `MERGE` command, and adds a `WHEN NOT MATCHED BY SOURCE` clause to let you provide behavior if a source row doesn't meet the conditions.
[`COPY`](https://www.postgresql.org/docs/17/sql-copy.html), used to efficiently bulk load data into PostgreSQL, has improved performance in PostgreSQL 17 when the source encoding matches the destination encoding. Additionally `COPY` has a new option, `IGNORE_ERROR`, that lets the copy proceed even if the there is an error inserting a row.
PostgreSQL 17 includes a built-in collation provider that provides similar semantics to the `C` collation provided by libc. This new collation is guaranteed to be immutable, ensuring that the return values of your sorts won't change regardless of what system your PostgreSQL installation runs on.
### Security Features
PostgreSQL 17 adds a new connection parameter, `sslnegotation`, which allows PostgreSQL to perform direct TLS handshakes when using [ALPN](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation), eliminating a network roundtrip. PostgreSQL is registered as `postgresql` in the ALPN directory.
This release introduces event triggers that execute during authentication, and in libpq includes a new API called `PQchangePassword` that automatically hashes passwords on the client-side to prevent accidental plaintext logging in the server.
PostgreSQL 17 adds a new [predefined role](https://www.postgresql.org/docs/17/predefined-roles.html) called `pg_maintain` that gives users permission to execute `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW`, `REINDEX`, and `LOCK TABLE` on all relations. This release also ensures the `search_path` is safe for maintenance operations like `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW` and `INDEX`. Finally, users can now use `ALTER SYSTEM` to set the values for unrecognized custom server variables.
### Backup and Export Management
PostgreSQL 17 introduces the ability to perform incremental backups using [`pg_basebackup`](https://www.postgresql.org/docs/17/app-pgbasebackup.html), and adds a new utility called [`pg_combinebackup`](https://www.postgresql.org/docs/17/app-pgcombinebackup.html) that's used as part of the backup restoration process. This release adds a new flag to [`pg_dump`](https://www.postgresql.org/docs/17/app-pgdump.html) called `--filter`, letting you specify a file that contains instructions on what objects to include/exclude from a dump.
### Monitoring
The [`EXPLAIN`](https://www.postgresql.org/docs/17/sql-explain.html) command, which provides information about query plans and execution, adds two new options: `SERIALIZE`, which shows the amount of time it takes to convert data for network transmission, and `MEMORY`, which reports optimizer memory usage. Additionally, `EXPLAIN` can now show how much time is spent for I/O block reads and writes.
PostgreSQL 17 normalizes the parameters for `CALL` in [`pg_stat_statements`](https://www.postgresql.org/docs/17/pgstatstatements.html), reducing the number of entries for frequently called stored procedures. Additionally, [`VACUUM` progress reporting](https://www.postgresql.org/docs/devel/progress-reporting.html#VACUUM-PROGRESS-REPORTING) now shows the progress of vacuuming indexes. PostgreSQL 17 also introduces a new view, [`pg_wait_events`](https://www.postgresql.org/docs/17/view-pg-wait-events.html), which provides descriptions about wait events and can be combined with `pg_stat_activity` to give more insight into an operation. Additionally, some information in the [`pg_stat_bgwriter`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-BGWRITER-VIEW) view is now split out into the new [`pg_stat_checkpointer`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-CHECKPOINTER-VIEW) view.
Additional Features
-------------------
Many other new features and improvements have been added to PostgreSQL 17. Many
of these may also be helpful for your use cases. Please see the
[release notes](https://www.postgresql.org/docs/17/release-17.html) for a
complete list of new and changed features:
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
Testing for Bugs & Compatibility
--------------------------------
The stability of each PostgreSQL release greatly depends on you, the community,
to test the upcoming version with your workloads and testing tools in order to
find bugs and regressions before the general availability of PostgreSQL 17. As
this is a Beta, minor changes to database behaviors, feature details, and APIs
are still possible. Your feedback and testing will help determine the final
tweaks on the new features, so please test in the near future. The quality of
user testing helps determine when we can make a final release.
A list of [open issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
is publicly available in the PostgreSQL wiki. You can
[report bugs](https://www.postgresql.org/account/submitbug/) using this form on
the PostgreSQL website:
[https://www.postgresql.org/account/submitbug/](https://www.postgresql.org/account/submitbug/)
Beta Schedule
-------------
This is the first beta release of version 17. The PostgreSQL Project will
release additional betas as required for testing, followed by one or more
release candidates, until the final release around September/October 2024. For further
information please see the [Beta Testing](https://www.postgresql.org/developer/beta/)
page.
Links
-----
* [Download](https://www.postgresql.org/download/)
* [Beta Testing Information](https://www.postgresql.org/developer/beta/)
* [PostgreSQL 17 Beta Release Notes](https://www.postgresql.org/docs/17/release-17.html)
* [PostgreSQL 17 Open Issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
* [Feature Matrix](https://www.postgresql.org/about/featurematrix/)
* [Submit a Bug](https://www.postgresql.org/account/submitbug/)
Attachments:
[text/plain] 17beta1.md (11.7K, ../../[email protected]/2-17beta1.md)
download | inline:
The PostgreSQL Global Development Group announces that the first beta release of
PostgreSQL 17 is now [available for download](https://www.postgresql.org/download/).
This release contains previews of all features that will be available when
PostgreSQL 17 is made generally available, though some details of the release
can change during the beta period.
You can find information about all of the features and changes found in
PostgreSQL 17 in the [release notes](https://www.postgresql.org/docs/17/release-17.html):
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
In the spirit of the open source PostgreSQL community, we strongly encourage you
to test the new features of PostgreSQL 17 on your systems to help us eliminate
bugs or other issues that may exist. While we do not advise you to run
PostgreSQL 17 Beta 1 in production environments, we encourage you to find ways
to run your typical application workloads against this beta release.
Your testing and feedback will help the community ensure that the PostgreSQL 17
release upholds our standards of delivering a stable, reliable release of the
world's most advanced open source relational database. Please read more about
our [beta testing process](https://www.postgresql.org/developer/beta/) and how
you can contribute:
[https://www.postgresql.org/developer/beta/](https://www.postgresql.org/developer/beta/)
PostgreSQL 17 Feature Highlights
--------------------------------
### Query and Operational Performance Improvements
PostgreSQL 17 builds on recent releases and continues to improve performance across the entire system. [Vacuum](https://www.postgresql.org/docs/17/routine-vacuuming.html), the PostgreSQL process responsible for reclaiming storage, has a new internal data structure that reduces memory usage and has shown up to a 6x improvement in overall time to complete its work. Additionally, the vacuum process no longer has a `1GB` limit on the memory it can use (controlled by [`maintenance_work_mem`](https://www.postgresql.org/docs/17/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM)), giving you the option to apply more resources to vacuuming.
PostgreSQL 17 can now use both planner statistics and the sort order of [common table expressions](https://www.postgresql.org/docs/17/queries-with.html) (aka [`WITH` queries](https://www.postgresql.org/docs/17/queries-with.html)) to further optimize these queries and help them to execute more quickly. Additionally, this release significantly improves execution time of queries that use the `IN` clause with a [B-tree index](https://www.postgresql.org/docs/17/indexes-types.html#INDEXES-TYPES-BTREE). Starting with this release, PostgreSQL can remove redundant `IS NOT NULL` statements from execution on columns that have a `NOT NULL` constraint, and no longer needs to do work on queries that contain an `IS NULL` clause on an `IS NOT NULL` column. Now as of PostgreSQL 17, you can now use parallel index builds for [BRIN](https://www.postgresql.org/docs/17/brin.html) indexes.
Workloads with highly concurrent changes can benefit from PostgreSQL 17 with an improvement to how [write-ahead log](https://www.postgresql.org/docs/17/wal-intro.html) ([WAL](https://www.postgresql.org/docs/17/wal-intro.html)) locks are managed, with some tests showing up to a 2x performance improvement.
Finally, PostgreSQL 17 adds more explicitly SIMD instructions, including AVX-512 support for the [`bit_count](https://www.postgresql.org/docs/17/functions-bitstring.html) function.
### Partitioned and distributed workloads enhancements
PostgreSQL 17 brings more flexibility to partition management, adding the ability to both split and merge partitions, and adds support to partitioned tables for identity columns and exclusion constraints. Additionally, the [PostgreSQL foreign data wrapper](https://www.postgresql.org/docs/17/postgres-fdw.html) ([`postgres_fdw`](https://www.postgresql.org/docs/17/postgres-fdw.html)) can realize performance benefits from queries with `EXISTS` and `IN` subqueries as these can now be pushed down to the remote server.
PostgreSQL 17 adds new capabilities to logical replication that make it simpler to use it in high availability workloads and on upgrades. Starting with PostgreSQL 17, you no longer need to drop [logical replication slots](https://www.postgresql.org/docs/17/logical-replication-subscription.html#LOGICAL-REPLICATION-SUBSCRIPTION-SLOT) when using [`pg_upgrade`](https://www.postgresql.org/docs/17/pgupgrade.html), allowing you to avoid resynchronizing data after an upgrade. Additionally, this release introduces failover control for logical replication, providing more control for managing PostgreSQL databases in high availability environments. PostgreSQL 17 also lets logical replication subscribers use `hash` indexes for lookups, and introduces the `pg_createsubscriber` command-line tool for adding logical replication on a replica using physical replication.
### Developer Experience
PostgreSQL 17 continues to build on the SQL/JSON standard, adding support for the `JSON_TABLE` features that can convert JSON to a standard PostgreSQL table, and SQL/JSON constructor (`JSON`, `JSON_SCALAR`, `JSON_SERIALIZE`) and query functions (`JSON_EXISTS`, `JSON_QUERY`, `JSON_VALUE`). Notably, these features were originally planned for the PostgreSQL 15 release but were reverted during the beta period due to design considerations, which is one reason we ask for you to help us test features during beta! Additionally, PostgreSQL 17 adds more functionality to its `jsonpath` implementation, including the ability to convert JSON values to different data types.
The [`MERGE`](https://www.postgresql.org/docs/17/sql-merge.html) command now supports the `RETURNING` clause, letting you further work with modified rows. You can also see what part of the `MERGE` command modified the row using the new `merge_action` function. PostgreSQL 17 also lets you update views using the `MERGE` command, and adds a `WHEN NOT MATCHED BY SOURCE` clause to let you provide behavior if a source row doesn't meet the conditions.
[`COPY`](https://www.postgresql.org/docs/17/sql-copy.html), used to efficiently bulk load data into PostgreSQL, has improved performance in PostgreSQL 17 when the source encoding matches the destination encoding. Additionally `COPY` has a new option, `IGNORE_ERROR`, that lets the copy proceed even if the there is an error inserting a row.
PostgreSQL 17 includes a built-in collation provider that provides similar semantics to the `C` collation provided by libc. This new collation is guaranteed to be immutable, ensuring that the return values of your sorts won't change regardless of what system your PostgreSQL installation runs on.
### Security Features
PostgreSQL 17 adds a new connection parameter, `sslnegotation`, which allows PostgreSQL to perform direct TLS handshakes when using [ALPN](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation), eliminating a network roundtrip. PostgreSQL is registered as `postgresql` in the ALPN directory.
This release introduces event triggers that execute during authentication, and in libpq includes a new API called `PQchangePassword` that automatically hashes passwords on the client-side to prevent accidental plaintext logging in the server.
PostgreSQL 17 adds a new [predefined role](https://www.postgresql.org/docs/17/predefined-roles.html) called `pg_maintain` that gives users permission to execute `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW`, `REINDEX`, and `LOCK TABLE` on all relations. This release also ensures the `search_path` is safe for maintenance operations like `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW` and `INDEX`. Finally, users can now use `ALTER SYSTEM` to set the values for unrecognized custom server variables.
### Backup and Export Management
PostgreSQL 17 introduces the ability to perform incremental backups using [`pg_basebackup`](https://www.postgresql.org/docs/17/app-pgbasebackup.html), and adds a new utility called [`pg_combinebackup`](https://www.postgresql.org/docs/17/app-pgcombinebackup.html) that's used as part of the backup restoration process. This release adds a new flag to [`pg_dump`](https://www.postgresql.org/docs/17/app-pgdump.html) called `--filter`, letting you specify a file that contains instructions on what objects to include/exclude from a dump.
### Monitoring
The [`EXPLAIN`](https://www.postgresql.org/docs/17/sql-explain.html) command, which provides information about query plans and execution, adds two new options: `SERIALIZE`, which shows the amount of time it takes to convert data for network transmission, and `MEMORY`, which reports optimizer memory usage. Additionally, `EXPLAIN` can now show how much time is spent for I/O block reads and writes.
PostgreSQL 17 normalizes the parameters for `CALL` in [`pg_stat_statements`](https://www.postgresql.org/docs/17/pgstatstatements.html), reducing the number of entries for frequently called stored procedures. Additionally, [`VACUUM` progress reporting](https://www.postgresql.org/docs/devel/progress-reporting.html#VACUUM-PROGRESS-REPORTING) now shows the progress of vacuuming indexes. PostgreSQL 17 also introduces a new view, [`pg_wait_events`](https://www.postgresql.org/docs/17/view-pg-wait-events.html), which provides descriptions about wait events and can be combined with `pg_stat_activity` to give more insight into an operation. Additionally, some information in the [`pg_stat_bgwriter`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-BGWRITER-VIEW) view is now split out into the new [`pg_stat_checkpointer`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-CHECKPOINTER-VIEW) view.
Additional Features
-------------------
Many other new features and improvements have been added to PostgreSQL 17. Many
of these may also be helpful for your use cases. Please see the
[release notes](https://www.postgresql.org/docs/17/release-17.html) for a
complete list of new and changed features:
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
Testing for Bugs & Compatibility
--------------------------------
The stability of each PostgreSQL release greatly depends on you, the community,
to test the upcoming version with your workloads and testing tools in order to
find bugs and regressions before the general availability of PostgreSQL 17. As
this is a Beta, minor changes to database behaviors, feature details, and APIs
are still possible. Your feedback and testing will help determine the final
tweaks on the new features, so please test in the near future. The quality of
user testing helps determine when we can make a final release.
A list of [open issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
is publicly available in the PostgreSQL wiki. You can
[report bugs](https://www.postgresql.org/account/submitbug/) using this form on
the PostgreSQL website:
[https://www.postgresql.org/account/submitbug/](https://www.postgresql.org/account/submitbug/)
Beta Schedule
-------------
This is the first beta release of version 17. The PostgreSQL Project will
release additional betas as required for testing, followed by one or more
release candidates, until the final release around September/October 2024. For further
information please see the [Beta Testing](https://www.postgresql.org/developer/beta/)
page.
Links
-----
* [Download](https://www.postgresql.org/download/)
* [Beta Testing Information](https://www.postgresql.org/developer/beta/)
* [PostgreSQL 17 Beta Release Notes](https://www.postgresql.org/docs/17/release-17.html)
* [PostgreSQL 17 Open Issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
* [Feature Matrix](https://www.postgresql.org/about/featurematrix/)
* [Submit a Bug](https://www.postgresql.org/account/submitbug/)
[application/pgp-signature] OpenPGP_signature.asc (840B, ../../[email protected]/3-OpenPGP_signature.asc)
download
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: PostgreSQL 17 Beta 1 release announcement draft
2024-05-16 01:45 PostgreSQL 17 Beta 1 release announcement draft Jonathan S. Katz <[email protected]>
@ 2024-05-16 02:42 ` David Rowley <[email protected]>
1 sibling, 0 replies; 4+ messages in thread
From: David Rowley @ 2024-05-16 02:42 UTC (permalink / raw)
To: Jonathan S. Katz <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Thanks for writing that up. It's always exciting to see this each year.
On Thu, 16 May 2024 at 13:45, Jonathan S. Katz <[email protected]> wrote:
> * Please indicate if you believe there's a notable omission, or if we
> should omit any of these callouts
I'd say the streaming read stuff added in b5a9b18cd and subsequent
commits like b7b0f3f27 and 041b96802 are worth a mention. I'd be happy
to see this over the IS NOT NULL qual stuff that I worked on in there
or even the AVX512 bit counting. Speeding up a backwater aggregate
function is nice, but IMO, not compatible with reducing the number
reads.
There's some benchmarking in a youtube video:
https://youtu.be/QAYzWAlxCYc?si=L0UT6Lrf067ZBv46&t=237
> * Please indicate if a description is confusing - I'm happy to rewrite
> to ensure it's clearer.
>
> Please provide feedback no later than Wed 2024-05-22 18:00 UTC.
The only other thing I saw from a quick read was a stray "the" in "the
copy proceed even if the there is an error inserting a row."
David
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: PostgreSQL 17 Beta 1 release announcement draft
2024-05-16 01:45 PostgreSQL 17 Beta 1 release announcement draft Jonathan S. Katz <[email protected]>
@ 2024-05-19 21:34 ` Jonathan S. Katz <[email protected]>
1 sibling, 0 replies; 4+ messages in thread
From: Jonathan S. Katz @ 2024-05-19 21:34 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
On 5/15/24 9:45 PM, Jonathan S. Katz wrote:
> Hi,
>
> Attached is a copy of the PostgreSQL 17 Beta 1 release announcement
> draft. This contains a user-facing summary of some of the features that
> will be available in the Beta, as well as a call to test. I've made an
> effort to group them logically around the different workflows they affect.
>
> A few notes:
>
> * The section with the features is not 80-char delimited. I will do that
> before the final copy
>
> * There is an explicit callout that we've added in the SQL/JSON features
> that were previously reverted in PG15. I want to ensure we're
> transparent about that, but also use it as a hook to get people testing.
>
> When reviewing:
>
> * Please check for correctness of feature descriptions, keeping in mind
> this is targeted for a general audience
>
> * Please indicate if you believe there's a notable omission, or if we
> should omit any of these callouts
>
> * Please indicate if a description is confusing - I'm happy to rewrite
> to ensure it's clearer.
>
> Please provide feedback no later than Wed 2024-05-22 18:00 UTC. As the
> beta release takes some extra effort, I want to ensure all changes are
> in with time to spare before release day.
Thanks for all the feedback to date. Please see the next revision.
Again, please provide feedback no later than 2024-05-22 18:00 UTC.
Thanks,
Jonathan
The PostgreSQL Global Development Group announces that the first beta release of
PostgreSQL 17 is now [available for download](https://www.postgresql.org/download/).
This release contains previews of all features that will be available when
PostgreSQL 17 is made generally available, though some details of the release
can change during the beta period.
You can find information about all of the PostgreSQL 17 features and changes in
the [release notes](https://www.postgresql.org/docs/17/release-17.html):
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
In the spirit of the open source PostgreSQL community, we strongly encourage you
to test the new features of PostgreSQL 17 on your systems to help us eliminate
bugs or other issues that may exist. While we do not advise you to run
PostgreSQL 17 Beta 1 in production environments, we encourage you to find ways
to run your typical application workloads against this beta release.
Your testing and feedback will help the community ensure that the PostgreSQL 17
release upholds our standards of delivering a stable, reliable release of the
world's most advanced open source relational database. Please read more about
our [beta testing process](https://www.postgresql.org/developer/beta/) and how
you can contribute:
[https://www.postgresql.org/developer/beta/](https://www.postgresql.org/developer/beta/)
PostgreSQL 17 Feature Highlights
--------------------------------
### Query and Operational Performance Improvements
PostgreSQL 17 builds on recent releases and continues to improve performance across the entire system. [Vacuum](https://www.postgresql.org/docs/17/routine-vacuuming.html), the PostgreSQL process responsible for reclaiming storage, has a new internal data structure that reduces memory usage and has shown up to a 6x improvement in overall time to complete its work. Additionally, the vacuum process no longer has a `1GB` limit on the memory it can use (controlled by [`maintenance_work_mem`](https://www.postgresql.org/docs/17/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM)), giving you the option to apply more resources to vacuuming.
This release introduces adds an interface to stream I/O, and can show performance improvements when performing sequential scans and running [`ANALYZE`](https://www.postgresql.org/docs/17/sql-analyze.html).
PostgreSQL 17 can now use both planner statistics and the sort order of [common table expressions](https://www.postgresql.org/docs/17/queries-with.html) (aka [`WITH` queries](https://www.postgresql.org/docs/17/queries-with.html)) to further optimize these queries and help them to execute more quickly. Additionally, this release significantly improves execution time of queries that use the `IN` clause with a [B-tree index](https://www.postgresql.org/docs/17/indexes-types.html#INDEXES-TYPES-BTREE). Starting with this release, PostgreSQL can remove redundant `IS NOT NULL` statements from execution on columns that have a `NOT NULL` constraint, and no longer needs to do work on queries that contain an `IS NULL` clause on an `IS NOT NULL` column. Now as of PostgreSQL 17, you can use parallel index builds for [BRIN](https://www.postgresql.org/docs/17/brin.html) indexes.
Workloads with highly concurrent changes can benefit from PostgreSQL 17 with an improvement to how [write-ahead log](https://www.postgresql.org/docs/17/wal-intro.html) ([WAL](https://www.postgresql.org/docs/17/wal-intro.html)) locks are managed, with some tests showing up to a 2x performance improvement.
Finally, PostgreSQL 17 adds more explicit SIMD instructions, including AVX-512 support for the [`bit_count](https://www.postgresql.org/docs/17/functions-bitstring.html) function.
### Partitioned and distributed workloads enhancements
PostgreSQL 17 brings more flexibility to partition management, adding the ability to both split and merge partitions, and adds support to partitioned tables for identity columns and exclusion constraints. Additionally, the [PostgreSQL foreign data wrapper](https://www.postgresql.org/docs/17/postgres-fdw.html) ([`postgres_fdw`](https://www.postgresql.org/docs/17/postgres-fdw.html)) can realize performance benefits from queries with `EXISTS` and `IN` subqueries as these can now be pushed down to the remote server.
PostgreSQL 17 adds new capabilities to logical replication that make it simpler to use it in high availability workloads and on upgrades. Starting with upgrades from PostgreSQL 17 to newer versions, you no longer need to drop [logical replication slots](https://www.postgresql.org/docs/17/logical-replication-subscription.html#LOGICAL-REPLICATION-SUBSCRI...) when using [`pg_upgrade`](https://www.postgresql.org/docs/17/pgupgrade.html), allowing you to avoid resynchronizing data after an upgrade. Additionally, this release introduces failover control for logical replication, providing more control for managing PostgreSQL databases in high availability environments. PostgreSQL 17 also lets logical replication subscribers use `hash` indexes for lookups, and introduces the `pg_createsubscriber` command-line tool for adding logical replication on a replica using physical replication.
### Developer Experience
PostgreSQL 17 continues to build on the SQL/JSON standard, adding support for the `JSON_TABLE` features that can convert JSON to a standard PostgreSQL table, and SQL/JSON constructor (`JSON`, `JSON_SCALAR`, `JSON_SERIALIZE`) and query functions (`JSON_EXISTS`, `JSON_QUERY`, `JSON_VALUE`). Notably, these features were originally planned for the PostgreSQL 15 release but were reverted during the beta period due to design considerations, which is one reason we ask for you to help us test features during beta! Additionally, PostgreSQL 17 adds more functionality to its `jsonpath` implementation, including the ability to convert JSON values to different data types.
The [`MERGE`](https://www.postgresql.org/docs/17/sql-merge.html) command now supports the `RETURNING` clause, letting you further work with modified rows. You can also see what part of the `MERGE` command modified the row using the new `merge_action` function. PostgreSQL 17 also lets you update views using the `MERGE` command, and adds a `WHEN NOT MATCHED BY SOURCE` clause to let you provide behavior if a source row doesn't meet the conditions.
[`COPY`](https://www.postgresql.org/docs/17/sql-copy.html) is used to efficiently bulk load data into PostgreSQL, and with PostgreSQL 17 shows a 2x performance improvement when loading large rows. Additionally, `COPY` has improved performance when the source encoding matches the destination encoding, and has a new option, `ON_ERROR`, that lets the copy proceed even if there is an error inserting a row.
PostgreSQL 17 includes a built-in collation provider that provides similar sorting semantics to the `C` collation except with `UTF-8` encoding rather than `SQL_ASCII`. This new collation is guaranteed to be immutable, ensuring that the return values of your sorts won't change regardless of what system your PostgreSQL installation runs on.
### Security Features
PostgreSQL 17 adds a new connection parameter, `sslnegotation`, which allows PostgreSQL to perform direct TLS handshakes when using [ALPN](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation), eliminating a network roundtrip. PostgreSQL is registered as `postgresql` in the ALPN directory.
This release introduces event triggers that execute during authentication, and in libpq includes a new API called `PQchangePassword` that automatically hashes passwords on the client-side to prevent accidental plaintext logging in the server.
PostgreSQL 17 adds a new [predefined role](https://www.postgresql.org/docs/17/predefined-roles.html) called `pg_maintain` that gives users permission to execute `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW`, `REINDEX`, and `LOCK TABLE` on all relations. This release also ensures the `search_path` is safe for maintenance operations like `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW` and `INDEX`. Finally, users can now use `ALTER SYSTEM` to set the values for unrecognized custom server variables.
### Backup and Export Management
PostgreSQL 17 introduces the ability to perform incremental backups using [`pg_basebackup`](https://www.postgresql.org/docs/17/app-pgbasebackup.html), and adds a new utility called [`pg_combinebackup`](https://www.postgresql.org/docs/17/app-pgcombinebackup.html) that's used as part of the backup restoration process. This release adds a new flag to [`pg_dump`](https://www.postgresql.org/docs/17/app-pgdump.html) called `--filter`, letting you specify a file that contains instructions on what objects to include/exclude from a dump.
### Monitoring
The [`EXPLAIN`](https://www.postgresql.org/docs/17/sql-explain.html) command, which provides information about query plans and execution, adds two new options: `SERIALIZE`, which shows the amount of time it takes to convert data for network transmission, and `MEMORY`, which reports optimizer memory usage. Additionally, `EXPLAIN` can now show how much time is spent for I/O block reads and writes.
PostgreSQL 17 normalizes the parameters for `CALL` in [`pg_stat_statements`](https://www.postgresql.org/docs/17/pgstatstatements.html), reducing the number of entries for frequently called stored procedures. Additionally, [`VACUUM` progress reporting](https://www.postgresql.org/docs/devel/progress-reporting.html#VACUUM-PROGRESS-REPORTING) now shows the progress of vacuuming indexes. PostgreSQL 17 also introduces a new view, [`pg_wait_events`](https://www.postgresql.org/docs/17/view-pg-wait-events.html), which provides descriptions about wait events and can be combined with `pg_stat_activity` to give more insight into why a session is blocked. Additionally, some information in the [`pg_stat_bgwriter`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-BGWRITER-VIEW) view is now split out into the new [`pg_stat_checkpointer`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-CHECKPOINTER-VIEW) view.
Additional Features
-------------------
Many other new features and improvements have been added to PostgreSQL 17. Many
of these may also be helpful for your use cases. Please see the
[release notes](https://www.postgresql.org/docs/17/release-17.html) for a
complete list of new and changed features:
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
Testing for Bugs & Compatibility
--------------------------------
The stability of each PostgreSQL release greatly depends on you, the community,
to test the upcoming version with your workloads and testing tools in order to
find bugs and regressions before the general availability of PostgreSQL 17. As
this is a Beta, minor changes to database behaviors, feature details, and APIs
are still possible. Your feedback and testing will help determine the final
tweaks on the new features, so please test in the near future. The quality of
user testing helps determine when we can make a final release.
A list of [open issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
is publicly available in the PostgreSQL wiki. You can
[report bugs](https://www.postgresql.org/account/submitbug/) using this form on
the PostgreSQL website:
[https://www.postgresql.org/account/submitbug/](https://www.postgresql.org/account/submitbug/)
Beta Schedule
-------------
This is the first beta release of version 17. The PostgreSQL Project will
release additional betas as required for testing, followed by one or more
release candidates, until the final release around September/October 2024. For further
information please see the [Beta Testing](https://www.postgresql.org/developer/beta/)
page.
Links
-----
* [Download](https://www.postgresql.org/download/)
* [Beta Testing Information](https://www.postgresql.org/developer/beta/)
* [PostgreSQL 17 Beta Release Notes](https://www.postgresql.org/docs/17/release-17.html)
* [PostgreSQL 17 Open Issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
* [Feature Matrix](https://www.postgresql.org/about/featurematrix/)
* [Submit a Bug](https://www.postgresql.org/account/submitbug/)
Attachments:
[text/plain] 17beta1.md (12.0K, ../../[email protected]/2-17beta1.md)
download | inline:
The PostgreSQL Global Development Group announces that the first beta release of
PostgreSQL 17 is now [available for download](https://www.postgresql.org/download/).
This release contains previews of all features that will be available when
PostgreSQL 17 is made generally available, though some details of the release
can change during the beta period.
You can find information about all of the PostgreSQL 17 features and changes in
the [release notes](https://www.postgresql.org/docs/17/release-17.html):
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
In the spirit of the open source PostgreSQL community, we strongly encourage you
to test the new features of PostgreSQL 17 on your systems to help us eliminate
bugs or other issues that may exist. While we do not advise you to run
PostgreSQL 17 Beta 1 in production environments, we encourage you to find ways
to run your typical application workloads against this beta release.
Your testing and feedback will help the community ensure that the PostgreSQL 17
release upholds our standards of delivering a stable, reliable release of the
world's most advanced open source relational database. Please read more about
our [beta testing process](https://www.postgresql.org/developer/beta/) and how
you can contribute:
[https://www.postgresql.org/developer/beta/](https://www.postgresql.org/developer/beta/)
PostgreSQL 17 Feature Highlights
--------------------------------
### Query and Operational Performance Improvements
PostgreSQL 17 builds on recent releases and continues to improve performance across the entire system. [Vacuum](https://www.postgresql.org/docs/17/routine-vacuuming.html), the PostgreSQL process responsible for reclaiming storage, has a new internal data structure that reduces memory usage and has shown up to a 6x improvement in overall time to complete its work. Additionally, the vacuum process no longer has a `1GB` limit on the memory it can use (controlled by [`maintenance_work_mem`](https://www.postgresql.org/docs/17/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM)), giving you the option to apply more resources to vacuuming.
This release introduces adds an interface to stream I/O, and can show performance improvements when performing sequential scans and running [`ANALYZE`](https://www.postgresql.org/docs/17/sql-analyze.html).
PostgreSQL 17 can now use both planner statistics and the sort order of [common table expressions](https://www.postgresql.org/docs/17/queries-with.html) (aka [`WITH` queries](https://www.postgresql.org/docs/17/queries-with.html)) to further optimize these queries and help them to execute more quickly. Additionally, this release significantly improves execution time of queries that use the `IN` clause with a [B-tree index](https://www.postgresql.org/docs/17/indexes-types.html#INDEXES-TYPES-BTREE). Starting with this release, PostgreSQL can remove redundant `IS NOT NULL` statements from execution on columns that have a `NOT NULL` constraint, and no longer needs to do work on queries that contain an `IS NULL` clause on an `IS NOT NULL` column. Now as of PostgreSQL 17, you can use parallel index builds for [BRIN](https://www.postgresql.org/docs/17/brin.html) indexes.
Workloads with highly concurrent changes can benefit from PostgreSQL 17 with an improvement to how [write-ahead log](https://www.postgresql.org/docs/17/wal-intro.html) ([WAL](https://www.postgresql.org/docs/17/wal-intro.html)) locks are managed, with some tests showing up to a 2x performance improvement.
Finally, PostgreSQL 17 adds more explicit SIMD instructions, including AVX-512 support for the [`bit_count](https://www.postgresql.org/docs/17/functions-bitstring.html) function.
### Partitioned and distributed workloads enhancements
PostgreSQL 17 brings more flexibility to partition management, adding the ability to both split and merge partitions, and adds support to partitioned tables for identity columns and exclusion constraints. Additionally, the [PostgreSQL foreign data wrapper](https://www.postgresql.org/docs/17/postgres-fdw.html) ([`postgres_fdw`](https://www.postgresql.org/docs/17/postgres-fdw.html)) can realize performance benefits from queries with `EXISTS` and `IN` subqueries as these can now be pushed down to the remote server.
PostgreSQL 17 adds new capabilities to logical replication that make it simpler to use it in high availability workloads and on upgrades. Starting with upgrades from PostgreSQL 17 to newer versions, you no longer need to drop [logical replication slots](https://www.postgresql.org/docs/17/logical-replication-subscription.html#LOGICAL-REPLICATION-SUBSCRIPTION-SLOT) when using [`pg_upgrade`](https://www.postgresql.org/docs/17/pgupgrade.html), allowing you to avoid resynchronizing data after an upgrade. Additionally, this release introduces failover control for logical replication, providing more control for managing PostgreSQL databases in high availability environments. PostgreSQL 17 also lets logical replication subscribers use `hash` indexes for lookups, and introduces the `pg_createsubscriber` command-line tool for adding logical replication on a replica using physical replication.
### Developer Experience
PostgreSQL 17 continues to build on the SQL/JSON standard, adding support for the `JSON_TABLE` features that can convert JSON to a standard PostgreSQL table, and SQL/JSON constructor (`JSON`, `JSON_SCALAR`, `JSON_SERIALIZE`) and query functions (`JSON_EXISTS`, `JSON_QUERY`, `JSON_VALUE`). Notably, these features were originally planned for the PostgreSQL 15 release but were reverted during the beta period due to design considerations, which is one reason we ask for you to help us test features during beta! Additionally, PostgreSQL 17 adds more functionality to its `jsonpath` implementation, including the ability to convert JSON values to different data types.
The [`MERGE`](https://www.postgresql.org/docs/17/sql-merge.html) command now supports the `RETURNING` clause, letting you further work with modified rows. You can also see what part of the `MERGE` command modified the row using the new `merge_action` function. PostgreSQL 17 also lets you update views using the `MERGE` command, and adds a `WHEN NOT MATCHED BY SOURCE` clause to let you provide behavior if a source row doesn't meet the conditions.
[`COPY`](https://www.postgresql.org/docs/17/sql-copy.html) is used to efficiently bulk load data into PostgreSQL, and with PostgreSQL 17 shows a 2x performance improvement when loading large rows. Additionally, `COPY` has improved performance when the source encoding matches the destination encoding, and has a new option, `ON_ERROR`, that lets the copy proceed even if there is an error inserting a row.
PostgreSQL 17 includes a built-in collation provider that provides similar sorting semantics to the `C` collation except with `UTF-8` encoding rather than `SQL_ASCII`. This new collation is guaranteed to be immutable, ensuring that the return values of your sorts won't change regardless of what system your PostgreSQL installation runs on.
### Security Features
PostgreSQL 17 adds a new connection parameter, `sslnegotation`, which allows PostgreSQL to perform direct TLS handshakes when using [ALPN](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation), eliminating a network roundtrip. PostgreSQL is registered as `postgresql` in the ALPN directory.
This release introduces event triggers that execute during authentication, and in libpq includes a new API called `PQchangePassword` that automatically hashes passwords on the client-side to prevent accidental plaintext logging in the server.
PostgreSQL 17 adds a new [predefined role](https://www.postgresql.org/docs/17/predefined-roles.html) called `pg_maintain` that gives users permission to execute `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW`, `REINDEX`, and `LOCK TABLE` on all relations. This release also ensures the `search_path` is safe for maintenance operations like `VACUUM`, `ANALYZE`, `CLUSTER`, `REFRESH MATERIALIZED VIEW` and `INDEX`. Finally, users can now use `ALTER SYSTEM` to set the values for unrecognized custom server variables.
### Backup and Export Management
PostgreSQL 17 introduces the ability to perform incremental backups using [`pg_basebackup`](https://www.postgresql.org/docs/17/app-pgbasebackup.html), and adds a new utility called [`pg_combinebackup`](https://www.postgresql.org/docs/17/app-pgcombinebackup.html) that's used as part of the backup restoration process. This release adds a new flag to [`pg_dump`](https://www.postgresql.org/docs/17/app-pgdump.html) called `--filter`, letting you specify a file that contains instructions on what objects to include/exclude from a dump.
### Monitoring
The [`EXPLAIN`](https://www.postgresql.org/docs/17/sql-explain.html) command, which provides information about query plans and execution, adds two new options: `SERIALIZE`, which shows the amount of time it takes to convert data for network transmission, and `MEMORY`, which reports optimizer memory usage. Additionally, `EXPLAIN` can now show how much time is spent for I/O block reads and writes.
PostgreSQL 17 normalizes the parameters for `CALL` in [`pg_stat_statements`](https://www.postgresql.org/docs/17/pgstatstatements.html), reducing the number of entries for frequently called stored procedures. Additionally, [`VACUUM` progress reporting](https://www.postgresql.org/docs/devel/progress-reporting.html#VACUUM-PROGRESS-REPORTING) now shows the progress of vacuuming indexes. PostgreSQL 17 also introduces a new view, [`pg_wait_events`](https://www.postgresql.org/docs/17/view-pg-wait-events.html), which provides descriptions about wait events and can be combined with `pg_stat_activity` to give more insight into why a session is blocked. Additionally, some information in the [`pg_stat_bgwriter`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-BGWRITER-VIEW) view is now split out into the new [`pg_stat_checkpointer`](https://www.postgresql.org/docs/17/monitoring-stats.html#MONITORING-PG-STAT-CHECKPOINTER-VIEW) view.
Additional Features
-------------------
Many other new features and improvements have been added to PostgreSQL 17. Many
of these may also be helpful for your use cases. Please see the
[release notes](https://www.postgresql.org/docs/17/release-17.html) for a
complete list of new and changed features:
[https://www.postgresql.org/docs/17/release-17.html](https://www.postgresql.org/docs/17/release-17.html)
Testing for Bugs & Compatibility
--------------------------------
The stability of each PostgreSQL release greatly depends on you, the community,
to test the upcoming version with your workloads and testing tools in order to
find bugs and regressions before the general availability of PostgreSQL 17. As
this is a Beta, minor changes to database behaviors, feature details, and APIs
are still possible. Your feedback and testing will help determine the final
tweaks on the new features, so please test in the near future. The quality of
user testing helps determine when we can make a final release.
A list of [open issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
is publicly available in the PostgreSQL wiki. You can
[report bugs](https://www.postgresql.org/account/submitbug/) using this form on
the PostgreSQL website:
[https://www.postgresql.org/account/submitbug/](https://www.postgresql.org/account/submitbug/)
Beta Schedule
-------------
This is the first beta release of version 17. The PostgreSQL Project will
release additional betas as required for testing, followed by one or more
release candidates, until the final release around September/October 2024. For further
information please see the [Beta Testing](https://www.postgresql.org/developer/beta/)
page.
Links
-----
* [Download](https://www.postgresql.org/download/)
* [Beta Testing Information](https://www.postgresql.org/developer/beta/)
* [PostgreSQL 17 Beta Release Notes](https://www.postgresql.org/docs/17/release-17.html)
* [PostgreSQL 17 Open Issues](https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items)
* [Feature Matrix](https://www.postgresql.org/about/featurematrix/)
* [Submit a Bug](https://www.postgresql.org/account/submitbug/)
[application/pgp-signature] OpenPGP_signature.asc (840B, ../../[email protected]/3-OpenPGP_signature.asc)
download
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2024-05-19 21:34 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-03-08 06:43 [PATCH 03/10] Make sure published XIDs are persistent Kyotaro Horiguchi <[email protected]>
2024-05-16 01:45 PostgreSQL 17 Beta 1 release announcement draft Jonathan S. Katz <[email protected]>
2024-05-16 02:42 ` Re: PostgreSQL 17 Beta 1 release announcement draft David Rowley <[email protected]>
2024-05-19 21:34 ` Re: PostgreSQL 17 Beta 1 release announcement draft Jonathan S. Katz <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox