Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osFUH-000631-RZ for pgsql-hackers@arkaria.postgresql.org; Tue, 08 Nov 2022 03:41:22 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1osFUG-00060d-N9 for pgsql-hackers@arkaria.postgresql.org; Tue, 08 Nov 2022 03:41:20 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osFUG-00060U-4X for pgsql-hackers@lists.postgresql.org; Tue, 08 Nov 2022 03:41:20 +0000 Received: from momjian.us ([72.94.173.45]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1osFUB-0008Fj-QG for pgsql-hackers@postgresql.org; Tue, 08 Nov 2022 03:41:19 +0000 Received: from bruce by momjian.us with local (Exim 4.94.2) (envelope-from ) id 1osFU3-00EJ9v-Fm; Mon, 07 Nov 2022 22:41:07 -0500 Date: Mon, 7 Nov 2022 22:41:07 -0500 From: Bruce Momjian To: Simon Riggs Cc: Laurenz Albe , Robert Treat , Alvaro Herrera , Erik Rijkers , Justin Pryzby , PostgreSQL Hackers Subject: Re: New docs chapter on Transaction Management and related changes Message-ID: References: <20221014084615.chvollxv3oz43oqu@alvherre.pgsql> <80a5fd9c080d66e62f90c9565e3e1b54e61f584c.camel@cybertec.at> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="fDF5tz/WMJiLgjfm" Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --fDF5tz/WMJiLgjfm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Nov 7, 2022 at 10:58:05AM +0000, Simon Riggs wrote: > What I've posted is the merged patch, i.e. your latest patch, plus > changes to RELEASE SAVEPOINT from you on Oct 16, plus changes based on > the later comments from Robert and I. Thanks. I have two changes to your patch. First, I agree "destroy" is the wrong word for this, but I don't think "subcommit" is good, for three reasons: 1. Release merges the non-aborted changes into the previous transaction _and_ frees their resources --- "subcommit" doesn't have both meanings, which I think means if we need a single word, we should use "release" and later define what that means. 2. The "subcommit" concept doesn't closely match the user-visible behavior, even though we use subtransactions to accomplish this. Release is more of a rollup/merge into the previously-active transaction/savepoint. 3. "subcommit" is an implementation detail that I don't think we should expose to users in the manual pages. I adjusted the first paragraph of RELEASE SAVEPOINT to highlight the above issues. My original patch had similar wording. The first attachment shows my changes to your patch, and the second attachment is my full patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Indecision is a decision. Inaction is an action. Mark Batterson --fDF5tz/WMJiLgjfm Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="patches.diff" diff --git a/doc/src/sgml/ref/release_savepoint.sgml b/doc/src/sgml/ref/release_savepoint.sgml index 00b57ffcbb..34299f5618 100644 --- a/doc/src/sgml/ref/release_savepoint.sgml +++ b/doc/src/sgml/ref/release_savepoint.sgml @@ -21,7 +21,7 @@ PostgreSQL documentation RELEASE SAVEPOINT - subcommit a previously defined savepoint + release a previously defined savepoint @@ -34,16 +34,13 @@ RELEASE [ SAVEPOINT ] savepoint_name Description - RELEASE SAVEPOINT will subcommit the subtransaction - established by the named savepoint, if one exists. This will release - any resources held by the subtransaction. If there were any - subtransactions of the named savepoint, these will also be subcommitted. - - - - Changes made after RELEASE SAVEPOINT will be in - the same transaction, and have the same transaction id, as changes - made before the named savepoint was created. + RELEASE SAVEPOINT releases the named savepoint and + all active savepoints that were created after the named savepoint, + and frees their resources. All changes made since the creation of the + savepoint, excluding rolled back savepoints changes, are merged into + the transaction or savepoint that was active when the named savepoint + was created. Changes made after RELEASE SAVEPOINT + will also be part of this active transaction or savepoint. @@ -55,7 +52,7 @@ RELEASE [ SAVEPOINT ] savepoint_name savepoint_name - The name of the savepoint to subcommit. + The name of the savepoint to release. diff --git a/doc/src/sgml/ref/rollback.sgml b/doc/src/sgml/ref/rollback.sgml index 02b118fc04..e0e7c61560 100644 --- a/doc/src/sgml/ref/rollback.sgml +++ b/doc/src/sgml/ref/rollback.sgml @@ -61,9 +61,6 @@ ROLLBACK [ WORK | TRANSACTION ] [ AND [ NO ] CHAIN ] linkend="sql-set-transaction"/>) as the just finished one. Otherwise, no new transaction is started. - - The SQL Standard describes this as a chained transaction. - --fDF5tz/WMJiLgjfm Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="xact.diff" diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 559eb898a9..6df0092f4c 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7225,12 +7225,14 @@ local0.* /var/log/postgresql %v - Virtual transaction ID (backendID/localXID) + Virtual transaction ID (backendID/localXID); see + no %x - Transaction ID (0 if none is assigned) + Transaction ID (0 if none is assigned); see + no diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index b030b36002..fdffba4442 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4992,7 +4992,8 @@ WHERE ... xmin and xmax. Transaction identifiers are 32-bit quantities. In some contexts, a 64-bit variant xid8 is used. Unlike xid values, xid8 values increase strictly - monotonically and cannot be reused in the lifetime of a database cluster. + monotonically and cannot be reused in the lifetime of a database + cluster. See for more details. diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index de450cd661..0d6be9a2fa 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -104,6 +104,7 @@ + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 6e0425cb3d..153587c4b8 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -24673,7 +24673,10 @@ SELECT collation for ('foo' COLLATE "de_DE"); Returns the current transaction's ID. It will assign a new one if the current transaction does not have one already (because it has not - performed any database updates). + performed any database updates); see for details. If executed in a + subtransaction this will return the top-level xid; see for details. @@ -24690,6 +24693,7 @@ SELECT collation for ('foo' COLLATE "de_DE"); ID is assigned yet. (It's best to use this variant if the transaction might otherwise be read-only, to avoid unnecessary consumption of an XID.) + If executed in a subtransaction this will return the top-level xid. @@ -24733,6 +24737,8 @@ SELECT collation for ('foo' COLLATE "de_DE"); Returns a current snapshot, a data structure showing which transaction IDs are now in-progress. + Only top-level xids are included in the snapshot; subxids are not + shown; see for details. @@ -24787,7 +24793,8 @@ SELECT collation for ('foo' COLLATE "de_DE"); Is the given transaction ID visible according to this snapshot (that is, was it completed before the snapshot was taken)? Note that this function will not give the correct answer for - a subtransaction ID. + a subtransaction ID (subxid); see for + details. @@ -24799,8 +24806,9 @@ SELECT collation for ('foo' COLLATE "de_DE"); wraps around every 4 billion transactions. However, the functions shown in use a 64-bit type xid8 that does not wrap around during the life - of an installation, and can be converted to xid by casting if - required. The data type pg_snapshot stores information about + of an installation and can be converted to xid by casting if + required; see for details. + The data type pg_snapshot stores information about transaction ID visibility at a particular moment in time. Its components are described in . pg_snapshot's textual representation is @@ -24846,7 +24854,7 @@ SELECT collation for ('foo' COLLATE "de_DE"); xmax and not in this list was already completed at the time of the snapshot, and thus is either visible or dead according to its commit status. This list does not include the transaction IDs of - subtransactions. + subtransactions (subxids). diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml index d6d0a3a814..fe138a47bb 100644 --- a/doc/src/sgml/glossary.sgml +++ b/doc/src/sgml/glossary.sgml @@ -1710,7 +1710,8 @@ 3 (values under that are reserved) and the epoch value is incremented by one. In some contexts, the epoch and xid values are - considered together as a single 64-bit value. + considered together as a single 64-bit value; see for more details. For more information, see diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index e5d622d514..186d7ffa11 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -918,7 +918,8 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser backend_xid xid - Top-level transaction identifier of this backend, if any. + Top-level transaction identifier of this backend, if any; see + . @@ -1890,7 +1891,8 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser virtualxid - Waiting to acquire a virtual transaction ID lock. + Waiting to acquire a virtual transaction ID lock; see + . diff --git a/doc/src/sgml/pgrowlocks.sgml b/doc/src/sgml/pgrowlocks.sgml index 2914bf6e6d..ad15cda668 100644 --- a/doc/src/sgml/pgrowlocks.sgml +++ b/doc/src/sgml/pgrowlocks.sgml @@ -57,7 +57,8 @@ pgrowlocks(text) returns setof record locker xid - Transaction ID of locker, or multixact ID if multitransaction + Transaction ID of locker, or multixact ID if + multitransaction; see multi diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 73439c049e..2e271862fc 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -271,6 +271,7 @@ break is not needed in a wider output rendering. &brin; &hash; &storage; + &transaction; &bki; &planstats; &backup-manifest; diff --git a/doc/src/sgml/ref/commit.sgml b/doc/src/sgml/ref/commit.sgml index 5f244cdd3c..53d830998c 100644 --- a/doc/src/sgml/ref/commit.sgml +++ b/doc/src/sgml/ref/commit.sgml @@ -62,6 +62,9 @@ COMMIT [ WORK | TRANSACTION ] [ AND [ NO ] CHAIN ] linkend="sql-set-transaction"/>) as the just finished one. Otherwise, no new transaction is started. + + The SQL Standard describes this as a chained transaction. + diff --git a/doc/src/sgml/ref/release_savepoint.sgml b/doc/src/sgml/ref/release_savepoint.sgml index daf8eb9a43..34299f5618 100644 --- a/doc/src/sgml/ref/release_savepoint.sgml +++ b/doc/src/sgml/ref/release_savepoint.sgml @@ -21,7 +21,7 @@ PostgreSQL documentation RELEASE SAVEPOINT - destroy a previously defined savepoint + release a previously defined savepoint @@ -34,23 +34,13 @@ RELEASE [ SAVEPOINT ] savepoint_name Description - RELEASE SAVEPOINT destroys a savepoint previously defined - in the current transaction. - - - - Destroying a savepoint makes it unavailable as a rollback point, - but it has no other user visible behavior. It does not undo the - effects of commands executed after the savepoint was established. - (To do that, see .) - Destroying a savepoint when - it is no longer needed allows the system to reclaim some resources - earlier than transaction end. - - - - RELEASE SAVEPOINT also destroys all savepoints that were - established after the named savepoint was established. + RELEASE SAVEPOINT releases the named savepoint and + all active savepoints that were created after the named savepoint, + and frees their resources. All changes made since the creation of the + savepoint, excluding rolled back savepoints changes, are merged into + the transaction or savepoint that was active when the named savepoint + was created. Changes made after RELEASE SAVEPOINT + will also be part of this active transaction or savepoint. @@ -62,7 +52,7 @@ RELEASE [ SAVEPOINT ] savepoint_name savepoint_name - The name of the savepoint to destroy. + The name of the savepoint to release. @@ -78,7 +68,7 @@ RELEASE [ SAVEPOINT ] savepoint_name It is not possible to release a savepoint when the transaction is in - an aborted state. + an aborted state, to do that use . @@ -93,7 +83,7 @@ RELEASE [ SAVEPOINT ] savepoint_name Examples - To establish and later destroy a savepoint: + To establish and later release a savepoint: BEGIN; INSERT INTO table1 VALUES (3); @@ -104,6 +94,36 @@ COMMIT; The above transaction will insert both 3 and 4. + + + A more complex example with multiple nested subtransactions: + +BEGIN; + INSERT INTO table1 VALUES (1); + SAVEPOINT sp1; + INSERT INTO table1 VALUES (2); + SAVEPOINT sp2; + INSERT INTO table1 VALUES (3); + RELEASE SAVEPOINT sp2; + INSERT INTO table1 VALUES (4))); -- generates an error + + In this example, the application requests the release of the savepoint + sp2, which inserted 3. This changes the insert's + transaction context to sp1. When the statement + attempting to insert value 4 generates an error, the insertion of 2 and + 4 are lost because they are in the same, now-rolled back savepoint, + and value 3 is in the same transaction context. The application can + now only choose one of these two commands, since all other commands + will be ignored with a warning: + + ROLLBACK; + ROLLBACK TO SAVEPOINT sp1; + + Choosing ROLLBACK will abort everything, including + value 1, whereas ROLLBACK TO SAVEPOINT sp1 will retain + value 1 and allow the transaction to continue. + + diff --git a/doc/src/sgml/ref/rollback.sgml b/doc/src/sgml/ref/rollback.sgml index 142f71e774..e0e7c61560 100644 --- a/doc/src/sgml/ref/rollback.sgml +++ b/doc/src/sgml/ref/rollback.sgml @@ -56,7 +56,7 @@ ROLLBACK [ WORK | TRANSACTION ] [ AND [ NO ] CHAIN ] AND CHAIN - If AND CHAIN is specified, a new transaction is + If AND CHAIN is specified, a new unaborted transaction is immediately started with the same transaction characteristics (see ) as the just finished one. Otherwise, no new transaction is started. diff --git a/doc/src/sgml/ref/rollback_to.sgml b/doc/src/sgml/ref/rollback_to.sgml index 27fa95cd1b..32c1bb9723 100644 --- a/doc/src/sgml/ref/rollback_to.sgml +++ b/doc/src/sgml/ref/rollback_to.sgml @@ -35,8 +35,9 @@ ROLLBACK [ WORK | TRANSACTION ] TO [ SAVEPOINT ] savepoint_name Roll back all commands that were executed after the savepoint was - established. The savepoint remains valid and can be rolled back to - again later, if needed. + established and then start a new subtransaction at the same transaction level. + The savepoint remains valid and can be rolled back to again later, + if needed. diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 7c716fe327..4d8bc659f2 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1416,7 +1416,8 @@ Virtual ID of the transaction targeted by the lock, - or null if the target is not a virtual transaction ID + or null if the target is not a virtual transaction ID; see + @@ -1425,8 +1426,8 @@ transactionid xid - ID of the transaction targeted by the lock, - or null if the target is not a transaction ID + ID of the transaction targeted by the lock, or null if the target + is not a transaction ID; diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml index 6a38b53744..f335305113 100644 --- a/doc/src/sgml/wal.sgml +++ b/doc/src/sgml/wal.sgml @@ -4,8 +4,9 @@ Reliability and the Write-Ahead Log - This chapter explains how the Write-Ahead Log is used to obtain - efficient, reliable operation. + This chapter explains how to control the reliability of + PostgreSQL, including details about the + Write-Ahead Log. @@ -909,4 +910,36 @@ seem to be a problem in practice. + + + + Two-Phase Transactions + + + PostgreSQL supports a two-phase commit (2PC) + protocol that allows multiple distributed systems to work together + in a transactional manner. The commands are PREPARE + TRANSACTION, COMMIT PREPARED and + ROLLBACK PREPARED. Two-phase transactions + are intended for use by external transaction management systems. + PostgreSQL follows the features and model + proposed by the X/Open XA standard, but does not implement some less + often used aspects. + + + + When the user executes PREPARE TRANSACTION, the + only possible next commands are COMMIT PREPARED + or ROLLBACK PREPARED. In general, this prepared + state is intended to be of very short duration, but external + availability issues might mean transactions stay in this state + for an extended interval. Short-lived prepared + transactions are stored only in shared memory and WAL. + Transactions that span checkpoints are recorded in the + pg_twophase directory. Currently-prepared + transactions can be inspected using pg_prepared_xacts. + + + diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml new file mode 100644 index 0000000000..a14cc4c012 --- /dev/null +++ b/doc/src/sgml/xact.sgml @@ -0,0 +1,172 @@ + + + + + Transaction Management + + + This chapter provides an overview of the internals of + PostgreSQL's transaction management system. + The word transaction is often abbreviated as "xact". + + + + + Transactions and Identifiers + + + Transactions can be created explicitly using BEGIN + and COMMIT, which creates a transaction block. + An SQL statement outside of a transaction block automatically uses + a single-statement transaction. + + + + Every transaction is identified by a unique + VirtualTransactionId (also called + virtualXID or vxid), which + is comprised of a backend ID (or backendID) + and a sequentially-assigned number local to each backend, known as + localXID. For example, the virtual transaction + ID 4/12532 has a backendID + of 4 and a localXID of + 12532. + + + + Once a transaction writes to the database, it is assigned a + non-virtual TransactionId (or xid), + e.g., 278394. Xids are assigned sequentially + using a global counter used by all databases within the + PostgreSQL cluster. This property is used by + the transaction system to order transactions by their first database + write, i.e., lower-numbered xids started writing before higher-numbered + xids. Of course, transactions might start in a different order. + + + + The internal transaction ID type xid is 32-bits wide + and wraps around every + 4 billion transactions. A 32-bit epoch is incremented during each + wrap around. There is also a 64-bit type xid8 which + includes this epoch and therefore does not wrap around during the + life of an installation and can be converted to xid by casting. + The functions in + return xid8 values. Xids are used as the + basis for PostgreSQL's MVCC concurrency mechanism, Hot Standby, and Read Replica servers. + + + + When a top-level transaction with a (non-virtual) xid commits, + it is marked as committed in the pg_xact + directory. Additional information is recorded in the + pg_commit_ts directory if is enabled. + + + + In addition to vxid and xid, + when a transaction is prepared for two-phase commit it + is also identified by a Global Transaction Identifier + (GID). GIDs are string literals up to 200 + bytes long, which must be unique amongst other currently prepared + transactions. The mapping of GID to xid is shown in pg_prepared_xacts. + + + + + + Transactions and Locking + + + Currently-executing transactions are shown in pg_locks + in columns virtualxid and + transactionid. Read-only + transactions will have virtualxids but + NULL transactionids, while read-write + transactions will have both as non-NULL. + + + + Lock waits on table-level locks are shown waiting for + virtualxid, while lock waits on row-level + locks are shown waiting for transactionid. + Row-level read and write locks are recorded directly in locked + rows and can be inspected using the + extension. Row-level read locks might also require the assignment + of multixact IDs (mxid). Mxids are recorded in + the pg_multixact directory. + + + + + + Subtransactions + + + Subtransactions are started inside transactions, allowing large + transactions to be broken into smaller units. Subtransactions can + commit or abort without affecting their parent transactions, allowing + parent transactions to continue. This allows errors to be handled + more easily, which is a common application development pattern. + The word subtransaction is often abbreviated as + subxact. + + + + Subtransactions can be started explicitly using the + SAVEPOINT command, but can also be started in + other ways, such as PL/pgSQL's EXCEPTION clause. + PL/Python and PL/TCL also support explicit subtransactions. + Subtransactions can also be started from other subtransactions. + The top-level transaction and its child subtransactions form a + hierarchy or tree, which is why we refer to the main transaction as + the top-level transaction. + + + + If a subtransaction is assigned a non-virtual transaction ID, + its transaction ID is referred to as a subxid. + Read-only subtransactions are not assigned subxids, but once they + attempt to write, they will be assigned one. This also causes all of + a subxid's parents, up to and including the top-level transaction, + to be assigned non-virtual transaction ids. We ensure that a parent + xid is always lower than any of its child subxids. + + + + The immediate parent xid of each subxid is recorded in the + pg_subtrans directory. No entry is made for + top-level xids since they do not have a parent, nor is an entry made + for read-only subtransactions. + + + + When a subtransaction commits, all of its committed child + subtransactions with subxids will also be considered subcommitted + in that transaction. When a subtransaction aborts, all of its child + subtransactions will also be considered aborted. + + + + When a top-level transaction with an xid commits, all of its + subcommitted child subtransactions are also persistently recorded + as committed in the pg_xact directory. If the + top-level transaction aborts, all its subtransactions are also aborted, + even if they were subcommitted. + + + + The more subtransactions each transaction keeps open (not rolled back + or released), the greater the transaction management overhead. Up to + 64 open subxids are cached in shared memory for each backend; after + that point, the overhead increases significantly since we must look + up subxid entries in pg_subtrans. + + + + --fDF5tz/WMJiLgjfm--