agora inbox for [email protected]
help / color / mirror / Atom feedRe: PATCH: Batch/pipelining support for libpq
40+ messages / 7 participants
[nested] [flat]
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-07 16:52 Daniel Verite <[email protected]>
0 siblings, 2 replies; 40+ messages in thread
From: Daniel Verite @ 2017-03-07 16:52 UTC (permalink / raw)
To: Vaishnavi Prabakaran <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
Vaishnavi Prabakaran wrote:
> Yes, I have created a new patch entry into the commitfest 2017-03 and
> attached the latest patch with this e-mail.
Please find attached a companion patch implementing the batch API in
pgbench, exposed as \beginbatch and \endbatch meta-commands
(without documentation).
The idea for now is to make it easier to exercise the API and test
how batching performs. I guess I'll submit the patch separately in
a future CF, depending on when/if the libpq patch goes in.
While developing this, I noted a few things with 0001-v4:
1. lack of initialization for count in PQbatchQueueCount.
Trivial fix:
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1874,7 +1874,7 @@ PQisBusy(PGconn *conn)
int
PQbatchQueueCount(PGconn *conn)
{
- int count;
+ int count = 0;
PGcommandQueueEntry *entry;
2. misleading error message in PQexecStart. It gets called by a few other
functions than PQexec, such as PQprepare. As I understand it, the intent
here is to forbid the synchronous functions in batch mode, so this error
message should not single out PQexec.
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status !=
PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot
PQexec in batch mode\n"));
+ return false;
+ }
+
3. In relation to #2, PQsendQuery() is not forbidden in batch mode
although I don't think it can work with it, as it's based on the old
protocol.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..9b2fce8 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ PQbatchBegin(st->con);
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2339,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2356,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2375,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3256,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/plain] pgbench-batch-mode-v1.patch (5.4K, ../../[email protected]/2-pgbench-batch-mode-v1.patch)
download | inline diff:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..9b2fce8 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ PQbatchBegin(st->con);
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2339,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2356,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2375,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3256,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-08 01:56 Craig Ringer <[email protected]>
parent: Daniel Verite <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Craig Ringer @ 2017-03-08 01:56 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Vaishnavi Prabakaran <[email protected]>; Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On 8 March 2017 at 00:52, Daniel Verite <[email protected]> wrote:
> Vaishnavi Prabakaran wrote:
>
>> Yes, I have created a new patch entry into the commitfest 2017-03 and
>> attached the latest patch with this e-mail.
>
> Please find attached a companion patch implementing the batch API in
> pgbench, exposed as \beginbatch and \endbatch meta-commands
> (without documentation).
>
> The idea for now is to make it easier to exercise the API and test
> how batching performs. I guess I'll submit the patch separately in
> a future CF, depending on when/if the libpq patch goes in.
That's great, thanks, and thanks also for the fixes. Any chance you
can attach your updated patch?
I looked at modifying psql to support batching when run
non-interactively, but it would've required major restructuring of its
control loop and I ran out of time. I didn't think of modifying
pgbench. Great to see.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-08 02:04 Vaishnavi Prabakaran <[email protected]>
parent: Daniel Verite <[email protected]>
1 sibling, 2 replies; 40+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-08 02:04 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On Wed, Mar 8, 2017 at 3:52 AM, Daniel Verite <[email protected]>
wrote:
> Vaishnavi Prabakaran wrote:
>
> > Yes, I have created a new patch entry into the commitfest 2017-03 and
> > attached the latest patch with this e-mail.
>
> Please find attached a companion patch implementing the batch API in
> pgbench, exposed as \beginbatch and \endbatch meta-commands
> (without documentation).
>
> The idea for now is to make it easier to exercise the API and test
> how batching performs. I guess I'll submit the patch separately in
> a future CF, depending on when/if the libpq patch goes in.
>
>
Thanks for the companion patch and here are some comments:
1. I see, below check is used to verify if the connection is not in batch
mode:
if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
But, it is better to use if (PQbatchStatus(st->con) ==
PQBATCH_MODE_OFF) for this verification. Reason is there is one more state
in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in aborted
status, this check will still assume that the connection is not in batch
mode.
In a same way, "if(PQbatchStatus(st->con) == PQBATCH_MODE_ON)" check is
better to be modified as "PQbatchStatus(st->con) != PQBATCH_MODE_OFF".
2. @@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData
*agg)
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
This is not required as below PQbatchBegin() internally does this
verification.
+ PQbatchBegin(st->con);
3. It is better to check the return value of PQbatchBegin() rather than
assuming success. E.g: PQbatchBegin() will return false if the connection
is in copy in/out/both states.
> While developing this, I noted a few things with 0001-v4:
>
> 1. lack of initialization for count in PQbatchQueueCount.
> Trivial fix:
>
> --- a/src/interfaces/libpq/fe-exec.c
> +++ b/src/interfaces/libpq/fe-exec.c
> @@ -1874,7 +1874,7 @@ PQisBusy(PGconn *conn)
> int
> PQbatchQueueCount(PGconn *conn)
> {
> - int count;
> + int count = 0;
> PGcommandQueueEntry *entry;
>
>
Thanks for your review and yes, Corrected.
> 2. misleading error message in PQexecStart. It gets called by a few other
> functions than PQexec, such as PQprepare. As I understand it, the intent
> here is to forbid the synchronous functions in batch mode, so this error
> message should not single out PQexec.
>
> @@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
> if (!conn)
> return false;
>
> + if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status !=
> PQBATCH_MODE_OFF)
> + {
> + printfPQExpBuffer(&conn->errorMessage,
> + libpq_gettext("cannot
> PQexec in batch mode\n"));
> + return false;
> + }
> +
>
>
Hmm, this error message goes with the flow of other error messages in the
same function. E.g: "PQexec not allowed during COPY BOTH" . But, anyways I
modified the
error message to be more generic.
> 3. In relation to #2, PQsendQuery() is not forbidden in batch mode
> although I don't think it can work with it, as it's based on the old
> protocol.
>
>
>
It is actually forbidden and you can see the error message "cannot
PQsendQuery in batch mode, use PQsendQueryParams" when you use
PQsendQuery().
Attached the updated patch.
Thanks & Regards,
Vaishnavi
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] 0001-Pipelining-batch-support-for-libpq-code-v5.patch (49.8K, ../../CAOoUkxRG=TUSbvitGV5H3_VBzgXLXxsaLeRTcTzWzsOS2JFHtg@mail.gmail.com/3-0001-Pipelining-batch-support-for-libpq-code-v5.patch)
download | inline diff:
---
doc/src/sgml/libpq.sgml | 511 +++++++++++++++++++++++++++++++
src/interfaces/libpq/.gitignore | 1 +
src/interfaces/libpq/Makefile | 5 +
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 590 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
10 files changed, 1188 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..899cc5d 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,517 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/.gitignore b/src/interfaces/libpq/.gitignore
index cb96af7..4c0d934 100644
--- a/src/interfaces/libpq/.gitignore
+++ b/src/interfaces/libpq/.gitignore
@@ -22,3 +22,4 @@
/encnames.c
/wchar.c
/libpq.rc
+/tmp_check
diff --git a/src/interfaces/libpq/Makefile b/src/interfaces/libpq/Makefile
index 4b1e552..8d5cf21 100644
--- a/src/interfaces/libpq/Makefile
+++ b/src/interfaces/libpq/Makefile
@@ -129,6 +129,11 @@ install: all installdirs install-lib
installcheck:
$(MAKE) -C test $@
+check: prove-check
+
+prove-check:
+ $(prove_check)
+
installdirs: installdirs-lib
$(MKDIR_P) '$(DESTDIR)$(includedir)' '$(DESTDIR)$(includedir_internal)' '$(DESTDIR)$(datadir)'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 65b7c31..f6530e7 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3417,6 +3417,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3428,6 +3447,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3460,6 +3480,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b551875..abae83c 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,300 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+
+ /* Parse any available data */
+ parseInput(conn);
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2220,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2425,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2626,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2644,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2666,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2691,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index e9b73a9..5792247 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -217,10 +217,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -229,7 +234,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -299,6 +305,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -386,6 +408,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -398,6 +421,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -690,6 +723,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
[application/octet-stream] 0002-Pipelining-batch-support-for-libpq-test-v3.patch (43.4K, ../../CAOoUkxRG=TUSbvitGV5H3_VBzgXLXxsaLeRTcTzWzsOS2JFHtg@mail.gmail.com/4-0002-Pipelining-batch-support-for-libpq-test-v3.patch)
download | inline diff:
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-09 21:43 Daniel Verite <[email protected]>
parent: Vaishnavi Prabakaran <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Daniel Verite @ 2017-03-09 21:43 UTC (permalink / raw)
To: Vaishnavi Prabakaran <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
Vaishnavi Prabakaran wrote:
> if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
>
> But, it is better to use if (PQbatchStatus(st->con) ==
> PQBATCH_MODE_OFF) for this verification. Reason is there is one more state
> in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in aborted
> status, this check will still assume that the connection is not in batch
> mode.
> In a same way, "if(PQbatchStatus(st->con) == PQBATCH_MODE_ON)" check is
> better to be modified as "PQbatchStatus(st->con) != PQBATCH_MODE_OFF".
Agreed, these two tests need to be changed to account for the
PQBATCH_MODE_ABORTED case. Fixed in new patch.
> 2. @@ -2207,7 +2227,47 @@ doCustom(TState *thread, CState *st, StatsData
> *agg)
> + if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
> + {
> + commandFailed(st, "already in batch mode");
> + break;
> + }
>
> This is not required as below PQbatchBegin() internally does this
> verification.
>
> + PQbatchBegin(st->con);
The point of this test is to specifically forbid a sequence like this
\beginbatch
...(no \endbatch)
\beginbatch
...
and according to the doc for PQbatchBegin, it looks like the return
code won't tell us:
"Causes a connection to enter batch mode if it is currently idle or
already in batch mode.
int PQbatchBegin(PGconn *conn);
Returns 1 for success"
My understanding is that "already in batch mode" is not an error.
"Returns 0 and has no effect if the connection is not currently
idle, i.e. it has a result ready, is waiting for more input from
the server, etc. This function does not actually send anything to
the server, it just changes the libpq connection state"
> 3. It is better to check the return value of PQbatchBegin() rather than
> assuming success. E.g: PQbatchBegin() will return false if the connection
> is in copy in/out/both states.
Given point #2 above, I think both tests are needed:
if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
{
commandFailed(st, "already in batch mode");
break;
}
if (PQbatchBegin(st->con) == 0)
{
commandFailed(st, "failed to start a batch");
break;
}
> > 3. In relation to #2, PQsendQuery() is not forbidden in batch mode
> > although I don't think it can work with it, as it's based on the old
> > protocol.
> >
> It is actually forbidden and you can see the error message "cannot
> PQsendQuery in batch mode, use PQsendQueryParams" when you use
> PQsendQuery().
Sorry for blaming the innocent piece of code, looking closer
it's pgbench that does not display the message.
With the simple query protocol, sendCommand() does essentially:
r = PQsendQuery(st->con, sql);
... other stuff...
if (r == 0)
{
if (debug)
fprintf(stderr, "client %d could not send %s\n",
st->id, command->argv[0]);
st->ecnt++;
return false;
}
So only in debug mode does it inform the user that it failed, and
even then, it does not display PQerrorMessage(st->con).
In non-debug mode it's opaque to the user. If it always fail, it appears
to loop forever. The caller has this comment that is relevant to the problem:
if (!sendCommand(st, command))
{
/*
* Failed. Stay in CSTATE_START_COMMAND state, to
* retry. ??? What the point or retrying? Should
* rather abort?
*/
return;
}
I think it doesn't bother anyone up to now because the immediate
failure modes of PQsendQuery() are resource allocation or protocol
failures that tend to never happen.
Anyway currently \beginbatch avoids the problem by checking
whether querymode == QUERY_SIMPLE, to fail gracefully
instead of letting the endless loop happen.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..f93673e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,51 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ if (PQbatchBegin(st->con) == 0)
+ {
+ commandFailed(st, "failed to start a batch");
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2343,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2360,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2379,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3260,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/plain] pgbench-batch-mode-v2.patch (5.5K, ../../[email protected]/2-pgbench-batch-mode-v2.patch)
download | inline diff:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..f93673e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -269,7 +269,8 @@ typedef enum
*
* CSTATE_START_COMMAND starts the execution of a command. On a SQL
* command, the command is sent to the server, and we move to
- * CSTATE_WAIT_RESULT state. On a \sleep meta-command, the timer is set,
+ * CSTATE_WAIT_RESULT state unless in batch mode.
+ * On a \sleep meta-command, the timer is set,
* and we enter the CSTATE_SLEEP state to wait for it to expire. Other
* meta-commands are executed immediately.
*
@@ -1882,11 +1883,24 @@ sendCommand(CState *st, Command *command)
if (commands[j]->type != SQL_COMMAND)
continue;
preparedStatementName(name, st->use_file, j);
- res = PQprepare(st->con, name,
- commands[j]->argv[0], commands[j]->argc - 1, NULL);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- fprintf(stderr, "%s", PQerrorMessage(st->con));
- PQclear(res);
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ {
+ res = PQprepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ PQclear(res);
+ }
+ else
+ {
+ /*
+ * In batch mode, we use asynchronous functions. If a server-side
+ * error occurs, it will be processed later among the other results.
+ */
+ if (!PQsendPrepare(st->con, name,
+ commands[j]->argv[0], commands[j]->argc - 1, NULL))
+ fprintf(stderr, "%s", PQerrorMessage(st->con));
+ }
}
st->prepared[st->use_file] = true;
}
@@ -2165,7 +2179,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
return;
}
else
- st->state = CSTATE_WAIT_RESULT;
+ {
+ /* Wait for results, unless in batch mode */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_OFF)
+ st->state = CSTATE_WAIT_RESULT;
+ else
+ st->state = CSTATE_END_COMMAND;
+ }
}
else if (command->type == META_COMMAND)
{
@@ -2207,7 +2227,51 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
else
{
- if (pg_strcasecmp(argv[0], "set") == 0)
+ if (pg_strcasecmp(argv[0], "beginbatch") == 0)
+ {
+ /*
+ * In batch mode, we use a workflow based on libpq batch
+ * functions.
+ */
+ if (querymode == QUERY_SIMPLE)
+ {
+ commandFailed(st, "cannot use batch mode with the simple query protocol");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_OFF)
+ {
+ commandFailed(st, "already in batch mode");
+ break;
+ }
+ if (PQbatchBegin(st->con) == 0)
+ {
+ commandFailed(st, "failed to start a batch");
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "endbatch") == 0)
+ {
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ {
+ commandFailed(st, "not in batch mode");
+ break;
+ }
+ if (!PQbatchQueueSync(st->con))
+ {
+ commandFailed(st, "failed to end the batch");
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQbatchEnd(st->con) == 0)
+ {
+ /* collect pending results before getting out of batch mode */
+ st->state = CSTATE_WAIT_RESULT;
+ break;
+ }
+ }
+ else if (pg_strcasecmp(argv[0], "set") == 0)
{
PgBenchExpr *expr = command->expr;
PgBenchValue result;
@@ -2279,6 +2343,7 @@ doCustom(TState *thread, CState *st, StatsData *agg)
}
break;
+
/*
* Wait for the current SQL command to complete
*/
@@ -2295,6 +2360,13 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
+ if (PQbatchStatus(st->con) == PQBATCH_MODE_ON &&
+ !PQbatchQueueProcess(st->con))
+ {
+ /* no complete result yet in batch mode*/
+ return;
+ }
+
/*
* Read and discard the query result;
*/
@@ -2307,7 +2379,22 @@ doCustom(TState *thread, CState *st, StatsData *agg)
/* OK */
PQclear(res);
discard_response(st);
- st->state = CSTATE_END_COMMAND;
+ /*
+ * In non-batch mode, only one result per command is expected.
+ * In batch mode, keep waiting for results until getting
+ * PGRES_BATCH_END.
+ */
+ if (PQbatchStatus(st->con) != PQBATCH_MODE_ON)
+ st->state = CSTATE_END_COMMAND;
+ break;
+ case PGRES_BATCH_END:
+ if (PQbatchEnd(st->con) == 1)
+ {
+ /* all results collected, exit out of command and batch mode */
+ st->state = CSTATE_END_COMMAND;
+ }
+ else
+ fprintf(stderr, "client %d to exit batch mode", st->id);
break;
default:
commandFailed(st, PQerrorMessage(st->con));
@@ -3173,6 +3260,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "beginbatch") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "endbatch") == 0 )
+ {
+ if (my_command->argc > 1)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "unexpected argument", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-10 13:52 Daniel Verite <[email protected]>
parent: Vaishnavi Prabakaran <[email protected]>
1 sibling, 1 reply; 40+ messages in thread
From: Daniel Verite @ 2017-03-10 13:52 UTC (permalink / raw)
To: Vaishnavi Prabakaran <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
Hi,
I notice that PQsetSingleRowMode() doesn't work when getting batch results.
The function is documented as:
" int PQsetSingleRowMode(PGconn *conn);
This function can only be called immediately after PQsendQuery or one
of its sibling functions, before any other operation on the connection
such as PQconsumeInput or PQgetResult"
But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
so whatever it was when sending the query gets lost, and besides
other queries might have been submitted in the meantime.
Also if trying to set that mode when fetching like this
while (QbatchQueueProcess(conn)) {
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed");
}
..
it might work the first time, but on the next iterations, conn->asyncStatus
might be PGASYNC_READY, which is a failure condition for
PQsetSingleRowMode(), so that won't do.
ISTM that the simplest fix would be that when in batch mode,
PQsetSingleRowMode() should register that the last submitted query
does request that mode, and when later QbatchQueueProcess dequeues
the batch entry for the corresponding query, this flag would be popped off
and set as the current mode.
Please find attached the suggested fix, against the v5 of the patch.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] fix-for-singlerowmode.diff (2.0K, ../../[email protected]/2-fix-for-singlerowmode.diff)
download | inline diff:
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 3c0be46..8ddf63d 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -1425,7 +1425,7 @@ PQmakePipelinedCommand(PGconn *conn)
}
entry->next = NULL;
entry->query = NULL;
-
+ entry->singleRowMode = false;
return entry;
}
@@ -1783,16 +1783,28 @@ PQsetSingleRowMode(PGconn *conn)
/*
* Only allow setting the flag when we have launched a query and not yet
* received any results.
+ * In batch mode, store the flag in the queue for applying it later.
*/
if (!conn)
return 0;
- if (conn->asyncStatus != PGASYNC_BUSY)
- return 0;
if (conn->queryclass != PGQUERY_SIMPLE &&
conn->queryclass != PGQUERY_EXTENDED)
return 0;
if (conn->result)
return 0;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (conn->asyncStatus != PGASYNC_BUSY)
+ return 0;
+ }
+ else
+ {
+ /* apply to the last submitted query in the batch, or fail */
+ if (conn->cmd_queue_tail != NULL)
+ conn->cmd_queue_tail->singleRowMode = true;
+ else
+ return 0;
+ }
/* OK, set flag */
conn->singleRowMode = true;
@@ -2120,9 +2132,8 @@ PQbatchQueueProcess(PGconn *conn)
* Initialize async result-accumulation state */
pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
-
+ /* Set single-row processing mode */
+ conn->singleRowMode = next_query->singleRowMode;
conn->last_query = next_query->query;
next_query->query = NULL;
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 33f212f..af4f753 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -315,6 +315,7 @@ typedef struct pgCommandQueueEntry
{
PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
char *query; /* SQL command, or NULL if unknown */
+ bool singleRowMode; /* apply single row mode to this query */
struct pgCommandQueueEntry *next;
} PGcommandQueueEntry;
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-13 00:54 Vaishnavi Prabakaran <[email protected]>
parent: Daniel Verite <[email protected]>
0 siblings, 2 replies; 40+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-13 00:54 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On Sat, Mar 11, 2017 at 12:52 AM, Daniel Verite <[email protected]>
wrote:
> Hi,
>
> I notice that PQsetSingleRowMode() doesn't work when getting batch results.
>
> The function is documented as:
> " int PQsetSingleRowMode(PGconn *conn);
>
> This function can only be called immediately after PQsendQuery or one
> of its sibling functions, before any other operation on the connection
> such as PQconsumeInput or PQgetResult"
>
> But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
> so whatever it was when sending the query gets lost, and besides
> other queries might have been submitted in the meantime.
>
> Also if trying to set that mode when fetching like this
>
> while (QbatchQueueProcess(conn)) {
> r = PQsetSingleRowMode(conn);
> if (r!=1) {
> fprintf(stderr, "PQsetSingleRowMode() failed");
> }
> ..
>
> it might work the first time, but on the next iterations, conn->asyncStatus
> might be PGASYNC_READY, which is a failure condition for
> PQsetSingleRowMode(), so that won't do.
>
Thanks for investigating the problem, and could you kindly explain what
"next iteration" you mean here? Because I don't see any problem in
following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
, PQgetResult(). Am I missing anything?
Please note that it is MUST to call PQgetResult immediately after
PQbatchQueueProcess() as documented.
>
> ISTM that the simplest fix would be that when in batch mode,
> PQsetSingleRowMode() should register that the last submitted query
> does request that mode, and when later QbatchQueueProcess dequeues
> the batch entry for the corresponding query, this flag would be popped off
> and set as the current mode.
>
> Please find attached the suggested fix, against the v5 of the patch.
>
Before going with this fix, I would like you to consider the option of
asking batch processing users(via documentation) to set single-row mode
before calling PQgetResult().
Either way we need to fix the documentation part, letting users know how
they can activate single-row mode while using batch processing.
Let me know your thoughts.
Best Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-13 01:32 Craig Ringer <[email protected]>
parent: Vaishnavi Prabakaran <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Craig Ringer @ 2017-03-13 01:32 UTC (permalink / raw)
To: Vaishnavi Prabakaran <[email protected]>; +Cc: Daniel Verite <[email protected]>; Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On 13 March 2017 at 08:54, Vaishnavi Prabakaran
<[email protected]> wrote:
> Before going with this fix, I would like you to consider the option of
> asking batch processing users(via documentation) to set single-row mode
> before calling PQgetResult().
> Either way we need to fix the documentation part, letting users know how
> they can activate single-row mode while using batch processing.
> Let me know your thoughts.
Thanks for looking into this, it's clear that I didn't cover the combo
of single-row-mode and batch mode adequately.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-13 17:19 Daniel Verite <[email protected]>
parent: Vaishnavi Prabakaran <[email protected]>
1 sibling, 1 reply; 40+ messages in thread
From: Daniel Verite @ 2017-03-13 17:19 UTC (permalink / raw)
To: Vaishnavi Prabakaran <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
Vaishnavi Prabakaran wrote:
> > while (QbatchQueueProcess(conn)) {
> > r = PQsetSingleRowMode(conn);
> > if (r!=1) {
> > fprintf(stderr, "PQsetSingleRowMode() failed");
> > }
> > ..
> Thanks for investigating the problem, and could you kindly explain what
> "next iteration" you mean here? Because I don't see any problem in
> following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
> , PQgetResult()
I mean the next iteration of the above while statement. Referring
to the doc, that would be the "next batch entry":
" To get the result of the first batch entry the client must call
PQbatchQueueProcess. It must then call PQgetResult and handle the
results until PQgetResult returns null (or would return null if
called). The result from the next batch entry may then be retrieved
using PQbatchQueueProcess and the cycle repeated"
Attached is a bare-bones testcase showing the problem.
As it is, it works, retrieving results for three "SELECT 1"
in the same batch. Now in order to use the single-row
fetch mode, consider adding this:
r = PQsetSingleRowMode(conn);
if (r!=1) {
fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
}
When inserted after the call to PQbatchQueueProcess,
which is what I understand you're saying works for you,
it fails for me when starting to get the results of the 2nd query
and after.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] test-singlerow-batch.c (981B, ../../[email protected]/2-test-singlerow-batch.c)
download
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-14 06:50 Vaishnavi Prabakaran <[email protected]>
parent: Daniel Verite <[email protected]>
0 siblings, 2 replies; 40+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-14 06:50 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite <[email protected]>
wrote:
>
> I mean the next iteration of the above while statement. Referring
> to the doc, that would be the "next batch entry":
>
> " To get the result of the first batch entry the client must call
> PQbatchQueueProcess. It must then call PQgetResult and handle the
> results until PQgetResult returns null (or would return null if
> called). The result from the next batch entry may then be retrieved
> using PQbatchQueueProcess and the cycle repeated"
>
> Attached is a bare-bones testcase showing the problem.
> As it is, it works, retrieving results for three "SELECT 1"
> in the same batch. Now in order to use the single-row
> fetch mode, consider adding this:
>
> r = PQsetSingleRowMode(conn);
> if (r!=1) {
> fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
> }
>
> When inserted after the call to PQbatchQueueProcess,
> which is what I understand you're saying works for you,
> it fails for me when starting to get the results of the 2nd query
> and after.
>
>
Thanks for explaining the issue in detail and yes, I do see the issue using
your attached test file.
And I think the problem is with the "parseInput" call at the end of
PQbatchQueueProcess().
I don't see the need for "parseInput" to cover the scope of
PQbatchQueueProcess(), also anyways we do it in PQgetResult().
This function call changes the asyncstatus of connection to READY(following
command complete message from backend), so eventually PQsetSingleRowMode()
is failing. So, attached the alternative fix for this issue.
Please share me your thoughts.
I would also like to hear Craig's opinion on it before applying this fix to
the original patch, just to make sure am not missing anything here.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] 0002-single-row-mode-fix.patch (702B, ../../CAOoUkxTrru01azpXBadqR-gEHxJ+VK5ReKEpO0ih22TyxBV2HA@mail.gmail.com/3-0002-single-row-mode-fix.patch)
download | inline diff:
From bc320c1f03afe2c614dc75e47aa10b39256df0a7 Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <[email protected]>
Date: Tue, 14 Mar 2017 17:34:32 +1100
Subject: [PATCH 2/2] single row mode fix
---
src/interfaces/libpq/fe-exec.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 3c0be46..129b49a 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -2155,7 +2155,7 @@ PQbatchQueueProcess(PGconn *conn)
conn->asyncStatus = PGASYNC_BUSY;
/* Parse any available data */
- parseInput(conn);
+ /* parseInput(conn); */
}
return true;
--
2.7.4.windows.1
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-16 02:42 Vaishnavi Prabakaran <[email protected]>
parent: Vaishnavi Prabakaran <[email protected]>
1 sibling, 0 replies; 40+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-16 02:42 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On Tue, Mar 14, 2017 at 5:50 PM, Vaishnavi Prabakaran <
[email protected]> wrote:
>
>
> On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite <[email protected]>
> wrote:
>
>>
>> I mean the next iteration of the above while statement. Referring
>> to the doc, that would be the "next batch entry":
>>
>> " To get the result of the first batch entry the client must call
>> PQbatchQueueProcess. It must then call PQgetResult and handle the
>> results until PQgetResult returns null (or would return null if
>> called). The result from the next batch entry may then be retrieved
>> using PQbatchQueueProcess and the cycle repeated"
>>
>> Attached is a bare-bones testcase showing the problem.
>> As it is, it works, retrieving results for three "SELECT 1"
>> in the same batch. Now in order to use the single-row
>> fetch mode, consider adding this:
>>
>> r = PQsetSingleRowMode(conn);
>> if (r!=1) {
>> fprintf(stderr, "PQsetSingleRowMode() failed for i=%d\n", i);
>> }
>>
>> When inserted after the call to PQbatchQueueProcess,
>> which is what I understand you're saying works for you,
>> it fails for me when starting to get the results of the 2nd query
>> and after.
>>
>>
>
> Thanks for explaining the issue in detail and yes, I do see the issue
> using your attached test file.
>
> And I think the problem is with the "parseInput" call at the end of
> PQbatchQueueProcess().
> I don't see the need for "parseInput" to cover the scope of
> PQbatchQueueProcess(), also anyways we do it in PQgetResult().
> This function call changes the asyncstatus of connection to
> READY(following command complete message from backend), so eventually
> PQsetSingleRowMode() is failing. So, attached the alternative fix for this
> issue.
>
Please share me your thoughts.
>
> I would also like to hear Craig's opinion on it before applying this fix
> to the original patch, just to make sure am not missing anything here.
>
>
Attached the code patch applied with the fix explained above and moving the
CF status seeking review.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/octet-stream] 0001-Pipelining-batch-support-for-libpq-code-v6.patch (49.5K, ../../CAOoUkxRvkPPcQjPWqM2Z1Dwi=ucnZ0oQ2tegryjPz++Qq+eZYQ@mail.gmail.com/3-0001-Pipelining-batch-support-for-libpq-code-v6.patch)
download | inline diff:
From 1d6efd751051399555ec8cb902500421e8f7468b Mon Sep 17 00:00:00 2001
From: Vaishnavi Prabakaran <[email protected]>
Date: Thu, 16 Mar 2017 13:32:08 +1100
Subject: [PATCH] Pipelining-batch-support-for-libpq-code-v6
---
doc/src/sgml/libpq.sgml | 519 +++++++++++++++++++++++++++++++
src/interfaces/libpq/exports.txt | 6 +
src/interfaces/libpq/fe-connect.c | 28 ++
src/interfaces/libpq/fe-exec.c | 587 ++++++++++++++++++++++++++++++++++--
src/interfaces/libpq/fe-protocol2.c | 6 +
src/interfaces/libpq/fe-protocol3.c | 15 +-
src/interfaces/libpq/libpq-fe.h | 24 +-
src/interfaces/libpq/libpq-int.h | 41 ++-
8 files changed, 1187 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 4bc5bf3..e82721e 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -4655,6 +4655,525 @@ int PQflush(PGconn *conn);
</sect1>
+ <sect1 id="libpq-batch-mode">
+ <title>Batch mode and query pipelining</title>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>batch mode</secondary>
+ </indexterm>
+
+ <indexterm zone="libpq-batch-mode">
+ <primary>libpq</primary>
+ <secondary>pipelining</secondary>
+ </indexterm>
+
+ <para>
+ <application>libpq</application> supports queueing up multiple queries into
+ a pipeline to be executed as a batch on the server. Batching queries allows
+ applications to avoid a client/server round-trip after each query to get
+ the results before issuing the next query.
+ </para>
+
+ <para>
+ An example of batch use may be found in the source distribution in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename>.
+ </para>
+
+ <sect2>
+ <title>When to use batching</title>
+
+ <para>
+ Much like asynchronous query mode, there is no performance disadvantage to
+ using batching and pipelining. It increases client application complexity
+ and extra caution is required to prevent client/server deadlocks but
+ offers considerable performance improvements.
+ </para>
+
+ <para>
+ Batching is most useful when the server is distant, i.e. network latency
+ (<quote>ping time</quote>) is high, and when many small operations are being performed in
+ rapid sequence. There is usually less benefit in using batches when each
+ query takes many multiples of the client/server round-trip time to execute.
+ A 100-statement operation run on a server 300ms round-trip-time away would take
+ 30 seconds in network latency alone without batching; with batching it may spend
+ as little as 0.3s waiting for results from the server.
+ </para>
+
+ <para>
+ Use batches when your application does lots of small
+ <literal>INSERT</literal>, <literal>UPDATE</literal> and
+ <literal>DELETE</literal> operations that can't easily be transformed into
+ operations on sets or into a
+ <link linkend="libpq-copy"><literal>COPY</literal></link> operation.
+ </para>
+
+ <para>
+ Batching is less useful when information from one operation is required by the
+ client before it knows enough to send the next operation. The client must
+ introduce a synchronisation point and wait for a full client/server
+ round-trip to get the results it needs. However, it's often possible to
+ adjust the client design to exchange the required information server-side.
+ Read-modify-write cycles are especially good candidates; for example:
+ <programlisting>
+ BEGIN;
+ SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
+ -- result: x=2
+ -- client adds 1 to x:
+ UPDATE mytable SET x = 3 WHERE id = 42;
+ COMMIT;
+ </programlisting>
+ could be much more efficiently done with:
+ <programlisting>
+ UPDATE mytable SET x = x + 1 WHERE id = 42;
+ </programlisting>
+ </para>
+
+ <note>
+ <para>
+ The batch API was introduced in PostgreSQL 10.0, but clients using PostgresSQL 10.0 version of libpq can
+ use batches on server versions 8.4 and newer. Batching works on any server
+ that supports the v3 extended query protocol.
+ </para>
+ </note>
+
+ </sect2>
+
+ <sect2 id="libpq-batch-using">
+ <title>Using batch mode</title>
+
+ <para>
+ To issue batches the application must switch
+ <application>libpq</application> into batch mode. Enter batch mode with <link
+ linkend="libpq-PQbatchBegin"><function>PQbatchBegin(conn)</function></link> or test
+ whether batch mode is active with <link
+ linkend="libpq-PQbatchStatus"><function>PQbatchStatus(conn)</function></link>. In batch mode only <link
+ linkend="libpq-async">asynchronous operations</link> are permitted, and
+ <literal>COPY</literal> is not recommended as it most likely will trigger failure in batch processing.
+ (The restriction on <literal>COPY</literal> is an implementation
+ limit; the PostgreSQL protocol and server can support batched <literal>COPY</literal>).
+ </para>
+
+ <para>
+ The client uses libpq's asynchronous query functions to dispatch work,
+ marking the end of each batch with <function>PQbatchQueueSync</function>.
+ Concurrently, it uses <function>PQgetResult</function> and
+ <function>PQbatchQueueProcess</function> to get results. It may eventually exit
+ batch mode with <function>PQbatchEnd</function> once all results are
+ processed.
+ </para>
+
+ <note>
+ <para>
+ It is best to use batch mode with <application>libpq</application> in
+ <link linkend="libpq-pqsetnonblocking">non-blocking mode</link>. If used in
+ blocking mode it is possible for a client/server deadlock to occur. The
+ client will block trying to send queries to the server, but the server will
+ block trying to send results from queries it has already processed to the
+ client. This only occurs when the client sends enough queries to fill its
+ output buffer and the server's receive buffer before switching to
+ processing input from the server, but it's hard to predict exactly when
+ that'll happen so it's best to always use non-blocking mode.
+ </para>
+ </note>
+
+ <sect3 id="libpq-batch-sending">
+ <title>Issuing queries</title>
+
+ <para>
+ After entering batch mode the application dispatches requests
+ using normal asynchronous <application>libpq</application> functions such as
+ <function>PQsendQueryParams</function>, <function>PQsendPrepare</function>,
+ <function>PQsendQueryPrepared</function>, <function>PQdescribePortal</function>,
+ <function>PQdescribePrepared</function>, <function>PQsendDescribePortal</function>,
+ <function>PQsendDescribePrepared</function>.
+ The asynchronous requests are followed by a <link
+ linkend="libpq-PQbatchQueueSync"><function>PQbatchQueueSync(conn)</function></link> call to mark
+ the end of the batch. The client <emphasis>does not</emphasis> need to call
+ <function>PQgetResult</function> immediately after dispatching each
+ operation. <link linkend="libpq-batch-results">Result processing</link>
+ is handled separately.
+ </para>
+
+ <para>
+ Batched operations will be executed by the server in the order the client
+ sends them. The server will send the results in the order the statements
+ executed. The server may begin executing the batch before all commands
+ in the batch are queued and the end of batch command is sent. If any
+ statement encounters an error the server aborts the current transaction and
+ skips processing the rest of the batch. Query processing resumes after the
+ end of the failed batch.
+ </para>
+
+ <para>
+ It's fine for one operation to depend on the results of a
+ prior one. One query may define a table that the next query in the same
+ batch uses; similarly, an application may create a named prepared statement
+ then execute it with later statements in the same batch.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-results">
+ <title>Processing results</title>
+
+ <para>
+ The client <link linkend="libpq-batch-interleave">interleaves result
+ processing with sending batch queries</link>, or for small batches may
+ process all results after sending the whole batch.
+ </para>
+
+ <para>
+ To get the result of the first batch entry the client must call <link
+ linkend="libpq-PQbatchQueueProcess"><function>PQbatchQueueProcess</function></link>. It must then call
+ <function>PQgetResult</function> and handle the results until
+ <function>PQgetResult</function> returns null (or would return null if
+ called). The result from the next batch entry may then be retrieved using
+ <function>PQbatchQueueProcess</function> and the cycle repeated. The
+ application handles individual statement results as normal.
+ </para>
+
+ <para>
+ To enter single-row mode, call <function>PQsetSingleRowMode</function> immediately after a
+ successful call of <function>PQbatchQueueProcess</function>. This mode selection is effective
+ only for the currently executing query. For more information on the use of <function>PQsetSingleRowMode
+ </function>, refer to <xref linkend="libpq-single-row-mode">.
+
+ </para>
+
+ <para>
+ <function>PQgetResult</function> behaves the same as for normal asynchronous
+ processing except that it may contain the new <type>PGresult</type> types
+ <literal>PGRES_BATCH_END</literal> and <literal>PGRES_BATCH_ABORTED</literal>.
+ <literal>PGRES_BATCH_END</literal> is reported exactly once for each
+ <function>PQbatchQueueSync</function> call at the corresponding point in
+ the result stream and at no other time. <literal>PGRES_BATCH_ABORTED</literal>
+ is emitted during error handling; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+ </para>
+
+ <para>
+ <function>PQisBusy</function>, <function>PQconsumeInput</function>, etc
+ operate as normal when processing batch results.
+ </para>
+
+ <para>
+ <application>libpq</application> does not provide any information to the
+ application about the query currently being processed. The application
+ must keep track of the order in which it sent queries and the expected
+ results. Applications will typically use a state machine or a FIFO queue
+ for this.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-errors">
+ <title>Error handling</title>
+
+ <para>
+ When a query in a batch causes an <literal>ERROR</literal> the server
+ skips processing all subsequent messages until the end-of-batch message.
+ The open transaction is aborted.
+ </para>
+
+ <para>
+ From the client perspective, after the client gets a
+ <literal>PGRES_FATAL_ERROR</literal> return from
+ <function>PQresultStatus</function> the batch is flagged as aborted.
+ <application>libpq</application> will report
+ <literal>PGRES_BATCH_ABORTED</literal> result for each remaining queued
+ operation in an aborted batch. The result for
+ <function>PQbatchQueueSync</function> is reported as
+ <literal>PGRES_BATCH_END</literal> to signal the end of the aborted batch
+ and resumption of normal result processing.
+ </para>
+
+ <para>
+ The client <emphasis>must</emphasis> process results with
+ <function>PQbatchQueueProcess(...)</function> and
+ <function>PQgetResult</function> during error recovery.
+ </para>
+
+ <para>
+ If the batch used an implicit transaction then operations that have
+ already executed are rolled back and operations that were queued for after
+ the failed operation are skipped entirely. The same behaviour holds if the
+ batch starts and commits a single explicit transaction (i.e. the first
+ statement is <literal>BEGIN</literal> and the last is
+ <literal>COMMIT</literal>) except that the session remains in an aborted
+ transaction state at the end of the batch. If a batch contains <emphasis>
+ multiple explicit transactions</emphasis>, all transactions that committed
+ prior to the error remain committed, the currently in-progress transaction
+ is aborted and all subsequent operations in the current and all later
+ transactions in the same batch are skipped completely.
+ </para>
+
+ <para>
+ The client must not assume that work is committed when it
+ <emphasis>sends</emphasis> a <literal>COMMIT</literal>, only when the
+ corresponding result is received to confirm the commit is complete.
+ Because errors arrive asynchronously the application needs to be able to
+ restart from the last <emphasis>received</emphasis> committed change and
+ resend work done after that point if something goes wrong.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-interleave">
+ <title>Interleaving result processing and query dispatch</title>
+
+ <para>
+ To avoid deadlocks on large batches the client should be structured around
+ a nonblocking I/O loop using a function like <function>select</function>,
+ <function>poll</function>, <function>epoll</function>,
+ <function>WaitForMultipleObjectEx</function>, etc.
+ </para>
+
+ <para>
+ The client application should generally maintain a queue of work still to
+ be dispatched and a queue of work that has been dispatched but not yet had
+ its results processed. When the socket is writable it should dispatch more
+ work. When the socket is readable it should read results and process them,
+ matching them up to the next entry in its expected results queue. Batches
+ should be scoped to logical units of work, usually (but not always) one
+ transaction per batch. There's no need to exit batch mode and re-enter it
+ between batches or to wait for one batch to finish before sending the next.
+ </para>
+
+ <para>
+ An example using <function>select()</function> and a simple state machine
+ to track sent and received work is in
+ <filename>src/test/modules/test_libpq/testlibpqbatch.c</filename> in the PostgreSQL
+ source distribution.
+ </para>
+
+ </sect3>
+
+ <sect3 id="libpq-batch-end">
+ <title>Ending batch mode</title>
+
+ <para>
+ Once all dispatched commands have had their results processed and the end batch
+ result has been consumed the application may return to non-batched mode with
+ <link linkend="libpq-PQbatchEnd"><function>PQbatchEnd(conn)</function></link>.
+ </para>
+ </sect3>
+
+ </sect2>
+
+ <sect2 id="libpq-funcs-batch">
+ <title>Functions associated with batch mode</title>
+
+ <variablelist>
+
+ <varlistentry id="libpq-PQbatchStatus">
+ <term>
+ <function>PQbatchStatus</function>
+ <indexterm>
+ <primary>PQbatchStatus</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns current batch mode status of the <application>libpq</application> connection.
+<synopsis>
+int PQbatchStatus(PGconn *conn);
+</synopsis>
+ </para>
+ <variablelist>
+ <varlistentry id="libpq-PQbatchStatus-1">
+ <term>
+ <literal>PQBATCH_MODE_ON</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ON</literal> if <application>libpq</application> connection is in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-2">
+ <term>
+ <literal>PQBATCH_MODE_OFF</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_OFF</literal> if <application>libpq</application> connection is not in <link
+ linkend="libpq-batch-mode">batch mode</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchStatus-3">
+ <term>
+ <literal>PQBATCH_MODE_ABORTED</literal>
+ </term>
+ <listitem>
+ <para>
+ Returns <literal>PQBATCH_MODE_ABORTED</literal> if <application>libpq</application> connection is in
+ aborted status. The aborted flag is cleared as soon as the result of the
+ <function>PQbatchQueueSync</function> at the end of the aborted batch is
+ processed. Clients don't usually need this function to verify aborted status
+ as they can tell that the batch is aborted from <literal>PGRES_BATCH_ABORTED</literal>
+ result codes.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchBegin">
+ <term>
+ <function>PQbatchBegin</function>
+ <indexterm>
+ <primary>PQbatchBegin</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to enter batch mode if it is currently idle or
+ already in batch mode.
+
+<synopsis>
+int PQbatchBegin(PGconn *conn);
+</synopsis>
+
+ </para>
+ <para>
+ Returns 1 for success. Returns 0 and has no
+ effect if the connection is not currently idle, i.e. it has a result
+ ready, is waiting for more input from the server, etc. This function
+ does not actually send anything to the server, it just changes the
+ <application>libpq</application> connection state.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchEnd">
+ <term>
+ <function>PQbatchEnd</function>
+ <indexterm>
+ <primary>PQbatchEnd</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes a connection to exit batch mode if it is currently in batch mode
+ with an empty queue and no pending results.
+<synopsis>
+int PQbatchEnd(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success.
+ Returns 1 and takes no action if not in batch mode. If the connection has
+ pending batch items in the queue for reading with
+ <function>PQbatchQueueProcess</function>, the current statement isn't finished
+ processing or there are results pending for collection with
+ <function>PQgetResult</function>, returns 0 and does nothing.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueSync">
+ <term>
+ <function>PQbatchQueueSync</function>
+ <indexterm>
+ <primary>PQbatchQueueSync</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Delimits the end of a set of a batched commands by sending a <link
+ linkend="protocol-flow-ext-query">sync message</link> and flushing
+ the send buffer. The end of a batch serves as
+ the delimiter of an implicit transaction and
+ an error recovery point; see <link linkend="libpq-batch-errors">
+ error handling</link>.
+
+<synopsis>
+int PQbatchQueueSync(PGconn *conn);
+</synopsis>
+ </para>
+ <para>Returns 1 for success. Returns 0 if the connection is not in batch mode
+ or sending a <link linkend="protocol-flow-ext-query">sync message</link> is failed.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueProcess">
+ <term>
+ <function>PQbatchQueueProcess</function>
+ <indexterm>
+ <primary>PQbatchQueueProcess</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Causes the connection to start processing the next queued query's
+ results.
+ </para>
+
+<synopsis>
+int PQbatchQueueProcess(PGconn *conn);
+</synopsis>
+
+ <para>
+ Returns 1 if a new query was popped from the result queue
+ for processing. Returns 0 and has no effect if there are no query results
+ pending, batch mode is not enabled, or if the query currently processed
+ is incomplete or still has pending results. Reason for these failures can
+ be verified with <function>PQbatchQueueCount</function>, <function>PQbatchStatus
+ </function> and <function>PQgetResult</function>.
+ See <link linkend="libpq-batch-results">processing results</link>.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQbatchQueueCount">
+ <term>
+ <function>PQbatchQueueCount</function>
+ <indexterm>
+ <primary>PQbatchQueueCount</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Returns the number of queries still in the queue for this batch, not
+ including any query that's currently having results being processed.
+ This is the number of times <function>PQbatchQueueProcess</function> has to be
+ called before the query queue is empty again.
+
+<synopsis>
+int PQbatchQueueCount(PGconn *conn);
+</synopsis>
+
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ </variablelist>
+
+ </sect2>
+
+ </sect1>
+
<sect1 id="libpq-single-row-mode">
<title>Retrieving Query Results Row-By-Row</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 21dd772..e9f81b3 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -171,3 +171,9 @@ PQsslAttributeNames 168
PQsslAttribute 169
PQsetErrorContextVisibility 170
PQresultVerboseErrorMessage 171
+PQbatchQueueCount 172
+PQbatchBegin 173
+PQbatchEnd 174
+PQbatchQueueSync 175
+PQbatchQueueProcess 176
+PQbatchStatus 177
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 27155f8..bb10b89 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -3459,6 +3459,25 @@ sendTerminateConn(PGconn *conn)
}
/*
+ * PQfreeCommandQueue
+ * Free all the entries of PGcommandQueueEntry queue passed.
+ */
+static void
+PQfreeCommandQueue(PGcommandQueueEntry *queue)
+{
+
+ while (queue != NULL)
+ {
+ PGcommandQueueEntry *prev = queue;
+
+ queue = queue->next;
+ if (prev->query)
+ free(prev->query);
+ free(prev);
+ }
+}
+
+/*
* closePGconn
* - properly close a connection to the backend
*
@@ -3470,6 +3489,7 @@ static void
closePGconn(PGconn *conn)
{
PGnotify *notify;
+ PGcommandQueueEntry *queue;
pgParameterStatus *pstatus;
sendTerminateConn(conn);
@@ -3502,6 +3522,14 @@ closePGconn(PGconn *conn)
free(prev);
}
conn->notifyHead = conn->notifyTail = NULL;
+ queue = conn->cmd_queue_head;
+ PQfreeCommandQueue(queue);
+ conn->cmd_queue_head = conn->cmd_queue_tail = NULL;
+
+ queue = conn->cmd_queue_recycle;
+ PQfreeCommandQueue(queue);
+
+ conn->cmd_queue_recycle = NULL;
pstatus = conn->pstatus;
while (pstatus != NULL)
{
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 9decd53..7c02f00 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -39,7 +39,9 @@ char *const pgresStatus[] = {
"PGRES_NONFATAL_ERROR",
"PGRES_FATAL_ERROR",
"PGRES_COPY_BOTH",
- "PGRES_SINGLE_TUPLE"
+ "PGRES_SINGLE_TUPLE",
+ "PGRES_BATCH_END",
+ "PGRES_BATCH_ABORTED"
};
/*
@@ -69,6 +71,9 @@ static PGresult *PQexecFinish(PGconn *conn);
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
+static PGcommandQueueEntry *PQmakePipelinedCommand(PGconn *conn);
+static void PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
+static void PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry);
/* ----------------
@@ -1108,7 +1113,7 @@ pqRowProcessor(PGconn *conn, const char **errmsgp)
conn->next_result = conn->result;
conn->result = res;
/* And mark the result ready to return */
- conn->asyncStatus = PGASYNC_READY;
+ conn->asyncStatus = PGASYNC_READY_MORE;
}
return 1;
@@ -1131,6 +1136,13 @@ fail:
int
PQsendQuery(PGconn *conn, const char *query)
{
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot PQsendQuery in batch mode, use PQsendQueryParams\n"));
+ return false;
+ }
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1229,6 +1241,10 @@ PQsendPrepare(PGconn *conn,
const char *stmtName, const char *query,
int nParams, const Oid *paramTypes)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
if (!PQsendQueryStart(conn))
return 0;
@@ -1287,18 +1303,34 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+ else
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
/* remember we are doing just a Parse */
- conn->queryclass = PGQUERY_PREPARE;
+ *queryclass = PGQUERY_PREPARE;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
- conn->last_query = strdup(query);
+ if (*last_query)
+ free(*last_query);
+ *last_query = strdup(query);
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1308,10 +1340,14 @@ PQsendPrepare(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1359,7 +1395,80 @@ PQsendQueryPrepared(PGconn *conn,
}
/*
- * Common startup code for PQsendQuery and sibling routines
+ * PQmakePipelinedCommand
+ * Get a new command queue entry, allocating it if required. Doesn't add it to
+ * the tail of the queue yet, use PQappendPipelinedCommand once the command has
+ * been written for that. If a command fails once it's called this, it should
+ * use PQrecyclePipelinedCommand to put it on the freelist or release it.
+ *
+ * If allocation fails sets the error message and returns null.
+ */
+static PGcommandQueueEntry *
+PQmakePipelinedCommand(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (conn->cmd_queue_recycle == NULL)
+ {
+ entry = (PGcommandQueueEntry *) malloc(sizeof(PGcommandQueueEntry));
+ if (entry == NULL)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory\n"));
+ return NULL;
+ }
+ }
+ else
+ {
+ entry = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry->next;
+ }
+ entry->next = NULL;
+ entry->query = NULL;
+
+ return entry;
+}
+
+/*
+ * PQappendPipelinedCommand
+ * Append a precreated command queue entry to the queue after it's been
+ * sent successfully.
+ */
+static void
+PQappendPipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (conn->cmd_queue_head == NULL)
+ conn->cmd_queue_head = entry;
+ else
+ conn->cmd_queue_tail->next = entry;
+ conn->cmd_queue_tail = entry;
+}
+
+/*
+ * PQrecyclePipelinedCommand
+ * Push a command queue entry onto the freelist. It must be a dangling entry
+ * with null next pointer and not referenced by any other entry's next pointer.
+ */
+static void
+PQrecyclePipelinedCommand(PGconn *conn, PGcommandQueueEntry * entry)
+{
+ if (entry == NULL)
+ return;
+ if (entry->next != NULL)
+ {
+ fprintf(stderr, "tried to recycle non-dangling command queue entry");
+ abort();
+ }
+ if (entry->query)
+ free(entry->query);
+
+ entry->next = conn->cmd_queue_recycle;
+ conn->cmd_queue_recycle = entry;
+}
+
+/*
+ * PQsendQueryStart
+ * Common startup code for PQsendQuery and sibling routines
*/
static bool
PQsendQueryStart(PGconn *conn)
@@ -1377,20 +1486,60 @@ PQsendQueryStart(PGconn *conn)
libpq_gettext("no connection to the server\n"));
return false;
}
- /* Can't send while already busy, either. */
- if (conn->asyncStatus != PGASYNC_IDLE)
+
+ /* Can't send while already busy, either, unless enqueuing for later */
+ if (conn->asyncStatus != PGASYNC_IDLE && conn->batch_status == PQBATCH_MODE_OFF)
{
printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("another command is already in progress\n"));
return false;
}
- /* initialize async result-accumulation state */
- pqClearAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * When enqueuing a message we don't change much of the connection
+ * state since it's already in use for the current command. The
+ * connection state will get updated when PQbatchQueueProcess(...)
+ * advances to start processing the queued message.
+ *
+ * Just make sure we can safely enqueue given the current connection
+ * state. We can enqueue behind another queue item, or behind a
+ * non-queue command (one that sends its own sync), but we can't
+ * enqueue if the connection is in a copy state.
+ */
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_QUEUED:
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* ok to queue */
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("cannot queue commands during COPY\n"));
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, idle state in batch mode"));
+ break;
+ }
+ }
+ else
+ {
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state
+ */
+ pqClearAsyncResult(conn);
- /* reset single-row processing mode */
- conn->singleRowMode = false;
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+ }
/* ready to send command message */
return true;
}
@@ -1414,6 +1563,10 @@ PQsendQueryGuts(PGconn *conn,
int resultFormat)
{
int i;
+ PGcommandQueueEntry *pipeCmd = NULL;
+ char **last_query;
+ PGQueryClass *queryclass;
+
/* This isn't gonna work on a 2.0 server */
if (PG_PROTOCOL_MAJOR(conn->pversion) < 3)
@@ -1423,6 +1576,23 @@ PQsendQueryGuts(PGconn *conn,
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ last_query = &pipeCmd->query;
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ last_query = &conn->last_query;
+ queryclass = &conn->queryclass;
+ }
+
+
/*
* We will send Parse (if needed), Bind, Describe Portal, Execute, Sync,
* using specified statement name and the unnamed portal.
@@ -1535,22 +1705,25 @@ PQsendQueryGuts(PGconn *conn,
pqPutMsgEnd(conn) < 0)
goto sendFailed;
- /* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are using extended query protocol */
- conn->queryclass = PGQUERY_EXTENDED;
+ *queryclass = PGQUERY_EXTENDED;
/* and remember the query text too, if possible */
/* if insufficient memory, last_query just winds up NULL */
- if (conn->last_query)
- free(conn->last_query);
+ if (*last_query)
+ free(*last_query);
if (command)
- conn->last_query = strdup(command);
+ *last_query = strdup(command);
else
- conn->last_query = NULL;
+ *last_query = NULL;
/*
* Give the data a push. In nonblock mode, don't complain if we're unable
@@ -1560,10 +1733,14 @@ PQsendQueryGuts(PGconn *conn,
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
@@ -1690,6 +1867,297 @@ PQisBusy(PGconn *conn)
return conn->asyncStatus == PGASYNC_BUSY;
}
+/*
+ * PQbatchQueueCount
+ * Return number of queries currently pending in batch mode
+ */
+int
+PQbatchQueueCount(PGconn *conn)
+{
+ int count = 0;
+ PGcommandQueueEntry *entry;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ return 0;
+
+ entry = conn->cmd_queue_head;
+ while (entry != NULL)
+ {
+ ++count;
+ entry = entry->next;
+ }
+ return count;
+}
+
+/*
+ * PQbatchStatus
+ * Returns current batch mode status
+ */
+int
+PQbatchStatus(PGconn *conn)
+{
+ if (!conn)
+ return FALSE;
+
+ return conn->batch_status;
+}
+
+/*
+ * PQbatchBegin
+ * Put an idle connection in batch mode. Commands submitted after this
+ * can be pipelined on the connection, there's no requirement to wait for
+ * one to finish before the next is dispatched.
+ *
+ * Queuing of new query or syncing during COPY is not allowed.
+ *
+ * A set of commands is terminated by a PQbatchQueueSync. Multiple sets of batched
+ * commands may be sent while in batch mode. Batch mode can be exited by
+ * calling PQbatchEnd() once all results are processed.
+ *
+ * This doesn't actually send anything on the wire, it just puts libpq
+ * into a state where it can pipeline work.
+ */
+int
+PQbatchBegin(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ return true;
+
+ if (conn->asyncStatus != PGASYNC_IDLE)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->asyncStatus = PGASYNC_QUEUED;
+
+ return true;
+}
+
+/*
+ * PQbatchEnd
+ * End batch mode and return to normal command mode.
+ *
+ * Has no effect unless the client has processed all results
+ * from all outstanding batches and the connection is idle.
+ *
+ * Returns true if batch mode ended.
+ */
+int
+PQbatchEnd(PGconn *conn)
+{
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return true;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* can't end batch while busy */
+ return false;
+ case PGASYNC_QUEUED:
+ break;
+ }
+
+ /* still work to process */
+ if (conn->cmd_queue_head != NULL)
+ return false;
+
+ conn->batch_status = PQBATCH_MODE_OFF;
+ conn->asyncStatus = PGASYNC_IDLE;
+
+ return true;
+}
+
+/*
+ * PQbatchQueueSync
+ * End a batch submission by sending a protocol sync. The connection will
+ * remain in batch mode and unavailable for new non-batch commands until all
+ * results from the batch are processed by the client.
+ *
+ * It's legal to start submitting another batch immediately, without waiting
+ * for the results of the current batch. There's no need to end batch mode
+ * and start it again.
+ *
+ * If a command in a batch fails, every subsequent command up to and including
+ * the PQbatchQueueSync command result gets set to PGRES_BATCH_ABORTED state. If the
+ * whole batch is processed without error, a PGresult with PGRES_BATCH_END is
+ * produced.
+ */
+int
+PQbatchQueueSync(PGconn *conn)
+{
+ PGcommandQueueEntry *entry;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ case PGASYNC_QUEUED:
+ /* can send sync to end this batch of cmds */
+ break;
+ }
+
+ entry = PQmakePipelinedCommand(conn);
+ if (entry == NULL)
+ return false; /* error msg already set */
+
+ entry->queryclass = PGQUERY_SYNC;
+ entry->query = NULL;
+
+ /* construct the Sync message */
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+
+ PQappendPipelinedCommand(conn, entry);
+
+ /* Should try to flush immediately if there's room */
+ PQflush(conn);
+
+ return true;
+
+sendFailed:
+ PQrecyclePipelinedCommand(conn, entry);
+ pqHandleSendFailure(conn);
+ return false;
+}
+
+/*
+ * PQbatchQueueProcess
+ * In batch mode, start processing the next query in the queue.
+ *
+ * Returns true if the next query was popped from the queue and can
+ * be processed by PQconsumeInput, PQgetResult, etc.
+ *
+ * Returns false if the current query isn't done yet, the connection
+ * is not in a batch, or there are no more queries to process.
+ */
+int
+PQbatchQueueProcess(PGconn *conn)
+{
+ PGcommandQueueEntry *next_query;
+
+ if (!conn)
+ return false;
+
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ return false;
+
+ switch (conn->asyncStatus)
+ {
+ case PGASYNC_COPY_IN:
+ case PGASYNC_COPY_OUT:
+ case PGASYNC_COPY_BOTH:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, COPY in batch mode"));
+ break;
+ case PGASYNC_READY:
+ case PGASYNC_READY_MORE:
+ case PGASYNC_BUSY:
+ /* client still has to process current query or results */
+ return false;
+ break;
+ case PGASYNC_IDLE:
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext_noop("internal error, IDLE in batch mode"));
+ break;
+ case PGASYNC_QUEUED:
+ /* next query please */
+ break;
+ }
+
+ if (conn->cmd_queue_head == NULL)
+ {
+ /*
+ * In batch mode but nothing left on the queue; caller can submit more
+ * work or PQbatchEnd() now.
+ */
+ return false;
+ }
+
+ /*
+ * Pop the next query from the queue and set up the connection state as if
+ * it'd just been dispatched from a non-batched call
+ */
+ next_query = conn->cmd_queue_head;
+ conn->cmd_queue_head = next_query->next;
+ next_query->next = NULL;
+
+ /* This command's results will come in immediately.
+ * Initialize async result-accumulation state */
+ pqClearAsyncResult(conn);
+
+ /* reset single-row processing mode */
+ conn->singleRowMode = false;
+
+
+ conn->last_query = next_query->query;
+ next_query->query = NULL;
+ conn->queryclass = next_query->queryclass;
+
+ PQrecyclePipelinedCommand(conn, next_query);
+
+ if (conn->batch_status == PQBATCH_MODE_ABORTED && conn->queryclass != PGQUERY_SYNC)
+ {
+ /*
+ * In an aborted batch we don't get anything from the server for each
+ * result; we're just discarding input until we get to the next sync
+ * from the server. The client needs to know its queries got aborted
+ * so we create a fake PGresult to return immediately from
+ * PQgetResult.
+ */
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_ABORTED);
+ if (!conn->result)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("out of memory"));
+ pqSaveErrorResult(conn);
+ }
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ {
+ /* allow parsing to continue */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+
+ return true;
+}
+
/*
* PQgetResult
@@ -1749,10 +2217,32 @@ PQgetResult(PGconn *conn)
switch (conn->asyncStatus)
{
case PGASYNC_IDLE:
+ case PGASYNC_QUEUED:
res = NULL; /* query is complete */
break;
case PGASYNC_READY:
res = pqPrepareAsyncResult(conn);
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ /*
+ * batched queries aren't followed by a Sync to put us back in
+ * PGASYNC_IDLE state, and when we do get a sync we could
+ * still have another batch coming after this one.
+ *
+ * The connection isn't idle since we can't submit new
+ * nonbatched commands. It isn't also busy since the current
+ * command is done and we need to process a new one.
+ */
+ conn->asyncStatus = PGASYNC_QUEUED;
+ }
+ else
+ {
+ /* Set the state back to BUSY, allowing parsing to proceed. */
+ conn->asyncStatus = PGASYNC_BUSY;
+ }
+ break;
+ case PGASYNC_READY_MORE:
+ res = pqPrepareAsyncResult(conn);
/* Set the state back to BUSY, allowing parsing to proceed. */
conn->asyncStatus = PGASYNC_BUSY;
break;
@@ -1932,6 +2422,13 @@ PQexecStart(PGconn *conn)
if (!conn)
return false;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ printfPQExpBuffer(&conn->errorMessage,
+ libpq_gettext("Synchronous command execution functions are not allowed in batch mode\n"));
+ return false;
+ }
+
/*
* Silently discard any prior query result that application didn't eat.
* This is probably poor design, but it's here for backward compatibility.
@@ -2126,6 +2623,9 @@ PQsendDescribePortal(PGconn *conn, const char *portal)
static int
PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
{
+ PGcommandQueueEntry *pipeCmd = NULL;
+ PGQueryClass *queryclass;
+
/* Treat null desc_target as empty string */
if (!desc_target)
desc_target = "";
@@ -2141,6 +2641,20 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
return 0;
}
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ pipeCmd = PQmakePipelinedCommand(conn);
+
+ if (pipeCmd == NULL)
+ return 0; /* error msg already set */
+
+ queryclass = &pipeCmd->queryclass;
+ }
+ else
+ {
+ queryclass = &conn->queryclass;
+ }
+
/* construct the Describe message */
if (pqPutMsgStart('D', false, conn) < 0 ||
pqPutc(desc_type, conn) < 0 ||
@@ -2149,15 +2663,18 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* construct the Sync message */
- if (pqPutMsgStart('S', false, conn) < 0 ||
- pqPutMsgEnd(conn) < 0)
- goto sendFailed;
+ if (conn->batch_status == PQBATCH_MODE_OFF)
+ {
+ if (pqPutMsgStart('S', false, conn) < 0 ||
+ pqPutMsgEnd(conn) < 0)
+ goto sendFailed;
+ }
/* remember we are doing a Describe */
- conn->queryclass = PGQUERY_DESCRIBE;
+ *queryclass = PGQUERY_DESCRIBE;
/* reset last-query string (not relevant now) */
- if (conn->last_query)
+ if (conn->last_query && conn->batch_status != PQBATCH_MODE_OFF)
{
free(conn->last_query);
conn->last_query = NULL;
@@ -2171,10 +2688,14 @@ PQsendDescribe(PGconn *conn, char desc_type, const char *desc_target)
goto sendFailed;
/* OK, it's launched! */
- conn->asyncStatus = PGASYNC_BUSY;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ PQappendPipelinedCommand(conn, pipeCmd);
+ else
+ conn->asyncStatus = PGASYNC_BUSY;
return 1;
sendFailed:
+ PQrecyclePipelinedCommand(conn, pipeCmd);
pqHandleSendFailure(conn);
return 0;
}
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index 3b0500f..c01f1a2 100644
--- a/src/interfaces/libpq/fe-protocol2.c
+++ b/src/interfaces/libpq/fe-protocol2.c
@@ -412,6 +412,12 @@ pqParseInput2(PGconn *conn)
{
char id;
+ if (conn->asyncStatus == PGASYNC_QUEUED || conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "internal error, attempt to read v2 protocol in batch mode");
+ abort();
+ }
+
/*
* Loop to parse successive complete messages available in the buffer.
*/
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 53776e2..e24d7ce 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -220,10 +220,18 @@ pqParseInput3(PGconn *conn)
return;
conn->asyncStatus = PGASYNC_READY;
break;
- case 'Z': /* backend is ready for new query */
+ case 'Z': /* sync response, backend is ready for new query */
if (getReadyForQuery(conn))
return;
- conn->asyncStatus = PGASYNC_IDLE;
+ if (conn->batch_status != PQBATCH_MODE_OFF)
+ {
+ conn->batch_status = PQBATCH_MODE_ON;
+ conn->result = PQmakeEmptyPGresult(conn,
+ PGRES_BATCH_END);
+ conn->asyncStatus = PGASYNC_READY;
+ }
+ else
+ conn->asyncStatus = PGASYNC_IDLE;
break;
case 'I': /* empty query */
if (conn->result == NULL)
@@ -880,6 +888,9 @@ pqGetErrorNotice3(PGconn *conn, bool isError)
PQExpBufferData workBuf;
char id;
+ if (isError && conn->batch_status != PQBATCH_MODE_OFF)
+ conn->batch_status = PQBATCH_MODE_ABORTED;
+
/*
* Since the fields might be pretty long, we create a temporary
* PQExpBuffer rather than using conn->workBuffer. workBuffer is intended
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 635af5b..737264d 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -95,7 +95,10 @@ typedef enum
PGRES_NONFATAL_ERROR, /* notice or warning message */
PGRES_FATAL_ERROR, /* query failed */
PGRES_COPY_BOTH, /* Copy In/Out data transfer in progress */
- PGRES_SINGLE_TUPLE /* single tuple from larger resultset */
+ PGRES_SINGLE_TUPLE, /* single tuple from larger resultset */
+ PGRES_BATCH_END, /* end of a batch of commands */
+ PGRES_BATCH_ABORTED, /* Command didn't run because of an abort
+ * earlier in a batch */
} ExecStatusType;
typedef enum
@@ -134,6 +137,17 @@ typedef enum
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+/*
+ * PQBatchStatus - Current status of batch mode
+ */
+
+typedef enum
+{
+ PQBATCH_MODE_OFF,
+ PQBATCH_MODE_ON,
+ PQBATCH_MODE_ABORTED
+} PQBatchStatus;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
@@ -425,6 +439,14 @@ extern PGresult *PQgetResult(PGconn *conn);
extern int PQisBusy(PGconn *conn);
extern int PQconsumeInput(PGconn *conn);
+/* Routines for batch mode management */
+extern int PQbatchStatus(PGconn *conn);
+extern int PQbatchQueueCount(PGconn *conn);
+extern int PQbatchBegin(PGconn *conn);
+extern int PQbatchEnd(PGconn *conn);
+extern int PQbatchQueueSync(PGconn *conn);
+extern int PQbatchQueueProcess(PGconn *conn);
+
/* LISTEN/NOTIFY support */
extern PGnotify *PQnotifies(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index b8ec341..33f212f 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -215,10 +215,15 @@ typedef enum
{
PGASYNC_IDLE, /* nothing's happening, dude */
PGASYNC_BUSY, /* query in progress */
- PGASYNC_READY, /* result ready for PQgetResult */
+ PGASYNC_READY, /* query done, waiting for client to fetch
+ * result */
+ PGASYNC_READY_MORE, /* query done, waiting for client to fetch
+ * result, More results expected from this
+ * query */
PGASYNC_COPY_IN, /* Copy In data transfer in progress */
PGASYNC_COPY_OUT, /* Copy Out data transfer in progress */
- PGASYNC_COPY_BOTH /* Copy In/Out data transfer in progress */
+ PGASYNC_COPY_BOTH, /* Copy In/Out data transfer in progress */
+ PGASYNC_QUEUED /* Current query done, more in queue */
} PGAsyncStatusType;
/* PGQueryClass tracks which query protocol we are now executing */
@@ -227,7 +232,8 @@ typedef enum
PGQUERY_SIMPLE, /* simple Query protocol (PQexec) */
PGQUERY_EXTENDED, /* full Extended protocol (PQexecParams) */
PGQUERY_PREPARE, /* Parse only (PQprepare) */
- PGQUERY_DESCRIBE /* Describe Statement or Portal */
+ PGQUERY_DESCRIBE, /* Describe Statement or Portal */
+ PGQUERY_SYNC /* A protocol sync to end a batch */
} PGQueryClass;
/* PGSetenvStatusType defines the state of the PQSetenv state machine */
@@ -297,6 +303,22 @@ typedef enum pg_conn_host_type
CHT_UNIX_SOCKET
} pg_conn_host_type;
+/* An entry in the pending command queue. Used by batch mode to keep track
+ * of the expected results of future commands we've dispatched.
+ *
+ * Note that entries in this list are reused by being zeroed and appended to
+ * the tail when popped off the head. The entry with null next pointer is not
+ * the end of the list of expected commands, that's the tail pointer in
+ * pg_conn.
+ */
+typedef struct pgCommandQueueEntry
+{
+ PGQueryClass queryclass; /* Query type; PGQUERY_SYNC for sync msg */
+ char *query; /* SQL command, or NULL if unknown */
+ struct pgCommandQueueEntry *next;
+} PGcommandQueueEntry;
+
+
/*
* pg_conn_host stores all information about one of possibly several hosts
* mentioned in the connection string. Derived by splitting the pghost
@@ -384,6 +406,7 @@ struct pg_conn
bool options_valid; /* true if OK to attempt connection */
bool nonblocking; /* whether this connection is using nonblock
* sending semantics */
+ PQBatchStatus batch_status; /* Batch(pipelining) mode status of connection */
bool singleRowMode; /* return current query result row-by-row? */
char copy_is_binary; /* 1 = copy binary, 0 = copy text */
int copy_already_done; /* # bytes already returned in COPY
@@ -396,6 +419,16 @@ struct pg_conn
int whichhost; /* host we're currently considering */
pg_conn_host *connhost; /* details about each possible host */
+ /*
+ * The command queue
+ *
+ * head is the next pending cmd, tail is where we append new commands.
+ * Freed entries for recycling go on the recycle linked list.
+ */
+ PGcommandQueueEntry *cmd_queue_head;
+ PGcommandQueueEntry *cmd_queue_tail;
+ PGcommandQueueEntry *cmd_queue_recycle;
+
/* Connection data */
pgsocket sock; /* FD for socket, PGINVALID_SOCKET if
* unconnected */
@@ -693,6 +726,8 @@ extern char *libpq_ngettext(const char *msgid, const char *msgid_plural, unsigne
#define libpq_ngettext(s, p, n) ((n) == 1 ? (s) : (p))
#endif
+#define libpq_gettext_noop(x) (x)
+
/*
* These macros are needed to let error-handling code be portable between
* Unix and Windows. (ugh)
--
2.7.4.windows.1
[application/octet-stream] 0002-Pipelining-batch-support-for-libpq-test-v3.patch (43.4K, ../../CAOoUkxRvkPPcQjPWqM2Z1Dwi=ucnZ0oQ2tegryjPz++Qq+eZYQ@mail.gmail.com/4-0002-Pipelining-batch-support-for-libpq-test-v3.patch)
download | inline diff:
---
src/test/modules/test_libpq/.gitignore | 5 +
src/test/modules/test_libpq/Makefile | 25 +
src/test/modules/test_libpq/README | 1 +
src/test/modules/test_libpq/t/001_libpq_async.pl | 26 +
src/test/modules/test_libpq/testlibpqbatch.c | 1608 ++++++++++++++++++++++
5 files changed, 1665 insertions(+)
create mode 100644 src/test/modules/test_libpq/.gitignore
create mode 100644 src/test/modules/test_libpq/Makefile
create mode 100644 src/test/modules/test_libpq/README
create mode 100644 src/test/modules/test_libpq/t/001_libpq_async.pl
create mode 100644 src/test/modules/test_libpq/testlibpqbatch.c
diff --git a/src/test/modules/test_libpq/.gitignore b/src/test/modules/test_libpq/.gitignore
new file mode 100644
index 0000000..11e8463
--- /dev/null
+++ b/src/test/modules/test_libpq/.gitignore
@@ -0,0 +1,5 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+/testlibpqbatch
diff --git a/src/test/modules/test_libpq/Makefile b/src/test/modules/test_libpq/Makefile
new file mode 100644
index 0000000..d907063
--- /dev/null
+++ b/src/test/modules/test_libpq/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_libpq/Makefile
+
+OBJS = testlibpqbatch.o
+PROGRAM = testlibpqbatch
+
+PG_CPPFLAGS = -I$(libpq_srcdir)
+PG_LIBS += $(libpq)
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_libpq
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+testlibpqbatch.o: testlibpqbatch.c
+testlibpqbatch: testlibpqbatch.o
+check: testlibpqbatch prove-check
+
+prove-check:
+ $(prove_check)
diff --git a/src/test/modules/test_libpq/README b/src/test/modules/test_libpq/README
new file mode 100644
index 0000000..d8174dd
--- /dev/null
+++ b/src/test/modules/test_libpq/README
@@ -0,0 +1 @@
+Test programs and libraries for libpq
diff --git a/src/test/modules/test_libpq/t/001_libpq_async.pl b/src/test/modules/test_libpq/t/001_libpq_async.pl
new file mode 100644
index 0000000..706b2de
--- /dev/null
+++ b/src/test/modules/test_libpq/t/001_libpq_async.pl
@@ -0,0 +1,26 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+use Cwd;
+
+my $node = get_new_node('main');
+$node->init;
+$node->start;
+
+my $port = $node->port;
+
+my $numrows = 10000;
+my @tests = qw(disallowed_in_batch simple_batch multi_batch batch_abort timings copyfailure);
+
+$ENV{PATH} = "$ENV{PATH}:" . getcwd();
+for my $testname (@tests)
+{
+ $node->command_ok(['testlibpqbatch', 'dbname=postgres', "$numrows", "$testname"],
+ "testlibpqbatch $testname");
+}
+
+#$node->stop('fast');
diff --git a/src/test/modules/test_libpq/testlibpqbatch.c b/src/test/modules/test_libpq/testlibpqbatch.c
new file mode 100644
index 0000000..9b72960
--- /dev/null
+++ b/src/test/modules/test_libpq/testlibpqbatch.c
@@ -0,0 +1,1608 @@
+/*
+ * src/test/modules/test_libpq/testlibpqbatch.c
+ *
+ *
+ * testlibpqbatch.c
+ * Test of batch execution functionality
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/time.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void exit_nicely(PGconn *conn);
+static void simple_batch(PGconn *conn);
+static void test_disallowed_in_batch(PGconn *conn);
+static void batch_insert_pipelined(PGconn *conn, int n_rows);
+static void batch_insert_sequential(PGconn *conn, int n_rows);
+static void batch_insert_copy(PGconn *conn, int n_rows);
+static void test_batch_abort(PGconn *conn);
+static void test_copyfailure(PGconn *conn);
+static const Oid INT4OID = 23;
+
+static const char *const drop_table_sql
+= "DROP TABLE IF EXISTS batch_demo";
+static const char *const create_table_sql
+= "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer);";
+static const char *const insert_sql
+= "INSERT INTO batch_demo(itemno) VALUES ($1);";
+
+/* max char length of an int32, plus sign and null terminator */
+#define MAXINTLEN 12
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+static void
+simple_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "simple batch... ");
+ fflush(stderr);
+
+ /*
+ * Enter batch mode and dispatch a set of operations, which we'll then
+ * process the results of as they come in.
+ *
+ * For a simple case we should be able to do this without interim
+ * processing of results since our out buffer will give us enough slush to
+ * work with and we won't block on sending. So blocking mode is fine.
+ */
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode\n");
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode with work in progress should fail, but succeeded\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending a batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * in batch mode we have to ask for the first result to be processed;
+ * until we do PQgetResult will return null:
+ */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* We can't PQbatchQueueProcess when there might still be pending results */
+ if (PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() should've failed with pending results: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ /*
+ * Even though we've processed the result there's still a sync to come and
+ * we can't exit batch mode yet
+ */
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ /* should now get an explicit sync result */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after end batch call\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_disallowed_in_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+
+ fprintf(stderr, "test error cases... ");
+ fflush(stderr);
+
+ if (PQisnonblocking(conn))
+ {
+ fprintf(stderr, "Expected blocking connection mode: %u\n", __LINE__);
+ goto fail;
+ }
+
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Unable to enter batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not activated properly\n");
+ goto fail;
+ }
+
+ /* PQexec should fail in batch mode */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "PQexec should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* So should PQsendQuery */
+ if (PQsendQuery(conn, "SELECT 1") != 0)
+ {
+ fprintf(stderr, "PQsendQuery should fail in batch mode but succeeded\n");
+ goto fail;
+ }
+
+ /* Entering batch mode when already in batch mode is OK */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "re-entering batch mode should be a no-op but failed\n");
+ goto fail;
+ }
+
+ if (PQisBusy(conn))
+ {
+ fprintf(stderr, "PQisBusy should return false when idle in batch, returned true\n");
+ goto fail;
+ }
+
+ /* ok, back to normal command mode */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "couldn't exit idle empty batch mode\n");
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Batch mode not terminated properly\n");
+ goto fail;
+ }
+
+ /* exiting batch mode when not in batch mode should be a no-op */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "batch mode exit when not in batch mode should succeed but failed\n");
+ goto fail;
+ }
+
+ /* can now PQexec again */
+ res = PQexec(conn, "SELECT 1");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "PQexec should succeed after exiting batch mode but failed with: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+static void
+test_copyfailure(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *create_sql = "CREATE UNLOGGED TABLE batch_demo(id serial primary key, itemno integer); INSERT INTO batch_demo VALUES(5,10); ";
+ const char *select_sql = "select id from batch_demo;";
+ const char *copy_sql = "copy batch_demo(id) to stdout;";
+ const char *copyfrom_sql = "copy batch_demo(itemno) FROM stdin;";
+ int ret;
+ char *copybuf;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode first time\n");
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, copy_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY TO query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY TO command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_OUT)
+ {
+ fprintf(stderr, "Wrong state during COPY TO command processing: %s %s\n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ }
+
+ for (;;)
+ {
+ ret = PQgetCopyData(conn, ©buf, 0);
+ if (ret < 0)
+ break; /* done or error */
+
+ if (copybuf)
+ {
+ fprintf(stderr, "COPYBUF: %s \n", copybuf);
+ PQfreemem(copybuf);
+ }
+ }
+
+ PQclear(res);
+ res = NULL;
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Finish of COPY TO command failed %s :%s", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed in sync command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ PQclear(res);
+ PQsetnonblocking(conn, 1);
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "Failed to enter batch mode\n");
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, copyfrom_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching COPY FROM query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQsendQueryParams(conn, select_sql, 0, NULL, NULL, NULL, NULL, 1))
+ {
+ fprintf(stderr, "dispatching SELECT query failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* Start processing the batch results */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed while processing COPY FROM command : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "Wrong state during COPY command processing: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY FROM command failed: %s \n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Expect a failure here */
+ if (PQresultStatus(res) == PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpectedly COPY FROM finished with %s: %s",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "Batch mode is not aborted after failure\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at SELECT command after copy : %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ /* Select query after copy should also fail */
+ if (PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "SELECT - Expected failure, got %s: %s \n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ /* Clean up all the error responses after COPY failure */
+ do
+ {
+ PQbatchQueueProcess(conn);
+ res = PQgetResult(conn);
+ fprintf(stderr, "Error status and message got from server due to COPY command failure are : %s %s \n", PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ } while (res != NULL);
+
+ PQclear(res);
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "Attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = PQexec(conn, select_sql);
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "\nExpected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ return;
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+multi_batch(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+
+ fprintf(stderr, "multi batch... ");
+ fflush(stderr);
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT $1", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second SELECT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /* OK, start processing the batch results */
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something in a batch before first PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQgetResult(conn) != NULL)
+ {
+ fprintf(stderr, "PQgetResult returned something extra after first result before PQbatchQueueProcess() call\n");
+ goto fail;
+ }
+
+ if (PQbatchEnd(conn))
+ {
+ fprintf(stderr, "exiting batch mode after query but before sync succeeded incorrectly\n");
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at sync after first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when sync result expected: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s instead of sync result, error: %s (line %u)\n",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn), __LINE__);
+ goto fail;
+ }
+
+ PQclear(res);
+ res = NULL;
+
+ /* second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch item\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (res == NULL)
+ {
+ fprintf(stderr, "PQgetResult returned null when there's a batch item: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second end batch\n",
+ PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+/*
+ * When an operation in a batch fails the rest of the batch is flushed. We
+ * still have to get results for each batch item, but the item will just be
+ * a PGRES_BATCH_ABORTED code.
+ *
+ * This intentionally doesn't use a transaction to wrap the batch. You should
+ * usually use an xact, but in this case we want to observe the effects of each
+ * statement.
+ */
+static void
+test_batch_abort(PGconn *conn)
+{
+ PGresult *res = NULL;
+ const char *dummy_params[1] = {"1"};
+ Oid dummy_param_oids[1] = {INT4OID};
+ int i;
+
+ fprintf(stderr, "aborted batch... ");
+ fflush(stderr);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+
+ /*
+ * Queue up a couple of small batches and process each without returning
+ * to command mode first. Make sure the second operation in the first
+ * batch ERRORs.
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "1";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching first INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "SELECT no_such_function($1)", 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching error select failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "2";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending first batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ dummy_params[0] = "3";
+ if (!PQsendQueryParams(conn, insert_sql, 1, dummy_param_oids,
+ dummy_params, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching second-batch insert failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQbatchQueueSync(conn))
+ {
+ fprintf(stderr, "Ending second batch failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * OK, start processing the batch results.
+ *
+ * We should get a tuples-ok for the first query, a fatal error, a batch
+ * aborted message for the second insert, a batch-end, then a command-ok
+ * and a batch-ok for the second batch operation.
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first batch item, error='%s'\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)),
+ res == NULL ? PQerrorMessage(conn) : PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* second query, caused error */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "Unexpected result code from second batch item. Wanted PGRES_FATAL_ERROR, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /*
+ * batch should now be aborted.
+ *
+ * Note that we could still queue more queries at this point if we wanted;
+ * they'd get added to a new third batch since we've already sent a
+ * second. The aborted flag relates only to the batch being received.
+ */
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* third query in batch, the second insert */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at third batch entry: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_ABORTED)
+ {
+ fprintf(stderr, "Unexpected result code from third batch item. Wanted PGRES_BATCH_ABORTED, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "batch should be flagged as aborted but isn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ /*
+ * The end of a failed batch is still a PGRES_BATCH_END so clients know to
+ * start processing results normally again and can tell the difference
+ * between skipped commands and the sync.
+ */
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code from first batch sync. Wanted PGRES_BATCH_END, got %s\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ if (PQbatchStatus(conn) == PQBATCH_MODE_ABORTED)
+ {
+ fprintf(stderr, "sync should've cleared the aborted flag but didn't\n");
+ goto fail;
+ }
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* the insert from the second batch */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at first entry in second batch: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "Unexpected result code %s from first item in second batch\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* the second batch sync */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "PQbatchQueueProcess() failed at second batch sync: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (((res = PQgetResult(conn)) == NULL) || PQresultStatus(res) != PGRES_BATCH_END)
+ {
+ fprintf(stderr, "Unexpected result code %s from second batch sync\n",
+ res == NULL ? "NULL" : PQresStatus(PQresultStatus(res)));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ /* We're still in a batch... */
+ if (PQbatchStatus(conn) == PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "Fell out of batch mode somehow\n");
+ goto fail;
+ }
+
+ /* until we end it, which we can safely do now */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQbatchStatus(conn) != PQBATCH_MODE_OFF)
+ {
+ fprintf(stderr, "exiting batch mode didn't seem to work\n");
+ goto fail;
+ }
+
+ fprintf(stderr, "ok\n");
+
+ /*
+ * Since we fired the batches off without a surrounding xact, the results
+ * should be:
+ *
+ * - Implicit xact started by server around 1st batch - First insert
+ * applied - Second statement aborted xact - Third insert skipped - Sync
+ * rolled back first implicit xact - Implicit xact created by server
+ * around 2nd batch - insert applied from 2nd batch - Sync commits 2nd
+ * xact
+ *
+ * So we should only have the value 3 that we inserted.
+ */
+ res = PQexec(conn, "SELECT itemno FROM batch_demo");
+
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "Expected tuples, got %s: %s",
+ PQresStatus(PQresultStatus(res)), PQerrorMessage(conn));
+ goto fail;
+ }
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ const char *val = PQgetvalue(res, i, 0);
+
+ if (strcmp(val, "3") != 0)
+ {
+ fprintf(stderr, "expected only insert with value 3, got %s", val);
+ goto fail;
+ }
+ }
+
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr, "expected 1 result, got %d", PQntuples(res));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+/* State machine enums for batch insert */
+typedef enum BatchInsertStep
+{
+ BI_BEGIN_TX,
+ BI_DROP_TABLE,
+ BI_CREATE_TABLE,
+ BI_PREPARE,
+ BI_INSERT_ROWS,
+ BI_COMMIT_TX,
+ BI_SYNC,
+ BI_DONE
+} BatchInsertStep;
+
+static void
+batch_insert_pipelined(PGconn *conn, int n_rows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+ BatchInsertStep send_step = BI_BEGIN_TX,
+ recv_step = BI_BEGIN_TX;
+ int rows_to_send,
+ rows_to_receive;
+
+ insert_params[0] = &insert_param_0[0];
+
+ rows_to_send = rows_to_receive = n_rows;
+
+ /*
+ * Do a batched insert into a table created at the start of the batch
+ */
+ if (!PQbatchBegin(conn))
+ {
+ fprintf(stderr, "failed to enter batch mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (!PQsendQueryParams(conn, "BEGIN",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "xact start failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent BEGIN\n");
+
+ send_step = BI_DROP_TABLE;
+
+ if (!PQsendQueryParams(conn, drop_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent DROP\n");
+
+ send_step = BI_CREATE_TABLE;
+
+ if (!PQsendQueryParams(conn, create_table_sql,
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stderr, "dispatching CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent CREATE\n");
+
+ send_step = BI_PREPARE;
+
+ if (!PQsendPrepare(conn, "my_insert", insert_sql, 1, insert_param_oids))
+ {
+ fprintf(stderr, "dispatching PREPARE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ fprintf(stdout, "sent PREPARE\n");
+
+ send_step = BI_INSERT_ROWS;
+
+ /*
+ * Now we start inserting. We'll be sending enough data that we could fill
+ * our out buffer, so to avoid deadlocking we need to enter nonblocking
+ * mode and consume input while we send more output. As results of each
+ * query are processed we should pop them to allow processing of the next
+ * query. There's no need to finish the batch before processing results.
+ */
+ if (PQsetnonblocking(conn, 1) != 0)
+ {
+ fprintf(stderr, "failed to set nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ while (recv_step != BI_DONE)
+ {
+ int sock;
+ fd_set input_mask;
+ fd_set output_mask;
+
+ sock = PQsocket(conn);
+
+ if (sock < 0)
+ break; /* shouldn't happen */
+
+ FD_ZERO(&input_mask);
+ FD_SET(sock, &input_mask);
+ FD_ZERO(&output_mask);
+ FD_SET(sock, &output_mask);
+
+ if (select(sock + 1, &input_mask, &output_mask, NULL, NULL) < 0)
+ {
+ fprintf(stderr, "select() failed: %s\n", strerror(errno));
+ exit_nicely(conn);
+ }
+
+ /*
+ * Process any results, so we keep the server's out buffer free
+ * flowing and it can continue to process input
+ */
+ if (FD_ISSET(sock, &input_mask))
+ {
+ PQconsumeInput(conn);
+
+ /* Read until we'd block if we tried to read */
+ while (!PQisBusy(conn) && recv_step < BI_DONE)
+ {
+ const char *cmdtag;
+ const char *description = NULL;
+ int status;
+ BatchInsertStep next_step;
+
+
+ res = PQgetResult(conn);
+
+ if (res == NULL)
+ {
+ /*
+ * No more results from this query, advance to the next
+ * result
+ */
+ if (!PQbatchQueueProcess(conn))
+ {
+ fprintf(stderr, "Expected next query result but unable to dequeue: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+ fprintf(stdout, "next query!\n");
+ continue;
+ }
+
+ status = PGRES_COMMAND_OK;
+ next_step = recv_step + 1;
+ switch (recv_step)
+ {
+ case BI_BEGIN_TX:
+ cmdtag = "BEGIN";
+ break;
+ case BI_DROP_TABLE:
+ cmdtag = "DROP TABLE";
+ break;
+ case BI_CREATE_TABLE:
+ cmdtag = "CREATE TABLE";
+ break;
+ case BI_PREPARE:
+ cmdtag = "";
+ description = "PREPARE";
+ break;
+ case BI_INSERT_ROWS:
+ cmdtag = "INSERT";
+ rows_to_receive--;
+ if (rows_to_receive > 0)
+ next_step = BI_INSERT_ROWS;
+ break;
+ case BI_COMMIT_TX:
+ cmdtag = "COMMIT";
+ break;
+ case BI_SYNC:
+ cmdtag = "";
+ description = "SYNC";
+ status = PGRES_BATCH_END;
+ break;
+ case BI_DONE:
+ /* unreachable */
+ abort();
+ }
+ if (description == NULL)
+ description = cmdtag;
+
+ fprintf(stderr, "At state %d (%s) expect tag '%s', result code %s, expect %d more rows, transition to %d\n",
+ recv_step, description, cmdtag, PQresStatus(status), rows_to_receive, next_step);
+
+ if (PQresultStatus(res) != status)
+ {
+ fprintf(stderr, "%s reported status %s, expected %s. Error msg is [%s]\n",
+ description, PQresStatus(PQresultStatus(res)), PQresStatus(status), PQerrorMessage(conn));
+ goto fail;
+ }
+ if (strncmp(PQcmdStatus(res), cmdtag, strlen(cmdtag)) != 0)
+ {
+ fprintf(stderr, "%s expected command tag '%s', got '%s'\n",
+ description, cmdtag, PQcmdStatus(res));
+ goto fail;
+ }
+
+ fprintf(stdout, "Got %s OK\n", cmdtag);
+
+ recv_step = next_step;
+
+ PQclear(res);
+ res = NULL;
+ }
+ }
+
+ /* Write more rows and/or the end batch message, if needed */
+ if (FD_ISSET(sock, &output_mask))
+ {
+ PQflush(conn);
+
+ if (send_step == BI_INSERT_ROWS)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", rows_to_send);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ if (PQsendQueryPrepared(conn, "my_insert",
+ 1, insert_params, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent row %d\n", rows_to_send);
+
+ rows_to_send--;
+ if (rows_to_send == 0)
+ send_step = BI_COMMIT_TX;
+ }
+ else
+ {
+ /*
+ * in nonblocking mode, so it's OK for an insert to fail
+ * to send
+ */
+ fprintf(stderr, "WARNING: failed to send insert #%d: %s\n",
+ rows_to_send, PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_COMMIT_TX)
+ {
+ if (PQsendQueryParams(conn, "COMMIT",
+ 0, NULL, NULL, NULL, NULL, 0))
+ {
+ fprintf(stdout, "sent COMMIT\n");
+ send_step = BI_SYNC;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: failed to send commit: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ else if (send_step == BI_SYNC)
+ {
+ if (PQbatchQueueSync(conn))
+ {
+ fprintf(stdout, "Dispatched end batch message\n");
+ send_step = BI_DONE;
+ }
+ else
+ {
+ fprintf(stderr, "WARNING: Ending a batch failed: %s\n",
+ PQerrorMessage(conn));
+ }
+ }
+ }
+
+ }
+
+ /* We've got the sync message and the batch should be done */
+ if (!PQbatchEnd(conn))
+ {
+ fprintf(stderr, "attempt to exit batch mode failed when it should've succeeded: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ if (PQsetnonblocking(conn, 0) != 0)
+ {
+ fprintf(stderr, "failed to clear nonblocking mode: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+
+static void
+batch_insert_sequential(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+ const char *insert_params[1];
+ Oid insert_param_oids[1] = {INT4OID};
+ char insert_param_0[MAXINTLEN];
+
+ insert_params[0] = &insert_param_0[0];
+
+ res = PQexec(conn, "BEGIN");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "BEGIN failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQprepare(conn, "my_insert2", insert_sql, 1, insert_param_oids);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "prepare failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ while (nrows > 0)
+ {
+ snprintf(&insert_param_0[0], MAXINTLEN, "%d", nrows);
+ insert_param_0[MAXINTLEN - 1] = '\0';
+
+ res = PQexecPrepared(conn, "my_insert2",
+ 1, insert_params, NULL, NULL, 0);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "INSERT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ nrows--;
+ }
+
+ res = PQexec(conn, "COMMIT");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COMMIT failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+batch_insert_copy(PGconn *conn, int nrows)
+{
+ PGresult *res = NULL;
+
+ res = PQexec(conn, drop_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "DROP TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+
+ res = PQexec(conn, create_table_sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "CREATE TABLE failed: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ res = PQexec(conn, "COPY batch_demo(itemno) FROM stdin");
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ {
+ fprintf(stderr, "COPY: %s\n", PQerrorMessage(conn));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ while (nrows > 0)
+ {
+ char buf[12 + 2];
+ int formatted = snprintf(&buf[0], 12 + 1, "%d\n", nrows);
+
+ if (formatted >= 12 + 1)
+ {
+ fprintf(stderr, "Buffer write truncated somehow\n");
+ goto fail;
+ }
+
+ if (PQputCopyData(conn, buf, formatted) != 1)
+ {
+ fprintf(stderr, "Write of COPY data failed: %s\n",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ nrows--;
+ }
+
+ if (PQputCopyEnd(conn, NULL) != 1)
+ {
+ fprintf(stderr, "Finishing COPY failed: %s",
+ PQerrorMessage(conn));
+ goto fail;
+ }
+
+ res = PQgetResult(conn);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "COPY finished with %s: %s\n",
+ PQresStatus(PQresultStatus(res)),
+ PQresultErrorMessage(res));
+ goto fail;
+ }
+ PQclear(res);
+ res = NULL;
+
+ return;
+
+fail:
+ PQclear(res);
+ exit_nicely(conn);
+}
+
+static void
+test_timings(PGconn *conn, int number_of_rows)
+{
+ struct timeval start_time,
+ end_time,
+ elapsed_time;
+
+ fprintf(stderr, "inserting %d rows batched then unbatched\n", number_of_rows);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_pipelined(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("batch insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_sequential(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("sequential insert elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ gettimeofday(&start_time, NULL);
+ batch_insert_copy(conn, number_of_rows);
+ gettimeofday(&end_time, NULL);
+ timersub(&end_time, &start_time, &elapsed_time);
+ printf("COPY elapsed: %ld.%06lds\n", elapsed_time.tv_sec, (long) elapsed_time.tv_usec);
+
+ fprintf(stderr, "Done.\n");
+}
+
+static void
+usage_exit(const char *progname)
+{
+ fprintf(stderr, "Usage: %s ['connstring' [number_of_rows [test_to_run]]]\n", progname);
+ fprintf(stderr, " tests: all|disallowed_in_batch|simple_batch|multi_batch|batch_abort|timings\n");
+ exit(1);
+}
+
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ int number_of_rows = 10000;
+
+ int run_disallowed_in_batch = 1,
+ run_simple_batch = 1,
+ run_multi_batch = 1,
+ run_batch_abort = 1,
+ run_timings = 1,
+ run_copyfailure = 1;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 4)
+ {
+ usage_exit(argv[0]);
+ }
+ if (argc > 3)
+ {
+ if (strcmp(argv[3], "all") != 0)
+ {
+ run_disallowed_in_batch = 0;
+ run_simple_batch = 0;
+ run_multi_batch = 0;
+ run_batch_abort = 0;
+ run_timings = 0;
+ run_copyfailure = 0;
+ if (strcmp(argv[3], "disallowed_in_batch") == 0)
+ run_disallowed_in_batch = 1;
+ else if (strcmp(argv[3], "simple_batch") == 0)
+ run_simple_batch = 1;
+ else if (strcmp(argv[3], "multi_batch") == 0)
+ run_multi_batch = 1;
+ else if (strcmp(argv[3], "batch_abort") == 0)
+ run_batch_abort = 1;
+ else if (strcmp(argv[3], "timings") == 0)
+ run_timings = 1;
+ else if (strcmp(argv[3], "copyfailure") == 0)
+ run_copyfailure = 1;
+ else
+ {
+ fprintf(stderr, "%s is not a recognized test name\n", argv[3]);
+ usage_exit(argv[0]);
+ }
+ }
+ }
+ if (argc > 2)
+ {
+ errno = 0;
+ number_of_rows = strtol(argv[2], NULL, 10);
+ if (errno)
+ {
+ fprintf(stderr, "couldn't parse '%s' as an integer or zero rows supplied: %s", argv[2], strerror(errno));
+ usage_exit(argv[0]);
+ }
+ if (number_of_rows <= 0)
+ {
+ fprintf(stderr, "number_of_rows must be positive");
+ usage_exit(argv[0]);
+ }
+ }
+ if (argc > 1)
+ {
+ conninfo = argv[1];
+ }
+ else
+ {
+ conninfo = "dbname = postgres";
+ }
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s\n",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ if (run_disallowed_in_batch)
+ test_disallowed_in_batch(conn);
+
+ if (run_simple_batch)
+ simple_batch(conn);
+
+ if (run_multi_batch)
+ multi_batch(conn);
+
+ if (run_batch_abort)
+ test_batch_abort(conn);
+
+ if (run_timings)
+ test_timings(conn, number_of_rows);
+
+ if (run_copyfailure)
+ test_copyfailure(conn);
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.7.4.windows.1
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-16 13:37 Daniel Verite <[email protected]>
parent: Vaishnavi Prabakaran <[email protected]>
1 sibling, 1 reply; 40+ messages in thread
From: Daniel Verite @ 2017-03-16 13:37 UTC (permalink / raw)
To: Vaishnavi Prabakaran <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
Vaishnavi Prabakaran wrote:
> So, attached the alternative fix for this issue.
> Please share me your thoughts.
I assume you prefer the alternative fix because it's simpler.
> I would also like to hear Craig's opinion on it before applying this fix
> to the original patch, just to make sure am not missing anything here.
+1
The main question is whether the predicates enforced
by PQsetSingleRowMode() apply in batch mode in all cases
when it's legit to call that function. Two predicates
that may be problematic are:
if (conn->asyncStatus != PGASYNC_BUSY)
return 0;
and
if (conn->result)
return 0;
The general case with batch mode is that, from the doc:
"The client interleaves result processing with sending batch queries"
Note that I've not even tested that here, I've tested
batching a bunch of queries in a first step and getting the results
in a second step.
I am not confident that the above predicates will be true
in all cases. Also your alternative fix assumes that we add
a user-visible exception to PQsetSingleRowMode in batch mode,
whereby it must not be called as currently documented:
"call PQsetSingleRowMode immediately after a successful call of
PQsendQuery (or a sibling function)"
My gut feeling is that it's not the right direction, I prefer making
the single-row a per-query attribute internally and keep
PQsetSingleRowMode's contract unchanged from the
user's perspective.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: PATCH: Batch/pipelining support for libpq
@ 2017-03-20 01:32 Vaishnavi Prabakaran <[email protected]>
parent: Daniel Verite <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Vaishnavi Prabakaran @ 2017-03-20 01:32 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Michael Paquier <[email protected]>; Prabakaran, Vaishnavi <[email protected]>; Craig Ringer <[email protected]>; Haribabu Kommi <[email protected]>; Tsunakawa, Takayuki <[email protected]>; Dmitry Igrishin <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Manuel Kniep <[email protected]>; [email protected] <[email protected]>; Iwata, Aya <[email protected]>
On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite <[email protected]>
wrote:
> Vaishnavi Prabakaran wrote:
>
> > So, attached the alternative fix for this issue.
> > Please share me your thoughts.
>
> I assume you prefer the alternative fix because it's simpler.
>
I would like add one more reason for this fix, I think "PQsetSingleRowMode"
should be called only when the result is ready to be processed and before
starting to consume result as it is documented currently as follows -
"To enter single-row mode, call PQsetSingleRowMode immediately after a
successful call of PQsendQuery (or a sibling function). This mode selection
is effective only for the currently executing query. Then call PQgetResult
repeatedly..."
I agree that first fix (you shared) will allow user to set single-row mode
after PQsendQuery, but it also allows user to set this mode at any time of
batch processing(not necessarily "immediately after PQsendQuery"), also
"mode selection is effective only for the currently executing query" will
be false. Please note that I don't see any problem with this deviation. I
like to outline that documentation here anyways needs an update/note.
Before going further, I would like to mention that I have modified the
documentation of batch processing( in v6 code patch) as below:
"To enter single-row mode, call PQsetSingleRowMode immediately after a
successful call of PQbatchQueueProcess. This mode selection is effective
only for the currently executing query. For more information on the
use of PQsetSingleRowMode
, refer to Section 33.6, “Retrieving Query Results Row-By-Row”. "
Please let me know if you think this is not enough but wanted to update
section 33.6 also?
>
> > I would also like to hear Craig's opinion on it before applying this fix
> > to the original patch, just to make sure am not missing anything here.
>
> +1
>
> The main question is whether the predicates enforced
> by PQsetSingleRowMode() apply in batch mode in all cases
> when it's legit to call that function. Two predicates
> that may be problematic are:
> if (conn->asyncStatus != PGASYNC_BUSY)
> return 0;
> and
> if (conn->result)
> return 0;
>
> The general case with batch mode is that, from the doc:
> "The client interleaves result processing with sending batch queries"
>
While sending batch queries in middle of result processing, only the query
is appended to the list of queries maintained for batch processing and no
current connection attribute impacting result processing will be changed.
So, calling the PQsetSingleRowMode in-between result processing will fail
as it tries to set single-row mode for currently executing query for which
result processing is already started.
Note that I've not even tested that here,
I've tested
> batching a bunch of queries in a first step and getting the results
> in a second step.
> I am not confident that the above predicates will be true
> in all cases.
Also your alternative fix assumes that we add
> a user-visible exception to PQsetSingleRowMode in batch mode,
> whereby it must not be called as currently documented:
> "call PQsetSingleRowMode immediately after a successful call of
> PQsendQuery (or a sibling function)"
My gut feeling is that it's not the right direction, I prefer making
> the single-row a per-query attribute internally and keep
> PQsetSingleRowMode's contract unchanged from the
> user's perspective.
>
>
Am going to include the test which you shared in the test patch. Please let
me know if you want to cover anymore specific cases to gain confidence.
Thanks & Regards,
Vaishnavi,
Fujitsu Australia.
^ permalink raw reply [nested|flat] 40+ messages in thread
* psql --csv and other parameters
@ 2018-11-28 07:09 Erik Rijkers <[email protected]>
0 siblings, 1 reply; 40+ messages in thread
From: Erik Rijkers @ 2018-11-28 07:09 UTC (permalink / raw)
To: pgsql-hackers
I don't know if this really is a bug but it seems wrong to me:
psql -A --csv -Xc "select * from pg_namespace order by 1"
gives a difference result than
psql --csv -A -Xc "select * from pg_namespace order by 1"
I would say both should give the same result, and
that result should be the same as from:
psql --csv -Xc "select * from pg_namespace order by 1"
Thanks,
Erik Rijkers
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: psql --csv and other parameters
@ 2018-11-28 13:38 Daniel Verite <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 1 reply; 40+ messages in thread
From: Daniel Verite @ 2018-11-28 13:38 UTC (permalink / raw)
To: Erik Rijkers <[email protected]>; +Cc: pgsql-hackers
Erik Rijkers wrote:
> I don't know if this really is a bug but it seems wrong to me:
>
> psql -A --csv -Xc "select * from pg_namespace order by 1"
>
> gives a difference result than
>
> psql --csv -A -Xc "select * from pg_namespace order by 1"
-A and --csv each select an output format, and since
there can be only one output format active, they are
mutually exclusive.
When options conflict, the rightmost on the command line wins,
so -A --cvs is the same as --csv, and --csv -A is the same as -A
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: psql --csv and other parameters
@ 2018-11-29 02:27 Michael Paquier <[email protected]>
parent: Daniel Verite <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Michael Paquier @ 2018-11-29 02:27 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Erik Rijkers <[email protected]>; pgsql-hackers
On Wed, Nov 28, 2018 at 02:38:04PM +0100, Daniel Verite wrote:
> -A and --csv each select an output format, and since
> there can be only one output format active, they are
> mutually exclusive.
>
> When options conflict, the rightmost on the command line wins,
> so -A --cvs is the same as --csv, and --csv -A is the same as -A
And that's what I would expect. A lot of folks like using aliases and
relying on the last option being defined as a winner instead of bumping
into an error if multiple overlapping options are defined.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v7 06/11] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-04-07 04:28 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-04-07 04:28 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it just
leads to unnecessary cacheline conflicts to have backends share a
cacheline with other backends PGXACT data (which also have frequently
changing xmins of course).
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/nbtree/nbtxlog.c | 2 +-
src/backend/access/transam/README | 4 +--
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 30 ++++++++---------
13 files changed, 54 insertions(+), 58 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 23d12c1f72f..3b3936249ab 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -95,6 +95,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -219,11 +224,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index 66ddbaa5c4a..2a5b5308644 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* InvisibleToEveryoneCheckFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on master and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 6e5ee3b443e..a21ee727ed4 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2185,7 +2185,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, bool *rightsib_empty)
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/nbtree/nbtxlog.c b/src/backend/access/nbtree/nbtxlog.c
index 431d7c3d709..d43eb21a3cf 100644
--- a/src/backend/access/nbtree/nbtxlog.c
+++ b/src/backend/access/nbtree/nbtxlog.c
@@ -928,7 +928,7 @@ btree_xlog_reuse_page(XLogReaderState *record)
*
* latestRemovedXid was the page's btpo.xact. The
* InvisibleToEveryoneCheckXid test in _bt_page_recyclable() conceptually
- * mirrors the pgxact->xmin > limitXmin test in
+ * mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on master and standby.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index be805a5660b..85c2625ec42 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -296,7 +296,7 @@ ensure that the C compiler does exactly what you tell it to.)
Another important activity that uses the shared ProcArray is
ComputeTransactionHorizons, which must determine lower bound for the oldest
xmin of any active MVCC snapshot, system-wide. Each individual backend
-advertises the smallest xmin of its own snapshots in MyPgXact->xmin, or zero
+advertises the smallest xmin of its own snapshots in MyProc->xmin, or zero
if it currently has no live snapshots (eg, if it's between transactions or
hasn't yet set a snapshot for a new transaction).
ComputeTransactionHorizons takes the MIN() of the valid xmin fields. It
@@ -335,7 +335,7 @@ Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots,
there is no such guarantee for the horizons computed by
ComputeTransactionHorizons. This is because we allow XID-less
-transactions to clear their MyPgXact->xmin asynchronously (without
+transactions to clear their MyProc->xmin asynchronously (without
taking ProcArrayLock), so one execution might see what had been the
oldest xmin, and another not. This is OK since the thresholds need
only be a valid lower bound. As noted above, we are already assuming
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2f7d4ed59a8..5867cc60f3e 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 094bf6139f0..b63697da456 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d7088d19fd6..667ebca4e23 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1948,7 +1948,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2077,7 +2077,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2119,7 +2119,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2132,9 +2132,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index a1823caf632..52822c74cff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -543,9 +543,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -565,9 +565,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -719,7 +719,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1468,7 +1468,7 @@ ComputeTransactionHorizons(ComputedHorizons *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1738,7 +1738,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1799,7 +1799,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -1951,8 +1951,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2072,7 +2072,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2125,7 +2125,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2136,7 +2136,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2148,7 +2148,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2161,7 +2161,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2169,20 +2168,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2807,7 +2804,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2893,7 +2890,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -2903,7 +2899,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 5aa19d3f781..66d25dba7f8 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1182233bf43..01f1c133014 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -477,7 +477,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -598,7 +598,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -855,7 +855,7 @@ bool
SnapshotSet(void)
{
/* can't be safe, because somehow xmin is not set */
- if (!TransactionIdIsValid(MyPgXact->xmin) && HistoricSnapshot == NULL)
+ if (!TransactionIdIsValid(MyProc->xmin) && HistoricSnapshot == NULL)
return false;
/*
@@ -971,13 +971,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -998,7 +998,7 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
TransactionXmin = InvalidTransactionId;
RecentXmin = InvalidTransactionId;
return;
@@ -1007,8 +1007,8 @@ SnapshotResetXmin(void)
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1155,13 +1155,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1847,7 +1847,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--filbd3fs2o265vea
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0007-snapshot-scalability-Move-in-progress-xids-to-Pro.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v9 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-04-08 11:34 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-04-08 11:34 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it just
leads to unnecessary cacheline ping-pong to have it in PGXACT.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
11 files changed, 50 insertions(+), 54 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 23d12c1f72f..3b3936249ab 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -95,6 +95,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -219,11 +224,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index af4731cff18..19f39e4dc0a 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on master and standby.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index 4e2178dabab..94d8f3fd0a2 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 2f7d4ed59a8..5867cc60f3e 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 2baca12c5f4..9d741aa03fa 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index d8989762d74..b15faa18194 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1948,7 +1948,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2077,7 +2077,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2119,7 +2119,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2132,9 +2132,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 58f119f9895..899f936925e 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -588,9 +588,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -610,9 +610,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -764,7 +764,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1553,7 +1553,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1825,7 +1825,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1887,7 +1887,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2039,8 +2039,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2160,7 +2160,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2213,7 +2213,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2224,7 +2224,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2236,7 +2236,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2249,7 +2249,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2257,20 +2256,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2895,7 +2892,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2981,7 +2978,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -2991,7 +2987,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 5aa19d3f781..66d25dba7f8 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index ba5d9615c79..e9d3e832c76 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--w7fggl772xdywjph
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0003-snapshot-scalability-Move-in-progress-xids-to-Pro.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v11 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 08f006f782e..286c9a9aec3 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -102,6 +102,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -224,11 +229,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index 3167305ac00..b6603cd73cf 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 9e6376f2c2b..c88ca4221a4 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2208,7 +2208,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index 4e2178dabab..94d8f3fd0a2 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 9b2e59bf0ec..ae7c1a4c172 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 2baca12c5f4..9d741aa03fa 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index fd370d52b66..06da4b4352a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1967,7 +1967,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2096,7 +2096,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2138,7 +2138,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2151,9 +2151,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index c011387ba90..980ca2cc2af 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -585,9 +585,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -607,9 +607,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -761,7 +761,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1550,7 +1550,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1822,7 +1822,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1884,7 +1884,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2036,8 +2036,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2157,7 +2157,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2210,7 +2210,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2221,7 +2221,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2233,7 +2233,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2246,7 +2246,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2254,20 +2253,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2892,7 +2889,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2978,7 +2975,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -2988,7 +2984,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--kdyesixubxkl3b7r
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v12 3/7] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 08f006f782e..286c9a9aec3 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -102,6 +102,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -224,11 +229,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index 3167305ac00..b6603cd73cf 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index d18b2722693..d567c51c6f2 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2315,7 +2315,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index 4e2178dabab..94d8f3fd0a2 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 9b2e59bf0ec..ae7c1a4c172 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 2baca12c5f4..9d741aa03fa 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index fd370d52b66..06da4b4352a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1967,7 +1967,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2096,7 +2096,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2138,7 +2138,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2151,9 +2151,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 20115e2f63f..164cf0cabc2 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -586,9 +586,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -608,9 +608,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -762,7 +762,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1560,7 +1560,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1835,7 +1835,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1897,7 +1897,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2049,8 +2049,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2170,7 +2170,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2223,7 +2223,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2234,7 +2234,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2246,7 +2246,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2259,7 +2259,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2267,20 +2266,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2906,7 +2903,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2992,7 +2989,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3002,7 +2998,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--pyx7ptyhkbjgwa43
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0004-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC.
@ 2020-07-15 22:35 Andres Freund <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw)
Now that xmin isn't needed for GetSnapshotData() anymore, it leads to
unnecessary cacheline ping-pong to have it in PGXACT as it is updated
more frequently than the other PGXACT members.
Author: Andres Freund
Reviewed-By: Robert Haas, Thomas Munro, David Rowley
Discussion: https://postgr.es/m/[email protected]
---
src/include/storage/proc.h | 10 +++---
src/backend/access/gist/gistxlog.c | 2 +-
src/backend/access/nbtree/nbtpage.c | 2 +-
src/backend/access/transam/README | 2 +-
src/backend/access/transam/twophase.c | 2 +-
src/backend/commands/indexcmds.c | 2 +-
src/backend/replication/logical/snapbuild.c | 6 ++--
src/backend/replication/walsender.c | 10 +++---
src/backend/storage/ipc/procarray.c | 36 +++++++++------------
src/backend/storage/ipc/sinvaladt.c | 2 +-
src/backend/storage/lmgr/proc.c | 4 +--
src/backend/utils/time/snapmgr.c | 28 ++++++++--------
12 files changed, 51 insertions(+), 55 deletions(-)
diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 52ff43cabaa..5e4b028a5f9 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -101,6 +101,11 @@ struct PGPROC
Latch procLatch; /* generic latch for process */
+ TransactionId xmin; /* minimal running XID as it was when we were
+ * starting our xact, excluding LAZY VACUUM:
+ * vacuum must not remove tuples deleted by
+ * xid >= xmin ! */
+
LocalTransactionId lxid; /* local id of top-level transaction currently
* being executed by this proc, if running;
* else InvalidLocalTransactionId */
@@ -223,11 +228,6 @@ typedef struct PGXACT
* executed by this proc, if running and XID
* is assigned; else InvalidTransactionId */
- TransactionId xmin; /* minimal running XID as it was when we were
- * starting our xact, excluding LAZY VACUUM:
- * vacuum must not remove tuples deleted by
- * xid >= xmin ! */
-
uint8 vacuumFlags; /* vacuum-related flags, see above */
bool overflowed;
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index a63b05388c5..dcd28f678b3 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -389,7 +389,7 @@ gistRedoPageReuse(XLogReaderState *record)
*
* latestRemovedXid was the page's deleteXid. The
* GlobalVisIsRemovableFullXid(deleteXid) test in gistPageRecyclable()
- * conceptually mirrors the pgxact->xmin > limitXmin test in
+ * conceptually mirrors the PGPROC->xmin > limitXmin test in
* GetConflictingVirtualXIDs(). Consequently, one XID value achieves the
* same exclusion effect on primary and standby.
*/
diff --git a/src/backend/access/nbtree/nbtpage.c b/src/backend/access/nbtree/nbtpage.c
index 74be3807bb7..7f392480ac0 100644
--- a/src/backend/access/nbtree/nbtpage.c
+++ b/src/backend/access/nbtree/nbtpage.c
@@ -2317,7 +2317,7 @@ _bt_unlink_halfdead_page(Relation rel, Buffer leafbuf, BlockNumber scanblkno,
* we're in VACUUM and would not otherwise have an XID. Having already
* updated links to the target, ReadNewTransactionId() suffices as an
* upper bound. Any scan having retained a now-stale link is advertising
- * in its PGXACT an xmin less than or equal to the value we read here. It
+ * in its PGPROC an xmin less than or equal to the value we read here. It
* will continue to do so, holding back the xmin horizon, for the duration
* of that scan.
*/
diff --git a/src/backend/access/transam/README b/src/backend/access/transam/README
index fffe0783295..c15b5540a09 100644
--- a/src/backend/access/transam/README
+++ b/src/backend/access/transam/README
@@ -331,7 +331,7 @@ necessary.
Note that while it is certain that two concurrent executions of
GetSnapshotData will compute the same xmin for their own snapshots, there is
no such guarantee for the horizons computed by ComputeXidHorizons. This is
-because we allow XID-less transactions to clear their MyPgXact->xmin
+because we allow XID-less transactions to clear their MyProc->xmin
asynchronously (without taking ProcArrayLock), so one execution might see
what had been the oldest xmin, and another not. This is OK since the
thresholds need only be a valid lower bound. As noted above, we are already
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 31f135f5ced..eb5f4680a3d 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -464,7 +464,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
/* We set up the gxact's VXID as InvalidBackendId/XID */
proc->lxid = (LocalTransactionId) xid;
pgxact->xid = xid;
- pgxact->xmin = InvalidTransactionId;
+ Assert(proc->xmin == InvalidTransactionId);
proc->delayChkpt = false;
pgxact->vacuumFlags = 0;
proc->pid = 0;
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 7819266a630..254dbcdce52 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -1535,7 +1535,7 @@ DefineIndex(Oid relationId,
StartTransactionCommand();
/* We should now definitely not be advertising any xmin. */
- Assert(MyPgXact->xmin == InvalidTransactionId);
+ Assert(MyProc->xmin == InvalidTransactionId);
/*
* The index is now valid in the sense that it contains all currently
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3089f0d5ddc..e9701ea7221 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -553,8 +553,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
elog(ERROR, "cannot build an initial slot snapshot, not all transactions are monitored anymore");
/* so we don't overwrite the existing value */
- if (TransactionIdIsValid(MyPgXact->xmin))
- elog(ERROR, "cannot build an initial slot snapshot when MyPgXact->xmin already is valid");
+ if (TransactionIdIsValid(MyProc->xmin))
+ elog(ERROR, "cannot build an initial slot snapshot when MyProc->xmin already is valid");
snap = SnapBuildBuildSnapshot(builder);
@@ -575,7 +575,7 @@ SnapBuildInitialSnapshot(SnapBuild *builder)
}
#endif
- MyPgXact->xmin = snap->xmin;
+ MyProc->xmin = snap->xmin;
/* allocate in transaction context */
newxip = (TransactionId *)
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 460ca3f947f..3f756b470af 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1964,7 +1964,7 @@ PhysicalReplicationSlotNewXmin(TransactionId feedbackXmin, TransactionId feedbac
ReplicationSlot *slot = MyReplicationSlot;
SpinLockAcquire(&slot->mutex);
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
/*
* For physical replication we don't need the interlock provided by xmin
@@ -2093,7 +2093,7 @@ ProcessStandbyHSFeedbackMessage(void)
if (!TransactionIdIsNormal(feedbackXmin)
&& !TransactionIdIsNormal(feedbackCatalogXmin))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
if (MyReplicationSlot != NULL)
PhysicalReplicationSlotNewXmin(feedbackXmin, feedbackCatalogXmin);
return;
@@ -2135,7 +2135,7 @@ ProcessStandbyHSFeedbackMessage(void)
* risk already since a VACUUM could already have determined the horizon.)
*
* If we're using a replication slot we reserve the xmin via that,
- * otherwise via the walsender's PGXACT entry. We can only track the
+ * otherwise via the walsender's PGPROC entry. We can only track the
* catalog xmin separately when using a slot, so we store the least of the
* two provided when not using a slot.
*
@@ -2148,9 +2148,9 @@ ProcessStandbyHSFeedbackMessage(void)
{
if (TransactionIdIsNormal(feedbackCatalogXmin)
&& TransactionIdPrecedes(feedbackCatalogXmin, feedbackXmin))
- MyPgXact->xmin = feedbackCatalogXmin;
+ MyProc->xmin = feedbackCatalogXmin;
else
- MyPgXact->xmin = feedbackXmin;
+ MyProc->xmin = feedbackXmin;
}
}
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 360e6e9da07..a016816ae86 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -587,9 +587,9 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
Assert(!TransactionIdIsValid(allPgXact[proc->pgprocno].xid));
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -609,9 +609,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
{
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
/* must be cleared with xid/xmin: */
pgxact->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
+ proc->xmin = InvalidTransactionId;
proc->delayChkpt = false; /* be sure this is cleared in abort */
proc->recoveryConflictPending = false;
@@ -763,7 +763,7 @@ ProcArrayClearTransaction(PGPROC *proc)
*/
pgxact->xid = InvalidTransactionId;
proc->lxid = InvalidLocalTransactionId;
- pgxact->xmin = InvalidTransactionId;
+ proc->xmin = InvalidTransactionId;
proc->recoveryConflictPending = false;
/* redundant, but just in case */
@@ -1563,7 +1563,7 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
/* Fetch xid just once - see GetNewTransactionId */
xid = UINT32_ACCESS_ONCE(pgxact->xid);
- xmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* Consider both the transaction's Xmin, and its Xid.
@@ -1838,7 +1838,7 @@ GetMaxSnapshotSubxidCount(void)
*
* We also update the following backend-global variables:
* TransactionXmin: the oldest xmin of any snapshot in use in the
- * current transaction (this is the same as MyPgXact->xmin).
+ * current transaction (this is the same as MyProc->xmin).
* RecentXmin: the xmin computed for the most recent snapshot. XIDs
* older than this are known not running any more.
*
@@ -1900,7 +1900,7 @@ GetSnapshotData(Snapshot snapshot)
/*
* It is sufficient to get shared lock on ProcArrayLock, even if we are
- * going to set MyPgXact->xmin.
+ * going to set MyProc->xmin.
*/
LWLockAcquire(ProcArrayLock, LW_SHARED);
@@ -2052,8 +2052,8 @@ GetSnapshotData(Snapshot snapshot)
replication_slot_xmin = procArray->replication_slot_xmin;
replication_slot_catalog_xmin = procArray->replication_slot_catalog_xmin;
- if (!TransactionIdIsValid(MyPgXact->xmin))
- MyPgXact->xmin = TransactionXmin = xmin;
+ if (!TransactionIdIsValid(MyProc->xmin))
+ MyProc->xmin = TransactionXmin = xmin;
LWLockRelease(ProcArrayLock);
@@ -2173,7 +2173,7 @@ GetSnapshotData(Snapshot snapshot)
}
/*
- * ProcArrayInstallImportedXmin -- install imported xmin into MyPgXact->xmin
+ * ProcArrayInstallImportedXmin -- install imported xmin into MyProc->xmin
*
* This is called when installing a snapshot imported from another
* transaction. To ensure that OldestXmin doesn't go backwards, we must
@@ -2226,7 +2226,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
/*
* Likewise, let's just make real sure its xmin does cover us.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (!TransactionIdIsNormal(xid) ||
!TransactionIdPrecedesOrEquals(xid, xmin))
continue;
@@ -2237,7 +2237,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
* GetSnapshotData first, we'll be overwriting a valid xmin here, so
* we don't check that.)
*/
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
break;
@@ -2249,7 +2249,7 @@ ProcArrayInstallImportedXmin(TransactionId xmin,
}
/*
- * ProcArrayInstallRestoredXmin -- install restored xmin into MyPgXact->xmin
+ * ProcArrayInstallRestoredXmin -- install restored xmin into MyProc->xmin
*
* This is like ProcArrayInstallImportedXmin, but we have a pointer to the
* PGPROC of the transaction from which we imported the snapshot, rather than
@@ -2262,7 +2262,6 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
{
bool result = false;
TransactionId xid;
- PGXACT *pgxact;
Assert(TransactionIdIsNormal(xmin));
Assert(proc != NULL);
@@ -2270,20 +2269,18 @@ ProcArrayInstallRestoredXmin(TransactionId xmin, PGPROC *proc)
/* Get lock so source xact can't end while we're doing this */
LWLockAcquire(ProcArrayLock, LW_SHARED);
- pgxact = &allPgXact[proc->pgprocno];
-
/*
* Be certain that the referenced PGPROC has an advertised xmin which is
* no later than the one we're installing, so that the system-wide xmin
* can't go backwards. Also, make sure it's running in the same database,
* so that the per-database xmin cannot go backwards.
*/
- xid = UINT32_ACCESS_ONCE(pgxact->xmin);
+ xid = UINT32_ACCESS_ONCE(proc->xmin);
if (proc->databaseId == MyDatabaseId &&
TransactionIdIsNormal(xid) &&
TransactionIdPrecedesOrEquals(xid, xmin))
{
- MyPgXact->xmin = TransactionXmin = xmin;
+ MyProc->xmin = TransactionXmin = xmin;
result = true;
}
@@ -2909,7 +2906,7 @@ GetCurrentVirtualXIDs(TransactionId limitXmin, bool excludeXmin0,
if (allDbs || proc->databaseId == MyDatabaseId)
{
/* Fetch xmin just once - might change on us */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
if (excludeXmin0 && !TransactionIdIsValid(pxmin))
continue;
@@ -2995,7 +2992,6 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
{
int pgprocno = arrayP->pgprocnos[index];
PGPROC *proc = &allProcs[pgprocno];
- PGXACT *pgxact = &allPgXact[pgprocno];
/* Exclude prepared transactions */
if (proc->pid == 0)
@@ -3005,7 +3001,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid)
proc->databaseId == dbOid)
{
/* Fetch xmin just once - can't change on us, but good coding */
- TransactionId pxmin = UINT32_ACCESS_ONCE(pgxact->xmin);
+ TransactionId pxmin = UINT32_ACCESS_ONCE(proc->xmin);
/*
* We ignore an invalid pxmin because this means that backend has
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index e5c115b92f2..ad048bc85fa 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -420,7 +420,7 @@ BackendIdGetTransactionIds(int backendID, TransactionId *xid, TransactionId *xmi
PGXACT *xact = &ProcGlobal->allPgXact[proc->pgprocno];
*xid = xact->xid;
- *xmin = xact->xmin;
+ *xmin = proc->xmin;
}
}
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e57fcd25388..de346cd87fc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -388,7 +388,7 @@ InitProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->pid = MyProcPid;
/* backendId, databaseId and roleId will be filled in later */
MyProc->backendId = InvalidBackendId;
@@ -572,7 +572,7 @@ InitAuxiliaryProcess(void)
MyProc->fpVXIDLock = false;
MyProc->fpLocalTransactionId = InvalidLocalTransactionId;
MyPgXact->xid = InvalidTransactionId;
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
MyProc->backendId = InvalidBackendId;
MyProc->databaseId = InvalidOid;
MyProc->roleId = InvalidOid;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 76578868cf9..689a3b6a597 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -27,11 +27,11 @@
* their lifetime is managed separately (as they live longer than one xact.c
* transaction).
*
- * These arrangements let us reset MyPgXact->xmin when there are no snapshots
+ * These arrangements let us reset MyProc->xmin when there are no snapshots
* referenced by this transaction, and advance it when the one with oldest
* Xmin is no longer referenced. For simplicity however, only registered
* snapshots not active snapshots participate in tracking which one is oldest;
- * we don't try to change MyPgXact->xmin except when the active-snapshot
+ * we don't try to change MyProc->xmin except when the active-snapshot
* stack is empty.
*
*
@@ -187,7 +187,7 @@ static ActiveSnapshotElt *OldestActiveSnapshot = NULL;
/*
* Currently registered Snapshots. Ordered in a heap by xmin, so that we can
- * quickly find the one with lowest xmin, to advance our MyPgXact->xmin.
+ * quickly find the one with lowest xmin, to advance our MyProc->xmin.
*/
static int xmin_cmp(const pairingheap_node *a, const pairingheap_node *b,
void *arg);
@@ -475,7 +475,7 @@ GetNonHistoricCatalogSnapshot(Oid relid)
/*
* Make sure the catalog snapshot will be accounted for in decisions
- * about advancing PGXACT->xmin. We could apply RegisterSnapshot, but
+ * about advancing PGPROC->xmin. We could apply RegisterSnapshot, but
* that would result in making a physical copy, which is overkill; and
* it would also create a dependency on some resource owner, which we
* do not want for reasons explained at the head of this file. Instead
@@ -596,7 +596,7 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid,
/* NB: curcid should NOT be copied, it's a local matter */
/*
- * Now we have to fix what GetSnapshotData did with MyPgXact->xmin and
+ * Now we have to fix what GetSnapshotData did with MyProc->xmin and
* TransactionXmin. There is a race condition: to make sure we are not
* causing the global xmin to go backwards, we have to test that the
* source transaction is still running, and that has to be done
@@ -950,13 +950,13 @@ xmin_cmp(const pairingheap_node *a, const pairingheap_node *b, void *arg)
/*
* SnapshotResetXmin
*
- * If there are no more snapshots, we can reset our PGXACT->xmin to InvalidXid.
+ * If there are no more snapshots, we can reset our PGPROC->xmin to InvalidXid.
* Note we can do this without locking because we assume that storing an Xid
* is atomic.
*
* Even if there are some remaining snapshots, we may be able to advance our
- * PGXACT->xmin to some degree. This typically happens when a portal is
- * dropped. For efficiency, we only consider recomputing PGXACT->xmin when
+ * PGPROC->xmin to some degree. This typically happens when a portal is
+ * dropped. For efficiency, we only consider recomputing PGPROC->xmin when
* the active snapshot stack is empty; this allows us not to need to track
* which active snapshot is oldest.
*
@@ -977,15 +977,15 @@ SnapshotResetXmin(void)
if (pairingheap_is_empty(&RegisteredSnapshots))
{
- MyPgXact->xmin = InvalidTransactionId;
+ MyProc->xmin = InvalidTransactionId;
return;
}
minSnapshot = pairingheap_container(SnapshotData, ph_node,
pairingheap_first(&RegisteredSnapshots));
- if (TransactionIdPrecedes(MyPgXact->xmin, minSnapshot->xmin))
- MyPgXact->xmin = minSnapshot->xmin;
+ if (TransactionIdPrecedes(MyProc->xmin, minSnapshot->xmin))
+ MyProc->xmin = minSnapshot->xmin;
}
/*
@@ -1132,13 +1132,13 @@ AtEOXact_Snapshot(bool isCommit, bool resetXmin)
/*
* During normal commit processing, we call ProcArrayEndTransaction() to
- * reset the MyPgXact->xmin. That call happens prior to the call to
+ * reset the MyProc->xmin. That call happens prior to the call to
* AtEOXact_Snapshot(), so we need not touch xmin here at all.
*/
if (resetXmin)
SnapshotResetXmin();
- Assert(resetXmin || MyPgXact->xmin == 0);
+ Assert(resetXmin || MyProc->xmin == 0);
}
@@ -1830,7 +1830,7 @@ TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
*/
if (old_snapshot_threshold == 0)
{
- if (TransactionIdPrecedes(latest_xmin, MyPgXact->xmin)
+ if (TransactionIdPrecedes(latest_xmin, MyProc->xmin)
&& TransactionIdFollows(latest_xmin, xlimit))
xlimit = latest_xmin;
--
2.25.0.114.g5b0ca878e0
--4il4azyv6rmablac
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v13-0003-snapshot-scalability-Introduce-dense-array-of-in.patch"
^ permalink raw reply [nested|flat] 40+ messages in thread
* Re: psql help message contains excessive indentations
@ 2023-09-18 14:39 Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 40+ messages in thread
From: Alvaro Herrera @ 2023-09-18 14:39 UTC (permalink / raw)
To: Yugo NAGATA <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]
On 2023-Sep-07, Yugo NAGATA wrote:
> Thank you for your explanation. I understood it. I thought of just
> imitating other places, and I didn't know each is a single translatable
> unit.
Thanks for reviewing, and Kyotaro for reporting. Pushed now.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Estoy de acuerdo contigo en que la verdad absoluta no existe...
El problema es que la mentira sí existe y tu estás mintiendo" (G. Lama)
^ permalink raw reply [nested|flat] 40+ messages in thread
end of thread, other threads:[~2023-09-18 14:39 UTC | newest]
Thread overview: 40+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-03-07 16:52 Re: PATCH: Batch/pipelining support for libpq Daniel Verite <[email protected]>
2017-03-08 01:56 ` Craig Ringer <[email protected]>
2017-03-08 02:04 ` Vaishnavi Prabakaran <[email protected]>
2017-03-09 21:43 ` Daniel Verite <[email protected]>
2017-03-10 13:52 ` Daniel Verite <[email protected]>
2017-03-13 00:54 ` Vaishnavi Prabakaran <[email protected]>
2017-03-13 01:32 ` Craig Ringer <[email protected]>
2017-03-13 17:19 ` Daniel Verite <[email protected]>
2017-03-14 06:50 ` Vaishnavi Prabakaran <[email protected]>
2017-03-16 02:42 ` Vaishnavi Prabakaran <[email protected]>
2017-03-16 13:37 ` Daniel Verite <[email protected]>
2017-03-20 01:32 ` Vaishnavi Prabakaran <[email protected]>
2018-11-28 07:09 psql --csv and other parameters Erik Rijkers <[email protected]>
2018-11-28 13:38 ` Re: psql --csv and other parameters Daniel Verite <[email protected]>
2018-11-29 02:27 ` Re: psql --csv and other parameters Michael Paquier <[email protected]>
2020-04-07 04:28 [PATCH v7 06/11] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-04-08 11:34 [PATCH v9 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v11 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v12 3/7] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2020-07-15 22:35 [PATCH v13 2/6] snapshot scalability: Move PGXACT->xmin back to PGPROC. Andres Freund <[email protected]>
2023-09-18 14:39 Re: psql help message contains excessive indentations Alvaro Herrera <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox