pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
82+ messages / 5 participants
[nested] [flat]
Commits from this thread:
2022-04-19 a62bff74b1 Fix aggregate logging of pgbench.
2022-04-10 b6b8824ee7 Doc: reformat descriptions of pgbench output formats.
2022-03-25 0f79caa3c6 Second attempt to fix pgbench TAP test.
2022-03-24 ac9c5dc5d4 Fix pgbench TAP test.
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
@ 2021-06-22 14:54 Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-22 18:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-23 08:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 3 replies; 82+ messages in thread
From: Yugo NAGATA @ 2021-06-22 14:54 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Fabien COELHO <coelho@cri.ensmp.fr>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hi hackers,
On Mon, 24 May 2021 11:29:10 +0900
Yugo NAGATA <nagata@sraoss.co.jp> wrote:
> Hi hackers,
>
> On Tue, 10 Mar 2020 09:48:23 +1300
> Thomas Munro <thomas.munro@gmail.com> wrote:
>
> > On Tue, Mar 10, 2020 at 8:43 AM Fabien COELHO <coelho@cri.ensmp.fr> wrote:
> > > >> Thank you very much! I'm going to send a new patch set until the end of
> > > >> this week (I'm sorry I was very busy in the release of Postgres Pro
> > > >> 11...).
> > > >
> > > > Is anyone interested in rebasing this, and summarising what needs to
> > > > be done to get it in? It's arguably a bug or at least quite
> > > > unfortunate that pgbench doesn't work with SERIALIZABLE, and I heard
> > > > that a couple of forks already ship Marina's patch set.
>
> I got interested in this and now looking into the patch and the past discussion.
> If anyone other won't do it and there are no objection, I would like to rebase
> this. Is that okay?
I rebased and fixed the previous patches (v11) rewtten by Marina Polyakova,
and attached the revised version (v12).
v12-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch
- a patch for the Variables structure (this is used to reset client
variables during the repeating of transactions after
serialization/deadlock failures).
v12-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch
- the main patch for handling client errors and repetition of
transactions with serialization/deadlock failures (see the detailed
description in the file).
These are the revised versions from v11-0002 and v11-0003. v11-0001
(for the RandomState structure) is not included because this has been
already committed (40923191944). V11-0004 (for a separate error reporting
function) is not included neither because pgbench now uses common logging
APIs (30a3e772b40).
In addition to rebase on master, I updated the patch according with the
review from Fabien COELHO [1] and discussions after this. Also, I added
some other fixes through my reviewing the previous patch.
[1] https://www.postgresql.org/message-id/alpine.DEB.2.21.1809081450100.10506%40lancre
Following are fixes according with Fabian's review.
> * Features
> As far as the actual retry feature is concerned, I'd say we are nearly
> there. However I have an issue with changing the behavior on meta command
> and other sql errors, which I find not desirable.
...
> I do not think that these changes of behavior are desirable. Meta command and
> miscellaneous SQL errors should result in immediatly aborting the whole run,
> because the client test code itself could not run correctly or the SQL sent
> was somehow wrong, which is also the client's fault, and the server
> performance bench does not make much sense in such conditions.
>
> ISTM that the focus of this patch should only be to handle some server
> runtime errors that can be retryed, but not to change pgbench behavior on
> other kind of errors. If these are to be changed, ISTM that it would be a
> distinct patch and would require some discussion, and possibly an option
> to enable it or not if some use case emerge. AFA this patch is concerned,
> I'd suggest to let that out.
Previously, all SQL and meta command errors could be retried, but I fixed
to allow only serialization & deadlock errors to be retried.
> Doc says "you cannot use an infinite number of retries without latency-limit..."
>
> Why should this be forbidden? At least if -T timeout takes precedent and
> shortens the execution, ISTM that there could be good reason to test that.
> Maybe it could be blocked only under -t if this would lead to an non-ending
> run.
I fixed to allow to use --max-tries with -T option even if latency-limit
is not used.
> As "--print-errors" is really for debug, maybe it could be named
> "--debug-errors". I'm not sure that having "--debug" implying this option
> is useful: As there are two distinct options, the user may be allowed
> to trigger one or the other as they wish?
print-errors was renamed to debug-errors.
> makeVariableValue error message is not for debug, but must be kept in all
> cases, and the false returned must result in an immediate abort. Same thing about
> lookupCreateVariable, an invalid name is a user error which warrants an immediate
> abort. Same thing again about coerce* functions or evalStandardFunc...
> Basically, most/all added "debug_level >= DEBUG_ERRORS" are not desirable.
"DEBUG_ERRORS" messages unrelated to serialization & deadlock errors were removed.
> sendRollback(): I'd suggest to simplify. The prepare/extended statement stuff is
> really about the transaction script, not dealing with errors, esp as there is no
> significant advantage in preparing a "ROLLBACK" statement which is short and has
> no parameters. I'd suggest to remove this function and just issue
> PQsendQuery("ROLLBACK;") in all cases.
Now, we just issue PQsendQuery("ROLLBACK;").
> In copyVariables, I'd simplify
>
> + if (source_var->svalue == NULL)
> + dest_var->svalue = NULL;
> + else
> + dest_var->svalue = pg_strdup(source_var->svalue);
>
>as:
> dest_var->value = (source_var->svalue == NULL) ? NULL : pg_strdup(source_var->svalue);
Fixed using a ternary operator.
> + if (sqlState) -> if (sqlState != NULL) ?
Fixed.
> Function getTransactionStatus name does not seem to correspond fully to what the
> function does. There is a passthru case which should be either avoided or
> clearly commented.
This was renamed to checkTransactionStatus according with [2].
[2] https://www.postgresql.org/message-id/c262e889315625e0fc0d77ca78fe2eac%40postgrespro.ru
> - commandFailed(st, "SQL", "perhaps the backend died while processing");
> + clientAborted(st,
> + "perhaps the backend died while processing");
>
> keep on one line?
This fix that replaced commandFailed with clientAborted was removed.
(See below)
> + if (doRetry(st, &now))
> + st->state = CSTATE_RETRY;
> + else
> + st->state = CSTATE_FAILURE;
>
> -> st->state = doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
Fixed using a ternary operator.
> * Comments
> "There're different types..." -> "There are different types..."
> "after the errors and"... -> "after errors and"...
> "the default value of max_tries is set to 1" -> "the default value
> of max_tries is 1"
> "We cannot retry the transaction" -> "We cannot retry a transaction"
> "may ultimately succeed or get a failure," -> "may ultimately succeed or fail,"
Fixed.
> Overall, the comment text in StatsData is very clear. However they are not
> clearly linked to the struct fields. I'd suggest that earch field when used
> should be quoted, so as to separate English from code, and the struct name
> should always be used explicitely when possible.
The comment in StatsData was fixed to clarify what each filed in this struct
represents.
> I'd insist in a comment that "cnt" does not include "skipped" transactions
> (anymore).
StatsData.cnt has a comment "number of successful transactions, not including
'skipped'", and CState.cnt has a comment "skipped and failed transactions are
also counted here".
> * Documentation:
> ISTM that there are too many "the":
> - "turns on the option ..." -> "turns on option ..."
> - "When the option ..." -> "When option ..."
> - "By default the option ..." -> "By default option ..."
> - "only if the option ..." -> "only if option ..."
> - "combined with the option ..." -> "combined with option ..."
> - "without the option ..." -> "without option ..."
The previous patch used a lot of "the option xxxx", but I fixed
them to "the xxxx option" because I found that the documentation
uses such way for referring to a certain option. For example,
- You can (and, for most purposes, probably should) increase the number
of rows by using the <option>-s</option> (scale factor) option.
- The prefix can be changed by using the <option>--log-prefix</option> option.
- If the <option>-j</option> option is 2 or higher, so that there are multiple
worker threads,
> - "is the sum of all the retries" -> "is the sum of all retries"
> "infinite" -> "unlimited"
> "not retried at all" -> "not retried" (maybe several times).
> "messages of all errors" -> "messages about all errors".
> "It is assumed that the scripts used do not contain" ->
> "It is assumed that pgbench scripts do not contai
Fixed.
Following are additional fixes based on my review on the previous patch.
* About error reporting
In the previous patch, commandFailed() was changed to report an error
that doesn't immediately abort the client, and clientAborted() was
added to report an abortion of the client. In the attached patch,
behaviors around errors other than serialization and deadlock are
not changed and such errors cause the client to abort, so commandFaile()
is used without any changes to report a client abortion, and commandError()
is added to report an error that can be retried under --debug-error.
* About progress reporting
In the previous patch, the number of failures was reported only when any
transaction was failed, and statistics of retry was reported only when
any transaction was retried. This means, the number of columns in the
reporting were different depending on the interval. This was odd and
harder to parse the output.
In the attached patch, the number of failures is always reported, and
the retry statistic is reported when max-tries is not 1.
* About result outputs
In the previous patch, the number of failed transaction, the number
of retried transaction, and the number of total retries were reported
as:
number of failures: 324 (3.240%)
...
number of retried: 5629 (56.290%)
number of retries: 103299
I think this was confusable. Especially, it was unclear for me what
"retried" and "retries" represent repectively. Therefore, in the
attached patch, they are reported as:
number of transactions failed: 324 (3.240%)
...
number of transactions retried: 5629 (56.290%)
number of total retries: 103299
which clarify that first two are the numbers of transactions and the
last one is the number of retries over all transactions.
* Abourt average connection time
In the previous patch, this was calculated as "conn_total_duration / total->cnt"
where conn_total_duration is the cumulated connection time sumed over threads and
total->cnt is the number of transaction that is successfully processed.
However, the average connection time could be overestimated because
conn_total_duration includes a connection time of failed transaction
due to serialization and deadlock errors. So, in the attached patch,
this is calculated as "conn_total_duration / total->cnt + failures".
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v12-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 82.0K) download
From ae18f7445eff881800a398c27c850806048b060f Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Fri, 28 May 2021 10:48:57 +0900
Subject: [PATCH v12 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-per-command, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --debug-errors.
---
doc/src/sgml/ref/pgbench.sgml | 399 +++++++-
src/bin/pgbench/pgbench.c | 952 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 217 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1480 insertions(+), 116 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 0c60077e1f..6811a6b29c 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -58,6 +58,7 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
@@ -65,11 +66,14 @@ tps = 896.967014 (without initial connection time)
</screen>
The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ settings. The seventh line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
The last line reports the number of transactions per second.
</para>
@@ -528,6 +532,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information about retrying such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -594,23 +609,29 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-r</option></term>
- <term><option>--report-latencies</option></term>
+ <term><option>--report-per-command</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -738,6 +759,26 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -748,6 +789,38 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; more over, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information about
+ retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--debug-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -943,8 +1016,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts">
+ <title id="transactions-and-scripts-title">What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1017,6 +1090,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, he will be aborted.
+ </para>
</note>
<para>
@@ -2207,7 +2285,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2228,6 +2306,17 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization_failure</literal> or
+ <literal>deadlock_failure</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information).
</para>
<para>
@@ -2256,6 +2345,24 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2271,7 +2378,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2285,7 +2392,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2293,21 +2409,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2319,13 +2439,44 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
+
+ <para>
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2339,27 +2490,64 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 10.870 ms
latency stddev = 7.341 ms
initial connection time = 30.954 ms
tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+statement latencies in milliseconds and failures:
+ 0.002 0 \set aid random(1, 100000 * :scale)
+ 0.005 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.001 0 \set delta random(-5000, 5000)
+ 0.326 0 BEGIN;
+ 0.603 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.454 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 5.528 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 7.335 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.371 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.212 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+number of transactions per client: 1000
+number of transactions actually processed: 9676/10000
+number of transactions failed: 324 (3.240%)
+number of serialization failures: 324 (3.240%)
+number of transactions retried: 5629 (56.290%)
+number of total retries: 103299
+maximum number of tries: 100
+number of transactions above the 100.0 ms latency limit: 21/9676 (0.217 %)
+latency average = 16.138 ms
+latency stddev = 21.017 ms
+tps = 413.650224 (including connections establishing)
+tps = 413.686560 (excluding connections establishing)
+statement latencies in milliseconds, failures and retries:
+ 0.002 0 0 \set aid random(1, 100000 * :scale)
+ 0.000 0 0 \set bid random(1, 1 * :scale)
+ 0.000 0 0 \set tid random(1, 10 * :scale)
+ 0.000 0 0 \set delta random(-5000, 5000)
+ 0.121 0 0 BEGIN;
+ 0.290 0 2 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.221 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 0.266 212 72127 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.222 112 31170 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.178 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.210 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2373,6 +2561,135 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries">
+ <title id="failures-and-retries-title">Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there're three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the current transaction is rolled back which also
+ includes setting the client variables as they were before the run of this
+ transaction (it is assumed that one transaction script contains only one
+ transaction; see <xref linkend="transactions-and-scripts"
+ endterm="transactions-and-scripts-title"/> for more information).
+ Transactions with serialization or deadlock errors are repeated after
+ rollbacks until they complete successfully or reach the maximum number of
+ tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of tries (specified by the <option>--latency-limit</option> option). If
+ the last transaction run fails, this transaction will be reported as failed.
+ </para>
+
+ <note>
+ <para>
+ Although without the <option>--max-tries</option> option the transaction
+ will never be retried after an error, use an unlimited number of tries
+ (<literal>--max-tries=0</literal>) and the <option>--latency-limit</option>
+ option or the <option>--time</option> to limit only the maximum time of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency for failed
+ transactions and commands is not computed separately.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions if it is non-zero.
+ If the total number of retried transactions is non-zero, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--debug-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8acda86cad..77888146e2 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -74,6 +74,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -273,9 +275,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 0;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -360,9 +387,65 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them)
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactinos) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -375,6 +458,30 @@ typedef struct RandomState
unsigned short xseed[3];
} RandomState;
+/*
+ * Data structure for repeating a transaction from the beginnning with the same
+ * parameters.
+ */
+typedef struct
+{
+ RandomState random_state; /* random seed */
+ Variables variables; /* client variables */
+} RetryState;
+
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
/* Various random sequences are initialized from this one. */
static RandomState base_random_sequence;
@@ -446,6 +553,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -494,8 +630,21 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ RetryState retry_state;
+ uint32 retries; /* how many times have we already retried the
+ * current transaction after a serialization or
+ * a deadlock error? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -590,6 +739,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -602,6 +754,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -616,6 +770,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool debug_errors = false; /* print debug messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -753,15 +909,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-per-command report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --debug-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1307,6 +1466,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1315,22 +1478,49 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 retries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ stats->retries += retries;
+ if (retries > 0)
+ stats->retried++;
+
+ switch (estatus)
{
- addToSimpleStats(&stats->latency, lat);
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2865,6 +3055,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2872,6 +3065,19 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ const Command *command = sql_script[st->use_file].commands[st->command];
+
+ Assert(command->type == SQL_COMMAND);
+ pg_log_error("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2979,6 +3185,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -3021,6 +3254,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3035,6 +3269,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3059,6 +3294,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3076,6 +3312,20 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (debug_errors)
+ commandError(st, PQerrorMessage(st->con));
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_ABORTED)
+ PQpipelineSync(st->con);
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3154,6 +3404,160 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+/*
+ * Clear the variables in the array. The array itself is not freed.
+ */
+static void
+clearVariables(Variables *variables)
+{
+ Variable *vars,
+ *var;
+ int nvars;
+
+ if (!variables)
+ return; /* nothing to do here */
+
+ vars = variables->vars;
+ nvars = variables->nvars;
+ for (var = vars; var - vars < nvars; ++var)
+ {
+ pg_free(var->name);
+ pg_free(var->svalue);
+ }
+
+ variables->nvars = 0;
+}
+
+/*
+ * Make a deep copy of variables array.
+ * Before copying the function frees the string fields of the destination
+ * variables and if necessary enlarges their array.
+ */
+static void
+copyVariables(Variables *dest, const Variables *source)
+{
+ Variable *dest_var;
+ const Variable *source_var;
+
+ if (!dest || !source || dest == source)
+ return; /* nothing to do here */
+
+ /*
+ * Clear the original variables and make sure that we have enough space for
+ * the new variables.
+ */
+ clearVariables(dest);
+ enlargeVariables(dest, source->nvars);
+
+ /* Make a deep copy of variables array */
+ for (source_var = source->vars, dest_var = dest->vars;
+ source_var - source->vars < source->nvars;
+ ++source_var, ++dest_var)
+ {
+ dest_var->name = pg_strdup(source_var->name);
+ dest_var->svalue = (source_var->svalue == NULL) ?
+ NULL : pg_strdup(source_var->svalue);
+ dest_var->value = source_var->value;
+ }
+ dest->nvars = source->nvars;
+ dest->vars_sorted = source->vars_sorted;
+}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries
+ * or time is over.
+ */
+ if ((max_tries && st->retries + 1 >= max_tries) || timer_exceeded)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Set in_tx_block to true if we are in a (failed) transaction block and false
+ * otherwise.
+ * Returns false on failure (broken connection or internal error).
+ */
+static bool
+checkTransactionStatus(PGconn *con, bool *in_tx_block)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ *in_tx_block = false;
+ break;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ *in_tx_block = true;
+ break;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ { /* there's something wrong */
+ pg_log_error("perhaps the backend died while processing");
+ return false;
+ }
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * If the latency limit is used, return a percentage of the current transaction
+ * latency from the latency limit. Otherwise return zero.
+ */
+static double
+getLatencyUsed(CState *st, pg_time_usec_t *now)
+{
+ if (!latency_limit)
+ return 0.0;
+
+ pg_time_now_lazy(now);
+ return (100.0 * (*now - st->txn_scheduled) / latency_limit);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3183,6 +3587,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
for (;;)
{
Command *command;
+ PGresult *res;
+ bool in_tx_block;
switch (st->state)
{
@@ -3191,6 +3597,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->retries = 0;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3227,6 +3637,14 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember its
+ * parameters: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->retry_state.random_state = st->cs_func_rs;
+ copyVariables(&st->retry_state.variables, &st->variables);
+
/* record transaction start time */
st->txn_begin = now;
@@ -3378,6 +3796,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3516,10 +3936,55 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ do
+ {
+ res = PQgetResult(st->con);
+ if (res)
+ PQclear(res);
+ } while (res);
+ /* Check if we can retry the error. */
+ st->state =
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ break;
+
/*
* Wait until sleep is done. This state is entered after a
* \sleep metacommand. The behavior is similar to
@@ -3562,6 +4027,132 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
CSTATE_START_COMMAND : CSTATE_SKIP_COMMAND;
break;
+ /*
+ * Clean up after an error.
+ */
+ case CSTATE_ERROR:
+
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
+
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (in_tx_block)
+ {
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else
+ {
+ /* Check if we can retry the error. */
+ st->state = doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ break;
+
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the retry. */
+ st->retries++;
+ if (report_per_command)
+ command->retries++;
+
+ /*
+ * Inform that the transaction will be retried after the error.
+ */
+ if (debug_errors)
+ {
+ fprintf(stderr,
+ "client %d repeats the transaction after the error (try %d",
+ st->id, st->retries);
+ if (max_tries)
+ fprintf(stderr, "/%d", max_tries);
+ if (latency_limit)
+ fprintf(stderr,
+ ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ fprintf(stderr, ")\n");
+ }
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->retry_state.random_state;
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Complete the failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ if (report_per_command)
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (debug_errors)
+ {
+ fprintf(stderr,
+ "client %d ends the failed transaction (try %d",
+ st->id, st->retries + 1);
+ if (max_tries)
+ fprintf(stderr, "/%d", max_tries);
+ if (latency_limit)
+ fprintf(stderr,
+ ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ else if (timer_exceeded)
+ fprintf(stderr,", the duration time is exceeded");
+ fprintf(stderr, ")\n");
+ }
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction except for a random state.
+ */
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
/*
* End of transaction (end of script, really).
*/
@@ -3576,6 +4167,29 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
Assert(conditional_stack_empty(st->cstack));
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (in_tx_block)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (is_connect)
{
finishCon(st);
@@ -3807,6 +4421,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization_failure";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock_failure";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal(stderr, "unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3851,6 +4502,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3861,6 +4520,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3868,22 +4531,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->retries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->retries);
fputc('\n', logfile);
}
}
@@ -3892,7 +4559,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3900,10 +4568,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3912,20 +4580,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->retries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3935,7 +4595,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->retries);
}
@@ -4782,6 +5443,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5450,7 +6113,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5477,23 +6142,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5506,8 +6178,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5516,6 +6188,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5575,9 +6253,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5594,35 +6273,65 @@ printResults(StatsData *total,
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
+ }
+
+ if (failures > 0)
+ {
+ printf("number of transactions failed: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ if (total->deadlock_failures)
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
}
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("number of total retries: " INT64_FORMAT "\n", total->retries);
+ }
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f %%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5648,7 +6357,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5667,6 +6376,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5676,25 +6388,60 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
+
+ if (failures > 0)
+ {
+ printf(" - number of transactions failed: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ if (total->deadlock_failures)
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - number of total retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5702,10 +6449,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5786,7 +6542,7 @@ main(int argc, char **argv)
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
- {"report-latencies", no_argument, NULL, 'r'},
+ {"report-per-command", no_argument, NULL, 'r'},
{"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
@@ -5808,6 +6564,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"debug-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -5845,6 +6604,7 @@ main(int argc, char **argv)
PGconn *con;
char *env;
+ bool retry = false; /* retry transactions with errors or not */
int exit_code = 0;
@@ -6176,6 +6936,36 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+
+ /*
+ * Always retry transactions with errors if this option is
+ * used. But if its value is 0, use the option
+ * --latency-limit to limit the number of tries.
+ */
+ retry = true;
+
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* debug-errors */
+ benchmarking_option_set = true;
+ debug_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6357,6 +7147,20 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (retry && !(latency_limit || duration > 0))
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ else if (!retry)
+ {
+ /* By default transactions with errors are not retried */
+ max_tries = 1;
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6565,6 +7369,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6713,7 +7521,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6802,7 +7611,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 923203ea51..4488cf926c 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,11 @@ use Config;
# start a pgbench specific server
my $node = get_new_node('main');
-$node->init;
+
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
+
$node->start;
# invoke pgbench, with parameters:
@@ -159,7 +163,8 @@ pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1225,6 +1230,214 @@ pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^\d \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+pgbench(
+ "-n -c 2 -t 1 -d --debug-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of transactions failed)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{number of total retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --debug-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of transactions failed)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{number of total retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index 9023fac52d..5bf9ab1f0e 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -179,6 +179,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index a562e28846..c304014f51 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index c64c655775..9c495072aa 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
v12-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.6K) download
From fd3e1e0203f8b2b0c500d9f1f9905d315e97b6f6 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v12 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 169 ++++++++++++++++++++++++--------------
1 file changed, 106 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e61055b6b7..8acda86cad 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -287,6 +287,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -304,6 +310,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -460,9 +484,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1418,39 +1440,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1582,21 +1604,43 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array. It is assumed that the sum of the number of current variables and the
+ * number of needed variables is less than or equal to (INT_MAX -
+ * VARIABLES_ALLOC_MARGIN).
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ /*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1608,23 +1652,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1633,12 +1671,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1656,12 +1695,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1676,12 +1715,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1740,7 +1780,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1761,7 +1801,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1776,12 +1816,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2647,7 +2688,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2717,7 +2758,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2748,7 +2789,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2809,7 +2850,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2861,7 +2902,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2872,7 +2913,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2919,7 +2960,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -3012,7 +3053,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3073,14 +3114,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3612,7 +3653,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3633,7 +3674,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3703,7 +3744,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3711,7 +3752,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -5993,7 +6034,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6333,19 +6374,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6380,11 +6421,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6393,30 +6434,32 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed =
((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) |
(((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) << 32);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-06-22 18:03 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-23 00:31 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2021-06-22 18:03 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Yugo-san,
Thanks a lot for continuing this work started by Marina!
I'm planning to review it for the July CF. I've just added an entry there:
https://commitfest.postgresql.org/33/3194/
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-22 18:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2021-06-23 00:31 ` Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 0 replies; 82+ messages in thread
From: Yugo NAGATA @ 2021-06-23 00:31 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Fabien,
On Tue, 22 Jun 2021 20:03:58 +0200 (CEST)
Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> Hello Yugo-san,
>
> Thanks a lot for continuing this work started by Marina!
>
> I'm planning to review it for the July CF. I've just added an entry there:
>
> https://commitfest.postgresql.org/33/3194/
Thanks!
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-06-23 08:38 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-25 11:01 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2021-06-23 08:38 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Yugo-san:
# About v12.1
This is a refactoring patch, which creates a separate structure for
holding variables. This will become handy in the next patch. There is also
a benefit from a software engineering point of view, so it has merit on
its own.
## Compilation
Patch applies cleanly, compiles, global & local checks pass.
## About the code
Fine.
I'm wondering whether we could use "vars" instead of "variables" as a
struct field name and function parameter name, so that is is shorter and
more distinct from the type name "Variables". What do you think?
## About comments
Remove the comment on enlargeVariables about "It is assumed …" the issue
of trying MAXINT vars is more than remote and is not worth mentioning. In
the same function, remove the comments about MARGIN, it is already on the
macro declaration, once is enough.
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-23 08:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2021-06-25 11:01 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-06-25 11:01 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
On Wed, 23 Jun 2021 10:38:43 +0200 (CEST)
Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> Hello Yugo-san:
>
> # About v12.1
>
> This is a refactoring patch, which creates a separate structure for
> holding variables. This will become handy in the next patch. There is also
> a benefit from a software engineering point of view, so it has merit on
> its own.
> ## Compilation
>
> Patch applies cleanly, compiles, global & local checks pass.
>
> ## About the code
>
> Fine.
>
> I'm wondering whether we could use "vars" instead of "variables" as a
> struct field name and function parameter name, so that is is shorter and
> more distinct from the type name "Variables". What do you think?
The struct "Variables" has a field named "vars" which is an array of
"Variable" type. I guess this is a reason why "variables" is used instead
of "vars" as a name of "Variables" type variable so that we could know
a variable's type is Variable or Variables. Also, in order to refer to
the field, we would use
vars->vars[vars->nvars]
and there are nested "vars". Could this make a codereader confused?
> ## About comments
>
> Remove the comment on enlargeVariables about "It is assumed $B!D(B" the issue
> of trying MAXINT vars is more than remote and is not worth mentioning. In
> the same function, remove the comments about MARGIN, it is already on the
> macro declaration, once is enough.
Sure. I'll remove them.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-23 08:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-25 11:01 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-06-26 08:20 ` Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 0 replies; 82+ messages in thread
From: Fabien COELHO @ 2021-06-26 08:20 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Yugo-san,
>> I'm wondering whether we could use "vars" instead of "variables" as a
>> struct field name and function parameter name, so that is is shorter and
>> more distinct from the type name "Variables". What do you think?
>
> The struct "Variables" has a field named "vars" which is an array of
> "Variable" type. I guess this is a reason why "variables" is used instead
> of "vars" as a name of "Variables" type variable so that we could know
> a variable's type is Variable or Variables. Also, in order to refer to
> the field, we would use
>
> vars->vars[vars->nvars]
>
> and there are nested "vars". Could this make a codereader confused?
Hmmm$B!D(J Probably. Let's keep "variables" then.
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-06-26 10:15 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2021-06-26 10:15 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Yugo-san,
# About v12.2
## Compilation
Patch seems to apply cleanly with "git apply", but does not compile on my
host: "undefined reference to `conditional_stack_reset'".
However it works better when using the "patch". I'm wondering why git
apply fails silently…
When compiling there are warnings about "pg_log_fatal", which does not
expect a FILE* on pgbench.c:4453. Remove the "stderr" argument.
Global and local checks ok.
> number of transactions failed: 324 (3.240%)
> ...
> number of transactions retried: 5629 (56.290%)
> number of total retries: 103299
I'd suggest: "number of failed transactions". "total number of retries" or
just "number of retries"?
## Feature
The overall code structure changes to implements the feature seems
reasonable to me, as we are at the 12th iteration of the patch.
Comments below are somehow about details and asking questions
about choices, and commenting…
## Documentation
There is a lot of documentation, which is good. I'll review these
separatly. It looks good, but having a native English speaker/writer
would really help!
Some output examples do not correspond to actual output for
the current version. In particular, there is always one TPS figure
given now, instead of the confusing two shown before.
## Comments
transactinos -> transactions.
## Code
By default max_tries = 0. Should not the initialization be 1,
as the documentation argues that it is the default?
Counter comments, missing + in the formula on the skipped line.
Given that we manage errors, ISTM that we should not necessarily
stop on other not retried errors, but rather count/report them and
possibly proceed. Eg with something like:
-- server side random fail
DO LANGUAGE plpgsql $$
BEGIN
IF RANDOM() < 0.1 THEN
RAISE EXCEPTION 'unlucky!';
END IF;
END;
$$;
Or:
-- client side random fail
BEGIN;
\if random(1, 10) <= 1
SELECT 1 +;
\else
SELECT 2;
\endif
COMMIT;
We could count the fail, rollback if necessary, and go on. What do you think?
Maybe such behavior would deserve an option.
--report-latencies -> --report-per-command: should we keep supporting
the previous option?
--failures-detailed: if we bother to run with handling failures, should
it always be on?
--debug-errors: I'm not sure we should want a special debug mode for that,
I'd consider integrating it with the standard debug, or just for development.
Also, should it use pg_log_debug?
doRetry: I'd separate the 3 no retries options instead of mixing max_tries and
timer_exceeeded, for clarity.
Tries vs retries: I'm at odds with having tries & retries and + 1 here
and there to handle that, which is a little bit confusing. I'm wondering whether
we could only count "tries" and adjust to report what we want later?
advanceConnectionState: ISTM that ERROR should logically be before others which
lead to it.
Variables management: it looks expensive, with copying and freeing variable arrays.
I'm wondering whether we should think of something more clever. Well, that would be
for some other patch.
"Accumulate the retries" -> "Count (re)tries"?
Currently, ISTM that the retry on error mode is implicitely always on.
Do we want that? I'd say yes, but maybe people could disagree.
## Tests
There are tests, good!
I'm wondering whether something simpler could be devised to trigger
serialization or deadlock errors, eg with a SEQUENCE and an \if.
See the attached files for generating deadlocks reliably (start with 2 clients).
What do you think? The PL/pgSQL minimal, it is really client-code
oriented.
Given that deadlocks are detected about every seconds, the test runs
would take some time. Let it be for now.
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2021-06-30 15:02 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 15:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 2 replies; 82+ messages in thread
From: Yugo NAGATA @ 2021-06-30 15:02 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Fabien,
On Sat, 26 Jun 2021 12:15:38 +0200 (CEST)
Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> Hello Yugo-san,
>
> # About v12.2
>
> ## Compilation
>
> Patch seems to apply cleanly with "git apply", but does not compile on my
> host: "undefined reference to `conditional_stack_reset'".
>
> However it works better when using the "patch". I'm wondering why git
> apply fails silently$B!D(B
Hmm, I don't know why your compiling fails... I can apply and complile
successfully using git.
> When compiling there are warnings about "pg_log_fatal", which does not
> expect a FILE* on pgbench.c:4453. Remove the "stderr" argument.
Ok.
> Global and local checks ok.
>
> > number of transactions failed: 324 (3.240%)
> > ...
> > number of transactions retried: 5629 (56.290%)
> > number of total retries: 103299
>
> I'd suggest: "number of failed transactions". "total number of retries" or
> just "number of retries"?
Ok. I fixed to use "number of failed transactions" and "total number of retries".
> ## Feature
>
> The overall code structure changes to implements the feature seems
> reasonable to me, as we are at the 12th iteration of the patch.
>
> Comments below are somehow about details and asking questions
> about choices, and commenting$B!D(B
>
> ## Documentation
>
> There is a lot of documentation, which is good. I'll review these
> separatly. It looks good, but having a native English speaker/writer
> would really help!
>
> Some output examples do not correspond to actual output for
> the current version. In particular, there is always one TPS figure
> given now, instead of the confusing two shown before.
Fixed.
> ## Comments
>
> transactinos -> transactions.
Fixed.
> ## Code
>
> By default max_tries = 0. Should not the initialization be 1,
> as the documentation argues that it is the default?
Ok. I fixed the default value to 1.
> Counter comments, missing + in the formula on the skipped line.
Fixed.
> Given that we manage errors, ISTM that we should not necessarily
> stop on other not retried errors, but rather count/report them and
> possibly proceed. Eg with something like:
>
> -- server side random fail
> DO LANGUAGE plpgsql $$
> BEGIN
> IF RANDOM() < 0.1 THEN
> RAISE EXCEPTION 'unlucky!';
> END IF;
> END;
> $$;
>
> Or:
>
> -- client side random fail
> BEGIN;
> \if random(1, 10) <= 1
> SELECT 1 +;
> \else
> SELECT 2;
> \endif
> COMMIT;
>
> We could count the fail, rollback if necessary, and go on. What do you think?
> Maybe such behavior would deserve an option.
This feature to count failures that could occur at runtime seems nice. However,
as discussed in [1], I think it is better to focus to only failures that can be
retried in this patch, and introduce the feature to handle other failures in a
separate patch.
[1] https://www.postgresql.org/message-id/alpine.DEB.2.21.1809121519590.13887%40lancre
> --report-latencies -> --report-per-command: should we keep supporting
> the previous option?
Ok. Although now the option is not only for latencies, considering users who
are using the existing option, I'm fine with this. I got back this to the
previous name.
> --failures-detailed: if we bother to run with handling failures, should
> it always be on?
If we print other failures that cannot be retried in future, it could a lot
of lines and might make some users who don't need details of failures annoyed.
Moreover, some users would always need information of detailed failures in log,
and others would need only total numbers of failures.
Currently we handle only serialization and deadlock failures, so the number of
lines printed and the number of columns of logging is not large even under the
failures-detail, but if we have a chance to handle other failures in future,
ISTM adding this option makes sense considering users who would like simple
outputs.
> --debug-errors: I'm not sure we should want a special debug mode for that,
> I'd consider integrating it with the standard debug, or just for development.
I think --debug is a debug option for telling users the pgbench's internal
behaviors, that is, which client is doing what. On other hand, --debug-errors
is for telling users what error caused a retry or a failure in detail. For
users who are not interested in pgbench's internal behavior (sending a command,
receiving a result, ... ) but interested in actual errors raised during running
script, this option seems useful.
> Also, should it use pg_log_debug?
If we use pg_log_debug, the message is printed only under --debug.
Therefore, I fixed to use pg_log_info instead of pg_log_error or fprintf.
> doRetry: I'd separate the 3 no retries options instead of mixing max_tries and
> timer_exceeeded, for clarity.
Ok. I fixed to separate them.
> Tries vs retries: I'm at odds with having tries & retries and + 1 here
> and there to handle that, which is a little bit confusing. I'm wondering whether
> we could only count "tries" and adjust to report what we want later?
I fixed to use "tries" instead of "retries" in CState. However, we still use
"retries" in StatsData and Command because the number of retries is printed
in the final result. Is it less confusing than the previous?
> advanceConnectionState: ISTM that ERROR should logically be before others which
> lead to it.
Sorry, I couldn't understand your suggestion. Is this about the order of case
statements or pg_log_error?
> Variables management: it looks expensive, with copying and freeing variable arrays.
> I'm wondering whether we should think of something more clever. Well, that would be
> for some other patch.
Well.., indeed there may be more efficient way. For example, instead of clearing all
vars in dest, it might be possible to copy or clear only the difference part between
dest and source and remaining unchanged part in dest. Anyway, I think this work should
be done in other patch.
> "Accumulate the retries" -> "Count (re)tries"?
Fixed.
> Currently, ISTM that the retry on error mode is implicitely always on.
> Do we want that? I'd say yes, but maybe people could disagree.
The default values of max-tries is 1, so the retry on error is off.
Failed transactions are retried only when the user wants it and
specifies a valid value to max-treis.
> ## Tests
>
> There are tests, good!
>
> I'm wondering whether something simpler could be devised to trigger
> serialization or deadlock errors, eg with a SEQUENCE and an \if.
>
> See the attached files for generating deadlocks reliably (start with 2 clients).
> What do you think? The PL/pgSQL minimal, it is really client-code
> oriented.
>
> Given that deadlocks are detected about every seconds, the test runs
> would take some time. Let it be for now.
Sorry, but I cannot find the attached file. I don't have a good idea
for a simpler test for now, but I can fix the test based on your idea
after getting the file.
I attached the patch updated according with your suggestion.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v13-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 81.5K) download
From 034db9dd6a44f76794a2f62f135b7543bddd23b2 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Fri, 28 May 2021 10:48:57 +0900
Subject: [PATCH v13 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-latencies, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --debug-errors.
---
doc/src/sgml/ref/pgbench.sgml | 396 +++++++-
src/bin/pgbench/pgbench.c | 960 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 217 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1487 insertions(+), 114 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 0c60077e1f..5290ff0945 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -58,6 +58,7 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
@@ -65,11 +66,14 @@ tps = 896.967014 (without initial connection time)
</screen>
The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ settings. The seventh line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
The last line reports the number of transactions per second.
</para>
@@ -528,6 +532,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information about retrying such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -594,11 +609,14 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
@@ -608,9 +626,12 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<term><option>--report-latencies</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -738,6 +759,26 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -748,6 +789,38 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; more over, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information about
+ retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--debug-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -943,8 +1016,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts">
+ <title id="transactions-and-scripts-title">What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1017,6 +1090,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, he will be aborted.
+ </para>
</note>
<para>
@@ -2207,7 +2285,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2228,6 +2306,17 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization_failure</literal> or
+ <literal>deadlock_failure</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information).
</para>
<para>
@@ -2256,6 +2345,24 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2271,7 +2378,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2285,7 +2392,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2293,21 +2409,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2319,13 +2439,44 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
+
+ <para>
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2339,27 +2490,63 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 10.870 ms
latency stddev = 7.341 ms
initial connection time = 30.954 ms
tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+statement latencies in milliseconds and failures:
+ 0.002 0 \set aid random(1, 100000 * :scale)
+ 0.005 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.001 0 \set delta random(-5000, 5000)
+ 0.326 0 BEGIN;
+ 0.603 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.454 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 5.528 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 7.335 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.371 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.212 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+number of transactions per client: 1000
+number of transactions actually processed: 9676/10000
+number of failed transactions: 324 (3.240%)
+number of serialization failures: 324 (3.240%)
+number of transactions retried: 5629 (56.290%)
+total number of retries: 103299
+maximum number of tries: 100
+number of transactions above the 100.0 ms latency limit: 21/9676 (0.217 %)
+latency average = 16.138 ms
+latency stddev = 21.017 ms
+tps = 413.686560 (without initial connection time)
+statement latencies in milliseconds, failures and retries:
+ 0.002 0 0 \set aid random(1, 100000 * :scale)
+ 0.000 0 0 \set bid random(1, 1 * :scale)
+ 0.000 0 0 \set tid random(1, 10 * :scale)
+ 0.000 0 0 \set delta random(-5000, 5000)
+ 0.121 0 0 BEGIN;
+ 0.290 0 2 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.221 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 0.266 212 72127 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.222 112 31170 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.178 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.210 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2373,6 +2560,135 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries">
+ <title id="failures-and-retries-title">Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there're three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the current transaction is rolled back which also
+ includes setting the client variables as they were before the run of this
+ transaction (it is assumed that one transaction script contains only one
+ transaction; see <xref linkend="transactions-and-scripts"
+ endterm="transactions-and-scripts-title"/> for more information).
+ Transactions with serialization or deadlock errors are repeated after
+ rollbacks until they complete successfully or reach the maximum number of
+ tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of tries (specified by the <option>--latency-limit</option> option). If
+ the last transaction run fails, this transaction will be reported as failed.
+ </para>
+
+ <note>
+ <para>
+ Although without the <option>--max-tries</option> option the transaction
+ will never be retried after an error, use an unlimited number of tries
+ (<literal>--max-tries=0</literal>) and the <option>--latency-limit</option>
+ option or the <option>--time</option> to limit only the maximum time of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency for failed
+ transactions and commands is not computed separately.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions if it is non-zero.
+ If the total number of retried transactions is non-zero, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--debug-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3629caba42..11482032ba 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -74,6 +74,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -273,9 +275,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 1;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -360,9 +387,65 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them) +
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactions) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -375,6 +458,30 @@ typedef struct RandomState
unsigned short xseed[3];
} RandomState;
+/*
+ * Data structure for repeating a transaction from the beginnning with the same
+ * parameters.
+ */
+typedef struct
+{
+ RandomState random_state; /* random seed */
+ Variables variables; /* client variables */
+} RetryState;
+
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
/* Various random sequences are initialized from this one. */
static RandomState base_random_sequence;
@@ -446,6 +553,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -494,8 +630,20 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ RetryState retry_state;
+ uint32 tries; /* how many times have we already tried the
+ * current transaction? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -590,6 +738,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -602,6 +753,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -616,6 +769,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool debug_errors = false; /* print debug messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -753,15 +908,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-latencies report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --debug-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1307,6 +1465,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1315,22 +1477,51 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 tries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ if (tries > 1)
+ {
+ stats->retries += (tries - 1);
+ stats->retried++;
+ }
+
+ switch (estatus)
{
- addToSimpleStats(&stats->latency, lat);
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2861,6 +3052,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2868,6 +3062,19 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ const Command *command = sql_script[st->use_file].commands[st->command];
+
+ Assert(command->type == SQL_COMMAND);
+ pg_log_info("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2975,6 +3182,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -3017,6 +3251,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3031,6 +3266,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3055,6 +3291,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3072,6 +3309,20 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (debug_errors)
+ commandError(st, PQerrorMessage(st->con));
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_ABORTED)
+ PQpipelineSync(st->con);
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3150,6 +3401,171 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+/*
+ * Clear the variables in the array. The array itself is not freed.
+ */
+static void
+clearVariables(Variables *variables)
+{
+ Variable *vars,
+ *var;
+ int nvars;
+
+ if (!variables)
+ return; /* nothing to do here */
+
+ vars = variables->vars;
+ nvars = variables->nvars;
+ for (var = vars; var - vars < nvars; ++var)
+ {
+ pg_free(var->name);
+ pg_free(var->svalue);
+ }
+
+ variables->nvars = 0;
+}
+
+/*
+ * Make a deep copy of variables array.
+ * Before copying the function frees the string fields of the destination
+ * variables and if necessary enlarges their array.
+ */
+static void
+copyVariables(Variables *dest, const Variables *source)
+{
+ Variable *dest_var;
+ const Variable *source_var;
+
+ if (!dest || !source || dest == source)
+ return; /* nothing to do here */
+
+ /*
+ * Clear the original variables and make sure that we have enough space for
+ * the new variables.
+ */
+ clearVariables(dest);
+ enlargeVariables(dest, source->nvars);
+
+ /* Make a deep copy of variables array */
+ for (source_var = source->vars, dest_var = dest->vars;
+ source_var - source->vars < source->nvars;
+ ++source_var, ++dest_var)
+ {
+ dest_var->name = pg_strdup(source_var->name);
+ dest_var->svalue = (source_var->svalue == NULL) ?
+ NULL : pg_strdup(source_var->svalue);
+ dest_var->value = source_var->value;
+ }
+ dest->nvars = source->nvars;
+ dest->vars_sorted = source->vars_sorted;
+}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries.
+ */
+ if (max_tries && st->tries >= max_tries)
+ return false;
+
+ /*
+ * We cannot retry the error if the benchmark duration is over.
+ */
+ if (timer_exceeded)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries
+ * or time is over.
+ */
+ if ((max_tries && st->tries >= max_tries) || timer_exceeded)
+ return false;
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Set in_tx_block to true if we are in a (failed) transaction block and false
+ * otherwise.
+ * Returns false on failure (broken connection or internal error).
+ */
+static bool
+checkTransactionStatus(PGconn *con, bool *in_tx_block)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ *in_tx_block = false;
+ break;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ *in_tx_block = true;
+ break;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ { /* there's something wrong */
+ pg_log_error("perhaps the backend died while processing");
+ return false;
+ }
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * If the latency limit is used, return a percentage of the current transaction
+ * latency from the latency limit. Otherwise return zero.
+ */
+static double
+getLatencyUsed(CState *st, pg_time_usec_t *now)
+{
+ if (!latency_limit)
+ return 0.0;
+
+ pg_time_now_lazy(now);
+ return (100.0 * (*now - st->txn_scheduled) / latency_limit);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3179,6 +3595,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
for (;;)
{
Command *command;
+ PGresult *res;
+ bool in_tx_block;
switch (st->state)
{
@@ -3187,6 +3605,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->tries = 1;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3223,6 +3645,14 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember its
+ * parameters: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->retry_state.random_state = st->cs_func_rs;
+ copyVariables(&st->retry_state.variables, &st->variables);
+
/* record transaction start time */
st->txn_begin = now;
@@ -3374,6 +3804,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3512,10 +3944,55 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ do
+ {
+ res = PQgetResult(st->con);
+ if (res)
+ PQclear(res);
+ } while (res);
+ /* Check if we can retry the error. */
+ st->state =
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ break;
+
/*
* Wait until sleep is done. This state is entered after a
* \sleep metacommand. The behavior is similar to
@@ -3558,6 +4035,144 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
CSTATE_START_COMMAND : CSTATE_SKIP_COMMAND;
break;
+ /*
+ * Clean up after an error.
+ */
+ case CSTATE_ERROR:
+
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
+
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (in_tx_block)
+ {
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else
+ {
+ /* Check if we can retry the error. */
+ st->state = doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ break;
+
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /*
+ * Inform that the transaction will be retried after the error.
+ */
+ if (debug_errors)
+ {
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d repeats the transaction after the error (try %d",
+ st->id, st->tries);
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+ if (latency_limit)
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+ }
+
+ /* Count tries and retries */
+ st->tries++;
+ if (report_per_command)
+ command->retries++;
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->retry_state.random_state;
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Complete the failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ if (report_per_command)
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (debug_errors)
+ {
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d ends the failed transaction (try %d",
+ st->id, st->tries);
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+ if (latency_limit)
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ else if (timer_exceeded)
+ appendPQExpBuffer(&buf, ", the duration time is exceeded");
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+ }
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction except for a random state.
+ */
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
/*
* End of transaction (end of script, really).
*/
@@ -3572,6 +4187,29 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
Assert(conditional_stack_empty(st->cstack));
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (in_tx_block)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (is_connect)
{
finishCon(st);
@@ -3803,6 +4441,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization_failure";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock_failure";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3847,6 +4522,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3857,6 +4540,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3864,22 +4551,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->tries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->tries - 1);
fputc('\n', logfile);
}
}
@@ -3888,7 +4579,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3896,10 +4588,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3908,20 +4600,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->tries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3931,7 +4615,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->tries);
}
@@ -4778,6 +5463,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5446,7 +6133,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5473,23 +6162,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5502,8 +6198,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5512,6 +6208,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5571,9 +6273,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5590,35 +6293,65 @@ printResults(StatsData *total,
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
}
+ if (failures > 0)
+ {
+ printf("number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ if (total->deadlock_failures)
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("total number of retries: " INT64_FORMAT "\n", total->retries);
+ }
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f %%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5644,7 +6377,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5663,6 +6396,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5672,25 +6408,60 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ if (failures > 0)
+ {
+ printf(" - number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ if (total->deadlock_failures)
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - total number of retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5698,10 +6469,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5804,6 +6584,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"debug-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -6172,6 +6955,28 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* debug-errors */
+ benchmarking_option_set = true;
+ debug_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6353,6 +7158,15 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (!latency_limit && duration <= 0)
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6561,6 +7375,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6709,7 +7527,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6798,7 +7617,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3aa9d5d753..afaa2b9d0b 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,11 @@ use Config;
# start a pgbench specific server
my $node = get_new_node('main');
-$node->init;
+
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
+
$node->start;
# invoke pgbench, with parameters:
@@ -159,7 +163,8 @@ pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1239,6 +1244,214 @@ pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^0 \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+pgbench(
+ "-n -c 2 -t 1 -d --debug-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of transactions failed)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{number of total retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --debug-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of transactions failed)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{number of total retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index 346a2667fc..56f7226c8e 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -178,6 +178,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index a562e28846..c304014f51 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index c64c655775..9c495072aa 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
v13-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.3K) download
From 0c506d7de82ffd82570aa751e641faa42be8dd84 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v13 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 163 +++++++++++++++++++++++---------------
1 file changed, 100 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 4aeccd93af..3629caba42 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -287,6 +287,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -304,6 +310,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -460,9 +484,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1418,39 +1440,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1582,21 +1604,37 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array.
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1608,23 +1646,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1633,12 +1665,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1656,12 +1689,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1676,12 +1709,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1740,7 +1774,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1761,7 +1795,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1776,12 +1810,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2649,7 +2684,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2719,7 +2754,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2750,7 +2785,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2811,7 +2846,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2863,7 +2898,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2874,7 +2909,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2921,7 +2956,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -3014,7 +3049,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3075,14 +3110,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3614,7 +3649,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3635,7 +3670,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3705,7 +3740,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3713,7 +3748,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -5995,7 +6030,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6335,19 +6370,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6382,11 +6417,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6395,30 +6430,32 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed =
((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) |
(((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) << 32);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-06-30 15:33 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-07 23:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2021-06-30 15:33 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Yugo-san,
Thanks for the update!
>> Patch seems to apply cleanly with "git apply", but does not compile on my
>> host: "undefined reference to `conditional_stack_reset'".
>>
>> However it works better when using the "patch". I'm wondering why git
>> apply fails silently$B!D(J
>
> Hmm, I don't know why your compiling fails... I can apply and complile
> successfully using git.
Hmmm. Strange!
>> Given that we manage errors, ISTM that we should not necessarily stop
>> on other not retried errors, but rather count/report them and possibly
>> proceed. Eg with something like: [...] We could count the fail,
>> rollback if necessary, and go on. What do you think? Maybe such
>> behavior would deserve an option.
>
> This feature to count failures that could occur at runtime seems nice. However,
> as discussed in [1], I think it is better to focus to only failures that can be
> retried in this patch, and introduce the feature to handle other failures in a
> separate patch.
Ok.
>> --report-latencies -> --report-per-command: should we keep supporting
>> the previous option?
>
> Ok. Although now the option is not only for latencies, considering users who
> are using the existing option, I'm fine with this. I got back this to the
> previous name.
Hmmm. I liked the new name! My point was whether we need to support the
old one as well for compatibility, or whether we should not bother. I'm
still wondering. As I think that the new name is better, I'd suggest to
keep it.
>> --failures-detailed: if we bother to run with handling failures, should
>> it always be on?
>
> If we print other failures that cannot be retried in future, it could a lot
> of lines and might make some users who don't need details of failures annoyed.
> Moreover, some users would always need information of detailed failures in log,
> and others would need only total numbers of failures.
Ok.
> Currently we handle only serialization and deadlock failures, so the number of
> lines printed and the number of columns of logging is not large even under the
> failures-detail, but if we have a chance to handle other failures in future,
> ISTM adding this option makes sense considering users who would like simple
> outputs.
Hmmm. What kind of failures could be managed with retries? I guess that on
a connection failure we can try to reconnect, but otherwise it is less
clear that other failures make sense to retry.
>> --debug-errors: I'm not sure we should want a special debug mode for that,
>> I'd consider integrating it with the standard debug, or just for development.
>
> I think --debug is a debug option for telling users the pgbench's internal
> behaviors, that is, which client is doing what. On other hand, --debug-errors
> is for telling users what error caused a retry or a failure in detail. For
> users who are not interested in pgbench's internal behavior (sending a command,
> receiving a result, ... ) but interested in actual errors raised during running
> script, this option seems useful.
Ok. The this is not really about debug per se, but a verbosity setting?
Maybe --verbose-errors would make more sense? I'm unsure. I'll think about
it.
>> Also, should it use pg_log_debug?
>
> If we use pg_log_debug, the message is printed only under --debug.
> Therefore, I fixed to use pg_log_info instead of pg_log_error or fprintf.
Ok, pg_log_info seems right.
>> Tries vs retries: I'm at odds with having tries & retries and + 1 here
>> and there to handle that, which is a little bit confusing. I'm wondering whether
>> we could only count "tries" and adjust to report what we want later?
>
> I fixed to use "tries" instead of "retries" in CState. However, we still use
> "retries" in StatsData and Command because the number of retries is printed
> in the final result. Is it less confusing than the previous?
I'm going to think about it.
>> advanceConnectionState: ISTM that ERROR should logically be before others which
>> lead to it.
>
> Sorry, I couldn't understand your suggestion. Is this about the order of case
> statements or pg_log_error?
My sentence got mixed up. My point was about the case order, so that they
are put in a more logical order when reading all the cases.
>> Currently, ISTM that the retry on error mode is implicitely always on.
>> Do we want that? I'd say yes, but maybe people could disagree.
>
> The default values of max-tries is 1, so the retry on error is off.
> Failed transactions are retried only when the user wants it and
> specifies a valid value to max-treis.
Ok. My point is that we do not stop on such errors, whereas before ISTM
that we would have stopped, so somehow the default behavior has changed
and the previous behavior cannot be reinstated with an option. Maybe that
is not bad, but this is a behavioral change which needs to be documented
and argumented.
>> See the attached files for generating deadlocks reliably (start with 2
>> clients). What do you think? The PL/pgSQL minimal, it is really
>> client-code oriented.
>
> Sorry, but I cannot find the attached file.
Sorry. Attached to this mail. The serialization stuff does not seem to
work as well as the deadlock one. Run with 2 clients.
--
Fabien.
Attachments:
deadlock_prep.sql (application/x-sql, 686B) download
deadlock.sql (application/x-sql, 814B) download
serializable.sql (application/x-sql, 691B) download
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 15:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2021-07-07 23:35 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 12:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-07 23:35 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Thomas Munro <thomas.munro@gmail.com>; Marina Polyakova <m.polyakova@postgrespro.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; pgsql-hackers; Teodor Sigaev <teodor@sigaev.ru>
Hello Fabien,
I attached the updated patch (v14)!
On Wed, 30 Jun 2021 17:33:24 +0200 (CEST)
Fabien COELHO <coelho@cri.ensmp.fr> wrote:
> >> --report-latencies -> --report-per-command: should we keep supporting
> >> the previous option?
> >
> > Ok. Although now the option is not only for latencies, considering users who
> > are using the existing option, I'm fine with this. I got back this to the
> > previous name.
>
> Hmmm. I liked the new name! My point was whether we need to support the
> old one as well for compatibility, or whether we should not bother. I'm
> still wondering. As I think that the new name is better, I'd suggest to
> keep it.
Ok. I misunderstood it. I returned the option name to report-per-command.
If we keep report-latencies, I can imagine the following choises:
- use report-latencies to print only latency information
- use report-latencies as alias of report-per-command for compatibility
and remove at an appropriate timing. (that is, treat as deprecated)
Among these, I prefer the latter because ISTM we would not need many options
for reporting information per command. However, actually, I wander that we
don't have to keep the previous one if we plan to remove it eventually.
> >> --failures-detailed: if we bother to run with handling failures, should
> >> it always be on?
> >
> > If we print other failures that cannot be retried in future, it could a lot
> > of lines and might make some users who don't need details of failures annoyed.
> > Moreover, some users would always need information of detailed failures in log,
> > and others would need only total numbers of failures.
>
> Ok.
>
> > Currently we handle only serialization and deadlock failures, so the number of
> > lines printed and the number of columns of logging is not large even under the
> > failures-detail, but if we have a chance to handle other failures in future,
> > ISTM adding this option makes sense considering users who would like simple
> > outputs.
>
> Hmmm. What kind of failures could be managed with retries? I guess that on
> a connection failure we can try to reconnect, but otherwise it is less
> clear that other failures make sense to retry.
Indeed, there would few failures that we should retry and I can not imagine
other than serialization , deadlock, and connection failures for now. However,
considering reporting the number of failed transaction and its causes in future,
as you said
> Given that we manage errors, ISTM that we should not necessarily
> stop on other not retried errors, but rather count/report them and
> possibly proceed.
, we could define more a few kind of failures. At least we can consider
meta-command and other SQL commands errors in addition to serialization,
deadlock, connection failures. So, the total number of kind of failures would
be five at least and reporting always all of them results a lot of lines and
columns in logging.
> >> --debug-errors: I'm not sure we should want a special debug mode for that,
> >> I'd consider integrating it with the standard debug, or just for development.
> >
> > I think --debug is a debug option for telling users the pgbench's internal
> > behaviors, that is, which client is doing what. On other hand, --debug-errors
> > is for telling users what error caused a retry or a failure in detail. For
> > users who are not interested in pgbench's internal behavior (sending a command,
> > receiving a result, ... ) but interested in actual errors raised during running
> > script, this option seems useful.
>
> Ok. The this is not really about debug per se, but a verbosity setting?
I think so.
> Maybe --verbose-errors would make more sense? I'm unsure. I'll think about
> it.
Agreed. This seems more proper than the previous one, so I fixed the name to
--verbose-errors.
> > Sorry, I couldn't understand your suggestion. Is this about the order of case
> > statements or pg_log_error?
>
> My sentence got mixed up. My point was about the case order, so that they
> are put in a more logical order when reading all the cases.
Ok. Considering the loical order, I moved WAIT_ROLLBACK_RESULT into
between ERROR and RETRY, because WAIT_ROLLBACK_RESULT comes atter ERROR state,
and RETRY comes after ERROR or WAIT_ROLLBACK_RESULT..
> >> Currently, ISTM that the retry on error mode is implicitely always on.
> >> Do we want that? I'd say yes, but maybe people could disagree.
> >
> > The default values of max-tries is 1, so the retry on error is off.
>
> > Failed transactions are retried only when the user wants it and
> > specifies a valid value to max-treis.
>
> Ok. My point is that we do not stop on such errors, whereas before ISTM
> that we would have stopped, so somehow the default behavior has changed
> and the previous behavior cannot be reinstated with an option. Maybe that
> is not bad, but this is a behavioral change which needs to be documented
> and argumented.
I understood. Indeed, there is a behavioural change about whether we abort
the client after some types of errors or not. Now, serialization / deadlock
errors don't cause the abortion and are recorded as failures whereas other
errors cause to abort the client.
If we would want to record other errors as failures in future, we would need
a new option to specify which type of failures (or all types of errors, maybe)
should be reported. Until that time, ISTM we can treat serialization and
deadlock as something special errors to be reported as failures.
I rewrote "Failures and Serialization/Deadlock Retries" section a bit to
emphasis that such errors are treated differently than other errors.
> >> See the attached files for generating deadlocks reliably (start with 2
> >> clients). What do you think? The PL/pgSQL minimal, it is really
> >> client-code oriented.
> >
> > Sorry, but I cannot find the attached file.
>
> Sorry. Attached to this mail. The serialization stuff does not seem to
> work as well as the deadlock one. Run with 2 clients.
Hmmm, your test didn't work well for me. Both tests got stuck in
pgbench_deadlock_wait() and pgbench didn't finish.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v14-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 81.9K) download
From feb0391c642a72568b45946f4aae19f8976fd713 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Mon, 7 Jun 2021 18:35:14 +0900
Subject: [PATCH v14 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-per-command, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --verbose-errors.
---
doc/src/sgml/ref/pgbench.sgml | 402 +++++++-
src/bin/pgbench/pgbench.c | 954 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 217 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1485 insertions(+), 116 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 0c60077e1f..0a3ccb3c92 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -58,6 +58,7 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
@@ -65,11 +66,14 @@ tps = 896.967014 (without initial connection time)
</screen>
The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ settings. The seventh line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
The last line reports the number of transactions per second.
</para>
@@ -528,6 +532,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information about retrying such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -594,23 +609,29 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-r</option></term>
- <term><option>--report-latencies</option></term>
+ <term><option>--report-per-command</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -738,6 +759,26 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -748,6 +789,38 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; moreover, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information about
+ retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--verbose-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -943,8 +1016,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts">
+ <title id="transactions-and-scripts-title">What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1017,6 +1090,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, he will be aborted.
+ </para>
</note>
<para>
@@ -2207,7 +2285,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2228,6 +2306,17 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization_failure</literal> or
+ <literal>deadlock_failure</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information).
</para>
<para>
@@ -2256,6 +2345,24 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2271,7 +2378,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2285,7 +2392,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2293,21 +2409,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2319,13 +2439,44 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
+
+ <para>
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2339,27 +2490,63 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 10.870 ms
latency stddev = 7.341 ms
initial connection time = 30.954 ms
tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+statement latencies in milliseconds and failures:
+ 0.002 0 \set aid random(1, 100000 * :scale)
+ 0.005 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.001 0 \set delta random(-5000, 5000)
+ 0.326 0 BEGIN;
+ 0.603 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.454 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 5.528 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 7.335 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.371 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.212 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+number of transactions per client: 1000
+number of transactions actually processed: 9676/10000
+number of failed transactions: 324 (3.240%)
+number of serialization failures: 324 (3.240%)
+number of transactions retried: 5629 (56.290%)
+total number of retries: 103299
+maximum number of tries: 100
+number of transactions above the 100.0 ms latency limit: 21/9676 (0.217 %)
+latency average = 16.138 ms
+latency stddev = 21.017 ms
+tps = 413.686560 (without initial connection time)
+statement latencies in milliseconds, failures and retries:
+ 0.002 0 0 \set aid random(1, 100000 * :scale)
+ 0.000 0 0 \set bid random(1, 1 * :scale)
+ 0.000 0 0 \set tid random(1, 10 * :scale)
+ 0.000 0 0 \set delta random(-5000, 5000)
+ 0.121 0 0 BEGIN;
+ 0.290 0 2 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.221 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 0.266 212 72127 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.222 112 31170 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.178 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.210 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2373,6 +2560,139 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries">
+ <title id="failures-and-retries-title">Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there're three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the client is not aborted. In such cases, the current
+ transaction is rolled back, which also includes setting the client variables
+ as they were before the run of this transaction (it is assumed that one
+ transaction script contains only one transaction; see
+ <xref linkend="transactions-and-scripts" endterm="transactions-and-scripts-title"/>
+ for more information). Transactions with serialization or deadlock errors are
+ repeated after rollbacks until they complete successfully or reach the maximum
+ number of tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of retries (specified by the <option>--latency-limit</option> option) / the end
+ of benchmark (specified by the <option>--time</option> option). If
+ the last trial run fails, this transaction will be reported as failed but
+ the client is not aborted and continue to work.
+ </para>
+
+ <note>
+ <para>
+ Without specifying the <option>--max-tries</option> option a transaction will
+ never be retried after a serialization or deadlock error because its default
+ values is 1. Use an unlimited number of tries (<literal>--max-tries=0</literal>)
+ and the <option>--latency-limit</option> option to limit only the maximum time
+ of tries. You can also use the <option>--time</option> to limit the benchmark
+ duration under an unlimited number of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency for failed
+ transactions and commands is not computed separately.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions if it is non-zero.
+ If the total number of retried transactions is non-zero, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--verbose-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3629caba42..1e3d024bde 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -74,6 +74,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -273,9 +275,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 1;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -360,9 +387,65 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them) +
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactions) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -375,6 +458,30 @@ typedef struct RandomState
unsigned short xseed[3];
} RandomState;
+/*
+ * Data structure for repeating a transaction from the beginnning with the same
+ * parameters.
+ */
+typedef struct
+{
+ RandomState random_state; /* random seed */
+ Variables variables; /* client variables */
+} RetryState;
+
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
/* Various random sequences are initialized from this one. */
static RandomState base_random_sequence;
@@ -446,6 +553,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -494,8 +630,20 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ RetryState retry_state;
+ uint32 tries; /* how many times have we already tried the
+ * current transaction? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -590,6 +738,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -602,6 +753,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -616,6 +769,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool verbose_errors = false; /* print verbose messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -753,15 +908,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-per-command report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --verbose-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1307,6 +1465,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1315,22 +1477,51 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 tries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ if (tries > 1)
{
- addToSimpleStats(&stats->latency, lat);
+ stats->retries += (tries - 1);
+ stats->retried++;
+ }
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ switch (estatus)
+ {
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
+
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2861,6 +3052,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2868,6 +3062,17 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ Assert(sql_script[st->use_file].commands[st->command]->type == SQL_COMMAND);
+ pg_log_info("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2975,6 +3180,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -3017,6 +3249,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3031,6 +3264,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3055,6 +3289,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3072,6 +3307,20 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (verbose_errors)
+ commandError(st, PQerrorMessage(st->con));
+ if (PQpipelineStatus(st->con) == PQ_PIPELINE_ABORTED)
+ PQpipelineSync(st->con);
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3150,6 +3399,165 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+/*
+ * Clear the variables in the array. The array itself is not freed.
+ */
+static void
+clearVariables(Variables *variables)
+{
+ Variable *vars,
+ *var;
+ int nvars;
+
+ if (!variables)
+ return; /* nothing to do here */
+
+ vars = variables->vars;
+ nvars = variables->nvars;
+ for (var = vars; var - vars < nvars; ++var)
+ {
+ pg_free(var->name);
+ pg_free(var->svalue);
+ }
+
+ variables->nvars = 0;
+}
+
+/*
+ * Make a deep copy of variables array.
+ * Before copying the function frees the string fields of the destination
+ * variables and if necessary enlarges their array.
+ */
+static void
+copyVariables(Variables *dest, const Variables *source)
+{
+ Variable *dest_var;
+ const Variable *source_var;
+
+ if (!dest || !source || dest == source)
+ return; /* nothing to do here */
+
+ /*
+ * Clear the original variables and make sure that we have enough space for
+ * the new variables.
+ */
+ clearVariables(dest);
+ enlargeVariables(dest, source->nvars);
+
+ /* Make a deep copy of variables array */
+ for (source_var = source->vars, dest_var = dest->vars;
+ source_var - source->vars < source->nvars;
+ ++source_var, ++dest_var)
+ {
+ dest_var->name = pg_strdup(source_var->name);
+ dest_var->svalue = (source_var->svalue == NULL) ?
+ NULL : pg_strdup(source_var->svalue);
+ dest_var->value = source_var->value;
+ }
+ dest->nvars = source->nvars;
+ dest->vars_sorted = source->vars_sorted;
+}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries.
+ */
+ if (max_tries && st->tries >= max_tries)
+ return false;
+
+ /*
+ * We cannot retry the error if the benchmark duration is over.
+ */
+ if (timer_exceeded)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Set in_tx_block to true if we are in a (failed) transaction block and false
+ * otherwise.
+ * Returns false on failure (broken connection or internal error).
+ */
+static bool
+checkTransactionStatus(PGconn *con, bool *in_tx_block)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ *in_tx_block = false;
+ break;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ *in_tx_block = true;
+ break;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ { /* there's something wrong */
+ pg_log_error("perhaps the backend died while processing");
+ return false;
+ }
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * If the latency limit is used, return a percentage of the current transaction
+ * latency from the latency limit. Otherwise return zero.
+ */
+static double
+getLatencyUsed(CState *st, pg_time_usec_t *now)
+{
+ if (!latency_limit)
+ return 0.0;
+
+ pg_time_now_lazy(now);
+ return (100.0 * (*now - st->txn_scheduled) / latency_limit);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3179,6 +3587,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
for (;;)
{
Command *command;
+ PGresult *res;
+ bool in_tx_block;
switch (st->state)
{
@@ -3187,6 +3597,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->tries = 1;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3223,6 +3637,14 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember its
+ * parameters: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->retry_state.random_state = st->cs_func_rs;
+ copyVariables(&st->retry_state.variables, &st->variables);
+
/* record transaction start time */
st->txn_begin = now;
@@ -3374,6 +3796,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3512,6 +3936,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
@@ -3558,6 +3984,187 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
CSTATE_START_COMMAND : CSTATE_SKIP_COMMAND;
break;
+ /*
+ * Clean up after an error.
+ */
+ case CSTATE_ERROR:
+
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
+
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (in_tx_block)
+ {
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else
+ {
+ /* Check if we can retry the error. */
+ st->state = doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ break;
+
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ do
+ {
+ res = PQgetResult(st->con);
+ if (res)
+ PQclear(res);
+ } while (res);
+ /* Check if we can retry the error. */
+ st->state =
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ break;
+
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /*
+ * Inform that the transaction will be retried after the error.
+ */
+ if (verbose_errors)
+ {
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d repeats the transaction after the error (try %d",
+ st->id, st->tries);
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+ if (latency_limit)
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+ }
+
+ /* Count tries and retries */
+ st->tries++;
+ if (report_per_command)
+ command->retries++;
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->retry_state.random_state;
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Complete the failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ if (report_per_command)
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (verbose_errors)
+ {
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d ends the failed transaction (try %d",
+ st->id, st->tries);
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+ if (latency_limit)
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ else if (timer_exceeded)
+ appendPQExpBuffer(&buf, ", the duration time is exceeded");
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+ }
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction except for a random state.
+ */
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
/*
* End of transaction (end of script, really).
*/
@@ -3572,6 +4179,29 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
Assert(conditional_stack_empty(st->cstack));
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (in_tx_block)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (is_connect)
{
finishCon(st);
@@ -3803,6 +4433,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization_failure";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock_failure";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3847,6 +4514,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3857,6 +4532,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3864,22 +4543,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->tries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->tries - 1);
fputc('\n', logfile);
}
}
@@ -3888,7 +4571,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3896,10 +4580,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3908,20 +4592,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->tries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3931,7 +4607,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->tries);
}
@@ -4778,6 +5455,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5446,7 +6125,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5473,23 +6154,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5502,8 +6190,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5512,6 +6200,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5571,9 +6265,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5590,35 +6285,65 @@ printResults(StatsData *total,
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
}
+ if (failures > 0)
+ {
+ printf("number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ if (total->deadlock_failures)
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("total number of retries: " INT64_FORMAT "\n", total->retries);
+ }
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f %%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5644,7 +6369,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5663,6 +6388,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5672,25 +6400,60 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ if (failures > 0)
+ {
+ printf(" - number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ if (total->deadlock_failures)
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - total number of retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5698,10 +6461,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5782,7 +6554,7 @@ main(int argc, char **argv)
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
- {"report-latencies", no_argument, NULL, 'r'},
+ {"report-per-command", no_argument, NULL, 'r'},
{"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
@@ -5804,6 +6576,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"verbose-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -6172,6 +6947,28 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* verbose-errors */
+ benchmarking_option_set = true;
+ verbose_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6353,6 +7150,15 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (!latency_limit && duration <= 0)
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6561,6 +7367,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6709,7 +7519,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6798,7 +7609,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3aa9d5d753..f3351b51cf 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,11 @@ use Config;
# start a pgbench specific server
my $node = get_new_node('main');
-$node->init;
+
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
+
$node->start;
# invoke pgbench, with parameters:
@@ -159,7 +163,8 @@ pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1239,6 +1244,214 @@ pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^0 \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+pgbench(
+ "-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of failed transactions)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{total number of retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --verbose-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of failed transactions)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{total number of retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index 346a2667fc..56f7226c8e 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -178,6 +178,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index a562e28846..c304014f51 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index c64c655775..9c495072aa 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
v14-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.3K) download
From e83a30aea551ed3af2e5a90bbff887953e717002 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v14 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 163 +++++++++++++++++++++++---------------
1 file changed, 100 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 4aeccd93af..3629caba42 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -287,6 +287,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -304,6 +310,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -460,9 +484,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1418,39 +1440,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1582,21 +1604,37 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array.
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1608,23 +1646,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1633,12 +1665,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1656,12 +1689,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1676,12 +1709,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1740,7 +1774,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1761,7 +1795,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1776,12 +1810,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2649,7 +2684,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2719,7 +2754,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2750,7 +2785,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2811,7 +2846,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2863,7 +2898,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2874,7 +2909,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2921,7 +2956,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -3014,7 +3049,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3075,14 +3110,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3614,7 +3649,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3635,7 +3670,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3705,7 +3740,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3713,7 +3748,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -5995,7 +6030,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6335,19 +6370,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6382,11 +6417,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6395,30 +6430,32 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed =
((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) |
(((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) << 32);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 15:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-07 23:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-10 12:54 ` Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-10 12:54 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
I have played with v14 patch. I previously complained that pgbench
always reported 9 errors (actually the number is always the number
specified by "-c" -1 in my case).
$ pgbench -p 11000 -c 10 -T 10 --max-tries=0 test
pgbench (15devel, server 13.3)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
duration: 10 s
number of transactions actually processed: 974
number of failed transactions: 9 (0.916%)
number of transactions retried: 651 (66.226%)
total number of retries: 8482
latency average = 101.317 ms (including failures)
initial connection time = 44.440 ms
tps = 97.796487 (without initial connection time)
To reduce the number of errors I provide "--max-tries=9000" because
pgbench reported 8482 errors.
$ pgbench -p 11000 -c 10 -T 10 --max-tries=9000 test
pgbench (15devel, server 13.3)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
duration: 10 s
number of transactions actually processed: 1133
number of failed transactions: 9 (0.788%)
number of transactions retried: 755 (66.112%)
total number of retries: 9278
maximum number of tries: 9000
latency average = 88.570 ms (including failures)
initial connection time = 23.384 ms
tps = 112.015219 (without initial connection time)
Unfortunately this didn't work. Still 9 errors because pgbench
terminated the last round of run.
Then I gave up to use -T, and switched to use -t. Number of
transactions for -t option was calculated by the total number of
transactions actually processed (1133) / number of clients (10) =
11.33. I rouned up 11.33 to 12, then multiply number of clients (10)
and got 120. The result:
$ pgbench -p 11000 -c 10 -t 120 --max-tries=9000 test
pgbench (15devel, server 13.3)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
number of transactions per client: 120
number of transactions actually processed: 1200/1200
number of transactions retried: 675 (56.250%)
total number of retries: 8524
maximum number of tries: 9000
latency average = 93.777 ms
initial connection time = 14.120 ms
tps = 106.635908 (without initial connection time)
Finally I was able to get a result without any errors. This is not a
super simple way to obtain pgbench results without errors, but
probably I can live with it.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-06-30 22:28 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2021-06-30 22:28 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> I attached the patch updated according with your suggestion.
v13 patches gave a compiler warning...
$ make >/dev/null
pgbench.c: In function ¡commandError¢:
pgbench.c:3071:17: warning: unused variable ¡command¢ [-Wunused-variable]
const Command *command = sql_script[st->use_file].commands[st->command];
^~~~~~~
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-01 00:03 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-05 09:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 2 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-01 00:03 UTC (permalink / raw)
To: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> v13 patches gave a compiler warning...
>
> $ make >/dev/null
> pgbench.c: In function ¡commandError¢:
> pgbench.c:3071:17: warning: unused variable ¡command¢ [-Wunused-variable]
> const Command *command = sql_script[st->use_file].commands[st->command];
> ^~~~~~~
There is a typo in the doc (more over -> moreover).
> of all transaction tries; more over, you cannot use an unlimited number
of all transaction tries; moreover, you cannot use an unlimited number
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-02 00:25 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-02 00:25 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
I have found an interesting result from patched pgbench (I have set
the isolation level to REPEATABLE READ):
$ pgbench -p 11000 -c 10 -T 30 --max-tries=0 test
pgbench (15devel, server 13.3)
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
duration: 30 s
number of transactions actually processed: 2586
number of failed transactions: 9 (0.347%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
number of transactions retried: 1892 (72.909%)
total number of retries: 21819
latency average = 115.551 ms (including failures)
initial connection time = 35.268 ms
tps = 86.241799 (without initial connection time)
I ran pgbench with 10 concurrent sessions. In this case pgbench always
reports 9 failed transactions regardless the setting of -T
option. This is because at the end of a pgbench session, only 1 out of
10 transaction succeeded but 9 transactions failed due to
serialization error without any chance to retry because -T expires.
This is a little bit disappointed because I wanted to see a result of
all transactions succeeded with retries. I tried -t instead of -T but
-t cannot be used with --max-tries=0.
Also I think this behavior is somewhat inconsistent with existing
behavior of pgbench. When pgbench runs without --max-tries option,
pgbench continues to run transactions even after -T expires:
$ time pgbench -p 11000 -T 10 -f pgbench.sql test
pgbench (15devel, server 13.3)
starting vacuum...end.
transaction type: pgbench.sql
scaling factor: 1
query mode: simple
number of clients: 1
number of threads: 1
duration: 10 s
number of transactions actually processed: 2
maximum number of tries: 1
latency average = 7009.006 ms
initial connection time = 8.045 ms
tps = 0.142674 (without initial connection time)
real 0m14.067s
user 0m0.010s
sys 0m0.004s
$ cat pgbench.sql
SELECT pg_sleep(7);
So pgbench does not stop transactions after 10 seconds passed but
waits for the last transaction completes. If we consistent with
behavior when --max-tries=0, shouldn't we retry until the last
transaction finishes?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-06 07:46 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-06 07:46 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello Ishii-san,
On Fri, 02 Jul 2021 09:25:03 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> I have found an interesting result from patched pgbench (I have set
> the isolation level to REPEATABLE READ):
>
> $ pgbench -p 11000 -c 10 -T 30 --max-tries=0 test
> pgbench (15devel, server 13.3)
> starting vacuum...end.
> transaction type: <builtin: TPC-B (sort of)>
> scaling factor: 1
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 30 s
> number of transactions actually processed: 2586
> number of failed transactions: 9 (0.347%)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> number of transactions retried: 1892 (72.909%)
> total number of retries: 21819
> latency average = 115.551 ms (including failures)
> initial connection time = 35.268 ms
> tps = 86.241799 (without initial connection time)
>
> I ran pgbench with 10 concurrent sessions. In this case pgbench always
> reports 9 failed transactions regardless the setting of -T
> option. This is because at the end of a pgbench session, only 1 out of
> 10 transaction succeeded but 9 transactions failed due to
> serialization error without any chance to retry because -T expires.
>
> This is a little bit disappointed because I wanted to see a result of
> all transactions succeeded with retries. I tried -t instead of -T but
> -t cannot be used with --max-tries=0.
>
> Also I think this behavior is somewhat inconsistent with existing
> behavior of pgbench. When pgbench runs without --max-tries option,
> pgbench continues to run transactions even after -T expires:
>
> $ time pgbench -p 11000 -T 10 -f pgbench.sql test
> pgbench (15devel, server 13.3)
> starting vacuum...end.
> transaction type: pgbench.sql
> scaling factor: 1
> query mode: simple
> number of clients: 1
> number of threads: 1
> duration: 10 s
> number of transactions actually processed: 2
> maximum number of tries: 1
> latency average = 7009.006 ms
> initial connection time = 8.045 ms
> tps = 0.142674 (without initial connection time)
>
> real 0m14.067s
> user 0m0.010s
> sys 0m0.004s
>
> $ cat pgbench.sql
> SELECT pg_sleep(7);
>
> So pgbench does not stop transactions after 10 seconds passed but
> waits for the last transaction completes. If we consistent with
> behavior when --max-tries=0, shouldn't we retry until the last
> transaction finishes?
I changed the previous patch to enable that the -T option can terminate
a retrying transaction and that we can specify --max-tries=0 without
--latency-limit if we have -T , according with the following comment.
> Doc says "you cannot use an infinite number of retries without latency-limit..."
>
> Why should this be forbidden? At least if -T timeout takes precedent and
> shortens the execution, ISTM that there could be good reason to test that.
> Maybe it could be blocked only under -t if this would lead to an non-ending
> run.
Indeed, as Ishii-san pointed out, some users might not want to terminate
retrying transactions due to -T. However, the actual negative effect is only
printing the number of failed transactions. The other result that users want to
know, such as tps, are almost not affected because they are measured for
transactions processed successfully. Actually, the percentage of failed
transaction is very little, only 0.347%.
In the existing behaviour, running transactions are never terminated due to
the -T option. However, ISTM that this would be based on an assumption
that a latency of each transaction is small and that a timing when we can
finish the benchmark would come soon. On the other hand, when transactions can
be retried unlimitedly, it may take a long time more than expected, and we can
not guarantee that this would finish successfully in limited time. Therefore,
terminating the benchmark by giving up to retry the transaction after time
expiration seems reasonable under unlimited retries. In the sense that we don't
terminate running transactions forcibly, this don't change the existing behaviour.
If you don't want to print the number of transactions failed due to -T, we can
fix to forbid to use -T without latency-limit under max-tries=0 for avoiding
possible never-ending-benchmark. In this case, users have to limit the number of
transaction retry by specifying latency-limit or max-tries (>0). However, if some
users would like to benchmark simply allowing unlimited retries, using -T and
max-tries=0 seems the most straight way, so I think it is better that they can be
used together.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-07 07:11 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-07 07:11 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> Indeed, as Ishii-san pointed out, some users might not want to terminate
> retrying transactions due to -T. However, the actual negative effect is only
> printing the number of failed transactions. The other result that users want to
> know, such as tps, are almost not affected because they are measured for
> transactions processed successfully. Actually, the percentage of failed
> transaction is very little, only 0.347%.
Well, "that's very little, let's ignore it" is not technically a right
direction IMO.
> In the existing behaviour, running transactions are never terminated due to
> the -T option. However, ISTM that this would be based on an assumption
> that a latency of each transaction is small and that a timing when we can
> finish the benchmark would come soon. On the other hand, when transactions can
> be retried unlimitedly, it may take a long time more than expected, and we can
> not guarantee that this would finish successfully in limited time.Therefore,
> terminating the benchmark by giving up to retry the transaction after time
> expiration seems reasonable under unlimited retries.
That's necessarily true in practice. By the time when -T is about to
expire, transactions are all finished in finite time as you can see
the result I showed. So it's reasonable that the very last cycle of
the benchmark will finish in finite time as well.
Of course if a benchmark cycle takes infinite time, this will be a
problem. However same thing can be said to non-retry
benchmarks. Theoretically it is possible that *one* benchmark cycle
takes forever. In this case the only solution will be just hitting ^C
to terminate pgbench. Why can't we have same assumption with
--max-tries=0 case?
> In the sense that we don't
> terminate running transactions forcibly, this don't change the existing behaviour.
This statement seems to be depending on your perosnal assumption.
I still don't understand why you think that --max-tries non 0 case
will *certainly* finish in finite time whereas --max-tries=0 case will
not.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-07 10:53 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-07 10:53 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Wed, 07 Jul 2021 16:11:23 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> > Indeed, as Ishii-san pointed out, some users might not want to terminate
> > retrying transactions due to -T. However, the actual negative effect is only
> > printing the number of failed transactions. The other result that users want to
> > know, such as tps, are almost not affected because they are measured for
> > transactions processed successfully. Actually, the percentage of failed
> > transaction is very little, only 0.347%.
>
> Well, "that's very little, let's ignore it" is not technically a right
> direction IMO.
Hmmm, It seems to me these failures are ignorable because with regard to failures
due to -T they occur only the last transaction of each client and do not affect
the result such as TPS and latency of successfully processed transactions.
(although I am not sure for what sense you use the word "technically"...)
However, maybe I am missing something. Could you please tell me what do you think
the actual harm for users about failures due to -D is?
> > In the existing behaviour, running transactions are never terminated due to
> > the -T option. However, ISTM that this would be based on an assumption
> > that a latency of each transaction is small and that a timing when we can
> > finish the benchmark would come soon. On the other hand, when transactions can
> > be retried unlimitedly, it may take a long time more than expected, and we can
> > not guarantee that this would finish successfully in limited time.Therefore,
> > terminating the benchmark by giving up to retry the transaction after time
> > expiration seems reasonable under unlimited retries.
>
> That's necessarily true in practice. By the time when -T is about to
> expire, transactions are all finished in finite time as you can see
> the result I showed. So it's reasonable that the very last cycle of
> the benchmark will finish in finite time as well.
Your script may finish in finite time, but others may not. However,
considering only serialization and deadlock errors, almost transactions
would finish in finite time eventually. In the previous version of the
patch, errors other than serialization or deadlock can be retried and
it causes unlimited retrying easily. Now, only the two kind of errors
can be retried, nevertheless, it is unclear for me that we can assume
that retying will finish in finite time. If we can assume it, maybe,
we can remove the restriction that --max-retries=0 must be used with
--latency-limit or -T.
> Of course if a benchmark cycle takes infinite time, this will be a
> problem. However same thing can be said to non-retry
> benchmarks. Theoretically it is possible that *one* benchmark cycle
> takes forever. In this case the only solution will be just hitting ^C
> to terminate pgbench. Why can't we have same assumption with
> --max-tries=0 case?
Indeed, it is possible an execution of a query takes a long or infinite
time. However, its cause would a problematic query in the custom script
or other problems occurs on the server side. These are not problem of
pgbench and, pgbench itself can't control either. On the other hand, the
unlimited number of tries is a behaviours specified by the pgbench option,
so I think pgbench itself should internally avoid problems caused from its
behaviours. That is, if max-tries=0 could cause infinite or much longer
benchmark time more than user expected due to too many retries, I think
pgbench should avoid it.
> > In the sense that we don't
> > terminate running transactions forcibly, this don't change the existing behaviour.
>
> This statement seems to be depending on your perosnal assumption.
Ok. If we regard that a transaction is still running even when it is under
retrying after an error, terminate of the retry may imply to terminate running
the transaction forcibly.
> I still don't understand why you think that --max-tries non 0 case
> will *certainly* finish in finite time whereas --max-tries=0 case will
> not.
I just mean that --max-tries greater than zero will prevent pgbench from retrying a
transaction forever.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-07 12:50 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-07 12:50 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> Well, "that's very little, let's ignore it" is not technically a right
>> direction IMO.
>
> Hmmm, It seems to me these failures are ignorable because with regard to failures
> due to -T they occur only the last transaction of each client and do not affect
> the result such as TPS and latency of successfully processed transactions.
> (although I am not sure for what sense you use the word "technically"...)
"My application button does not respond once in 100 times. It's just
1% error rate. You should ignore it." I would say this attitude is not
technically correct.
> However, maybe I am missing something. Could you please tell me what do you think
> the actual harm for users about failures due to -D is?
I don't know why you are referring to -D.
>> That's necessarily true in practice. By the time when -T is about to
>> expire, transactions are all finished in finite time as you can see
>> the result I showed. So it's reasonable that the very last cycle of
>> the benchmark will finish in finite time as well.
>
> Your script may finish in finite time, but others may not.
That's why I said "practically". In other words "in most cases the
scenario will finish in finite time".
> Indeed, it is possible an execution of a query takes a long or infinite
> time. However, its cause would a problematic query in the custom script
> or other problems occurs on the server side. These are not problem of
> pgbench and, pgbench itself can't control either. On the other hand, the
> unlimited number of tries is a behaviours specified by the pgbench option,
> so I think pgbench itself should internally avoid problems caused from its
> behaviours. That is, if max-tries=0 could cause infinite or much longer
> benchmark time more than user expected due to too many retries, I think
> pgbench should avoid it.
I would say that's user's responsibility to avoid infinite running
benchmarking. Remember, pgbench is a tool for serious users, not for
novice users.
Or, we should terminate the last cycle of benchmark regardless it is
retrying or not if -T expires. This will make pgbench behaves much
more consistent.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-07 13:46 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-07 13:46 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Wed, 07 Jul 2021 21:50:16 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> >> Well, "that's very little, let's ignore it" is not technically a right
> >> direction IMO.
> >
> > Hmmm, It seems to me these failures are ignorable because with regard to failures
> > due to -T they occur only the last transaction of each client and do not affect
> > the result such as TPS and latency of successfully processed transactions.
> > (although I am not sure for what sense you use the word "technically"...)
>
> "My application button does not respond once in 100 times. It's just
> 1% error rate. You should ignore it." I would say this attitude is not
> technically correct.
I cannot understand what you want to say. Reporting the number of transactions
that is failed intentionally can be treated as same as he error rate on your
application's button?
> > However, maybe I am missing something. Could you please tell me what do you think
> > the actual harm for users about failures due to -D is?
>
> I don't know why you are referring to -D.
Sorry. It's just a typo as you can imagine.
I am asking you what do you think the actual harm for users due to termination of
retrying by the -T option is.
> >> That's necessarily true in practice. By the time when -T is about to
> >> expire, transactions are all finished in finite time as you can see
> >> the result I showed. So it's reasonable that the very last cycle of
> >> the benchmark will finish in finite time as well.
> >
> > Your script may finish in finite time, but others may not.
>
> That's why I said "practically". In other words "in most cases the
> scenario will finish in finite time".
Sure.
> > Indeed, it is possible an execution of a query takes a long or infinite
> > time. However, its cause would a problematic query in the custom script
> > or other problems occurs on the server side. These are not problem of
> > pgbench and, pgbench itself can't control either. On the other hand, the
> > unlimited number of tries is a behaviours specified by the pgbench option,
> > so I think pgbench itself should internally avoid problems caused from its
> > behaviours. That is, if max-tries=0 could cause infinite or much longer
> > benchmark time more than user expected due to too many retries, I think
> > pgbench should avoid it.
>
> I would say that's user's responsibility to avoid infinite running
> benchmarking. Remember, pgbench is a tool for serious users, not for
> novice users.
Of course, users themselves should be careful of problematic script, but it
would be better that pgbench itself avoids problems if pgbench can beforehand.
> Or, we should terminate the last cycle of benchmark regardless it is
> retrying or not if -T expires. This will make pgbench behaves much
> more consistent.
Hmmm, indeed this might make the behaviour a bit consistent, but I am not
sure such behavioural change benefit users.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-10 17:19 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2021-07-10 17:19 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello,
> Of course, users themselves should be careful of problematic script, but it
> would be better that pgbench itself avoids problems if pgbench can beforehand.
>
>> Or, we should terminate the last cycle of benchmark regardless it is
>> retrying or not if -T expires. This will make pgbench behaves much
>> more consistent.
I would tend to agree with this behavior, that is not to start any new
transaction or transaction attempt once -T has expired.
I'm a little hesitant about how to count and report such unfinished
because of bench timeout transactions, though. Not counting them seems to
be the best option.
> Hmmm, indeed this might make the behaviour a bit consistent, but I am not
> sure such behavioural change benefit users.
The user benefit would be that if they asked for a 100s benchmark, pgbench
does a reasonable effort not to overshot that?
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2021-07-13 04:00 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-13 04:00 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: nagata@sraoss.co.jp; ishii@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>>> Or, we should terminate the last cycle of benchmark regardless it is
>>> retrying or not if -T expires. This will make pgbench behaves much
>>> more consistent.
>
> I would tend to agree with this behavior, that is not to start any new
> transaction or transaction attempt once -T has expired.
>
> I'm a little hesitant about how to count and report such unfinished
> because of bench timeout transactions, though. Not counting them seems
> to be the best option.
I agree.
>> Hmmm, indeed this might make the behaviour a bit consistent, but I am
>> not
>> sure such behavioural change benefit users.
>
> The user benefit would be that if they asked for a 100s benchmark,
> pgbench does a reasonable effort not to overshot that?
Right.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-13 04:32 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-13 04:32 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Tue, 13 Jul 2021 13:00:49 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> >>> Or, we should terminate the last cycle of benchmark regardless it is
> >>> retrying or not if -T expires. This will make pgbench behaves much
> >>> more consistent.
> >
> > I would tend to agree with this behavior, that is not to start any new
> > transaction or transaction attempt once -T has expired.
That is the behavior in the latest patch. Once -T has expired, any new
transaction or retry does not start.
IIUC, Ishii-san's proposal was changing the pgbench's behavior when -T has
expired to terminate any running transactions immediately regardless retrying.
I am not sure we should do it in this patch. If we would like this change,
it would be done in another patch as an improvement of the -T option.
> > I'm a little hesitant about how to count and report such unfinished
> > because of bench timeout transactions, though. Not counting them seems
> > to be the best option.
>
> I agree.
I also agree. Although I couldn't get an answer what does he think the actual
harm for users due to termination of retrying by the -T option is, I guess it just
complained about reporting the termination of retrying as failures. Therefore,
I will fix to finish the benchmark when the time is over during retrying, that is,
change the state to CSTATE_FINISHED instead of CSTATE_ERROR in such cases.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-13 05:35 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2021-07-13 05:35 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: ishii@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> > I would tend to agree with this behavior, that is not to start any new
>> > transaction or transaction attempt once -T has expired.
>
> That is the behavior in the latest patch. Once -T has expired, any new
> transaction or retry does not start.
Actually v14 has not changed the behavior in this regard as explained
in different email:
> $ pgbench -p 11000 -c 10 -T 10 --max-tries=0 test
> pgbench (15devel, server 13.3)
> starting vacuum...end.
> transaction type: <builtin: TPC-B (sort of)>
> scaling factor: 1
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 10 s
> number of transactions actually processed: 974
> number of failed transactions: 9 (0.916%)
> number of transactions retried: 651 (66.226%)
> total number of retries: 8482
> latency average = 101.317 ms (including failures)
> initial connection time = 44.440 ms
> tps = 97.796487 (without initial connection time)
>> > I'm a little hesitant about how to count and report such unfinished
>> > because of bench timeout transactions, though. Not counting them seems
>> > to be the best option.
>>
>> I agree.
>
> I also agree. Although I couldn't get an answer what does he think the actual
> harm for users due to termination of retrying by the -T option is, I guess it just
> complained about reporting the termination of retrying as failures. Therefore,
> I will fix to finish the benchmark when the time is over during retrying, that is,
> change the state to CSTATE_FINISHED instead of CSTATE_ERROR in such cases.
I guess Fabien wanted it differently. Suppose "-c 10 and -T 30" and we
have 100 success transactions by time 25. At time 25 pgbench starts
next benchmark cycle and by time 30 there are 10 failing transactions
(because they are retrying). pgbench stops the execution at time
30. According your proposal (change the state to CSTATE_FINISHED
instead of CSTATE_ERROR) the total number of success transactions will
be 100 + 10 = 110, right? I guess Fabien wants to have the number to
be 100 rather than 110.
Fabien,
Please correct me if you think differently.
Also actually I have explained the harm number of times but you have
kept on ignoring it because "it's subtle". My request has been pretty
simple.
> number of failed transactions: 9 (0.916%)
I don't like this and want to have the failed transactions to be 0.
Who wants a benchmark result having errors?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-13 06:50 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-13 06:50 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Tue, 13 Jul 2021 14:35:00 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> >> > I would tend to agree with this behavior, that is not to start any new
> >> > transaction or transaction attempt once -T has expired.
> >
> > That is the behavior in the latest patch. Once -T has expired, any new
> > transaction or retry does not start.
>
> Actually v14 has not changed the behavior in this regard as explained
> in different email:
Right. Both of v13 and v14 doen't start any new transaction or retry once
-T has expired.
> >> > I'm a little hesitant about how to count and report such unfinished
> >> > because of bench timeout transactions, though. Not counting them seems
> >> > to be the best option.
> >>
> >> I agree.
> >
> > I also agree. Although I couldn't get an answer what does he think the actual
> > harm for users due to termination of retrying by the -T option is, I guess it just
> > complained about reporting the termination of retrying as failures. Therefore,
> > I will fix to finish the benchmark when the time is over during retrying, that is,
> > change the state to CSTATE_FINISHED instead of CSTATE_ERROR in such cases.
>
> I guess Fabien wanted it differently. Suppose "-c 10 and -T 30" and we
> have 100 success transactions by time 25. At time 25 pgbench starts
> next benchmark cycle and by time 30 there are 10 failing transactions
> (because they are retrying). pgbench stops the execution at time
> 30. According your proposal (change the state to CSTATE_FINISHED
> instead of CSTATE_ERROR) the total number of success transactions will
> be 100 + 10 = 110, right?
No. The last failed transaction is not counted because CSTATE_END_TX is
bypassed, so please don't worry.
> Also actually I have explained the harm number of times but you have
> kept on ignoring it because "it's subtle". My request has been pretty
> simple.
>
> > number of failed transactions: 9 (0.916%)
>
> I don't like this and want to have the failed transactions to be 0.
> Who wants a benchmark result having errors?
I was asking you because I would like to confirm what you really complained
about; whether the problem is that retrying transaction is terminated by -T
option, or that pgbench reports it as the number of failed transactions? But
now, I understood this is the latter that you don't want to count the temination
of retrying as failures. Thanks.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-16 03:05 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-16 03:05 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello,
I attached the updated patch.
On Tue, 13 Jul 2021 15:50:52 +0900
Yugo NAGATA <nagata@sraoss.co.jp> wrote:
> > >> > I'm a little hesitant about how to count and report such unfinished
> > >> > because of bench timeout transactions, though. Not counting them seems
> > >> > to be the best option.
> > > I will fix to finish the benchmark when the time is over during retrying, that is,
> > > change the state to CSTATE_FINISHED instead of CSTATE_ERROR in such cases.
Done.
(I wrote CSTATE_ERROR, but correctly it is CSTATE_FAILURE.)
Now, once the timer is expired during retrying a failed transaction, pgbench never start
a new transaction for retry. If the transaction successes, it will counted in the result.
Otherwise, if the transaction fails again, it is not counted.
In addition, I fixed to work well with pipeline mode. Previously, pipeline mode was not
enough considered, ROLLBACK was not sent correctly. I fixed to handle errors in pipeline
mode properly, and now it works.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v15-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 82.5K) download
From 1cd3519f3a1cfbffe7bcce35fd6f12da566625b1 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Mon, 7 Jun 2021 18:35:14 +0900
Subject: [PATCH v15 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-per-command, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --verbose-errors.
---
doc/src/sgml/ref/pgbench.sgml | 402 +++++++-
src/bin/pgbench/pgbench.c | 974 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 217 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1505 insertions(+), 116 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 0c60077e1f..0a3ccb3c92 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -58,6 +58,7 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
@@ -65,11 +66,14 @@ tps = 896.967014 (without initial connection time)
</screen>
The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ settings. The seventh line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
The last line reports the number of transactions per second.
</para>
@@ -528,6 +532,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information about retrying such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -594,23 +609,29 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-r</option></term>
- <term><option>--report-latencies</option></term>
+ <term><option>--report-per-command</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -738,6 +759,26 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -748,6 +789,38 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; moreover, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information about
+ retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--verbose-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -943,8 +1016,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts">
+ <title id="transactions-and-scripts-title">What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1017,6 +1090,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, he will be aborted.
+ </para>
</note>
<para>
@@ -2207,7 +2285,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2228,6 +2306,17 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization_failure</literal> or
+ <literal>deadlock_failure</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information).
</para>
<para>
@@ -2256,6 +2345,24 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2271,7 +2378,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2285,7 +2392,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2293,21 +2409,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2319,13 +2439,44 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
+
+ <para>
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2339,27 +2490,63 @@ number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+maximum number of tries: 1
latency average = 10.870 ms
latency stddev = 7.341 ms
initial connection time = 30.954 ms
tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+statement latencies in milliseconds and failures:
+ 0.002 0 \set aid random(1, 100000 * :scale)
+ 0.005 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.001 0 \set delta random(-5000, 5000)
+ 0.326 0 BEGIN;
+ 0.603 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.454 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 5.528 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 7.335 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.371 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.212 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+number of transactions per client: 1000
+number of transactions actually processed: 9676/10000
+number of failed transactions: 324 (3.240%)
+number of serialization failures: 324 (3.240%)
+number of transactions retried: 5629 (56.290%)
+total number of retries: 103299
+maximum number of tries: 100
+number of transactions above the 100.0 ms latency limit: 21/9676 (0.217 %)
+latency average = 16.138 ms
+latency stddev = 21.017 ms
+tps = 413.686560 (without initial connection time)
+statement latencies in milliseconds, failures and retries:
+ 0.002 0 0 \set aid random(1, 100000 * :scale)
+ 0.000 0 0 \set bid random(1, 1 * :scale)
+ 0.000 0 0 \set tid random(1, 10 * :scale)
+ 0.000 0 0 \set delta random(-5000, 5000)
+ 0.121 0 0 BEGIN;
+ 0.290 0 2 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.221 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 0.266 212 72127 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.222 112 31170 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.178 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.210 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2373,6 +2560,139 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries">
+ <title id="failures-and-retries-title">Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there're three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the client is not aborted. In such cases, the current
+ transaction is rolled back, which also includes setting the client variables
+ as they were before the run of this transaction (it is assumed that one
+ transaction script contains only one transaction; see
+ <xref linkend="transactions-and-scripts" endterm="transactions-and-scripts-title"/>
+ for more information). Transactions with serialization or deadlock errors are
+ repeated after rollbacks until they complete successfully or reach the maximum
+ number of tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of retries (specified by the <option>--latency-limit</option> option) / the end
+ of benchmark (specified by the <option>--time</option> option). If
+ the last trial run fails, this transaction will be reported as failed but
+ the client is not aborted and continue to work.
+ </para>
+
+ <note>
+ <para>
+ Without specifying the <option>--max-tries</option> option a transaction will
+ never be retried after a serialization or deadlock error because its default
+ values is 1. Use an unlimited number of tries (<literal>--max-tries=0</literal>)
+ and the <option>--latency-limit</option> option to limit only the maximum time
+ of tries. You can also use the <option>--time</option> to limit the benchmark
+ duration under an unlimited number of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency for failed
+ transactions and commands is not computed separately.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions if it is non-zero.
+ If the total number of retried transactions is non-zero, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--verbose-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index c4c2fd3566..9e022c25cd 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -74,6 +74,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -273,9 +275,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 1;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -360,9 +387,65 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them) +
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactions) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -381,6 +464,30 @@ typedef struct RandomState
unsigned short xseed[3];
} RandomState;
+/*
+ * Data structure for repeating a transaction from the beginnning with the same
+ * parameters.
+ */
+typedef struct
+{
+ RandomState random_state; /* random seed */
+ Variables variables; /* client variables */
+} RetryState;
+
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
/* Various random sequences are initialized from this one. */
static RandomState base_random_sequence;
@@ -452,6 +559,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -500,8 +636,20 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ RetryState retry_state;
+ uint32 tries; /* how many times have we already tried the
+ * current transaction? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -596,6 +744,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -608,6 +759,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -622,6 +775,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool verbose_errors = false; /* print verbose messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -759,15 +914,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-per-command report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --verbose-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1313,6 +1471,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1321,22 +1483,51 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 tries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ if (tries > 1)
+ {
+ stats->retries += (tries - 1);
+ stats->retried++;
+ }
+
+ switch (estatus)
{
- addToSimpleStats(&stats->latency, lat);
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2867,6 +3058,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2874,6 +3068,17 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ Assert(sql_script[st->use_file].commands[st->command]->type == SQL_COMMAND);
+ pg_log_info("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2981,6 +3186,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -3023,6 +3255,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3037,6 +3270,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3061,6 +3295,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3078,6 +3313,18 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (verbose_errors)
+ commandError(st, PQerrorMessage(st->con));
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3156,6 +3403,159 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+/*
+ * Clear the variables in the array. The array itself is not freed.
+ */
+static void
+clearVariables(Variables *variables)
+{
+ Variable *vars,
+ *var;
+ int nvars;
+
+ if (!variables)
+ return; /* nothing to do here */
+
+ vars = variables->vars;
+ nvars = variables->nvars;
+ for (var = vars; var - vars < nvars; ++var)
+ {
+ pg_free(var->name);
+ pg_free(var->svalue);
+ }
+
+ variables->nvars = 0;
+}
+
+/*
+ * Make a deep copy of variables array.
+ * Before copying the function frees the string fields of the destination
+ * variables and if necessary enlarges their array.
+ */
+static void
+copyVariables(Variables *dest, const Variables *source)
+{
+ Variable *dest_var;
+ const Variable *source_var;
+
+ if (!dest || !source || dest == source)
+ return; /* nothing to do here */
+
+ /*
+ * Clear the original variables and make sure that we have enough space for
+ * the new variables.
+ */
+ clearVariables(dest);
+ enlargeVariables(dest, source->nvars);
+
+ /* Make a deep copy of variables array */
+ for (source_var = source->vars, dest_var = dest->vars;
+ source_var - source->vars < source->nvars;
+ ++source_var, ++dest_var)
+ {
+ dest_var->name = pg_strdup(source_var->name);
+ dest_var->svalue = (source_var->svalue == NULL) ?
+ NULL : pg_strdup(source_var->svalue);
+ dest_var->value = source_var->value;
+ }
+ dest->nvars = source->nvars;
+ dest->vars_sorted = source->vars_sorted;
+}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries.
+ */
+ if (max_tries && st->tries >= max_tries)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Set in_tx_block to true if we are in a (failed) transaction block and false
+ * otherwise.
+ * Returns false on failure (broken connection or internal error).
+ */
+static bool
+checkTransactionStatus(PGconn *con, bool *in_tx_block)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ *in_tx_block = false;
+ break;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ *in_tx_block = true;
+ break;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ { /* there's something wrong */
+ pg_log_error("perhaps the backend died while processing");
+ return false;
+ }
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return false;
+ }
+
+ /* OK */
+ return true;
+}
+
+/*
+ * If the latency limit is used, return a percentage of the current transaction
+ * latency from the latency limit. Otherwise return zero.
+ */
+static double
+getLatencyUsed(CState *st, pg_time_usec_t *now)
+{
+ if (!latency_limit)
+ return 0.0;
+
+ pg_time_now_lazy(now);
+ return (100.0 * (*now - st->txn_scheduled) / latency_limit);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3185,6 +3585,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
for (;;)
{
Command *command;
+ PGresult *res;
+ bool in_tx_block;
switch (st->state)
{
@@ -3193,6 +3595,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->tries = 1;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3229,6 +3635,14 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember its
+ * parameters: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->retry_state.random_state = st->cs_func_rs;
+ copyVariables(&st->retry_state.variables, &st->variables);
+
/* record transaction start time */
st->txn_begin = now;
@@ -3380,6 +3794,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3518,6 +3934,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
@@ -3564,6 +3982,215 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
CSTATE_START_COMMAND : CSTATE_SKIP_COMMAND;
break;
+ /*
+ * Clean up after an error.
+ */
+ case CSTATE_ERROR:
+
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
+
+ /* Read and discard until a sync point in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ bool is_sync = false;
+ do {
+ res = PQgetResult(st->con);
+ if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
+ is_sync = true;
+ PQclear(res);
+ } while (!is_sync);
+ }
+
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (in_tx_block)
+ {
+ /* Exit pipeline mode before rollback */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (PQexitPipelineMode(st->con) != 1)
+ {
+ pg_log_error("client %d aborted: failed to exit pipeline mode for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ }
+
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else
+ {
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ break;
+
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ do
+ {
+ res = PQgetResult(st->con);
+ if (res)
+ PQclear(res);
+ } while (res);
+
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ break;
+
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /*
+ * Inform that the transaction will be retried after the error.
+ */
+ if (verbose_errors)
+ {
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d repeats the transaction after the error (try %d",
+ st->id, st->tries);
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+ if (latency_limit)
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+ }
+
+ /* Count tries and retries */
+ st->tries++;
+ if (report_per_command)
+ command->retries++;
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->retry_state.random_state;
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Complete the failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ if (report_per_command)
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (verbose_errors)
+ {
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d ends the failed transaction (try %d",
+ st->id, st->tries);
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+ if (latency_limit)
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ getLatencyUsed(st, &now));
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+ }
+
+ /*
+ * Reset the execution parameters as they were at the beginning
+ * of the transaction except for a random state.
+ */
+ copyVariables(&st->variables, &st->retry_state.variables);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
/*
* End of transaction (end of script, really).
*/
@@ -3578,6 +4205,29 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
Assert(conditional_stack_empty(st->cstack));
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ if (!checkTransactionStatus(st->con, &in_tx_block))
+ {
+ /*
+ * There's something wrong...
+ * It is assumed that the function checkTransactionStatus
+ * has already printed a more detailed error message.
+ */
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (in_tx_block)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (is_connect)
{
finishCon(st);
@@ -3809,6 +4459,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization_failure";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock_failure";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3856,6 +4543,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3866,6 +4561,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3873,22 +4572,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->tries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->tries - 1);
fputc('\n', logfile);
}
}
@@ -3897,7 +4600,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3905,10 +4609,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3917,20 +4621,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->tries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3940,7 +4636,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->tries);
}
@@ -4787,6 +5484,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5455,7 +6154,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5482,23 +6183,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5512,8 +6220,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5522,6 +6230,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5581,9 +6295,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5600,35 +6315,65 @@ printResults(StatsData *total,
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
}
+ if (failures > 0)
+ {
+ printf("number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ if (total->deadlock_failures)
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("total number of retries: " INT64_FORMAT "\n", total->retries);
+ }
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f %%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5654,7 +6399,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5673,6 +6418,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5682,25 +6430,60 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ if (failures > 0)
+ {
+ printf(" - number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ if (total->deadlock_failures)
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (total->retried > 0)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - total number of retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5708,10 +6491,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5792,7 +6584,7 @@ main(int argc, char **argv)
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
- {"report-latencies", no_argument, NULL, 'r'},
+ {"report-per-command", no_argument, NULL, 'r'},
{"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
@@ -5814,6 +6606,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"verbose-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -6190,6 +6985,28 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* verbose-errors */
+ benchmarking_option_set = true;
+ verbose_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6371,6 +7188,15 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (!latency_limit && duration <= 0)
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6579,6 +7405,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6734,7 +7564,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6823,7 +7654,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 3aa9d5d753..f3351b51cf 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,11 @@ use Config;
# start a pgbench specific server
my $node = get_new_node('main');
-$node->init;
+
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
+
$node->start;
# invoke pgbench, with parameters:
@@ -159,7 +163,8 @@ pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1239,6 +1244,214 @@ pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^0 \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+pgbench(
+ "-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of failed transactions)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{total number of retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --verbose-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{^((?!number of failed transactions)(.|\n))*$},
+ qr{number of transactions retried: 1\b}, qr{total number of retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index 346a2667fc..56f7226c8e 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -178,6 +178,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index a562e28846..c304014f51 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index c64c655775..9c495072aa 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
v15-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.3K) download
From 36882744153c3642e4c75e101cd31b7f8a65a893 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v15 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 163 +++++++++++++++++++++++---------------
1 file changed, 100 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 364b5a2e47..c4c2fd3566 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -287,6 +287,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -304,6 +310,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -466,9 +490,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1424,39 +1446,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1588,21 +1610,37 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array.
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1614,23 +1652,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1639,12 +1671,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1662,12 +1695,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1682,12 +1715,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1746,7 +1780,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1767,7 +1801,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1782,12 +1816,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2655,7 +2690,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2725,7 +2760,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2756,7 +2791,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2817,7 +2852,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2869,7 +2904,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2880,7 +2915,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2927,7 +2962,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -3020,7 +3055,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3081,14 +3116,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3620,7 +3655,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3641,7 +3676,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3711,7 +3746,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3719,7 +3754,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -6013,7 +6048,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6353,19 +6388,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6400,11 +6435,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6413,30 +6448,32 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed =
((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) |
(((uint64) pg_jrand48(base_random_sequence.xseed) & 0xFFFFFFFF) << 32);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2021-07-19 18:04 ` Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2021-07-19 18:04 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> I attached the updated patch.
# About pgbench error handling v15
Patches apply cleanly. Compilation, global and local tests ok.
- v15.1: refactoring is a definite improvement.
Good, even if it is not very useful (see below).
While restructuring, maybe predefined variables could be make readonly
so that a script which would update them would fail, which would be a
good thing. Maybe this is probably material for an independent patch.
- v15.2: see detailed comments below
# Doc
Doc build is ok.
ISTM that "number of tries" line would be better placed between the
#threads and #transactions lines. What do you think?
Aggregate logging description: "{ failures | ... }" seems misleading
because it suggests we have one or the other, whereas it can also be
empty. I suggest: "{ | failures | ... }" to show the empty case.
Having a full example with retries in the doc is a good thing, and
illustrates in passing that running with a number of clients on a small
scale does not make much sense because of the contention on
tellers/branches. I'd wonder whether the number of tries is set too high,
though, ISTM that an application should give up before 100? I like that
the feature it is also limited by latency limit.
Minor editing:
"there're" -> "there are".
"the --time" -> "the --time option".
The overall English seems good, but I'm not a native speaker. As I already said, a native
speaker proofreading would be nice.
From a technical writing point of view, maybe the documentation could be improved a bit,
but I'm not a ease on that subject. Some comments:
"The latency for failed transactions and commands is not computed separately." is unclear,
please use a positive sentence to tell what is true instead of what is not and the reader
has to guess. Maybe: "The latency figures include failed transactions which have reached
the maximum number of tries or the transaction latency limit.".
"The main report contains the number of failed transactions if it is non-zero." ISTM that
this is a pain for scripts which would like to process these reports data, because the data
may or may not be there. I'm sure to write such scripts, which explains my concern:-)
"If the total number of retried transactions is non-zero…" should it rather be "not one",
because zero means unlimited retries?
The section describing the various type of errors that can occur is a good addition.
Option "--report-latencies" changed to "--report-per-commands": I'm fine with this change.
# FEATURES
--failures-detailed: I'm not convinced that this option should not always be on, but
this is not very important, so let it be.
--verbose-errors: I still think this is only for debugging, but let it be.
Copying variables: ISTM that we should not need to save the variables
states… no clearing, no copying should be needed. The restarted
transaction simply overrides the existing variables which is what the
previous version was doing anyway. The scripts should write their own
variables before using them, and if they don't then it is the user
problem. This is important for performance, because it means that after a
client has executed all scripts once the variable array is stable and does
not incur significant maintenance costs. The only thing that needs saving
for retry is the speudo-random generator state. This suggest simplifying
or removing "RetryState".
# CODE
The semantics of "cnt" is changed. Ok, the overall counters and their
relationships make sense, and it simplifies the reporting code. Good.
In readCommandResponse: ISTM that PGRES_NONFATAL_ERROR is not needed and
could be dealt with the default case. We are only interested in
serialization/deadlocks which are fatal errors?
doRetry: for consistency, given the assert, ISTM that it should return
false if duration has expired, by testing end_time or timer_exceeded.
checkTransactionStatus: this function does several things at once with 2
booleans, which make it not very clear to me. Maybe it would be clearer if
it would just return an enum (in trans, not in trans, conn error, other
error). Another reason to do that is that on connection error pgbench
could try to reconnect, which would be an interesting later extension, so
let's pave the way for that. Also, I do not think that the function
should print out a message, it should be the caller decision to do that.
verbose_errors: there is more or less repeated code under RETRY and
FAILURE, which should be factored out in a separate function. The
advanceConnectionFunction is long enough. Once this is done, there is no
need for a getLatencyUsed function.
I'd put cleaning up the pipeline in a function. I do not understand why
the pipeline mode is not exited in all cases, the code checks for the
pipeline status twice in a few lines. I'd put this cleanup in the sync
function as well, report to the caller (advanceConnectionState) if there
was an error, which would be managed there.
WAIT_ROLLBACK_RESULT: consumming results in a while could be a function to
avoid code repetition (there and in the "error:" label in
readCommandResponse). On the other hand, I'm not sure why the loop is
needed: we can only get there by submitting a "ROLLBACK" command, so there
should be only one result anyway?
report_per_command: please always count retries and failures of commands
even if they will not be reported in the end, the code will be simpler and
more efficient.
doLog: the format has changed, including a new string on failures which
replace the time field. Hmmm. Cannot say I like it much, but why not. ISTM
that the string could be shorten to "deadlock" or "serialization". ISTM
that the documentation example should include a line with a failure, to
make it clear what to expect.
I'm okay with always getting computing thread stats.
# COMMENTS
struct StatsData comment is helpful.
- "failed transactions" -> "unsuccessfully retried transactions"?
- 'cnt' decomposition: first term is field 'retried'? if so say it
explicitely?
"Complete the failed transaction" sounds strange: If it failed, it could
not complete? I'd suggest "Record a failed transaction".
# TESTS
I suggested to simplify the tests by using conditionals & sequences. You
reported that you got stuck. Hmmm.
I tried again my tests which worked fine when started with 2 clients,
otherwise they get stuck because the first client waits for the other one
which does not exists (the point is to generate deadlocks and other
errors). Maybe this is your issue?
Could you try with:
psql < deadlock_prep.sql
pgbench -t 4 -c 2 -f deadlock.sql
# note: each deadlock detection takes 1 second
psql < deadlock_prep.sql
pgbench -t 10 -c 2 -f serializable.sql
# very quick 50% serialization errors
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-02-18 14:45 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-02 09:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 82+ messages in thread
From: Yugo NAGATA @ 2022-02-18 14:45 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello Fabien,
Thank you so much for your review.
Sorry for the late reply. I've stopped working on it due to other
jobs but I came back again. I attached the updated patch. I would
appreciate it if you could review this again.
On Mon, 19 Jul 2021 20:04:23 +0200 (CEST)
Fabien COELHO <coelho@cri.ensmp.fr> wrote:
> # About pgbench error handling v15
>
> Patches apply cleanly. Compilation, global and local tests ok.
>
> - v15.1: refactoring is a definite improvement.
> Good, even if it is not very useful (see below).
Ok, we don't need to save variables in order to implement the
retry feature on pbench as you suggested. Well, should we completely
separate these two patches and should I fix v15.2 to not rely v15.1?
> While restructuring, maybe predefined variables could be make readonly
> so that a script which would update them would fail, which would be a
> good thing. Maybe this is probably material for an independent patch.
Yes, It shoule be for an independent patch.
> - v15.2: see detailed comments below
>
> # Doc
>
> Doc build is ok.
>
> ISTM that "number of tries" line would be better placed between the
> #threads and #transactions lines. What do you think?
Agreed. Fixed.
> Aggregate logging description: "{ failures | ... }" seems misleading
> because it suggests we have one or the other, whereas it can also be
> empty. I suggest: "{ | failures | ... }" to show the empty case.
The description is correct because either "failures" or "both of
serialization_failures and deadlock_failures" should appear in aggregate
logging. If "failures" was printed only when any transaction failed,
each line in aggregate logging could have different numbers of columns
and which would make it difficult to parse the results.
> I'd wonder whether the number of tries is set too high,
> though, ISTM that an application should give up before 100?
Indeed, max-tries=100 seems too high for practical system.
Also, I noticed that sum of latencies of each command (= 15.839 ms)
is significantly larger than the latency average (= 10.870 ms)
because "per commands" results in the documentation were fixed.
So, I retook a measurement on my machine for more accurate documentation. I
used max-tries=10.
> Minor editing:
>
> "there're" -> "there are".
>
> "the --time" -> "the --time option".
Fixed.
> "The latency for failed transactions and commands is not computed separately." is unclear,
> please use a positive sentence to tell what is true instead of what is not and the reader
> has to guess. Maybe: "The latency figures include failed transactions which have reached
> the maximum number of tries or the transaction latency limit.".
I'm not the original author of this description, but I guess this means "The latency is
measured only for successful transactions and commands but not for failed transactions
or commands.".
> "The main report contains the number of failed transactions if it is non-zero." ISTM that
> this is a pain for scripts which would like to process these reports data, because the data
> may or may not be there. I'm sure to write such scripts, which explains my concern:-)
I agree with you. I fixed the behavior to report the the number of failed transactions
always regardless with if it is non-zero or not.
> "If the total number of retried transactions is non-zero$B!D(B" should it rather be "not one",
> because zero means unlimited retries?
I guess that this means the actual number of retried transaction not the max-tries, so
"non-zero" was correct. However, for the same reason above, I fixed the behavior to
report the the retry statistics always regardeless with the actual retry numbers.
>
> # FEATURES
> Copying variables: ISTM that we should not need to save the variables
> states$B!D(B no clearing, no copying should be needed. The restarted
> transaction simply overrides the existing variables which is what the
> previous version was doing anyway. The scripts should write their own
> variables before using them, and if they don't then it is the user
> problem. This is important for performance, because it means that after a
> client has executed all scripts once the variable array is stable and does
> not incur significant maintenance costs. The only thing that needs saving
> for retry is the speudo-random generator state. This suggest simplifying
> or removing "RetryState".
Yes. The variables states is not necessary because we retry the
whole script. It was necessary in the initial patch because it
planned to retry one transaction included in the script. I removed
RetryState and copyVariables.
> # CODE
> In readCommandResponse: ISTM that PGRES_NONFATAL_ERROR is not needed and
> could be dealt with the default case. We are only interested in
> serialization/deadlocks which are fatal errors?
We need PGRES_NONFATAL_ERROR to save st->estatus. It is used outside
readCommandResponse to determine whether we should abort or not.
> doRetry: for consistency, given the assert, ISTM that it should return
> false if duration has expired, by testing end_time or timer_exceeded.
Ok. I fixed doRetry to check timer_exceeded again.
> checkTransactionStatus: this function does several things at once with 2
> booleans, which make it not very clear to me. Maybe it would be clearer if
> it would just return an enum (in trans, not in trans, conn error, other
> error). Another reason to do that is that on connection error pgbench
> could try to reconnect, which would be an interesting later extension, so
> let's pave the way for that. Also, I do not think that the function
> should print out a message, it should be the caller decision to do that.
OK. I added a new enum type TStatus and I fixed the function to return it.
Also, I changed the function name to getTransactionStatus because the
actual check is done by the caller.
> verbose_errors: there is more or less repeated code under RETRY and
> FAILURE, which should be factored out in a separate function. The
> advanceConnectionFunction is long enough. Once this is done, there is no
> need for a getLatencyUsed function.
OK. I made a function to print verbose error messages and removed the
getLatencyUsed function.
> I'd put cleaning up the pipeline in a function. I do not understand why
> the pipeline mode is not exited in all cases, the code checks for the
> pipeline status twice in a few lines. I'd put this cleanup in the sync
> function as well, report to the caller (advanceConnectionState) if there
> was an error, which would be managed there.
I fixed to exit the pipeline whenever we have an error in a pipeline mode.
Also, I added a PQpipelineSync call which was forgotten in the previous patch.
> WAIT_ROLLBACK_RESULT: consumming results in a while could be a function to
> avoid code repetition (there and in the "error:" label in
> readCommandResponse). On the other hand, I'm not sure why the loop is
> needed: we can only get there by submitting a "ROLLBACK" command, so there
> should be only one result anyway?
Right. We should receive just one PGRES_COMMAND_OK and null following it.
I eliminated the loop.
> report_per_command: please always count retries and failures of commands
> even if they will not be reported in the end, the code will be simpler and
> more efficient.
Ok. I fixed to count retries and failures of commands even if
report_per_command is false.
> doLog: the format has changed, including a new string on failures which
> replace the time field. Hmmm. Cannot say I like it much, but why not. ISTM
> that the string could be shorten to "deadlock" or "serialization". ISTM
> that the documentation example should include a line with a failure, to
> make it clear what to expect.
I fixed getResultString to return "deadlock" or "serialization" instead of
"deadlock_failure" or "serialization_failure". Also, I added an output
example to the documentation.
> I'm okay with always getting computing thread stats.
>
> # COMMENTS
>
> struct StatsData comment is helpful.
> - "failed transactions" -> "unsuccessfully retried transactions"?
This seems an accurate description. However, "failed transaction" is
short and simple, and it is used in several places, so instead of
replacing them I added the following statement to define it:
"failed transaction is defined as unsuccessfully retried transactions."
> - 'cnt' decomposition: first term is field 'retried'? if so say it
> explicitely?
No. 'retreid' includes unsuccessfully retreid transactions, but 'cnt'
includes only successfully retried transactions.
> "Complete the failed transaction" sounds strange: If it failed, it could
> not complete? I'd suggest "Record a failed transaction".
Sounds good. Fixed.
> # TESTS
>
> I suggested to simplify the tests by using conditionals & sequences. You
> reported that you got stuck. Hmmm.
>
> I tried again my tests which worked fine when started with 2 clients,
> otherwise they get stuck because the first client waits for the other one
> which does not exists (the point is to generate deadlocks and other
> errors). Maybe this is your issue?
That seems to be right. It got stuck when I used -T option rather than -t,
it was because, I guess, the number of transactions on each thread was
different.
> Could you try with:
>
> psql < deadlock_prep.sql
> pgbench -t 4 -c 2 -f deadlock.sql
> # note: each deadlock detection takes 1 second
>
> psql < deadlock_prep.sql
> pgbench -t 10 -c 2 -f serializable.sql
> # very quick 50% serialization errors
That works. However, it still gets hang when --max-tries = 2,
so maybe I would not think we can use it for testing the retry
feature....
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v16-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 81.7K) download
From 1762b812942a8d1b97ca07e269b0e41ff4189cce Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Mon, 7 Jun 2021 18:35:14 +0900
Subject: [PATCH v16 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-per-command, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --verbose-errors.
---
doc/src/sgml/ref/pgbench.sgml | 435 ++++++++-
src/bin/pgbench/pgbench.c | 900 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 215 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1458 insertions(+), 120 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index be1896fa99..d1f10f6835 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -56,8 +56,10 @@ scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
+maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+number of failed transactions: 0 (0.000%)
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
@@ -65,11 +67,18 @@ tps = 896.967014 (without initial connection time)
</screen>
The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ settings.
+ The seventh line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
+ The eighth line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the number of failed transactions due to
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
The last line reports the number of transactions per second.
</para>
@@ -531,6 +540,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information about retrying such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -597,23 +617,29 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-r</option></term>
- <term><option>--report-latencies</option></term>
+ <term><option>--report-per-command</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -741,6 +767,26 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -751,6 +797,38 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; moreover, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information about
+ retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--verbose-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -948,8 +1026,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts">
+ <title id="transactions-and-scripts-title">What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1022,6 +1100,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, he will be aborted.
+ </para>
</note>
<para>
@@ -2212,7 +2295,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2233,6 +2316,17 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization</literal> or
+ <literal>deadlock</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information).
</para>
<para>
@@ -2261,6 +2355,41 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
+ <para>
+ If <option>--failures-detailed</option> option is used, the type of
+ failure is reported in the <replaceable>time</replaceable> like this:
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 serialization 0 1499414498 84905 9
+2 0 serialization 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2276,7 +2405,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2290,7 +2419,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2298,21 +2436,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2324,13 +2466,44 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
+
+ <para>
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2342,29 +2515,67 @@ scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
+maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
-latency average = 10.870 ms
-latency stddev = 7.341 ms
-initial connection time = 30.954 ms
-tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+number of failed transactions: 0 (0.000%)
+number of transactions above the 50.0 ms latency limit: 1311/10000 (13.110 %)
+latency average = 28.488 ms
+latency stddev = 21.009 ms
+initial connection time = 69.068 ms
+tps = 346.224794 (without initial connection time)
+statement latencies in milliseconds and failures:
+ 0.012 0 \set aid random(1, 100000 * :scale)
+ 0.002 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.002 0 \set delta random(-5000, 5000)
+ 0.319 0 BEGIN;
+ 0.834 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.641 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 11.126 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 12.961 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.634 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.957 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+maximum number of tries: 10
+number of transactions per client: 1000
+number of transactions actually processed: 6317/10000
+number of failed transactions: 3683 (36.830%)
+number of transactions retried: 7667 (76.670%)
+total number of retries: 45339
+number of transactions above the 50.0 ms latency limit: 106/6317 (1.678 %)
+latency average = 17.016 ms
+latency stddev = 13.283 ms
+initial connection time = 45.017 ms
+tps = 186.792667 (without initial connection time)
+statement latencies in milliseconds, failures and retries:
+ 0.006 0 0 \set aid random(1, 100000 * :scale)
+ 0.001 0 0 \set bid random(1, 1 * :scale)
+ 0.001 0 0 \set tid random(1, 10 * :scale)
+ 0.001 0 0 \set delta random(-5000, 5000)
+ 0.385 0 0 BEGIN;
+ 0.773 0 1 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.624 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 1.098 320 3762 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.582 3363 41576 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.465 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.933 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2378,6 +2589,140 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries">
+ <title id="failures-and-retries-title">Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there are three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the client is not aborted. In such cases, the current
+ transaction is rolled back, which also includes setting the client variables
+ as they were before the run of this transaction (it is assumed that one
+ transaction script contains only one transaction; see
+ <xref linkend="transactions-and-scripts" endterm="transactions-and-scripts-title"/>
+ for more information). Transactions with serialization or deadlock errors are
+ repeated after rollbacks until they complete successfully or reach the maximum
+ number of tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of retries (specified by the <option>--latency-limit</option> option) / the end
+ of benchmark (specified by the <option>--time</option> option). If
+ the last trial run fails, this transaction will be reported as failed but
+ the client is not aborted and continue to work.
+ </para>
+
+ <note>
+ <para>
+ Without specifying the <option>--max-tries</option> option a transaction will
+ never be retried after a serialization or deadlock error because its default
+ values is 1. Use an unlimited number of tries (<literal>--max-tries=0</literal>)
+ and the <option>--latency-limit</option> option to limit only the maximum time
+ of tries. You can also use the <option>--time</option> option to limit the
+ benchmark duration under an unlimited number of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency is measured
+ only for successful transactions and commands but not for failed transactions
+ or commands.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions. If the
+ <option>--max-tries</option> option is not equal to 1, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--verbose-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3743e36dac..da7dad8f65 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -76,6 +76,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -275,9 +277,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 1;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -362,9 +389,66 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them) +
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction is defined as unsuccessfully retried transactions.
+ * It can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactions) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -375,6 +459,31 @@ typedef struct StatsData
*/
pg_time_usec_t epoch_shift;
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
+/*
+ * Transaction status at the end of a command.
+ */
+typedef enum TStatus
+{
+ TSTATUS_IDLE,
+ TSTATUS_IN_BLOCK,
+ TSTATUS_CONN_ERROR,
+ TSTATUS_OTHER_ERROR
+} TStatus;
+
/* Various random sequences are initialized from this one. */
static pg_prng_state base_random_sequence;
@@ -446,6 +555,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -494,8 +632,20 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ pg_prng_state random_state; /* random state */
+ uint32 tries; /* how many times have we already tried the
+ * current transaction? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -590,6 +740,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -602,6 +755,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -616,6 +771,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool verbose_errors = false; /* print verbose messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -753,15 +910,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-per-command report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --verbose-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1287,6 +1447,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1295,22 +1459,51 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 tries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ if (tries > 1)
{
- addToSimpleStats(&stats->latency, lat);
+ stats->retries += (tries - 1);
+ stats->retried++;
+ }
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ switch (estatus)
+ {
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
+
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2841,6 +3034,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2848,6 +3044,17 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ Assert(sql_script[st->use_file].commands[st->command]->type == SQL_COMMAND);
+ pg_log_info("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2955,6 +3162,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -2997,6 +3231,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3011,6 +3246,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3035,6 +3271,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3052,6 +3289,18 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (verbose_errors)
+ commandError(st, PQerrorMessage(st->con));
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3130,6 +3379,126 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries.
+ */
+ if (max_tries && st->tries >= max_tries)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+
+ /*
+ * We cannot retry the error if the benchmark duration is over.
+ */
+ if (timer_exceeded)
+ return false;
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Get the transaction status at the end of a command especially for
+ * checking if we are in a (failed) transaction block.
+ */
+static TStatus
+getTransactionStatus(PGconn *con)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ return TSTATUS_IDLE;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ return TSTATUS_IN_BLOCK;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ return TSTATUS_CONN_ERROR;
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return TSTATUS_OTHER_ERROR;
+ }
+
+ /* not reached */
+ Assert(false);
+ return TSTATUS_OTHER_ERROR;
+}
+
+/*
+ * Print verbose messages of an error
+ */
+static void
+printVerboseErrorMessages(CState *st, pg_time_usec_t *now, bool is_retry)
+{
+ PQExpBufferData buf;
+
+ initPQExpBuffer(&buf);
+
+ printfPQExpBuffer(&buf, "client %d ", st->id);
+ appendPQExpBuffer(&buf, "%s",
+ (is_retry ?
+ "repeats the transaction after the error" :
+ "ends the failed transaction"));
+ appendPQExpBuffer(&buf, " (try %d", st->tries);
+
+ /* Print max_tries if it is not unlimitted. */
+ if (max_tries)
+ appendPQExpBuffer(&buf, "/%d", max_tries);
+
+ /*
+ * If the latency limit is used, print a percentage of the current transaction
+ * latency from the latency limit.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ appendPQExpBuffer(&buf, ", %.3f%% of the maximum time of tries was used",
+ (100.0 * (*now - st->txn_scheduled) / latency_limit));
+ }
+ appendPQExpBuffer(&buf, ")\n");
+
+ pg_log_info("%s", buf.data);
+
+ termPQExpBuffer(&buf);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3159,6 +3528,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
for (;;)
{
Command *command;
+ PGresult *res;
+ TStatus tstatus;
switch (st->state)
{
@@ -3167,6 +3538,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->tries = 1;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3207,6 +3582,13 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember the
+ * random state: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->random_state = st->cs_func_rs;
+
/* record transaction start time */
st->txn_begin = now;
@@ -3363,6 +3745,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3508,6 +3892,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
@@ -3554,6 +3940,179 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
CSTATE_START_COMMAND : CSTATE_SKIP_COMMAND;
break;
+ /*
+ * Clean up after an error.
+ */
+ case CSTATE_ERROR:
+
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
+
+ /* Read and discard until a sync point in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ /* send a sync */
+ if (!PQpipelineSync(st->con))
+ {
+ pg_log_error("client %d aborted: failed to send a pipeline sync",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ /* receive PGRES_PIPELINE_SYNC and null fullowing it */
+ for(;;)
+ {
+ res = PQgetResult(st->con);
+ if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
+ {
+ PQclear(res);
+ res = PQgetResult(st->con);
+ Assert(res == NULL);
+ break;
+ }
+ PQclear(res);
+ }
+
+ /* exit pipline */
+ if (PQexitPipelineMode(st->con) != 1)
+ {
+ pg_log_error("client %d aborted: failed to exit pipeline mode for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
+
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+ tstatus = getTransactionStatus(st->con);
+ if (tstatus == TSTATUS_IN_BLOCK)
+ {
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else if (tstatus == TSTATUS_IDLE)
+ {
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ else
+ {
+ if (tstatus == TSTATUS_CONN_ERROR)
+ pg_log_error("perhaps the backend died while processing");
+
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ break;
+
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ /* null must be returned */
+ res = PQgetResult(st->con);
+ Assert(res == NULL);
+
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ break;
+
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /*
+ * Inform that the transaction will be retried after the error.
+ */
+ if (verbose_errors)
+ printVerboseErrorMessages(st, &now, true);
+
+ /* Count tries and retries */
+ st->tries++;
+ command->retries++;
+
+ /*
+ * Reset the random state as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->random_state;
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Record a failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (verbose_errors)
+ printVerboseErrorMessages(st, &now, false);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
/*
* End of transaction (end of script, really).
*/
@@ -3568,6 +4127,28 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
Assert(conditional_stack_empty(st->cstack));
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ tstatus = getTransactionStatus(st->con);
+ if (tstatus == TSTATUS_IN_BLOCK)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (tstatus != TSTATUS_IDLE)
+ {
+ if (tstatus == TSTATUS_CONN_ERROR)
+ pg_log_error("perhaps the backend died while processing");
+
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
if (is_connect)
{
pg_time_usec_t start = now;
@@ -3816,6 +4397,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3863,6 +4481,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3873,6 +4499,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3880,22 +4510,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->tries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->tries - 1);
fputc('\n', logfile);
}
}
@@ -3904,7 +4538,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3912,10 +4547,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3924,20 +4559,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->tries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3947,7 +4574,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->tries);
}
@@ -4806,6 +5434,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5474,7 +6104,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5501,23 +6133,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5531,8 +6170,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5541,6 +6180,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5600,9 +6245,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5615,39 +6261,67 @@ printResults(StatsData *total,
printf("query mode: %s\n", QUERYMODE[querymode]);
printf("number of clients: %d\n", nclients);
printf("number of threads: %d\n", nthreads);
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
if (duration <= 0)
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
+ }
+
+ printf("number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ if (total->deadlock_failures)
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("total number of retries: " INT64_FORMAT "\n", total->retries);
}
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f %%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5673,7 +6347,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5692,6 +6366,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5701,25 +6378,57 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ printf(" - number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
+
+ if (failures_detailed)
+ {
+ if (total->serialization_failures)
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ if (total->deadlock_failures)
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - total number of retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5727,10 +6436,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5810,7 +6528,7 @@ main(int argc, char **argv)
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
- {"report-latencies", no_argument, NULL, 'r'},
+ {"report-per-command", no_argument, NULL, 'r'},
{"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
@@ -5832,6 +6550,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"verbose-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -6185,6 +6906,28 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* verbose-errors */
+ benchmarking_option_set = true;
+ verbose_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6366,6 +7109,15 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (!latency_limit && duration <= 0)
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6578,6 +7330,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6724,7 +7480,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6813,7 +7570,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 8b03900f32..d07bf665a3 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,9 @@ use Config;
# start a pgbench specific server
my $node = PostgreSQL::Test::Cluster->new('main');
-$node->init;
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
$node->start;
# tablespace for testing, because partitioned tables cannot use pg_default
@@ -111,7 +113,8 @@ $node->pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1200,6 +1203,214 @@ $node->pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^0 \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+$node->pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+$node->pgbench(
+ "-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{number of transactions retried: 1\b},
+ qr{total number of retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+$node->pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --verbose-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{number of transactions retried: 1\b},
+ qr{total number of retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index acad19edd0..a5074c70d9 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -188,6 +188,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index 0bf877e895..5a94664989 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index b28189471c..fa53d86501 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
v16-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.2K) download
From 12eb5ff63cb92adeee86937104e2cbb274337289 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v16 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 163 +++++++++++++++++++++++---------------
1 file changed, 100 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f166a77e3a..3743e36dac 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -289,6 +289,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -306,6 +312,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -460,9 +484,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1398,39 +1420,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1562,21 +1584,37 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array.
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1588,23 +1626,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1613,12 +1645,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1636,12 +1669,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1656,12 +1689,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1720,7 +1754,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1741,7 +1775,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1756,12 +1790,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2629,7 +2664,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2699,7 +2734,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2730,7 +2765,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2791,7 +2826,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2843,7 +2878,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2854,7 +2889,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2901,7 +2936,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -2994,7 +3029,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3055,14 +3090,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3627,7 +3662,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3648,7 +3683,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3718,7 +3753,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3726,7 +3761,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -6020,7 +6055,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6348,19 +6383,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6398,11 +6433,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6411,28 +6446,30 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed = pg_prng_uint64(&base_random_sequence);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-02 09:03 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-02 11:41 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-02 09:03 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; ishii@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hi Yugo and Fabien,
It seems the patch is ready for committer except below. Do you guys
want to do more on below?
>> # TESTS
>>
>> I suggested to simplify the tests by using conditionals & sequences. You
>> reported that you got stuck. Hmmm.
>>
>> I tried again my tests which worked fine when started with 2 clients,
>> otherwise they get stuck because the first client waits for the other one
>> which does not exists (the point is to generate deadlocks and other
>> errors). Maybe this is your issue?
>
> That seems to be right. It got stuck when I used -T option rather than -t,
> it was because, I guess, the number of transactions on each thread was
> different.
>
>> Could you try with:
>>
>> psql < deadlock_prep.sql
>> pgbench -t 4 -c 2 -f deadlock.sql
>> # note: each deadlock detection takes 1 second
>>
>> psql < deadlock_prep.sql
>> pgbench -t 10 -c 2 -f serializable.sql
>> # very quick 50% serialization errors
>
> That works. However, it still gets hang when --max-tries = 2,
> so maybe I would not think we can use it for testing the retry
> feature....
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-02 09:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-02 11:41 ` Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 0 replies; 82+ messages in thread
From: Fabien COELHO @ 2022-03-02 11:41 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello Tatsuo-san,
> It seems the patch is ready for committer except below. Do you guys want
> to do more on below?
I'm planning a new review of this significant patch, possibly over the
next week-end, or the next.
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-12 14:54 ` Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2022-03-12 14:54 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello Yugo-san,
About Pgbench error handling v16:
This patch set needs a minor rebase because of 506035b0. Otherwise, patch
compiles, global and local "make check" are ok. Doc generation is ok.
This patch is in good shape, the code and comments are clear.
Some minor remarks below, including typos and a few small suggestions.
## About v16-1
This refactoring patch adds a struct for managing pgbench variables, instead of
mixing fields into the client state (CState) struct.
Patch compiles, global and local "make check" are both ok.
Although this patch is not necessary to add the feature, I'm fine with it as
improves pgbench source code readability.
## About v16-2
This last patch adds handling of serialization and deadlock errors to pgbench
transactions. This feature is desirable because it enlarge performance testing
options, and makes pgbench behave more like a database client application.
Possible future extension enabled by this patch include handling deconnections
errors by trying to reconnect, for instance.
The documentation is clear and well written, at least for my non-native speaker
eyes and ears.
English: "he will be aborted" -> "it will be aborted".
I'm fine with renaming --report-latencies to --report-per-command as the later
is clearer about what the options does.
I'm still not sure I like the "failure detailed" option, ISTM that the report
could be always detailed. That would remove some complexity and I do not think
that people executing a bench with error handling would mind having the details.
No big deal.
printVerboseErrorMessages: I'd make the buffer static and initialized only once
so that there is no significant malloc/free cycle involved when calling the function.
advanceConnectionState: I'd really prefer not to add new variables (res, status)
in the loop scope, and only declare them when actually needed in the state branches,
so as to avoid any unwanted interaction between states.
typo: "fullowing" -> "following"
Pipeline cleaning: the advance function is already soooo long, I'd put that in a
separate function and call it.
I think that the report should not remove data when they are 0, otherwise it makes
it harder to script around it (in failures_detailed on line 6284).
The test cover the different cases. I tried to suggest a simpler approach
in a previous round, but it seems not so simple to do so. They could be
simplified later, if possible.
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-03-19 18:23 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-19 18:23 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello Fabien,
On Sat, 12 Mar 2022 15:54:54 +0100 (CET)
Fabien COELHO <coelho@cri.ensmp.fr> wrote:
> Hello Yugo-san,
>
> About Pgbench error handling v16:
Thank you for your review! I attached the updated patches.
> This patch set needs a minor rebase because of 506035b0. Otherwise, patch
> compiles, global and local "make check" are ok. Doc generation is ok.
I rebased it.
> ## About v16-2
> English: "he will be aborted" -> "it will be aborted".
Fixed.
> I'm still not sure I like the "failure detailed" option, ISTM that the report
> could be always detailed. That would remove some complexity and I do not think
> that people executing a bench with error handling would mind having the details.
> No big deal.
I didn't change it because I think those who don't expect any failures using a
well designed script may not need details of failures. I think reporting such
details will be required only for benchmarks where any failures are expected.
> printVerboseErrorMessages: I'd make the buffer static and initialized only once
> so that there is no significant malloc/free cycle involved when calling the function.
OK. I fixed printVerboseErrorMessages to use a static variable.
> advanceConnectionState: I'd really prefer not to add new variables (res, status)
> in the loop scope, and only declare them when actually needed in the state branches,
> so as to avoid any unwanted interaction between states.
I fixed to declare the variables in the case statement blocks.
> typo: "fullowing" -> "following"
fixed.
> Pipeline cleaning: the advance function is already soooo long, I'd put that in a
> separate function and call it.
Ok. I made a new function "discardUntilSync" for the pipeline cleaning.
> I think that the report should not remove data when they are 0, otherwise it makes
> it harder to script around it (in failures_detailed on line 6284).
I fixed to report both serialization and deadlock failures always even when
they are 0.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v17-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.2K) download
From b4360d3c03013c86e1e62247f1c3c1378aacc38d Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v17 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 163 +++++++++++++++++++++++---------------
1 file changed, 100 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 000ffc4a5c..ab2c5dfc5f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -289,6 +289,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -306,6 +312,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -460,9 +484,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1398,39 +1420,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1562,21 +1584,37 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array.
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1588,23 +1626,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1613,12 +1645,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1636,12 +1669,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1656,12 +1689,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1720,7 +1754,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1741,7 +1775,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1756,12 +1790,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2629,7 +2664,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2699,7 +2734,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2730,7 +2765,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2791,7 +2826,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2843,7 +2878,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2854,7 +2889,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2901,7 +2936,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -2994,7 +3029,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3055,14 +3090,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3627,7 +3662,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3648,7 +3683,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3718,7 +3753,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3726,7 +3761,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -6020,7 +6055,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6348,19 +6383,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6398,11 +6433,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6411,28 +6446,30 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed = pg_prng_uint64(&base_random_sequence);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
v17-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 83.2K) download
From e5f646ef87641a67cc0003b5ec7c42c0e0beb6fe Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Mon, 7 Jun 2021 18:35:14 +0900
Subject: [PATCH v17 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-per-command, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --verbose-errors.
---
doc/src/sgml/ref/pgbench.sgml | 435 ++++++++-
src/bin/pgbench/pgbench.c | 965 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 215 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1499 insertions(+), 144 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index be1896fa99..49f57bda61 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -56,8 +56,10 @@ scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
+maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+number of failed transactions: 0 (0.000%)
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
@@ -65,11 +67,18 @@ tps = 896.967014 (without initial connection time)
</screen>
The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ settings.
+ The seventh line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
+ The eighth line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the number of failed transactions due to
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information).
The last line reports the number of transactions per second.
</para>
@@ -531,6 +540,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information about retrying such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -597,23 +617,29 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-r</option></term>
- <term><option>--report-latencies</option></term>
+ <term><option>--report-per-command</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -741,6 +767,26 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -751,6 +797,38 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; moreover, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information about
+ retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--verbose-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -948,8 +1026,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts">
+ <title id="transactions-and-scripts-title">What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1022,6 +1100,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, it will be aborted.
+ </para>
</note>
<para>
@@ -2212,7 +2295,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2233,6 +2316,17 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization</literal> or
+ <literal>deadlock</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
+ for more information).
</para>
<para>
@@ -2261,6 +2355,41 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
+ <para>
+ If <option>--failures-detailed</option> option is used, the type of
+ failure is reported in the <replaceable>time</replaceable> like this:
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 serialization 0 1499414498 84905 9
+2 0 serialization 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2276,7 +2405,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2290,7 +2419,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2298,21 +2436,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2324,13 +2466,44 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
+
+ <para>
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"
+ endterm="failures-and-retries-title"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2342,29 +2515,67 @@ scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
+maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
-latency average = 10.870 ms
-latency stddev = 7.341 ms
-initial connection time = 30.954 ms
-tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+number of failed transactions: 0 (0.000%)
+number of transactions above the 50.0 ms latency limit: 1311/10000 (13.110 %)
+latency average = 28.488 ms
+latency stddev = 21.009 ms
+initial connection time = 69.068 ms
+tps = 346.224794 (without initial connection time)
+statement latencies in milliseconds and failures:
+ 0.012 0 \set aid random(1, 100000 * :scale)
+ 0.002 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.002 0 \set delta random(-5000, 5000)
+ 0.319 0 BEGIN;
+ 0.834 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.641 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 11.126 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 12.961 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.634 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.957 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+maximum number of tries: 10
+number of transactions per client: 1000
+number of transactions actually processed: 6317/10000
+number of failed transactions: 3683 (36.830%)
+number of transactions retried: 7667 (76.670%)
+total number of retries: 45339
+number of transactions above the 50.0 ms latency limit: 106/6317 (1.678 %)
+latency average = 17.016 ms
+latency stddev = 13.283 ms
+initial connection time = 45.017 ms
+tps = 186.792667 (without initial connection time)
+statement latencies in milliseconds, failures and retries:
+ 0.006 0 0 \set aid random(1, 100000 * :scale)
+ 0.001 0 0 \set bid random(1, 1 * :scale)
+ 0.001 0 0 \set tid random(1, 10 * :scale)
+ 0.001 0 0 \set delta random(-5000, 5000)
+ 0.385 0 0 BEGIN;
+ 0.773 0 1 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.624 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 1.098 320 3762 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.582 3363 41576 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.465 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.933 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2378,6 +2589,140 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries">
+ <title id="failures-and-retries-title">Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there are three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the client is not aborted. In such cases, the current
+ transaction is rolled back, which also includes setting the client variables
+ as they were before the run of this transaction (it is assumed that one
+ transaction script contains only one transaction; see
+ <xref linkend="transactions-and-scripts" endterm="transactions-and-scripts-title"/>
+ for more information). Transactions with serialization or deadlock errors are
+ repeated after rollbacks until they complete successfully or reach the maximum
+ number of tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of retries (specified by the <option>--latency-limit</option> option) / the end
+ of benchmark (specified by the <option>--time</option> option). If
+ the last trial run fails, this transaction will be reported as failed but
+ the client is not aborted and continue to work.
+ </para>
+
+ <note>
+ <para>
+ Without specifying the <option>--max-tries</option> option a transaction will
+ never be retried after a serialization or deadlock error because its default
+ values is 1. Use an unlimited number of tries (<literal>--max-tries=0</literal>)
+ and the <option>--latency-limit</option> option to limit only the maximum time
+ of tries. You can also use the <option>--time</option> option to limit the
+ benchmark duration under an unlimited number of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency is measured
+ only for successful transactions and commands but not for failed transactions
+ or commands.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions. If the
+ <option>--max-tries</option> option is not equal to 1, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--verbose-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ab2c5dfc5f..7080d2a795 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -76,6 +76,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -275,9 +277,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 1;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -362,9 +389,66 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them) +
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction is defined as unsuccessfully retried transactions.
+ * It can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactions) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -375,6 +459,31 @@ typedef struct StatsData
*/
pg_time_usec_t epoch_shift;
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
+/*
+ * Transaction status at the end of a command.
+ */
+typedef enum TStatus
+{
+ TSTATUS_IDLE,
+ TSTATUS_IN_BLOCK,
+ TSTATUS_CONN_ERROR,
+ TSTATUS_OTHER_ERROR
+} TStatus;
+
/* Various random sequences are initialized from this one. */
static pg_prng_state base_random_sequence;
@@ -446,6 +555,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -494,8 +632,20 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ pg_prng_state random_state; /* random state */
+ uint32 tries; /* how many times have we already tried the
+ * current transaction? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -590,6 +740,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -602,6 +755,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -616,6 +771,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool verbose_errors = false; /* print verbose messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -753,15 +910,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-per-command report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --verbose-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1287,6 +1447,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1295,22 +1459,51 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 tries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ if (tries > 1)
+ {
+ stats->retries += (tries - 1);
+ stats->retried++;
+ }
+
+ switch (estatus)
{
- addToSimpleStats(&stats->latency, lat);
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2841,6 +3034,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2848,6 +3044,17 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ Assert(sql_script[st->use_file].commands[st->command]->type == SQL_COMMAND);
+ pg_log_info("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2955,6 +3162,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -2997,6 +3231,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3011,6 +3246,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3035,6 +3271,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3052,6 +3289,18 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (verbose_errors)
+ commandError(st, PQerrorMessage(st->con));
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3130,6 +3379,165 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries.
+ */
+ if (max_tries && st->tries >= max_tries)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+
+ /*
+ * We cannot retry the error if the benchmark duration is over.
+ */
+ if (timer_exceeded)
+ return false;
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Read results and discard it until a sync point.
+ */
+static int
+discardUntilSync(CState *st)
+{
+ /* send a sync */
+ if (!PQpipelineSync(st->con))
+ {
+ pg_log_error("client %d aborted: failed to send a pipeline sync",
+ st->id);
+ return 0;
+ }
+
+ /* receive PGRES_PIPELINE_SYNC and null following it */
+ for(;;)
+ {
+ PGresult *res = PQgetResult(st->con);
+ if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
+ {
+ PQclear(res);
+ res = PQgetResult(st->con);
+ Assert(res == NULL);
+ break;
+ }
+ PQclear(res);
+ }
+
+ /* exit pipline */
+ if (PQexitPipelineMode(st->con) != 1)
+ {
+ pg_log_error("client %d aborted: failed to exit pipeline mode for rolling back the failed transaction",
+ st->id);
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Get the transaction status at the end of a command especially for
+ * checking if we are in a (failed) transaction block.
+ */
+static TStatus
+getTransactionStatus(PGconn *con)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ return TSTATUS_IDLE;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ return TSTATUS_IN_BLOCK;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ return TSTATUS_CONN_ERROR;
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return TSTATUS_OTHER_ERROR;
+ }
+
+ /* not reached */
+ Assert(false);
+ return TSTATUS_OTHER_ERROR;
+}
+
+/*
+ * Print verbose messages of an error
+ */
+static void
+printVerboseErrorMessages(CState *st, pg_time_usec_t *now, bool is_retry)
+{
+ static PQExpBuffer buf = NULL;
+
+ if (buf == NULL)
+ buf = createPQExpBuffer();
+ else
+ resetPQExpBuffer(buf);
+
+ printfPQExpBuffer(buf, "client %d ", st->id);
+ appendPQExpBuffer(buf, "%s",
+ (is_retry ?
+ "repeats the transaction after the error" :
+ "ends the failed transaction"));
+ appendPQExpBuffer(buf, " (try %d", st->tries);
+
+ /* Print max_tries if it is not unlimitted. */
+ if (max_tries)
+ appendPQExpBuffer(buf, "/%d", max_tries);
+
+ /*
+ * If the latency limit is used, print a percentage of the current transaction
+ * latency from the latency limit.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ appendPQExpBuffer(buf, ", %.3f%% of the maximum time of tries was used",
+ (100.0 * (*now - st->txn_scheduled) / latency_limit));
+ }
+ appendPQExpBuffer(buf, ")\n");
+
+ pg_log_info("%s", buf->data);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3167,6 +3575,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->tries = 1;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3207,6 +3619,13 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember the
+ * random state: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->random_state = st->cs_func_rs;
+
/* record transaction start time */
st->txn_begin = now;
@@ -3363,6 +3782,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3508,6 +3929,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
@@ -3555,44 +3978,223 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
break;
/*
- * End of transaction (end of script, really).
+ * Clean up after an error.
*/
- case CSTATE_END_TX:
+ case CSTATE_ERROR:
+ {
+ TStatus tstatus;
- /* transaction finished: calculate latency and do log */
- processXactStats(thread, st, &now, false, agg);
+ Assert(st->estatus != ESTATUS_NO_ERROR);
- /*
- * missing \endif... cannot happen if CheckConditional was
- * okay
- */
- Assert(conditional_stack_empty(st->cstack));
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
- if (is_connect)
- {
- pg_time_usec_t start = now;
+ /* Read and discard until a sync point in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (!discardUntilSync(st))
+ {
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
- pg_time_now_lazy(&start);
- finishCon(st);
- now = pg_time_now();
- thread->conn_duration += now - start;
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+ tstatus = getTransactionStatus(st->con);
+ if (tstatus == TSTATUS_IN_BLOCK)
+ {
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else if (tstatus == TSTATUS_IDLE)
+ {
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ else
+ {
+ if (tstatus == TSTATUS_CONN_ERROR)
+ pg_log_error("perhaps the backend died while processing");
+
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ break;
}
- if ((st->cnt >= nxacts && duration <= 0) || timer_exceeded)
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
{
- /* script completed */
- st->state = CSTATE_FINISHED;
+ PGresult *res;
+
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ /* null must be returned */
+ res = PQgetResult(st->con);
+ Assert(res == NULL);
+
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
break;
}
- /* next transaction (script) */
- st->state = CSTATE_CHOOSE_SCRIPT;
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
/*
- * Ensure that we always return on this point, so as to avoid
- * an infinite loop if the script only contains meta commands.
+ * Inform that the transaction will be retried after the error.
*/
- return;
+ if (verbose_errors)
+ printVerboseErrorMessages(st, &now, true);
+
+ /* Count tries and retries */
+ st->tries++;
+ command->retries++;
+
+ /*
+ * Reset the random state as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->random_state;
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Record a failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (verbose_errors)
+ printVerboseErrorMessages(st, &now, false);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
+ /*
+ * End of transaction (end of script, really).
+ */
+ case CSTATE_END_TX:
+ {
+ TStatus tstatus;
+
+ /* transaction finished: calculate latency and do log */
+ processXactStats(thread, st, &now, false, agg);
+
+ /*
+ * missing \endif... cannot happen if CheckConditional was
+ * okay
+ */
+ Assert(conditional_stack_empty(st->cstack));
+
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ tstatus = getTransactionStatus(st->con);
+ if (tstatus == TSTATUS_IN_BLOCK)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (tstatus != TSTATUS_IDLE)
+ {
+ if (tstatus == TSTATUS_CONN_ERROR)
+ pg_log_error("perhaps the backend died while processing");
+
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (is_connect)
+ {
+ pg_time_usec_t start = now;
+
+ pg_time_now_lazy(&start);
+ finishCon(st);
+ now = pg_time_now();
+ thread->conn_duration += now - start;
+ }
+
+ if ((st->cnt >= nxacts && duration <= 0) || timer_exceeded)
+ {
+ /* script completed */
+ st->state = CSTATE_FINISHED;
+ break;
+ }
+
+ /* next transaction (script) */
+ st->state = CSTATE_CHOOSE_SCRIPT;
+
+ /*
+ * Ensure that we always return on this point, so as to avoid
+ * an infinite loop if the script only contains meta commands.
+ */
+ return;
+ }
/*
* Final states. Close the connection if it's still open.
@@ -3816,6 +4418,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3863,6 +4502,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3873,6 +4520,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3880,22 +4531,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->tries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->tries - 1);
fputc('\n', logfile);
}
}
@@ -3904,7 +4559,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3912,10 +4568,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3924,20 +4580,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->tries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3947,7 +4595,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->tries);
}
@@ -4806,6 +5455,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5474,7 +6125,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5501,23 +6154,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5531,8 +6191,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5541,6 +6201,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5600,9 +6266,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5615,39 +6282,65 @@ printResults(StatsData *total,
printf("query mode: %s\n", QUERYMODE[querymode]);
printf("number of clients: %d\n", nclients);
printf("number of threads: %d\n", nthreads);
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
if (duration <= 0)
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
+ }
+
+ printf("number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("total number of retries: " INT64_FORMAT "\n", total->retries);
}
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f%%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5673,7 +6366,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5692,6 +6385,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5701,25 +6397,55 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
+
+ printf(" - number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
+
+ if (failures_detailed)
+ {
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - total number of retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5727,10 +6453,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5810,7 +6545,7 @@ main(int argc, char **argv)
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
- {"report-latencies", no_argument, NULL, 'r'},
+ {"report-per-command", no_argument, NULL, 'r'},
{"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
@@ -5832,6 +6567,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"verbose-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -6185,6 +6923,28 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* verbose-errors */
+ benchmarking_option_set = true;
+ verbose_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6366,6 +7126,15 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (!latency_limit && duration <= 0)
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6578,6 +7347,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6724,7 +7497,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6813,7 +7587,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index f1341092fe..d173ceae7a 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,9 @@ use Config;
# start a pgbench specific server
my $node = PostgreSQL::Test::Cluster->new('main');
-$node->init;
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
$node->start;
# tablespace for testing, because partitioned tables cannot use pg_default
@@ -109,7 +111,8 @@ $node->pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1198,6 +1201,214 @@ $node->pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^0 \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+$node->pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+$node->pgbench(
+ "-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{number of transactions retried: 1\b},
+ qr{total number of retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+$node->pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --verbose-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{number of transactions retried: 1\b},
+ qr{total number of retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index acad19edd0..a5074c70d9 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -188,6 +188,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index 0bf877e895..5a94664989 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index b28189471c..fa53d86501 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-20 00:52 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 02:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 2 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-20 00:52 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; ishii@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hi Yugo,
I have looked into the patch and I noticed that <xref
linkend=... endterm=...> is used in pgbench.sgml. e.g.
<xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
AFAIK this is the only place where "endterm" is used. In other places
"link" tag is used instead:
<link linkend="failures-and-retries">Failures and Serialization/Deadlock Retries</link>
Note that the rendered result is identical. Do we want to use the link tag as well?
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-20 02:53 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-20 02:53 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hi Yugo,
I tested with serialization error scenario by setting:
default_transaction_isolation = 'repeatable read'
The result was:
$ pgbench -t 10 -c 10 --max-tries=10 test
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
maximum number of tries: 10
number of transactions per client: 10
number of transactions actually processed: 100/100
number of failed transactions: 0 (0.000%)
number of transactions retried: 35 (35.000%)
total number of retries: 74
latency average = 5.306 ms
initial connection time = 15.575 ms
tps = 1884.516810 (without initial connection time)
I had hard time to understand what those numbers mean:
number of transactions retried: 35 (35.000%)
total number of retries: 74
It seems "total number of retries" matches with the number of ERRORs
reported in PostgreSQL. Good. What I am not sure is "number of
transactions retried". What does this mean?
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 02:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-20 07:11 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:10 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-20 07:11 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> Hi Yugo,
>
> I tested with serialization error scenario by setting:
> default_transaction_isolation = 'repeatable read'
> The result was:
>
> $ pgbench -t 10 -c 10 --max-tries=10 test
> transaction type: <builtin: TPC-B (sort of)>
> scaling factor: 10
> query mode: simple
> number of clients: 10
> number of threads: 1
> maximum number of tries: 10
> number of transactions per client: 10
> number of transactions actually processed: 100/100
> number of failed transactions: 0 (0.000%)
> number of transactions retried: 35 (35.000%)
> total number of retries: 74
> latency average = 5.306 ms
> initial connection time = 15.575 ms
> tps = 1884.516810 (without initial connection time)
>
> I had hard time to understand what those numbers mean:
> number of transactions retried: 35 (35.000%)
> total number of retries: 74
>
> It seems "total number of retries" matches with the number of ERRORs
> reported in PostgreSQL. Good. What I am not sure is "number of
> transactions retried". What does this mean?
Oh, ok. I see it now. It turned out that "number of transactions
retried" does not actually means the number of transactions
rtried. Suppose pgbench exectutes following in a session:
BEGIN; -- transaction A starts
:
(ERROR)
ROLLBACK; -- transaction A aborts
(retry)
BEGIN; -- transaction B starts
:
(ERROR)
ROLLBACK; -- transaction B aborts
(retry)
BEGIN; -- transaction C starts
:
END; -- finally succeeds
In this case "total number of retries:" = 2 and "number of
transactions retried:" = 1. In this patch transactions A, B and C are
regarded as "same" transaction, so the retried transaction count
becomes 1. But it's confusing to use the language "transaction" here
because A, B and C are different transactions. I would think it's
better to use different language instead of "transaction", something
like "cycle"? i.e.
number of cycles retried: 35 (35.000%)
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 02:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-22 00:10 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 00:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-22 00:10 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Sun, 20 Mar 2022 16:11:43 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> > Hi Yugo,
> >
> > I tested with serialization error scenario by setting:
> > default_transaction_isolation = 'repeatable read'
> > The result was:
> >
> > $ pgbench -t 10 -c 10 --max-tries=10 test
> > transaction type: <builtin: TPC-B (sort of)>
> > scaling factor: 10
> > query mode: simple
> > number of clients: 10
> > number of threads: 1
> > maximum number of tries: 10
> > number of transactions per client: 10
> > number of transactions actually processed: 100/100
> > number of failed transactions: 0 (0.000%)
> > number of transactions retried: 35 (35.000%)
> > total number of retries: 74
> > latency average = 5.306 ms
> > initial connection time = 15.575 ms
> > tps = 1884.516810 (without initial connection time)
> >
> > I had hard time to understand what those numbers mean:
> > number of transactions retried: 35 (35.000%)
> > total number of retries: 74
> >
> > It seems "total number of retries" matches with the number of ERRORs
> > reported in PostgreSQL. Good. What I am not sure is "number of
> > transactions retried". What does this mean?
>
> Oh, ok. I see it now. It turned out that "number of transactions
> retried" does not actually means the number of transactions
> rtried. Suppose pgbench exectutes following in a session:
>
> BEGIN; -- transaction A starts
> :
> (ERROR)
> ROLLBACK; -- transaction A aborts
>
> (retry)
>
> BEGIN; -- transaction B starts
> :
> (ERROR)
> ROLLBACK; -- transaction B aborts
>
> (retry)
>
> BEGIN; -- transaction C starts
> :
> END; -- finally succeeds
>
> In this case "total number of retries:" = 2 and "number of
> transactions retried:" = 1. In this patch transactions A, B and C are
> regarded as "same" transaction, so the retried transaction count
> becomes 1. But it's confusing to use the language "transaction" here
> because A, B and C are different transactions. I would think it's
> better to use different language instead of "transaction", something
> like "cycle"? i.e.
>
> number of cycles retried: 35 (35.000%)
In the original patch by Marina Polyakova it was "number of retried",
but I changed it to "number of transactions retried" is because I felt
it was confusing with "number of retries". I chose the word "transaction"
because a transaction ends in any one of successful commit , skipped, or
failure, after possible retries.
Well, I agree with that it is somewhat confusing wording. If we can find
nice word to resolve the confusion, I don't mind if we change the word.
Maybe, we can use "executions" as well as "cycles". However, I am not sure
that the situation is improved by using such word because what such word
exactly means seems to be still unclear for users.
Another idea is instead reporting only "the number of successfully
retried transactions" that does not include "failed transactions",
that is, transactions failed after retries, like this;
number of transactions actually processed: 100/100
number of failed transactions: 0 (0.000%)
number of successfully retried transactions: 35 (35.000%)
total number of retries: 74
The meaning is clear and there seems to be no confusion.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 02:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:10 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-22 00:46 ` Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-22 00:46 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: ishii@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> On Sun, 20 Mar 2022 16:11:43 +0900 (JST)
> Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
>
>> > Hi Yugo,
>> >
>> > I tested with serialization error scenario by setting:
>> > default_transaction_isolation = 'repeatable read'
>> > The result was:
>> >
>> > $ pgbench -t 10 -c 10 --max-tries=10 test
>> > transaction type: <builtin: TPC-B (sort of)>
>> > scaling factor: 10
>> > query mode: simple
>> > number of clients: 10
>> > number of threads: 1
>> > maximum number of tries: 10
>> > number of transactions per client: 10
>> > number of transactions actually processed: 100/100
>> > number of failed transactions: 0 (0.000%)
>> > number of transactions retried: 35 (35.000%)
>> > total number of retries: 74
>> > latency average = 5.306 ms
>> > initial connection time = 15.575 ms
>> > tps = 1884.516810 (without initial connection time)
>> >
>> > I had hard time to understand what those numbers mean:
>> > number of transactions retried: 35 (35.000%)
>> > total number of retries: 74
>> >
>> > It seems "total number of retries" matches with the number of ERRORs
>> > reported in PostgreSQL. Good. What I am not sure is "number of
>> > transactions retried". What does this mean?
>>
>> Oh, ok. I see it now. It turned out that "number of transactions
>> retried" does not actually means the number of transactions
>> rtried. Suppose pgbench exectutes following in a session:
>>
>> BEGIN; -- transaction A starts
>> :
>> (ERROR)
>> ROLLBACK; -- transaction A aborts
>>
>> (retry)
>>
>> BEGIN; -- transaction B starts
>> :
>> (ERROR)
>> ROLLBACK; -- transaction B aborts
>>
>> (retry)
>>
>> BEGIN; -- transaction C starts
>> :
>> END; -- finally succeeds
>>
>> In this case "total number of retries:" = 2 and "number of
>> transactions retried:" = 1. In this patch transactions A, B and C are
>> regarded as "same" transaction, so the retried transaction count
>> becomes 1. But it's confusing to use the language "transaction" here
>> because A, B and C are different transactions. I would think it's
>> better to use different language instead of "transaction", something
>> like "cycle"? i.e.
>>
>> number of cycles retried: 35 (35.000%)
I realized that the same argument can be applied even to "number of
transactions actually processed" because with the retry feature,
"transaction" could comprise multiple transactions.
But if we go forward and replace those "transactions" with "cycles"
(or whatever) altogether, probably it could bring enough confusion to
users who have been using pgbench. Probably we should give up the
language changing and redefine "transaction" when the retry feature is
enabled instead like "when retry feature is enabled, each transaction
can be consisted of multiple transactions retried."
> In the original patch by Marina Polyakova it was "number of retried",
> but I changed it to "number of transactions retried" is because I felt
> it was confusing with "number of retries". I chose the word "transaction"
> because a transaction ends in any one of successful commit , skipped, or
> failure, after possible retries.
Ok.
> Well, I agree with that it is somewhat confusing wording. If we can find
> nice word to resolve the confusion, I don't mind if we change the word.
> Maybe, we can use "executions" as well as "cycles". However, I am not sure
> that the situation is improved by using such word because what such word
> exactly means seems to be still unclear for users.
>
> Another idea is instead reporting only "the number of successfully
> retried transactions" that does not include "failed transactions",
> that is, transactions failed after retries, like this;
>
> number of transactions actually processed: 100/100
> number of failed transactions: 0 (0.000%)
> number of successfully retried transactions: 35 (35.000%)
> total number of retries: 74
>
> The meaning is clear and there seems to be no confusion.
Thank you for the suggestion. But I think it would better to leave it
as it is because of the reason I mentioned above.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-22 00:08 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-22 00:08 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hi Ishii-san,
On Sun, 20 Mar 2022 09:52:06 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> Hi Yugo,
>
> I have looked into the patch and I noticed that <xref
> linkend=... endterm=...> is used in pgbench.sgml. e.g.
>
> <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
>
> AFAIK this is the only place where "endterm" is used. In other places
> "link" tag is used instead:
Thank you for pointing out it.
I've checked other places using <xref/> referring to <refsect2>, and found
that "xreflabel"s are used in such <refsect2> tags. So, I'll fix it
in this style.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-22 08:27 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-22 08:27 UTC (permalink / raw)
To: Yugo NAGATA <nagata@sraoss.co.jp>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Tue, 22 Mar 2022 09:08:15 +0900
Yugo NAGATA <nagata@sraoss.co.jp> wrote:
> Hi Ishii-san,
>
> On Sun, 20 Mar 2022 09:52:06 +0900 (JST)
> Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
>
> > Hi Yugo,
> >
> > I have looked into the patch and I noticed that <xref
> > linkend=... endterm=...> is used in pgbench.sgml. e.g.
> >
> > <xref linkend="failures-and-retries" endterm="failures-and-retries-title"/>
> >
> > AFAIK this is the only place where "endterm" is used. In other places
> > "link" tag is used instead:
>
> Thank you for pointing out it.
>
> I've checked other places using <xref/> referring to <refsect2>, and found
> that "xreflabel"s are used in such <refsect2> tags. So, I'll fix it
> in this style.
I attached the updated patch. I also fixed the following paragraph which I had
forgotten to fix in the previous patch.
The first seven lines report some of the most important parameter settings.
The sixth line reports the maximum number of tries for transactions with
serialization or deadlock errors
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments: download all (2 files, .tar.gz)
v18-0001-Pgbench-errors-use-the-Variables-structure-for-c.patch (text/x-diff, 15.2K) download
From b9f993e81836c4379478809da0e690023c319038 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 26 May 2021 16:58:36 +0900
Subject: [PATCH v18 1/2] Pgbench errors: use the Variables structure for
client variables
This is most important when it is used to reset client variables during the
repeating of transactions after serialization/deadlock failures.
Don't allocate Variable structs one by one. Instead, add a constant margin each
time it overflows.
---
src/bin/pgbench/pgbench.c | 163 +++++++++++++++++++++++---------------
1 file changed, 100 insertions(+), 63 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 000ffc4a5c..ab2c5dfc5f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -289,6 +289,12 @@ const char *progname;
volatile bool timer_exceeded = false; /* flag from signal handler */
+/*
+ * We don't want to allocate variables one by one; for efficiency, add a
+ * constant margin each time it overflows.
+ */
+#define VARIABLES_ALLOC_MARGIN 8
+
/*
* Variable definitions.
*
@@ -306,6 +312,24 @@ typedef struct
PgBenchValue value; /* actual variable's value */
} Variable;
+/*
+ * Data structure for client variables.
+ */
+typedef struct
+{
+ Variable *vars; /* array of variable definitions */
+ int nvars; /* number of variables */
+
+ /*
+ * The maximum number of variables that we can currently store in 'vars'
+ * without having to reallocate more space. We must always have max_vars >=
+ * nvars.
+ */
+ int max_vars;
+
+ bool vars_sorted; /* are variables sorted by name? */
+} Variables;
+
#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
@@ -460,9 +484,7 @@ typedef struct
int command; /* command number in script */
/* client variables */
- Variable *variables; /* array of variable definitions */
- int nvariables; /* number of variables */
- bool vars_sorted; /* are variables sorted by name? */
+ Variables variables;
/* various times about current transaction in microseconds */
pg_time_usec_t txn_scheduled; /* scheduled start time of transaction */
@@ -1398,39 +1420,39 @@ compareVariableNames(const void *v1, const void *v2)
/* Locate a variable by name; returns NULL if unknown */
static Variable *
-lookupVariable(CState *st, char *name)
+lookupVariable(Variables *variables, char *name)
{
Variable key;
/* On some versions of Solaris, bsearch of zero items dumps core */
- if (st->nvariables <= 0)
+ if (variables->nvars <= 0)
return NULL;
/* Sort if we have to */
- if (!st->vars_sorted)
+ if (!variables->vars_sorted)
{
- qsort((void *) st->variables, st->nvariables, sizeof(Variable),
+ qsort((void *) variables->vars, variables->nvars, sizeof(Variable),
compareVariableNames);
- st->vars_sorted = true;
+ variables->vars_sorted = true;
}
/* Now we can search */
key.name = name;
return (Variable *) bsearch((void *) &key,
- (void *) st->variables,
- st->nvariables,
+ (void *) variables->vars,
+ variables->nvars,
sizeof(Variable),
compareVariableNames);
}
/* Get the value of a variable, in string form; returns NULL if unknown */
static char *
-getVariable(CState *st, char *name)
+getVariable(Variables *variables, char *name)
{
Variable *var;
char stringform[64];
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
return NULL; /* not found */
@@ -1562,21 +1584,37 @@ valid_variable_name(const char *name)
return true;
}
+/*
+ * Make sure there is enough space for 'needed' more variable in the variables
+ * array.
+ */
+static void
+enlargeVariables(Variables *variables, int needed)
+{
+ /* total number of variables required now */
+ needed += variables->nvars;
+
+ if (variables->max_vars < needed)
+ {
+ variables->max_vars = needed + VARIABLES_ALLOC_MARGIN;
+ variables->vars = (Variable *)
+ pg_realloc(variables->vars, variables->max_vars * sizeof(Variable));
+ }
+}
+
/*
* Lookup a variable by name, creating it if need be.
* Caller is expected to assign a value to the variable.
* Returns NULL on failure (bad name).
*/
static Variable *
-lookupCreateVariable(CState *st, const char *context, char *name)
+lookupCreateVariable(Variables *variables, const char *context, char *name)
{
Variable *var;
- var = lookupVariable(st, name);
+ var = lookupVariable(variables, name);
if (var == NULL)
{
- Variable *newvars;
-
/*
* Check for the name only when declaring a new variable to avoid
* overhead.
@@ -1588,23 +1626,17 @@ lookupCreateVariable(CState *st, const char *context, char *name)
}
/* Create variable at the end of the array */
- if (st->variables)
- newvars = (Variable *) pg_realloc(st->variables,
- (st->nvariables + 1) * sizeof(Variable));
- else
- newvars = (Variable *) pg_malloc(sizeof(Variable));
-
- st->variables = newvars;
+ enlargeVariables(variables, 1);
- var = &newvars[st->nvariables];
+ var = &(variables->vars[variables->nvars]);
var->name = pg_strdup(name);
var->svalue = NULL;
/* caller is expected to initialize remaining fields */
- st->nvariables++;
+ variables->nvars++;
/* we don't re-sort the array till we have to */
- st->vars_sorted = false;
+ variables->vars_sorted = false;
}
return var;
@@ -1613,12 +1645,13 @@ lookupCreateVariable(CState *st, const char *context, char *name)
/* Assign a string value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariable(CState *st, const char *context, char *name, const char *value)
+putVariable(Variables *variables, const char *context, char *name,
+ const char *value)
{
Variable *var;
char *val;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1636,12 +1669,12 @@ putVariable(CState *st, const char *context, char *name, const char *value)
/* Assign a value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableValue(CState *st, const char *context, char *name,
+putVariableValue(Variables *variables, const char *context, char *name,
const PgBenchValue *value)
{
Variable *var;
- var = lookupCreateVariable(st, context, name);
+ var = lookupCreateVariable(variables, context, name);
if (!var)
return false;
@@ -1656,12 +1689,13 @@ putVariableValue(CState *st, const char *context, char *name,
/* Assign an integer value to a variable, creating it if need be */
/* Returns false on failure (bad name) */
static bool
-putVariableInt(CState *st, const char *context, char *name, int64 value)
+putVariableInt(Variables *variables, const char *context, char *name,
+ int64 value)
{
PgBenchValue val;
setIntValue(&val, value);
- return putVariableValue(st, context, name, &val);
+ return putVariableValue(variables, context, name, &val);
}
/*
@@ -1720,7 +1754,7 @@ replaceVariable(char **sql, char *param, int len, char *value)
}
static char *
-assignVariables(CState *st, char *sql)
+assignVariables(Variables *variables, char *sql)
{
char *p,
*name,
@@ -1741,7 +1775,7 @@ assignVariables(CState *st, char *sql)
continue;
}
- val = getVariable(st, name);
+ val = getVariable(variables, name);
free(name);
if (val == NULL)
{
@@ -1756,12 +1790,13 @@ assignVariables(CState *st, char *sql)
}
static void
-getQueryParams(CState *st, const Command *command, const char **params)
+getQueryParams(Variables *variables, const Command *command,
+ const char **params)
{
int i;
for (i = 0; i < command->argc - 1; i++)
- params[i] = getVariable(st, command->argv[i + 1]);
+ params[i] = getVariable(variables, command->argv[i + 1]);
}
static char *
@@ -2629,7 +2664,7 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
{
Variable *var;
- if ((var = lookupVariable(st, expr->u.variable.varname)) == NULL)
+ if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
{
pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
return false;
@@ -2699,7 +2734,7 @@ getMetaCommand(const char *cmd)
* Return true if succeeded, or false on error.
*/
static bool
-runShellCommand(CState *st, char *variable, char **argv, int argc)
+runShellCommand(Variables *variables, char *variable, char **argv, int argc)
{
char command[SHELL_COMMAND_SIZE];
int i,
@@ -2730,7 +2765,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
{
arg = argv[i] + 1; /* a string literal starting with colons */
}
- else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ else if ((arg = getVariable(variables, argv[i] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[i]);
return false;
@@ -2791,7 +2826,7 @@ runShellCommand(CState *st, char *variable, char **argv, int argc)
pg_log_error("%s: shell command must return an integer (not \"%s\")", argv[0], res);
return false;
}
- if (!putVariableInt(st, "setshell", variable, retval))
+ if (!putVariableInt(variables, "setshell", variable, retval))
return false;
pg_log_debug("%s: shell parameter name: \"%s\", value: \"%s\"", argv[0], argv[1], res);
@@ -2843,7 +2878,7 @@ sendCommand(CState *st, Command *command)
char *sql;
sql = pg_strdup(command->argv[0]);
- sql = assignVariables(st, sql);
+ sql = assignVariables(&st->variables, sql);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQuery(st->con, sql);
@@ -2854,7 +2889,7 @@ sendCommand(CState *st, Command *command)
const char *sql = command->argv[0];
const char *params[MAX_ARGS];
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
pg_log_debug("client %d sending %s", st->id, sql);
r = PQsendQueryParams(st->con, sql, command->argc - 1,
@@ -2901,7 +2936,7 @@ sendCommand(CState *st, Command *command)
st->prepared[st->use_file] = true;
}
- getQueryParams(st, command, params);
+ getQueryParams(&st->variables, command, params);
preparedStatementName(name, st->use_file, st->command);
pg_log_debug("client %d sending %s", st->id, name);
@@ -2994,7 +3029,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
varname = psprintf("%s%s", varprefix, varname);
/* store last row result as a string */
- if (!putVariable(st, meta == META_ASET ? "aset" : "gset", varname,
+ if (!putVariable(&st->variables, meta == META_ASET ? "aset" : "gset", varname,
PQgetvalue(res, ntuples - 1, fld)))
{
/* internal error */
@@ -3055,14 +3090,14 @@ error:
* of delay, in microseconds. Returns true on success, false on error.
*/
static bool
-evaluateSleep(CState *st, int argc, char **argv, int *usecs)
+evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
{
char *var;
int usec;
if (*argv[1] == ':')
{
- if ((var = getVariable(st, argv[1] + 1)) == NULL)
+ if ((var = getVariable(variables, argv[1] + 1)) == NULL)
{
pg_log_error("%s: undefined variable \"%s\"", argv[0], argv[1] + 1);
return false;
@@ -3627,7 +3662,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
* latency will be recorded in CSTATE_SLEEP state, not here, after the
* delay has elapsed.)
*/
- if (!evaluateSleep(st, argc, argv, &usec))
+ if (!evaluateSleep(&st->variables, argc, argv, &usec))
{
commandFailed(st, "sleep", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3648,7 +3683,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_ABORTED;
}
- if (!putVariableValue(st, argv[0], argv[1], &result))
+ if (!putVariableValue(&st->variables, argv[0], argv[1], &result))
{
commandFailed(st, "set", "assignment of meta-command failed");
return CSTATE_ABORTED;
@@ -3718,7 +3753,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SETSHELL)
{
- if (!runShellCommand(st, argv[1], argv + 2, argc - 2))
+ if (!runShellCommand(&st->variables, argv[1], argv + 2, argc - 2))
{
commandFailed(st, "setshell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -3726,7 +3761,7 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
}
else if (command->meta == META_SHELL)
{
- if (!runShellCommand(st, NULL, argv + 1, argc - 1))
+ if (!runShellCommand(&st->variables, NULL, argv + 1, argc - 1))
{
commandFailed(st, "shell", "execution of meta-command failed");
return CSTATE_ABORTED;
@@ -6020,7 +6055,7 @@ main(int argc, char **argv)
}
*p++ = '\0';
- if (!putVariable(&state[0], "option", optarg, p))
+ if (!putVariable(&state[0].variables, "option", optarg, p))
exit(1);
}
break;
@@ -6348,19 +6383,19 @@ main(int argc, char **argv)
int j;
state[i].id = i;
- for (j = 0; j < state[0].nvariables; j++)
+ for (j = 0; j < state[0].variables.nvars; j++)
{
- Variable *var = &state[0].variables[j];
+ Variable *var = &state[0].variables.vars[j];
if (var->value.type != PGBT_NO_VALUE)
{
- if (!putVariableValue(&state[i], "startup",
+ if (!putVariableValue(&state[i].variables, "startup",
var->name, &var->value))
exit(1);
}
else
{
- if (!putVariable(&state[i], "startup",
+ if (!putVariable(&state[i].variables, "startup",
var->name, var->svalue))
exit(1);
}
@@ -6398,11 +6433,11 @@ main(int argc, char **argv)
* :scale variables normally get -s or database scale, but don't override
* an explicit -D switch
*/
- if (lookupVariable(&state[0], "scale") == NULL)
+ if (lookupVariable(&state[0].variables, "scale") == NULL)
{
for (i = 0; i < nclients; i++)
{
- if (!putVariableInt(&state[i], "startup", "scale", scale))
+ if (!putVariableInt(&state[i].variables, "startup", "scale", scale))
exit(1);
}
}
@@ -6411,28 +6446,30 @@ main(int argc, char **argv)
* Define a :client_id variable that is unique per connection. But don't
* override an explicit -D switch.
*/
- if (lookupVariable(&state[0], "client_id") == NULL)
+ if (lookupVariable(&state[0].variables, "client_id") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "client_id", i))
+ if (!putVariableInt(&state[i].variables, "startup", "client_id", i))
exit(1);
}
/* set default seed for hash functions */
- if (lookupVariable(&state[0], "default_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "default_seed") == NULL)
{
uint64 seed = pg_prng_uint64(&base_random_sequence);
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "default_seed", (int64) seed))
+ if (!putVariableInt(&state[i].variables, "startup", "default_seed",
+ (int64) seed))
exit(1);
}
/* set random seed unless overwritten */
- if (lookupVariable(&state[0], "random_seed") == NULL)
+ if (lookupVariable(&state[0].variables, "random_seed") == NULL)
{
for (i = 0; i < nclients; i++)
- if (!putVariableInt(&state[i], "startup", "random_seed", random_seed))
+ if (!putVariableInt(&state[i].variables, "startup", "random_seed",
+ random_seed))
exit(1);
}
--
2.17.1
v18-0002-Pgbench-errors-and-serialization-deadlock-retrie.patch (text/x-diff, 82.9K) download
From 686a2beb6b4fd20e3f21c544ffbacb22782c3409 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Mon, 7 Jun 2021 18:35:14 +0900
Subject: [PATCH v18 2/2] Pgbench errors and serialization/deadlock retries
Client's run is aborted in case of a serious error, for example, the
connection with the database server was lost or the end of script reached
without completing the last transaction. In addition, if an execution of SQL
or meta command fails for reasons other than serialization or deadlock errors,
the client is aborted. Otherwise, if an SQL fails with serialization or
deadlock errors, the current transaction is rolled back which also
includes setting the client variables as they were before the run of this
transaction (it is assumed that one transaction script contains only one
transaction).
Transactions with serialization or deadlock errors are repeated after
rollbacks until they complete successfully or reach the maximum number of
tries (specified by the --max-tries option) / the maximum time of tries
(specified by the --latency-limit option). These options can be combined
together; more over, you cannot use an unlimited number of tries (--max-tries=0)
without the --latency-limit option or the --time option. By default the option
--max-tries is set to 1 and transactions with serialization/deadlock errors
are not retried. If the last transaction run fails, this transaction will be
reported as failed, and the client variables will be set as they were before
the first run of this transaction.
If there're retries and/or failures their statistics are printed in the
progress, in the transaction / aggregation logs and in the end with other
results (all and for each script). Also retries and failures are printed
per-command with average latencies if you use the appropriate benchmarking
option (--report-per-command, -r). If you want to group failures by basic types
(serialization failures / deadlock failures), use the option --failures-detailed.
If you want to distinguish all errors and failures (errors without retrying) by
type including which limit for retries was violated and how far it was exceeded
for the serialization/deadlock failures, use the options --verbose-errors.
---
doc/src/sgml/ref/pgbench.sgml | 431 ++++++++-
src/bin/pgbench/pgbench.c | 965 +++++++++++++++++--
src/bin/pgbench/t/001_pgbench_with_server.pl | 215 ++++-
src/bin/pgbench/t/002_pgbench_no_server.pl | 10 +
src/fe_utils/conditional.c | 16 +-
src/include/fe_utils/conditional.h | 2 +
6 files changed, 1494 insertions(+), 145 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index be1896fa99..ebdb4b3f46 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -56,20 +56,29 @@ scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
+maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
+number of failed transactions: 0 (0.000%)
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
tps = 896.967014 (without initial connection time)
</screen>
- The first six lines report some of the most important parameter
- settings. The next line reports the number of transactions completed
+ The first seven lines report some of the most important parameter
+ settings.
+ The sixth line reports the maximum number of tries for transactions with
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"/>
+ for more information).
+ The eighth line reports the number of transactions completed
and intended (the latter being just the product of number of clients
and number of transactions per client); these will be equal unless the run
- failed before completion. (In <option>-T</option> mode, only the actual
- number of transactions is printed.)
+ failed before completion or some SQL command(s) failed. (In
+ <option>-T</option> mode, only the actual number of transactions is printed.)
+ The next line reports the number of failed transactions due to
+ serialization or deadlock errors (see <xref linkend="failures-and-retries"/>
+ for more information).
The last line reports the number of transactions per second.
</para>
@@ -531,6 +540,17 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
at all. They are counted and reported separately as
<firstterm>skipped</firstterm>.
</para>
+ <para>
+ When the <option>--max-tries</option> option is used, the transaction with
+ serialization or deadlock error cannot be retried if the total time of
+ all its tries is greater than <replaceable>limit</replaceable> ms. To
+ limit only the time of tries and not their number, use
+ <literal>--max-tries=0</literal>. By default option
+ <option>--max-tries</option> is set to 1 and transactions with
+ serialization/deadlock errors are not retried. See <xref
+ linkend="failures-and-retries"/> for more information about retrying
+ such transactions.
+ </para>
</listitem>
</varlistentry>
@@ -597,23 +617,29 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<para>
Show progress report every <replaceable>sec</replaceable> seconds. The report
includes the time since the beginning of the run, the TPS since the
- last report, and the transaction latency average and standard
- deviation since the last report. Under throttling (<option>-R</option>),
- the latency is computed with respect to the transaction scheduled
- start time, not the actual transaction beginning time, thus it also
- includes the average schedule lag time.
+ last report, and the transaction latency average, standard deviation,
+ and the number of failed transactions since the last report. Under
+ throttling (<option>-R</option>), the latency is computed with respect
+ to the transaction scheduled start time, not the actual transaction
+ beginning time, thus it also includes the average schedule lag time.
+ When <option>--max-tries</option> is used to enable transactions retries
+ after serialization/deadlock errors, the report includes the number of
+ retried transactions and the sum of all retries.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-r</option></term>
- <term><option>--report-latencies</option></term>
+ <term><option>--report-per-command</option></term>
<listitem>
<para>
- Report the average per-statement latency (execution time from the
- perspective of the client) of each command after the benchmark
- finishes. See below for details.
+ Report the following statistics for each command after the benchmark
+ finishes: the average per-statement latency (execution time from the
+ perspective of the client), the number of failures and the number of
+ retries after serialization or deadlock errors in this command. The
+ report displays retry statistics only if the
+ <option>--max-tries</option> option is not equal to 1.
</para>
</listitem>
</varlistentry>
@@ -741,6 +767,25 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--failures-detailed</option></term>
+ <listitem>
+ <para>
+ Report failures in per-transaction and aggregation logs, as well as in
+ the main and per-script reports, grouped by the following types:
+ <itemizedlist>
+ <listitem>
+ <para>serialization failures;</para>
+ </listitem>
+ <listitem>
+ <para>deadlock failures;</para>
+ </listitem>
+ </itemizedlist>
+ See <xref linkend="failures-and-retries"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--log-prefix=<replaceable>prefix</replaceable></option></term>
<listitem>
@@ -751,6 +796,36 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--max-tries=<replaceable>number_of_tries</replaceable></option></term>
+ <listitem>
+ <para>
+ Enable retries for transactions with serialization/deadlock errors and
+ set the maximum number of these tries. This option can be combined with
+ the <option>--latency-limit</option> option which limits the total time
+ of all transaction tries; moreover, you cannot use an unlimited number
+ of tries (<literal>--max-tries=0</literal>) without
+ <option>--latency-limit</option> or <option>--time</option>.
+ The default value is 1 and transactions with serialization/deadlock
+ errors are not retried. See <xref linkend="failures-and-retries"/>
+ for more information about retrying such transactions.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--verbose-errors</option></term>
+ <listitem>
+ <para>
+ Print messages about all errors and failures (errors without retrying)
+ including which limit for retries was violated and how far it was
+ exceeded for the serialization/deadlock failures. (Note that in this
+ case the output can be significantly increased.).
+ See <xref linkend="failures-and-retries"/> for more information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--progress-timestamp</option></term>
<listitem>
@@ -948,8 +1023,8 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<refsect1>
<title>Notes</title>
- <refsect2>
- <title>What Is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
+ <refsect2 id="transactions-and-scripts" xreflabel="What is the "Transaction" Actually Performed in pgbench?">
+ <title>What is the <quote>Transaction</quote> Actually Performed in <application>pgbench</application>?</title>
<para>
<application>pgbench</application> executes test scripts chosen randomly
@@ -1022,6 +1097,11 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
both old and new versions of <application>pgbench</application>, be sure to write
each SQL command on a single line ending with a semicolon.
</para>
+ <para>
+ It is assumed that pgbench scripts do not contain incomplete blocks of SQL
+ transactions. If at runtime the client reaches the end of the script without
+ completing the last transaction block, it will be aborted.
+ </para>
</note>
<para>
@@ -2212,7 +2292,7 @@ END;
The format of the log is:
<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional>
+<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2233,6 +2313,16 @@ END;
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
+ <replaceable>retries</replaceable> is the sum of all retries after the
+ serialization or deadlock errors during the current script execution. It is
+ present only if the <option>--max-tries</option> option is not equal to 1.
+ If the transaction ends with a failure, its <replaceable>time</replaceable>
+ will be reported as <literal>failed</literal>. If you use the
+ <option>--failures-detailed</option> option, the
+ <replaceable>time</replaceable> of the failed transaction will be reported as
+ <literal>serialization</literal> or
+ <literal>deadlock</literal> depending on the type of failure (see
+ <xref linkend="failures-and-retries"/> for more information).
</para>
<para>
@@ -2261,6 +2351,41 @@ END;
were already late before they were even started.
</para>
+ <para>
+ The following example shows a snippet of a log file with failures and
+ retries, with the maximum number of tries set to 10 (note the additional
+ <replaceable>retries</replaceable> column):
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 failed 0 1499414498 84905 9
+2 0 failed 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
+ <para>
+ If <option>--failures-detailed</option> option is used, the type of
+ failure is reported in the <replaceable>time</replaceable> like this:
+<screen>
+3 0 47423 0 1499414498 34501 3
+3 1 8333 0 1499414498 42848 0
+3 2 8358 0 1499414498 51219 0
+4 0 72345 0 1499414498 59433 6
+1 3 41718 0 1499414498 67879 4
+1 4 8416 0 1499414498 76311 0
+3 3 33235 0 1499414498 84469 3
+0 0 serialization 0 1499414498 84905 9
+2 0 serialization 0 1499414498 86248 9
+3 4 8307 0 1499414498 92788 0
+</screen>
+ </para>
+
<para>
When running a long test on hardware that can handle a lot of transactions,
the log files can become very large. The <option>--sampling-rate</option> option
@@ -2276,7 +2401,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
@@ -2290,7 +2415,16 @@ END;
transaction latencies within the interval,
<replaceable>min_latency</replaceable> is the minimum latency within the interval,
and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval.
+ <replaceable>max_latency</replaceable> is the maximum latency within the interval,
+ <replaceable>failures</replaceable> is the number of transactions that ended
+ with a failed SQL command within the interval. If you use option
+ <option>--failures-detailed</option>, instead of the sum of all failed
+ transactions you will get more detailed statistics for the failed
+ transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
@@ -2298,21 +2432,25 @@ END;
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
- The very last field, <replaceable>skipped</replaceable>,
+ The next field, <replaceable>skipped</replaceable>,
is only present if the <option>--latency-limit</option> option is used, too.
It counts the number of transactions skipped because they would have
started too late.
+ The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
+ fields are present only if the <option>--max-tries</option> option is not
+ equal to 1. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval.
Each transaction is counted in the interval when it was committed.
</para>
<para>
Here is some example output:
<screen>
-1345828501 5601 1542744 483552416 61 2573
-1345828503 7884 1979812 565806736 60 1479
-1345828505 7208 1979422 567277552 59 1391
-1345828507 7685 1980268 569784714 60 1398
-1345828509 7073 1979779 573489941 236 1411
+1345828501 5601 1542744 483552416 61 2573 0
+1345828503 7884 1979812 565806736 60 1479 0
+1345828505 7208 1979422 567277552 59 1391 0
+1345828507 7685 1980268 569784714 60 1398 0
+1345828509 7073 1979779 573489941 236 1411 0
</screen></para>
<para>
@@ -2324,13 +2462,42 @@ END;
</refsect2>
<refsect2>
- <title>Per-Statement Latencies</title>
+ <title>Per-Statement Report</title>
+
+ <para>
+ With the <option>-r</option> option, <application>pgbench</application>
+ collects the following statistics for each statement:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>latency</literal> — elapsed transaction time for each
+ statement. <application>pgbench</application> reports an average value
+ of all successful runs of the statement.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of failures in this statement. See
+ <xref linkend="failures-and-retries"/> for more information.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ The number of retries after a serialization or a deadlock error in this
+ statement. See <xref linkend="failures-and-retries"/> for more information.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ The report displays retry statistics only if the <option>--max-tries</option>
+ option is not equal to 1.
+ </para>
<para>
- With the <option>-r</option> option, <application>pgbench</application> collects
- the elapsed transaction time of each statement executed by every
- client. It then reports an average of those values, referred to
- as the latency for each statement, after the benchmark has finished.
+ All values are computed for each statement executed by every client and are
+ reported after the benchmark has finished.
</para>
<para>
@@ -2342,29 +2509,67 @@ scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
+maximum number of tries: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
-latency average = 10.870 ms
-latency stddev = 7.341 ms
-initial connection time = 30.954 ms
-tps = 907.949122 (without initial connection time)
-statement latencies in milliseconds:
- 0.001 \set aid random(1, 100000 * :scale)
- 0.001 \set bid random(1, 1 * :scale)
- 0.001 \set tid random(1, 10 * :scale)
- 0.000 \set delta random(-5000, 5000)
- 0.046 BEGIN;
- 0.151 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
- 0.107 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
- 4.241 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
- 5.245 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
- 0.102 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
- 0.974 END;
+number of failed transactions: 0 (0.000%)
+number of transactions above the 50.0 ms latency limit: 1311/10000 (13.110 %)
+latency average = 28.488 ms
+latency stddev = 21.009 ms
+initial connection time = 69.068 ms
+tps = 346.224794 (without initial connection time)
+statement latencies in milliseconds and failures:
+ 0.012 0 \set aid random(1, 100000 * :scale)
+ 0.002 0 \set bid random(1, 1 * :scale)
+ 0.002 0 \set tid random(1, 10 * :scale)
+ 0.002 0 \set delta random(-5000, 5000)
+ 0.319 0 BEGIN;
+ 0.834 0 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.641 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 11.126 0 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 12.961 0 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.634 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.957 0 END;
</screen>
+
+ Another example of output for the default script using serializable default
+ transaction isolation level (<command>PGOPTIONS='-c
+ default_transaction_isolation=serializable' pgbench ...</command>):
+<screen>
+starting vacuum...end.
+transaction type: <builtin: TPC-B (sort of)>
+scaling factor: 1
+query mode: simple
+number of clients: 10
+number of threads: 1
+maximum number of tries: 10
+number of transactions per client: 1000
+number of transactions actually processed: 6317/10000
+number of failed transactions: 3683 (36.830%)
+number of transactions retried: 7667 (76.670%)
+total number of retries: 45339
+number of transactions above the 50.0 ms latency limit: 106/6317 (1.678 %)
+latency average = 17.016 ms
+latency stddev = 13.283 ms
+initial connection time = 45.017 ms
+tps = 186.792667 (without initial connection time)
+statement latencies in milliseconds, failures and retries:
+ 0.006 0 0 \set aid random(1, 100000 * :scale)
+ 0.001 0 0 \set bid random(1, 1 * :scale)
+ 0.001 0 0 \set tid random(1, 10 * :scale)
+ 0.001 0 0 \set delta random(-5000, 5000)
+ 0.385 0 0 BEGIN;
+ 0.773 0 1 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
+ 0.624 0 0 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
+ 1.098 320 3762 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
+ 0.582 3363 41576 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
+ 0.465 0 0 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
+ 1.933 0 0 END;
+ </screen>
</para>
<para>
- If multiple script files are specified, the averages are reported
+ If multiple script files are specified, all statistics are reported
separately for each script file.
</para>
@@ -2378,6 +2583,140 @@ statement latencies in milliseconds:
</para>
</refsect2>
+ <refsect2 id="failures-and-retries" xreflabel="Failures and Serialization/Deadlock Retries">
+ <title>Failures and Serialization/Deadlock Retries</title>
+
+ <para>
+ When executing <application>pgbench</application>, there are three main types
+ of errors:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Errors of the main program. They are the most serious and always result
+ in an immediate exit from the <application>pgbench</application> with
+ the corresponding error message. They include:
+ <itemizedlist>
+ <listitem>
+ <para>
+ errors at the beginning of the <application>pgbench</application>
+ (e.g. an invalid option value);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors in the initialization mode (e.g. the query to create
+ tables for built-in scripts fails);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ errors before starting threads (e.g. we could not connect to the
+ database server / the syntax error in the meta command / thread
+ creation failure);
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ internal <application>pgbench</application> errors (which are
+ supposed to never occur...).
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Errors when the thread manages its clients (e.g. the client could not
+ start a connection to the database server / the socket for connecting
+ the client to the database server has become invalid). In such cases
+ all clients of this thread stop while other threads continue to work.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Direct client errors. They lead to immediate exit from the
+ <application>pgbench</application> with the corresponding error message
+ only in the case of an internal <application>pgbench</application>
+ error (which are supposed to never occur...). Otherwise in the worst
+ case they only lead to the abortion of the failed client while other
+ clients continue their run (but some client errors are handled without
+ an abortion of the client and reported separately, see below). Later in
+ this section it is assumed that the discussed errors are only the
+ direct client errors and they are not internal
+ <application>pgbench</application> errors.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Client's run is aborted in case of a serious error, for example, the
+ connection with the database server was lost or the end of script reached
+ without completing the last transaction. In addition, if an execution of SQL
+ or meta command fails for reasons other than serialization or deadlock errors,
+ the client is aborted. Otherwise, if an SQL fails with serialization or
+ deadlock errors, the client is not aborted. In such cases, the current
+ transaction is rolled back, which also includes setting the client variables
+ as they were before the run of this transaction (it is assumed that one
+ transaction script contains only one transaction; see
+ <xref linkend="transactions-and-scripts"/> for more information).
+ Transactions with serialization or deadlock errors are repeated after
+ rollbacks until they complete successfully or reach the maximum
+ number of tries (specified by the <option>--max-tries</option> option) / the maximum
+ time of retries (specified by the <option>--latency-limit</option> option) / the end
+ of benchmark (specified by the <option>--time</option> option). If
+ the last trial run fails, this transaction will be reported as failed but
+ the client is not aborted and continue to work.
+ </para>
+
+ <note>
+ <para>
+ Without specifying the <option>--max-tries</option> option a transaction will
+ never be retried after a serialization or deadlock error because its default
+ values is 1. Use an unlimited number of tries (<literal>--max-tries=0</literal>)
+ and the <option>--latency-limit</option> option to limit only the maximum time
+ of tries. You can also use the <option>--time</option> option to limit the
+ benchmark duration under an unlimited number of tries.
+ </para>
+ <para>
+ Be careful when repeating scripts that contain multiple transactions: the
+ script is always retried completely, so the successful transactions can be
+ performed several times.
+ </para>
+ <para>
+ Be careful when repeating transactions with shell commands. Unlike the
+ results of SQL commands, the results of shell commands are not rolled back,
+ except for the variable value of the <command>\setshell</command> command.
+ </para>
+ </note>
+
+ <para>
+ The latency of a successful transaction includes the entire time of
+ transaction execution with rollbacks and retries. The latency is measured
+ only for successful transactions and commands but not for failed transactions
+ or commands.
+ </para>
+
+ <para>
+ The main report contains the number of failed transactions. If the
+ <option>--max-tries</option> option is not equal to 1, the main report also
+ contains the statistics related to retries: the total number of retried
+ transactions and total number of retries. The per-script report inherits all
+ these fields from the main report. The per-statement report displays retry
+ statistics only if the <option>--max-tries</option> option is not equal to 1.
+ </para>
+
+ <para>
+ If you want to group failures by basic types in per-transaction and
+ aggregation logs, as well as in the main and per-script reports, use the
+ <option>--failures-detailed</option> option. If you also want to distinguish
+ all errors and failures (errors without retrying) by type including which
+ limit for retries was violated and how far it was exceeded for the
+ serialization/deadlock failures, use the <option>--verbose-errors</option>
+ option.
+ </para>
+ </refsect2>
+
<refsect2>
<title>Good Practices</title>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ab2c5dfc5f..7080d2a795 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -76,6 +76,8 @@
#define M_PI 3.14159265358979323846
#endif
+#define ERRCODE_T_R_SERIALIZATION_FAILURE "40001"
+#define ERRCODE_T_R_DEADLOCK_DETECTED "40P01"
#define ERRCODE_UNDEFINED_TABLE "42P01"
/*
@@ -275,9 +277,34 @@ bool progress_timestamp = false; /* progress report with Unix time */
int nclients = 1; /* number of clients */
int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
-bool report_per_command; /* report per-command latencies */
+bool report_per_command = false; /* report per-command latencies, retries
+ * after errors and failures (errors
+ * without retrying) */
int main_pid; /* main process id used in log filename */
+/*
+ * There are different types of restrictions for deciding that the current
+ * transaction with a serialization/deadlock error can no longer be retried and
+ * should be reported as failed:
+ * - max_tries (--max-tries) can be used to limit the number of tries;
+ * - latency_limit (-L) can be used to limit the total time of tries;
+ * - duration (-T) can be used to limit the total benchmark time.
+ *
+ * They can be combined together, and you need to use at least one of them to
+ * retry the transactions with serialization/deadlock errors. If none of them is
+ * used, the default value of max_tries is 1 and such transactions will not be
+ * retried.
+ */
+
+/*
+ * We cannot retry a transaction after the serialization/deadlock error if its
+ * number of tries reaches this maximum; if its value is zero, it is not used.
+ */
+uint32 max_tries = 1;
+
+bool failures_detailed = false; /* whether to group failures in reports
+ * or logs by basic types */
+
const char *pghost = NULL;
const char *pgport = NULL;
const char *username = NULL;
@@ -362,9 +389,66 @@ typedef int64 pg_time_usec_t;
typedef struct StatsData
{
pg_time_usec_t start_time; /* interval start time, for aggregates */
- int64 cnt; /* number of transactions, including skipped */
+
+ /*
+ * Transactions are counted depending on their execution and outcome. First
+ * a transaction may have started or not: skipped transactions occur under
+ * --rate and --latency-limit when the client is too late to execute them.
+ * Secondly, a started transaction may ultimately succeed or fail, possibly
+ * after some retries when --max-tries is not one. Thus
+ *
+ * the number of all transactions =
+ * 'skipped' (it was too late to execute them) +
+ * 'cnt' (the number of successful transactions) +
+ * failed (the number of failed transactions).
+ *
+ * A successful transaction can have several unsuccessful tries before a
+ * successful run. Thus
+ *
+ * 'cnt' (the number of successful transactions) =
+ * successfully retried transactions (they got a serialization or a
+ * deadlock error(s), but were
+ * successfully retried from the very
+ * beginning) +
+ * directly successful transactions (they were successfully completed on
+ * the first try).
+ *
+ * A failed transaction is defined as unsuccessfully retried transactions.
+ * It can be one of two types:
+ *
+ * failed (the number of failed transactions) =
+ * 'serialization_failures' (they got a serialization error and were not
+ * successfully retried) +
+ * 'deadlock_failures' (they got a deadlock error and were not successfully
+ * retried).
+ *
+ * If the transaction was retried after a serialization or a deadlock error
+ * this does not guarantee that this retry was successful. Thus
+ *
+ * 'retries' (number of retries) =
+ * number of retries in all retried transactions =
+ * number of retries in (successfully retried transactions +
+ * failed transactions);
+ *
+ * 'retried' (number of all retried transactions) =
+ * successfully retried transactions +
+ * failed transactions.
+ */
+ int64 cnt; /* number of successful transactions, not
+ * including 'skipped' */
int64 skipped; /* number of transactions skipped under --rate
* and --latency-limit */
+ int64 retries; /* number of retries after a serialization or a
+ * deadlock error in all the transactions */
+ int64 retried; /* number of all transactions that were retried
+ * after a serialization or a deadlock error
+ * (perhaps the last try was unsuccessful) */
+ int64 serialization_failures; /* number of transactions that were not
+ * successfully retried after a
+ * serialization error */
+ int64 deadlock_failures; /* number of transactions that were not
+ * successfully retried after a deadlock
+ * error */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -375,6 +459,31 @@ typedef struct StatsData
*/
pg_time_usec_t epoch_shift;
+/*
+ * Error status for errors during script execution.
+ */
+typedef enum EStatus
+{
+ ESTATUS_NO_ERROR = 0,
+ ESTATUS_META_COMMAND_ERROR,
+
+ /* SQL errors */
+ ESTATUS_SERIALIZATION_ERROR,
+ ESTATUS_DEADLOCK_ERROR,
+ ESTATUS_OTHER_SQL_ERROR
+} EStatus;
+
+/*
+ * Transaction status at the end of a command.
+ */
+typedef enum TStatus
+{
+ TSTATUS_IDLE,
+ TSTATUS_IN_BLOCK,
+ TSTATUS_CONN_ERROR,
+ TSTATUS_OTHER_ERROR
+} TStatus;
+
/* Various random sequences are initialized from this one. */
static pg_prng_state base_random_sequence;
@@ -446,6 +555,35 @@ typedef enum
CSTATE_END_COMMAND,
CSTATE_SKIP_COMMAND,
+ /*
+ * States for failed commands.
+ *
+ * If the SQL/meta command fails, in CSTATE_ERROR clean up after an error:
+ * - clear the conditional stack;
+ * - if we have an unterminated (possibly failed) transaction block, send
+ * the rollback command to the server and wait for the result in
+ * CSTATE_WAIT_ROLLBACK_RESULT. If something goes wrong with rolling back,
+ * go to CSTATE_ABORTED.
+ *
+ * But if everything is ok we are ready for future transactions: if this is
+ * a serialization or deadlock error and we can re-execute the transaction
+ * from the very beginning, go to CSTATE_RETRY; otherwise go to
+ * CSTATE_FAILURE.
+ *
+ * In CSTATE_RETRY report an error, set the same parameters for the
+ * transaction execution as in the previous tries and process the first
+ * transaction command in CSTATE_START_COMMAND.
+ *
+ * In CSTATE_FAILURE report a failure, set the parameters for the
+ * transaction execution as they were before the first run of this
+ * transaction (except for a random state) and go to CSTATE_END_TX to
+ * complete this transaction.
+ */
+ CSTATE_ERROR,
+ CSTATE_WAIT_ROLLBACK_RESULT,
+ CSTATE_RETRY,
+ CSTATE_FAILURE,
+
/*
* CSTATE_END_TX performs end-of-transaction processing. It calculates
* latency, and logs the transaction. In --connect mode, it closes the
@@ -494,8 +632,20 @@ typedef struct
bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
+ /*
+ * For processing failures and repeating transactions with serialization or
+ * deadlock errors:
+ */
+ EStatus estatus; /* the error status of the current transaction
+ * execution; this is ESTATUS_NO_ERROR if there were
+ * no errors */
+ pg_prng_state random_state; /* random state */
+ uint32 tries; /* how many times have we already tried the
+ * current transaction? */
+
/* per client collected stats */
- int64 cnt; /* client transaction count, for -t */
+ int64 cnt; /* client transaction count, for -t; skipped and
+ * failed transactions are also counted here */
} CState;
/*
@@ -590,6 +740,9 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
* aset do gset on all possible queries of a combined query (\;).
* expr Parsed expression, if needed.
* stats Time spent in this command.
+ * retries Number of retries after a serialization or deadlock error in the
+ * current command.
+ * failures Number of errors in the current command that were not retried.
*/
typedef struct Command
{
@@ -602,6 +755,8 @@ typedef struct Command
char *varprefix;
PgBenchExpr *expr;
SimpleStats stats;
+ int64 retries;
+ int64 failures;
} Command;
typedef struct ParsedScript
@@ -616,6 +771,8 @@ static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
static int64 total_weight = 0;
+static bool verbose_errors = false; /* print verbose messages of all errors */
+
/* Builtin test scripts */
typedef struct BuiltinScript
{
@@ -753,15 +910,18 @@ usage(void)
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
- " -r, --report-latencies report average latency per command\n"
+ " -r, --report-per-command report latencies, failures and retries per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
" (default: \"pgbench_log\")\n"
+ " --max-tries=NUM max number of tries to run transaction (default: 1)\n"
+ " --verbose-errors print messages of all errors\n"
" --progress-timestamp use Unix epoch timestamps for progress\n"
" --random-seed=SEED set random seed (\"time\", \"rand\", integer)\n"
" --sampling-rate=NUM fraction of transactions to log (e.g., 0.01 for 1%%)\n"
@@ -1287,6 +1447,10 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->start_time = start;
sd->cnt = 0;
sd->skipped = 0;
+ sd->retries = 0;
+ sd->retried = 0;
+ sd->serialization_failures = 0;
+ sd->deadlock_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1295,22 +1459,51 @@ initStats(StatsData *sd, pg_time_usec_t start)
* Accumulate one additional item into the given stats object.
*/
static void
-accumStats(StatsData *stats, bool skipped, double lat, double lag)
+accumStats(StatsData *stats, bool skipped, double lat, double lag,
+ EStatus estatus, int64 tries)
{
- stats->cnt++;
-
+ /* Record the skipped transaction */
if (skipped)
{
/* no latency to record on skipped transactions */
stats->skipped++;
+ return;
}
- else
+
+ /*
+ * Record the number of retries regardless of whether the transaction was
+ * successful or failed.
+ */
+ if (tries > 1)
+ {
+ stats->retries += (tries - 1);
+ stats->retried++;
+ }
+
+ switch (estatus)
{
- addToSimpleStats(&stats->latency, lat);
+ /* Record the successful transaction */
+ case ESTATUS_NO_ERROR:
+ stats->cnt++;
- /* and possibly the same for schedule lag */
- if (throttle_delay)
- addToSimpleStats(&stats->lag, lag);
+ addToSimpleStats(&stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ addToSimpleStats(&stats->lag, lag);
+ break;
+
+ /* Record the failed transaction */
+ case ESTATUS_SERIALIZATION_ERROR:
+ stats->serialization_failures++;
+ break;
+ case ESTATUS_DEADLOCK_ERROR:
+ stats->deadlock_failures++;
+ break;
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
}
}
@@ -2841,6 +3034,9 @@ preparedStatementName(char *buffer, int file, int state)
sprintf(buffer, "P%d_%d", file, state);
}
+/*
+ * Report the abortion of the client when processing SQL commands.
+ */
static void
commandFailed(CState *st, const char *cmd, const char *message)
{
@@ -2848,6 +3044,17 @@ commandFailed(CState *st, const char *cmd, const char *message)
st->id, st->command, cmd, st->use_file, message);
}
+/*
+ * Report the error in the command while the script is executing.
+ */
+static void
+commandError(CState *st, const char *message)
+{
+ Assert(sql_script[st->use_file].commands[st->command]->type == SQL_COMMAND);
+ pg_log_info("client %d got an error in command %d (SQL) of script %d; %s",
+ st->id, st->command, st->use_file, message);
+}
+
/* return a script number with a weighted choice. */
static int
chooseScript(TState *thread)
@@ -2955,6 +3162,33 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Get the error status from the error code.
+ */
+static EStatus
+getSQLErrorStatus(const char *sqlState)
+{
+ if (sqlState != NULL)
+ {
+ if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
+ return ESTATUS_SERIALIZATION_ERROR;
+ else if (strcmp(sqlState, ERRCODE_T_R_DEADLOCK_DETECTED) == 0)
+ return ESTATUS_DEADLOCK_ERROR;
+ }
+
+ return ESTATUS_OTHER_SQL_ERROR;
+}
+
+/*
+ * Returns true if this type of error can be retried.
+ */
+static bool
+canRetryError(EStatus estatus)
+{
+ return (estatus == ESTATUS_SERIALIZATION_ERROR ||
+ estatus == ESTATUS_DEADLOCK_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -2997,6 +3231,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
{
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, 0);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
break;
@@ -3011,6 +3246,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* under \gset, report the error */
pg_log_error("client %d script %d command %d query %d: expected one row, got %d",
st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
else if (meta == META_ASET && ntuples <= 0)
@@ -3035,6 +3271,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
/* internal error */
pg_log_error("client %d script %d command %d query %d: error storing into variable %s",
st->id, st->use_file, st->command, qrynum, varname);
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
goto error;
}
@@ -3052,6 +3289,18 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
PQerrorMessage(st->con));
break;
+ case PGRES_NONFATAL_ERROR:
+ case PGRES_FATAL_ERROR:
+ st->estatus = getSQLErrorStatus(
+ PQresultErrorField(res, PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus))
+ {
+ if (verbose_errors)
+ commandError(st, PQerrorMessage(st->con));
+ goto error;
+ }
+ /* fall through */
+
default:
/* anything else is unexpected */
pg_log_error("client %d script %d aborted in command %d query %d: %s",
@@ -3130,6 +3379,165 @@ evaluateSleep(Variables *variables, int argc, char **argv, int *usecs)
return true;
}
+
+/*
+ * Returns true if the error can be retried.
+ */
+static bool
+doRetry(CState *st, pg_time_usec_t *now)
+{
+ Assert(st->estatus != ESTATUS_NO_ERROR);
+
+ /* We can only retry serialization or deadlock errors. */
+ if (!canRetryError(st->estatus))
+ return false;
+
+ /*
+ * We must have at least one option to limit the retrying of transactions
+ * that got an error.
+ */
+ Assert(max_tries || latency_limit || duration > 0);
+
+ /*
+ * We cannot retry the error if we have reached the maximum number of tries.
+ */
+ if (max_tries && st->tries >= max_tries)
+ return false;
+
+ /*
+ * We cannot retry the error if we spent too much time on this transaction.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ if (*now - st->txn_scheduled > latency_limit)
+ return false;
+ }
+
+ /*
+ * We cannot retry the error if the benchmark duration is over.
+ */
+ if (timer_exceeded)
+ return false;
+
+ /* OK */
+ return true;
+}
+
+/*
+ * Read results and discard it until a sync point.
+ */
+static int
+discardUntilSync(CState *st)
+{
+ /* send a sync */
+ if (!PQpipelineSync(st->con))
+ {
+ pg_log_error("client %d aborted: failed to send a pipeline sync",
+ st->id);
+ return 0;
+ }
+
+ /* receive PGRES_PIPELINE_SYNC and null following it */
+ for(;;)
+ {
+ PGresult *res = PQgetResult(st->con);
+ if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
+ {
+ PQclear(res);
+ res = PQgetResult(st->con);
+ Assert(res == NULL);
+ break;
+ }
+ PQclear(res);
+ }
+
+ /* exit pipline */
+ if (PQexitPipelineMode(st->con) != 1)
+ {
+ pg_log_error("client %d aborted: failed to exit pipeline mode for rolling back the failed transaction",
+ st->id);
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Get the transaction status at the end of a command especially for
+ * checking if we are in a (failed) transaction block.
+ */
+static TStatus
+getTransactionStatus(PGconn *con)
+{
+ PGTransactionStatusType tx_status;
+
+ tx_status = PQtransactionStatus(con);
+ switch (tx_status)
+ {
+ case PQTRANS_IDLE:
+ return TSTATUS_IDLE;
+ case PQTRANS_INTRANS:
+ case PQTRANS_INERROR:
+ return TSTATUS_IN_BLOCK;
+ case PQTRANS_UNKNOWN:
+ /* PQTRANS_UNKNOWN is expected given a broken connection */
+ if (PQstatus(con) == CONNECTION_BAD)
+ return TSTATUS_CONN_ERROR;
+ /* fall through */
+ case PQTRANS_ACTIVE:
+ default:
+ /*
+ * We cannot find out whether we are in a transaction block or not.
+ * Internal error which should never occur.
+ */
+ pg_log_error("unexpected transaction status %d", tx_status);
+ return TSTATUS_OTHER_ERROR;
+ }
+
+ /* not reached */
+ Assert(false);
+ return TSTATUS_OTHER_ERROR;
+}
+
+/*
+ * Print verbose messages of an error
+ */
+static void
+printVerboseErrorMessages(CState *st, pg_time_usec_t *now, bool is_retry)
+{
+ static PQExpBuffer buf = NULL;
+
+ if (buf == NULL)
+ buf = createPQExpBuffer();
+ else
+ resetPQExpBuffer(buf);
+
+ printfPQExpBuffer(buf, "client %d ", st->id);
+ appendPQExpBuffer(buf, "%s",
+ (is_retry ?
+ "repeats the transaction after the error" :
+ "ends the failed transaction"));
+ appendPQExpBuffer(buf, " (try %d", st->tries);
+
+ /* Print max_tries if it is not unlimitted. */
+ if (max_tries)
+ appendPQExpBuffer(buf, "/%d", max_tries);
+
+ /*
+ * If the latency limit is used, print a percentage of the current transaction
+ * latency from the latency limit.
+ */
+ if (latency_limit)
+ {
+ pg_time_now_lazy(now);
+ appendPQExpBuffer(buf, ", %.3f%% of the maximum time of tries was used",
+ (100.0 * (*now - st->txn_scheduled) / latency_limit));
+ }
+ appendPQExpBuffer(buf, ")\n");
+
+ pg_log_info("%s", buf->data);
+}
+
/*
* Advance the state machine of a connection.
*/
@@ -3167,6 +3575,10 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->use_file = chooseScript(thread);
Assert(conditional_stack_empty(st->cstack));
+ /* reset transaction variables to default values */
+ st->estatus = ESTATUS_NO_ERROR;
+ st->tries = 1;
+
pg_log_debug("client %d executing script \"%s\"",
st->id, sql_script[st->use_file].desc);
@@ -3207,6 +3619,13 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
memset(st->prepared, 0, sizeof(st->prepared));
}
+ /*
+ * It is the first try to run this transaction. Remember the
+ * random state: maybe it will get an error and we will need to
+ * run it again.
+ */
+ st->random_state = st->cs_func_rs;
+
/* record transaction start time */
st->txn_begin = now;
@@ -3363,6 +3782,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
* - else CSTATE_END_COMMAND
*/
st->state = executeMetaCommand(st, &now);
+ if (st->state == CSTATE_ABORTED)
+ st->estatus = ESTATUS_META_COMMAND_ERROR;
}
/*
@@ -3508,6 +3929,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
+ else if (canRetryError(st->estatus))
+ st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
break;
@@ -3555,44 +3978,223 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
break;
/*
- * End of transaction (end of script, really).
+ * Clean up after an error.
*/
- case CSTATE_END_TX:
+ case CSTATE_ERROR:
+ {
+ TStatus tstatus;
- /* transaction finished: calculate latency and do log */
- processXactStats(thread, st, &now, false, agg);
+ Assert(st->estatus != ESTATUS_NO_ERROR);
- /*
- * missing \endif... cannot happen if CheckConditional was
- * okay
- */
- Assert(conditional_stack_empty(st->cstack));
+ /* Clear the conditional stack */
+ conditional_stack_reset(st->cstack);
- if (is_connect)
- {
- pg_time_usec_t start = now;
+ /* Read and discard until a sync point in pipeline mode */
+ if (PQpipelineStatus(st->con) != PQ_PIPELINE_OFF)
+ {
+ if (!discardUntilSync(st))
+ {
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ }
- pg_time_now_lazy(&start);
- finishCon(st);
- now = pg_time_now();
- thread->conn_duration += now - start;
+ /*
+ * Check if we have a (failed) transaction block or not, and
+ * roll it back if any.
+ */
+ tstatus = getTransactionStatus(st->con);
+ if (tstatus == TSTATUS_IN_BLOCK)
+ {
+ /* Try to rollback a (failed) transaction block. */
+ if (!PQsendQuery(st->con, "ROLLBACK"))
+ {
+ pg_log_error("client %d aborted: failed to send sql command for rolling back the failed transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ else
+ st->state = CSTATE_WAIT_ROLLBACK_RESULT;
+ }
+ else if (tstatus == TSTATUS_IDLE)
+ {
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ }
+ else
+ {
+ if (tstatus == TSTATUS_CONN_ERROR)
+ pg_log_error("perhaps the backend died while processing");
+
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ }
+ break;
}
- if ((st->cnt >= nxacts && duration <= 0) || timer_exceeded)
+ /*
+ * Wait for the rollback command to complete
+ */
+ case CSTATE_WAIT_ROLLBACK_RESULT:
{
- /* script completed */
- st->state = CSTATE_FINISHED;
+ PGresult *res;
+
+ pg_log_debug("client %d receiving", st->id);
+ if (!PQconsumeInput(st->con))
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ if (PQisBusy(st->con))
+ return; /* don't have the whole result yet */
+
+ /*
+ * Read and discard the query result;
+ */
+ res = PQgetResult(st->con);
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK:
+ /* OK */
+ PQclear(res);
+ /* null must be returned */
+ res = PQgetResult(st->con);
+ Assert(res == NULL);
+
+ /*
+ * If time is over, we're done;
+ * otherwise, check if we can retry the error.
+ */
+ st->state = timer_exceeded ? CSTATE_FINISHED :
+ doRetry(st, &now) ? CSTATE_RETRY : CSTATE_FAILURE;
+ break;
+ default:
+ pg_log_error("client %d aborted while rolling back the transaction after an error; %s",
+ st->id, PQerrorMessage(st->con));
+ PQclear(res);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
break;
}
- /* next transaction (script) */
- st->state = CSTATE_CHOOSE_SCRIPT;
+ /*
+ * Retry the transaction after an error.
+ */
+ case CSTATE_RETRY:
+ command = sql_script[st->use_file].commands[st->command];
/*
- * Ensure that we always return on this point, so as to avoid
- * an infinite loop if the script only contains meta commands.
+ * Inform that the transaction will be retried after the error.
*/
- return;
+ if (verbose_errors)
+ printVerboseErrorMessages(st, &now, true);
+
+ /* Count tries and retries */
+ st->tries++;
+ command->retries++;
+
+ /*
+ * Reset the random state as they were at the beginning
+ * of the transaction.
+ */
+ st->cs_func_rs = st->random_state;
+
+ /* Process the first transaction command. */
+ st->command = 0;
+ st->estatus = ESTATUS_NO_ERROR;
+ st->state = CSTATE_START_COMMAND;
+ break;
+
+ /*
+ * Record a failed transaction.
+ */
+ case CSTATE_FAILURE:
+ command = sql_script[st->use_file].commands[st->command];
+
+ /* Accumulate the failure. */
+ command->failures++;
+
+ /*
+ * Inform that the failed transaction will not be retried.
+ */
+ if (verbose_errors)
+ printVerboseErrorMessages(st, &now, false);
+
+ /* End the failed transaction. */
+ st->state = CSTATE_END_TX;
+ break;
+
+ /*
+ * End of transaction (end of script, really).
+ */
+ case CSTATE_END_TX:
+ {
+ TStatus tstatus;
+
+ /* transaction finished: calculate latency and do log */
+ processXactStats(thread, st, &now, false, agg);
+
+ /*
+ * missing \endif... cannot happen if CheckConditional was
+ * okay
+ */
+ Assert(conditional_stack_empty(st->cstack));
+
+ /*
+ * We must complete all the transaction blocks that were
+ * started in this script.
+ */
+ tstatus = getTransactionStatus(st->con);
+ if (tstatus == TSTATUS_IN_BLOCK)
+ {
+ pg_log_error("client %d aborted: end of script reached without completing the last transaction",
+ st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+ else if (tstatus != TSTATUS_IDLE)
+ {
+ if (tstatus == TSTATUS_CONN_ERROR)
+ pg_log_error("perhaps the backend died while processing");
+
+ pg_log_error("client %d aborted while receiving the transaction status", st->id);
+ st->state = CSTATE_ABORTED;
+ break;
+ }
+
+ if (is_connect)
+ {
+ pg_time_usec_t start = now;
+
+ pg_time_now_lazy(&start);
+ finishCon(st);
+ now = pg_time_now();
+ thread->conn_duration += now - start;
+ }
+
+ if ((st->cnt >= nxacts && duration <= 0) || timer_exceeded)
+ {
+ /* script completed */
+ st->state = CSTATE_FINISHED;
+ break;
+ }
+
+ /* next transaction (script) */
+ st->state = CSTATE_CHOOSE_SCRIPT;
+
+ /*
+ * Ensure that we always return on this point, so as to avoid
+ * an infinite loop if the script only contains meta commands.
+ */
+ return;
+ }
/*
* Final states. Close the connection if it's still open.
@@ -3816,6 +4418,43 @@ executeMetaCommand(CState *st, pg_time_usec_t *now)
return CSTATE_END_COMMAND;
}
+/*
+ * Return the number fo failed transactions.
+ */
+static int64
+getFailures(const StatsData *stats)
+{
+ return (stats->serialization_failures +
+ stats->deadlock_failures);
+}
+
+/*
+ * Return a string constant representing the result of a transaction
+ * that is not successfully processed.
+ */
+static const char *
+getResultString(bool skipped, EStatus estatus)
+{
+ if (skipped)
+ return "skipped";
+ else if (failures_detailed)
+ {
+ switch (estatus)
+ {
+ case ESTATUS_SERIALIZATION_ERROR:
+ return "serialization";
+ case ESTATUS_DEADLOCK_ERROR:
+ return "deadlock";
+ default:
+ /* internal error which should never occur */
+ pg_log_fatal("unexpected error status: %d", estatus);
+ exit(1);
+ }
+ }
+ else
+ return "failed";
+}
+
/*
* Print log entry after completing one transaction.
*
@@ -3863,6 +4502,14 @@ doLog(TState *thread, CState *st,
agg->latency.sum2,
agg->latency.min,
agg->latency.max);
+
+ if (failures_detailed)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->serialization_failures,
+ agg->deadlock_failures);
+ else
+ fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
+
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
@@ -3873,6 +4520,10 @@ doLog(TState *thread, CState *st,
if (latency_limit)
fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
+ if (max_tries != 1)
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ agg->retried,
+ agg->retries);
fputc('\n', logfile);
/* reset data and move to next interval */
@@ -3880,22 +4531,26 @@ doLog(TState *thread, CState *st,
}
/* accumulate the current transaction */
- accumStats(agg, skipped, latency, lag);
+ accumStats(agg, skipped, latency, lag, st->estatus, st->tries);
}
else
{
/* no, print raw transactions */
- if (skipped)
- fprintf(logfile, "%d " INT64_FORMAT " skipped %d " INT64_FORMAT " "
- INT64_FORMAT,
- st->id, st->cnt, st->use_file, now / 1000000, now % 1000000);
- else
+ if (!skipped && st->estatus == ESTATUS_NO_ERROR)
fprintf(logfile, "%d " INT64_FORMAT " %.0f %d " INT64_FORMAT " "
INT64_FORMAT,
st->id, st->cnt, latency, st->use_file,
now / 1000000, now % 1000000);
+ else
+ fprintf(logfile, "%d " INT64_FORMAT " %s %d " INT64_FORMAT " "
+ INT64_FORMAT,
+ st->id, st->cnt, getResultString(skipped, st->estatus),
+ st->use_file, now / 1000000, now % 1000000);
+
if (throttle_delay)
fprintf(logfile, " %.0f", lag);
+ if (max_tries != 1)
+ fprintf(logfile, " %d", st->tries - 1);
fputc('\n', logfile);
}
}
@@ -3904,7 +4559,8 @@ doLog(TState *thread, CState *st,
* Accumulate and report statistics at end of a transaction.
*
* (This is also called when a transaction is late and thus skipped.
- * Note that even skipped transactions are counted in the "cnt" fields.)
+ * Note that even skipped and failed transactions are counted in the CState
+ * "cnt" field.)
*/
static void
processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
@@ -3912,10 +4568,10 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
{
double latency = 0.0,
lag = 0.0;
- bool thread_details = progress || throttle_delay || latency_limit,
- detailed = thread_details || use_log || per_script_stats;
+ bool detailed = progress || throttle_delay || latency_limit ||
+ use_log || per_script_stats;
- if (detailed && !skipped)
+ if (detailed && !skipped && st->estatus == ESTATUS_NO_ERROR)
{
pg_time_now_lazy(now);
@@ -3924,20 +4580,12 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
lag = st->txn_begin - st->txn_scheduled;
}
- if (thread_details)
- {
- /* keep detailed thread stats */
- accumStats(&thread->stats, skipped, latency, lag);
+ /* keep detailed thread stats */
+ accumStats(&thread->stats, skipped, latency, lag, st->estatus, st->tries);
- /* count transactions over the latency limit, if needed */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
- else
- {
- /* no detailed stats, just count */
- thread->stats.cnt++;
- }
+ /* count transactions over the latency limit, if needed */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
/* client stat is just counting */
st->cnt++;
@@ -3947,7 +4595,8 @@ processXactStats(TState *thread, CState *st, pg_time_usec_t *now,
/* XXX could use a mutex here, but we choose not to */
if (per_script_stats)
- accumStats(&sql_script[st->use_file].stats, skipped, latency, lag);
+ accumStats(&sql_script[st->use_file].stats, skipped, latency, lag,
+ st->estatus, st->tries);
}
@@ -4806,6 +5455,8 @@ create_sql_command(PQExpBuffer buf, const char *source)
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
+ my_command->retries = 0;
+ my_command->failures = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->varprefix = NULL; /* allocated later, if needed */
my_command->expr = NULL;
@@ -5474,7 +6125,9 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
{
/* generate and show report */
pg_time_usec_t run = now - *last_report;
- int64 ntx;
+ int64 cnt,
+ failures,
+ retried;
double tps,
total_run,
latency,
@@ -5501,23 +6154,30 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
mergeSimpleStats(&cur.lag, &threads[i].stats.lag);
cur.cnt += threads[i].stats.cnt;
cur.skipped += threads[i].stats.skipped;
+ cur.retries += threads[i].stats.retries;
+ cur.retried += threads[i].stats.retried;
+ cur.serialization_failures +=
+ threads[i].stats.serialization_failures;
+ cur.deadlock_failures += threads[i].stats.deadlock_failures;
}
/* we count only actually executed transactions */
- ntx = (cur.cnt - cur.skipped) - (last->cnt - last->skipped);
+ cnt = cur.cnt - last->cnt;
total_run = (now - test_start) / 1000000.0;
- tps = 1000000.0 * ntx / run;
- if (ntx > 0)
+ tps = 1000000.0 * cnt / run;
+ if (cnt > 0)
{
- latency = 0.001 * (cur.latency.sum - last->latency.sum) / ntx;
- sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / ntx;
+ latency = 0.001 * (cur.latency.sum - last->latency.sum) / cnt;
+ sqlat = 1.0 * (cur.latency.sum2 - last->latency.sum2) / cnt;
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (cur.lag.sum - last->lag.sum) / ntx;
+ lag = 0.001 * (cur.lag.sum - last->lag.sum) / cnt;
}
else
{
latency = sqlat = stdev = lag = 0;
}
+ failures = getFailures(&cur) - getFailures(last);
+ retried = cur.retried - last->retried;
if (progress_timestamp)
{
@@ -5531,8 +6191,8 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
}
fprintf(stderr,
- "progress: %s, %.1f tps, lat %.3f ms stddev %.3f",
- tbuf, tps, latency, stdev);
+ "progress: %s, %.1f tps, lat %.3f ms stddev %.3f, " INT64_FORMAT " failed",
+ tbuf, tps, latency, stdev, failures);
if (throttle_delay)
{
@@ -5541,6 +6201,12 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
fprintf(stderr, ", " INT64_FORMAT " skipped",
cur.skipped - last->skipped);
}
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ fprintf(stderr,
+ ", " INT64_FORMAT " retried, " INT64_FORMAT " retries",
+ retried, cur.retries - last->retries);
fprintf(stderr, "\n");
*last = cur;
@@ -5600,9 +6266,10 @@ printResults(StatsData *total,
int64 latency_late)
{
/* tps is about actually executed transactions during benchmarking */
- int64 ntx = total->cnt - total->skipped;
+ int64 failures = getFailures(total);
+ int64 total_cnt = total->cnt + total->skipped + failures;
double bench_duration = PG_TIME_GET_DOUBLE(total_duration);
- double tps = ntx / bench_duration;
+ double tps = total->cnt / bench_duration;
/* Report test parameters. */
printf("transaction type: %s\n",
@@ -5615,39 +6282,65 @@ printResults(StatsData *total,
printf("query mode: %s\n", QUERYMODE[querymode]);
printf("number of clients: %d\n", nclients);
printf("number of threads: %d\n", nthreads);
+
+ if (max_tries)
+ printf("maximum number of tries: %d\n", max_tries);
+
if (duration <= 0)
{
printf("number of transactions per client: %d\n", nxacts);
printf("number of transactions actually processed: " INT64_FORMAT "/%d\n",
- ntx, nxacts * nclients);
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
printf("number of transactions actually processed: " INT64_FORMAT "\n",
- ntx);
+ total->cnt);
+ }
+
+ printf("number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ failures, 100.0 * failures / total_cnt);
+
+ if (failures_detailed)
+ {
+ printf("number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->serialization_failures,
+ 100.0 * total->serialization_failures / total_cnt);
+ printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->deadlock_failures,
+ 100.0 * total->deadlock_failures / total_cnt);
+ }
+
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ {
+ printf("number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ total->retried, 100.0 * total->retried / total_cnt);
+ printf("total number of retries: " INT64_FORMAT "\n", total->retries);
}
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (total->cnt <= 0)
+ if (total->cnt + total->skipped <= 0)
return;
if (throttle_delay && latency_limit)
printf("number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
- total->skipped, 100.0 * total->skipped / total->cnt);
+ total->skipped, 100.0 * total->skipped / total_cnt);
if (latency_limit)
printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT "/" INT64_FORMAT " (%.3f%%)\n",
- latency_limit / 1000.0, latency_late, ntx,
- (ntx > 0) ? 100.0 * latency_late / ntx : 0.0);
+ latency_limit / 1000.0, latency_late, total->cnt,
+ (total->cnt > 0) ? 100.0 * latency_late / total->cnt : 0.0);
if (throttle_delay || progress || latency_limit)
printSimpleStats("latency", &total->latency);
else
{
/* no measurement, show average latency computed from run time */
- printf("latency average = %.3f ms\n",
- 0.001 * total_duration * nclients / total->cnt);
+ printf("latency average = %.3f ms%s\n",
+ 0.001 * total_duration * nclients / total_cnt,
+ failures > 0 ? " (including failures)" : "");
}
if (throttle_delay)
@@ -5673,7 +6366,7 @@ printResults(StatsData *total,
*/
if (is_connect)
{
- printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / total->cnt);
+ printf("average connection time = %.3f ms\n", 0.001 * conn_total_duration / (total->cnt + failures));
printf("tps = %f (including reconnection times)\n", tps);
}
else
@@ -5692,6 +6385,9 @@ printResults(StatsData *total,
if (per_script_stats)
{
StatsData *sstats = &sql_script[i].stats;
+ int64 script_failures = getFailures(sstats);
+ int64 script_total_cnt =
+ sstats->cnt + sstats->skipped + script_failures;
printf("SQL script %d: %s\n"
" - weight: %d (targets %.1f%% of total)\n"
@@ -5701,25 +6397,55 @@ printResults(StatsData *total,
100.0 * sql_script[i].weight / total_weight,
sstats->cnt,
100.0 * sstats->cnt / total->cnt,
- (sstats->cnt - sstats->skipped) / bench_duration);
+ sstats->cnt / bench_duration);
+
+ printf(" - number of failed transactions: " INT64_FORMAT " (%.3f%%)\n",
+ script_failures,
+ 100.0 * script_failures / script_total_cnt);
+
+ if (failures_detailed)
+ {
+ printf(" - number of serialization failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->serialization_failures,
+ (100.0 * sstats->serialization_failures /
+ script_total_cnt));
+ printf(" - number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->deadlock_failures,
+ (100.0 * sstats->deadlock_failures /
+ script_total_cnt));
+ }
- if (throttle_delay && latency_limit && sstats->cnt > 0)
+ /* it can be non-zero only if max_tries is not equal to one */
+ if (max_tries != 1)
+ {
+ printf(" - number of transactions retried: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->retried,
+ 100.0 * sstats->retried / script_total_cnt);
+ printf(" - total number of retries: " INT64_FORMAT "\n",
+ sstats->retries);
+ }
+
+ if (throttle_delay && latency_limit && script_total_cnt > 0)
printf(" - number of transactions skipped: " INT64_FORMAT " (%.3f%%)\n",
sstats->skipped,
- 100.0 * sstats->skipped / sstats->cnt);
+ 100.0 * sstats->skipped / script_total_cnt);
printSimpleStats(" - latency", &sstats->latency);
}
- /* Report per-command latencies */
+ /*
+ * Report per-command statistics: latencies, retries after errors,
+ * failures (errors without retrying).
+ */
if (report_per_command)
{
Command **commands;
- if (per_script_stats)
- printf(" - statement latencies in milliseconds:\n");
- else
- printf("statement latencies in milliseconds:\n");
+ printf("%sstatement latencies in milliseconds%s:\n",
+ per_script_stats ? " - " : "",
+ (max_tries == 1 ?
+ " and failures" :
+ ", failures and retries"));
for (commands = sql_script[i].commands;
*commands != NULL;
@@ -5727,10 +6453,19 @@ printResults(StatsData *total,
{
SimpleStats *cstats = &(*commands)->stats;
- printf(" %11.3f %s\n",
- (cstats->count > 0) ?
- 1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->first_line);
+ if (max_tries == 1)
+ printf(" %11.3f %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->first_line);
+ else
+ printf(" %11.3f %10" INT64_MODIFIER "d %10" INT64_MODIFIER "d %s\n",
+ (cstats->count > 0) ?
+ 1000.0 * cstats->sum / cstats->count : 0.0,
+ (*commands)->failures,
+ (*commands)->retries,
+ (*commands)->first_line);
}
}
}
@@ -5810,7 +6545,7 @@ main(int argc, char **argv)
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
- {"report-latencies", no_argument, NULL, 'r'},
+ {"report-per-command", no_argument, NULL, 'r'},
{"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
@@ -5832,6 +6567,9 @@ main(int argc, char **argv)
{"show-script", required_argument, NULL, 10},
{"partitions", required_argument, NULL, 11},
{"partition-method", required_argument, NULL, 12},
+ {"failures-detailed", no_argument, NULL, 13},
+ {"max-tries", required_argument, NULL, 14},
+ {"verbose-errors", no_argument, NULL, 15},
{NULL, 0, NULL, 0}
};
@@ -6185,6 +6923,28 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 13: /* failures-detailed */
+ benchmarking_option_set = true;
+ failures_detailed = true;
+ break;
+ case 14: /* max-tries */
+ {
+ int32 max_tries_arg = atoi(optarg);
+
+ if (max_tries_arg < 0)
+ {
+ pg_log_fatal("invalid number of maximum tries: \"%s\"", optarg);
+ exit(1);
+ }
+
+ benchmarking_option_set = true;
+ max_tries = (uint32) max_tries_arg;
+ }
+ break;
+ case 15: /* verbose-errors */
+ benchmarking_option_set = true;
+ verbose_errors = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -6366,6 +7126,15 @@ main(int argc, char **argv)
exit(1);
}
+ if (!max_tries)
+ {
+ if (!latency_limit && duration <= 0)
+ {
+ pg_log_fatal("an unlimited number of transaction tries can only be used with --latency-limit or a duration (-T)");
+ exit(1);
+ }
+ }
+
/*
* save main process id in the global variable because process id will be
* changed after fork.
@@ -6578,6 +7347,10 @@ main(int argc, char **argv)
mergeSimpleStats(&stats.lag, &thread->stats.lag);
stats.cnt += thread->stats.cnt;
stats.skipped += thread->stats.skipped;
+ stats.retries += thread->stats.retries;
+ stats.retried += thread->stats.retried;
+ stats.serialization_failures += thread->stats.serialization_failures;
+ stats.deadlock_failures += thread->stats.deadlock_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
@@ -6724,7 +7497,8 @@ threadRun(void *arg)
if (min_usec > this_usec)
min_usec = this_usec;
}
- else if (st->state == CSTATE_WAIT_RESULT)
+ else if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/*
* waiting for result from server - nothing to do unless the
@@ -6813,7 +7587,8 @@ threadRun(void *arg)
{
CState *st = &state[i];
- if (st->state == CSTATE_WAIT_RESULT)
+ if (st->state == CSTATE_WAIT_RESULT ||
+ st->state == CSTATE_WAIT_ROLLBACK_RESULT)
{
/* don't call advanceConnectionState unless data is available */
int sock = PQsocket(st->con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index f1341092fe..d173ceae7a 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -11,7 +11,9 @@ use Config;
# start a pgbench specific server
my $node = PostgreSQL::Test::Cluster->new('main');
-$node->init;
+# Set to untranslated messages, to be able to compare program output with
+# expected strings.
+$node->init(extra => [ '--locale', 'C' ]);
$node->start;
# tablespace for testing, because partitioned tables cannot use pg_default
@@ -109,7 +111,8 @@ $node->pgbench(
qr{builtin: TPC-B},
qr{clients: 2\b},
qr{processed: 10/10},
- qr{mode: simple}
+ qr{mode: simple},
+ qr{maximum number of tries: 1}
],
[qr{^$}],
'pgbench tpcb-like');
@@ -1198,6 +1201,214 @@ $node->pgbench(
check_pgbench_logs($bdir, '001_pgbench_log_3', 1, 10, 10,
qr{^0 \d{1,2} \d+ \d \d+ \d+$});
+# abortion of the client if the script contains an incomplete transaction block
+$node->pgbench(
+ '--no-vacuum', 2, [ qr{processed: 1/10} ],
+ [ qr{client 0 aborted: end of script reached without completing the last transaction} ],
+ 'incomplete transaction block',
+ { '001_pgbench_incomplete_transaction_block' => q{BEGIN;SELECT 1;} });
+
+# Test the concurrent update in the table row and deadlocks.
+
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE first_client_table (value integer); '
+ . 'CREATE UNLOGGED TABLE xy (x integer, y integer); '
+ . 'INSERT INTO xy VALUES (1, 2);');
+
+# Serialization error and retry
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
+
+# Check that we have a serialization error and the same random value of the
+# delta variable in the next try
+my $err_pattern =
+ "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ . "ERROR: could not serialize access due to concurrent update\\b.*"
+ . "\\g1";
+
+$node->pgbench(
+ "-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
+ 0,
+ [ qr{processed: 2/2\b}, qr{number of transactions retried: 1\b},
+ qr{total number of retries: 1\b} ],
+ [ qr/$err_pattern/s ],
+ 'concurrent update with retrying',
+ {
+ '001_pgbench_serialization' => q{
+-- What's happening:
+-- The first client starts the transaction with the isolation level Repeatable
+-- Read:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+--
+-- The second client starts a similar transaction with the same isolation level:
+--
+-- BEGIN;
+-- UPDATE xy SET y = ... WHERE x = 1;
+-- <waiting for the first client>
+--
+-- The first client commits its transaction, and the second client gets a
+-- serialization error.
+
+\set delta random(-5000, 5000)
+
+-- The second client will stop here
+SELECT pg_advisory_lock(0);
+
+-- Start transaction with concurrent update
+BEGIN;
+UPDATE xy SET y = y + :delta WHERE x = 1 AND pg_advisory_lock(1) IS NOT NULL;
+
+-- Wait for the second client
+DO $$
+DECLARE
+ exists boolean;
+ waiters integer;
+BEGIN
+ -- The second client always comes in second, and the number of rows in the
+ -- table first_client_table reflect this. Here the first client inserts a row,
+ -- so the second client will see a non-empty table when repeating the
+ -- transaction after the serialization error.
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF NOT exists THEN
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ -- And wait until the second client tries to get the same lock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 1::bigint) AND NOT granted;
+ IF waiters = 1 THEN
+ INSERT INTO first_client_table VALUES (1);
+
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+ END IF;
+END$$;
+
+COMMIT;
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+
+$node->safe_psql('postgres', 'DELETE FROM first_client_table;');
+
+local $ENV{PGOPTIONS} = "-c default_transaction_isolation=read\\ committed";
+
+# Deadlock error and retry
+
+# Check that we have a deadlock error
+$err_pattern =
+ "client (0|1) got an error in command (3|5) \\(SQL\\) of script 0; "
+ . "ERROR: deadlock detected\\b";
+
+$node->pgbench(
+ "-n -c 2 -t 1 --max-tries 2 --verbose-errors",
+ 0,
+ [ qr{processed: 2/2\b}, qr{number of transactions retried: 1\b},
+ qr{total number of retries: 1\b} ],
+ [ qr{$err_pattern} ],
+ 'deadlock with retrying',
+ {
+ '001_pgbench_deadlock' => q{
+-- What's happening:
+-- The first client gets the lock 2.
+-- The second client gets the lock 3 and tries to get the lock 2.
+-- The first client tries to get the lock 3 and one of them gets a deadlock
+-- error.
+--
+-- A client that does not get a deadlock error must hold a lock at the
+-- transaction start. Thus in the end it releases all of its locks before the
+-- client with the deadlock error starts a retry (we do not want any errors
+-- again).
+
+-- Since the client with the deadlock error has not released the blocking locks,
+-- let's do this here.
+SELECT pg_advisory_unlock_all();
+
+-- The second client and the client with the deadlock error stop here
+SELECT pg_advisory_lock(0);
+SELECT pg_advisory_lock(1);
+
+-- The second client and the client with the deadlock error always come after
+-- the first and the number of rows in the table first_client_table reflects
+-- this. Here the first client inserts a row, so in the future the table is
+-- always non-empty.
+DO $$
+DECLARE
+ exists boolean;
+BEGIN
+ SELECT EXISTS (SELECT * FROM first_client_table) INTO STRICT exists;
+ IF exists THEN
+ -- We are the second client or the client with the deadlock error
+
+ -- The first client will take care by itself of this lock (see below)
+ PERFORM pg_advisory_unlock(0);
+
+ PERFORM pg_advisory_lock(3);
+
+ -- The second client can get a deadlock here
+ PERFORM pg_advisory_lock(2);
+ ELSE
+ -- We are the first client
+
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (1);
+
+ PERFORM pg_advisory_lock(2);
+ END IF;
+END$$;
+
+DO $$
+DECLARE
+ num_rows integer;
+ waiters integer;
+BEGIN
+ -- Check if we are the first client
+ SELECT COUNT(*) FROM first_client_table INTO STRICT num_rows;
+ IF num_rows = 1 THEN
+ -- This code should not be used in a new transaction after an error
+ INSERT INTO first_client_table VALUES (2);
+
+ -- Let the second client begin
+ PERFORM pg_advisory_unlock(0);
+ PERFORM pg_advisory_unlock(1);
+
+ -- Make sure the second client is ready for deadlock
+ LOOP
+ SELECT COUNT(*) INTO STRICT waiters FROM pg_locks WHERE
+ locktype = 'advisory' AND
+ objsubid = 1 AND
+ ((classid::bigint << 32) | objid::bigint = 2::bigint) AND
+ NOT granted;
+
+ IF waiters = 1 THEN
+ -- Exit loop
+ EXIT;
+ END IF;
+ END LOOP;
+
+ PERFORM pg_advisory_lock(0);
+ -- And the second client took care by itself of the lock 1
+ END IF;
+END$$;
+
+-- The first client can get a deadlock here
+SELECT pg_advisory_lock(3);
+
+SELECT pg_advisory_unlock_all();
+}
+ });
+
+# Clean up
+$node->safe_psql('postgres', 'DROP TABLE first_client_table, xy;');
+
+
# done
$node->safe_psql('postgres', 'DROP TABLESPACE regress_pgbench_tap_1_ts');
$node->stop;
diff --git a/src/bin/pgbench/t/002_pgbench_no_server.pl b/src/bin/pgbench/t/002_pgbench_no_server.pl
index acad19edd0..a5074c70d9 100644
--- a/src/bin/pgbench/t/002_pgbench_no_server.pl
+++ b/src/bin/pgbench/t/002_pgbench_no_server.pl
@@ -188,6 +188,16 @@ my @options = (
'-i --partition-method=hash',
[qr{partition-method requires greater than zero --partitions}]
],
+ [
+ 'bad maximum number of tries',
+ '--max-tries -10',
+ [qr{invalid number of maximum tries: "-10"}]
+ ],
+ [
+ 'an infinite number of tries',
+ '--max-tries 0',
+ [qr{an unlimited number of transaction tries can only be used with --latency-limit or a duration}]
+ ],
# logging sub-options
[
diff --git a/src/fe_utils/conditional.c b/src/fe_utils/conditional.c
index 0bf877e895..5a94664989 100644
--- a/src/fe_utils/conditional.c
+++ b/src/fe_utils/conditional.c
@@ -24,13 +24,25 @@ conditional_stack_create(void)
}
/*
- * destroy stack
+ * Destroy all the elements from the stack. The stack itself is not freed.
*/
void
-conditional_stack_destroy(ConditionalStack cstack)
+conditional_stack_reset(ConditionalStack cstack)
{
+ if (!cstack)
+ return; /* nothing to do here */
+
while (conditional_stack_pop(cstack))
continue;
+}
+
+/*
+ * destroy stack
+ */
+void
+conditional_stack_destroy(ConditionalStack cstack)
+{
+ conditional_stack_reset(cstack);
free(cstack);
}
diff --git a/src/include/fe_utils/conditional.h b/src/include/fe_utils/conditional.h
index b28189471c..fa53d86501 100644
--- a/src/include/fe_utils/conditional.h
+++ b/src/include/fe_utils/conditional.h
@@ -73,6 +73,8 @@ typedef struct ConditionalStackData *ConditionalStack;
extern ConditionalStack conditional_stack_create(void);
+extern void conditional_stack_reset(ConditionalStack cstack);
+
extern void conditional_stack_destroy(ConditionalStack cstack);
extern int conditional_stack_depth(ConditionalStack cstack);
--
2.17.1
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-22 09:44 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-22 09:44 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: ishii@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> I've checked other places using <xref/> referring to <refsect2>, and found
>> that "xreflabel"s are used in such <refsect2> tags. So, I'll fix it
>> in this style.
>
> I attached the updated patch. I also fixed the following paragraph which I had
> forgotten to fix in the previous patch.
>
> The first seven lines report some of the most important parameter settings.
> The sixth line reports the maximum number of tries for transactions with
> serialization or deadlock errors
Thank you for the updated patch. I think the patches look good and now
it's ready for commit. If there's no objection, I would like to
commit/push the patches.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-23 10:08 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-23 10:08 UTC (permalink / raw)
To: ishii@sraoss.co.jp; +Cc: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> I attached the updated patch. I also fixed the following paragraph which I had
>> forgotten to fix in the previous patch.
>>
>> The first seven lines report some of the most important parameter settings.
>> The sixth line reports the maximum number of tries for transactions with
>> serialization or deadlock errors
>
> Thank you for the updated patch. I think the patches look good and now
> it's ready for commit. If there's no objection, I would like to
> commit/push the patches.
The patch Pushed. Thank you!
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-23 18:26 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-03-23 18:26 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> The patch Pushed. Thank you!
My hoary animal prairiedog doesn't like this [1]:
# Failed test 'concurrent update with retrying stderr /(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\\g1)/'
# at t/001_pgbench_with_server.pl line 1229.
# 'pgbench: pghost: /tmp/nhghgwAoki pgport: 58259 nclients: 2 nxacts: 1 dbName: postgres
...
# pgbench: client 0 got an error in command 3 (SQL) of script 0; ERROR: could not serialize access due to concurrent update
...
# '
# doesn't match '(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\\g1)'
# Looks like you failed 1 test of 425.
I'm not sure what the "\\b.*\\g1" part of this regex is meant to
accomplish, but it seems to be assuming more than it should
about the output format of TAP messages.
regards, tom lane
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prairiedog&dt=2022-03-23%2013%3A21%3A44
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-24 02:43 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-24 02:43 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Wed, 23 Mar 2022 14:26:54 -0400
Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> > The patch Pushed. Thank you!
>
> My hoary animal prairiedog doesn't like this [1]:
>
> # Failed test 'concurrent update with retrying stderr /(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\\g1)/'
> # at t/001_pgbench_with_server.pl line 1229.
> # 'pgbench: pghost: /tmp/nhghgwAoki pgport: 58259 nclients: 2 nxacts: 1 dbName: postgres
> ...
> # pgbench: client 0 got an error in command 3 (SQL) of script 0; ERROR: could not serialize access due to concurrent update
> ...
> # '
> # doesn't match '(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\\g1)'
> # Looks like you failed 1 test of 425.
>
> I'm not sure what the "\\b.*\\g1" part of this regex is meant to
> accomplish, but it seems to be assuming more than it should
> about the output format of TAP messages.
I have edited the test code from the original patch by mistake, but
I could not realize because the test works in my machine without any
errors somehow.
I attached a patch to fix the test as was in the original patch, where
backreferences are used to check retry of the same query.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments:
fix_pgbench_test.patch (text/x-diff, 761B) download
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index d173ceae7a..3eb5905e5a 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -1222,7 +1222,8 @@ local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
# Check that we have a serialization error and the same random value of the
# delta variable in the next try
my $err_pattern =
- "client (0|1) got an error in command 3 \\(SQL\\) of script 0; "
+ "(client (0|1) sending UPDATE xy SET y = y \\+ -?\\d+\\b).*"
+ . "client \\g2 got an error in command 3 \\(SQL\\) of script 0; "
. "ERROR: could not serialize access due to concurrent update\\b.*"
. "\\g1";
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-24 04:49 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-24 04:49 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: tgl@sss.pgh.pa.us; ishii@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> My hoary animal prairiedog doesn't like this [1]:
>>
>> # Failed test 'concurrent update with retrying stderr /(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\\g1)/'
>> # at t/001_pgbench_with_server.pl line 1229.
>> # 'pgbench: pghost: /tmp/nhghgwAoki pgport: 58259 nclients: 2 nxacts: 1 dbName: postgres
>> ...
>> # pgbench: client 0 got an error in command 3 (SQL) of script 0; ERROR: could not serialize access due to concurrent update
>> ...
>> # '
>> # doesn't match '(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\\g1)'
>> # Looks like you failed 1 test of 425.
>>
>> I'm not sure what the "\\b.*\\g1" part of this regex is meant to
>> accomplish, but it seems to be assuming more than it should
>> about the output format of TAP messages.
>
> I have edited the test code from the original patch by mistake, but
> I could not realize because the test works in my machine without any
> errors somehow.
>
> I attached a patch to fix the test as was in the original patch, where
> backreferences are used to check retry of the same query.
My machine (Ubuntu 20) did not complain either. Maybe perl version
difference? Any way, the fix pushed. Let's see how prairiedog feels.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-24 18:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-03-24 18:02 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
>> My hoary animal prairiedog doesn't like this [1]:
> My machine (Ubuntu 20) did not complain either. Maybe perl version
> difference? Any way, the fix pushed. Let's see how prairiedog feels.
Still not happy. After some digging in man pages, I believe the
problem is that its old version of Perl does not understand "\gN"
backreferences. Is there a good reason to be using that rather
than the traditional "\N" backref notation?
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-24 23:50 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-24 23:50 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: ishii@sraoss.co.jp; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> My machine (Ubuntu 20) did not complain either. Maybe perl version
>> difference? Any way, the fix pushed. Let's see how prairiedog feels.
>
> Still not happy. After some digging in man pages, I believe the
> problem is that its old version of Perl does not understand "\gN"
> backreferences. Is there a good reason to be using that rather
> than the traditional "\N" backref notation?
I don't see a reason to use "\gN" either. Actually after applying
attached patch, my machine is still happy with pgbench test.
Yugo?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
fix_pgbench_test_v2.patch (text/x-patch, 582B) download
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 60cae1e843..22a23489e8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -1224,7 +1224,7 @@ my $err_pattern =
"(client (0|1) sending UPDATE xy SET y = y \\+ -?\\d+\\b).*"
. "client \\g2 got an error in command 3 \\(SQL\\) of script 0; "
. "ERROR: could not serialize access due to concurrent update\\b.*"
- . "\\g1";
+ . "\\1";
$node->pgbench(
"-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-25 00:05 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-03-25 00:05 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> I don't see a reason to use "\gN" either. Actually after applying
> attached patch, my machine is still happy with pgbench test.
Note that the \\g2 just above also needs to be changed.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-25 00:14 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:07 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 2 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-25 00:14 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: ishii@sraoss.co.jp; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> Note that the \\g2 just above also needs to be changed.
Oops. Thanks. New patch attached. Test has passed on my machine.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
fix_pgbench_test_v3.patch (text/x-patch, 763B) download
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 60cae1e843..ca71f968dc 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -1222,9 +1222,9 @@ local $ENV{PGOPTIONS} = "-c default_transaction_isolation=repeatable\\ read";
# delta variable in the next try
my $err_pattern =
"(client (0|1) sending UPDATE xy SET y = y \\+ -?\\d+\\b).*"
- . "client \\g2 got an error in command 3 \\(SQL\\) of script 0; "
+ . "client \\2 got an error in command 3 \\(SQL\\) of script 0; "
. "ERROR: could not serialize access due to concurrent update\\b.*"
- . "\\g1";
+ . "\\1";
$node->pgbench(
"-n -c 2 -t 1 -d --verbose-errors --max-tries 2",
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-25 01:07 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 01:24 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-03-25 01:07 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> Oops. Thanks. New patch attached. Test has passed on my machine.
I reproduced the failure on another machine with perl 5.8.8,
and I can confirm that this patch fixes it.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:07 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-25 01:24 ` Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-25 01:24 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: ishii@sraoss.co.jp; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> I reproduced the failure on another machine with perl 5.8.8,
> and I can confirm that this patch fixes it.
Thank you for the test. I have pushed the patch.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-25 01:12 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-25 01:12 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Fri, 25 Mar 2022 09:14:00 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> > Note that the \\g2 just above also needs to be changed.
>
> Oops. Thanks. New patch attached. Test has passed on my machine.
This patch works for me. I think it is ok to use \N instead of \gN.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-25 01:25 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-25 01:25 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> Oops. Thanks. New patch attached. Test has passed on my machine.
>
> This patch works for me. I think it is ok to use \N instead of \gN.
Thanks. Patch pushed.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-26 16:36 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-03-26 16:36 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> Thanks. Patch pushed.
This patch has caused the PDF documentation to fail to build cleanly:
[WARN] FOUserAgent - The contents of fo:block line 1 exceed the available area in the inline-progression direction by more than 50 points. (See position 125066:375)
It's complaining about this:
<synopsis>
<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
which runs much too wide in HTML format too, even though that toolchain
doesn't tell you so.
We could silence the warning by inserting an arbitrary line break or two,
or refactoring the syntax description into multiple parts. Either way
seems to create a risk of confusion.
TBH, I think the *real* problem is that the complexity of this log format
has blown past "out of hand". Can't we simplify it? Who is really going
to use all these numbers? I pity the poor sucker who tries to write a
log analysis tool that will handle all the variants.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-27 06:28 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 02:29 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 2 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-27 06:28 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: ishii@sraoss.co.jp; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> This patch has caused the PDF documentation to fail to build cleanly:
>
> [WARN] FOUserAgent - The contents of fo:block line 1 exceed the available area in the inline-progression direction by more than 50 points. (See position 125066:375)
>
> It's complaining about this:
>
> <synopsis>
> <replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
> </synopsis>
>
> which runs much too wide in HTML format too, even though that toolchain
> doesn't tell you so.
Yeah.
> We could silence the warning by inserting an arbitrary line break or two,
> or refactoring the syntax description into multiple parts. Either way
> seems to create a risk of confusion.
I think we can fold the line nicely. Here is the rendered image.
Before:
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
After:
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
{ failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
Note that before it was like this:
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]
So newly added items are "{ failures | serialization_failures deadlock_failures }" and " [ retried retries ]".
> TBH, I think the *real* problem is that the complexity of this log format
> has blown past "out of hand". Can't we simplify it? Who is really going
> to use all these numbers? I pity the poor sucker who tries to write a
> log analysis tool that will handle all the variants.
Well, the extra logging items above only appear when the retry feature
is enabled. For those who do not use the feature the only new logging
item is "failures". For those who use the feature, the extra logging
items are apparently necessary. For example if we write an application
using repeatable read or serializable transaction isolation mode,
retrying failed transactions due to srialization error is an essential
technique. Also the retry rate of transactions will deeply affect the
performance and in such use cases the newly added items will be
precisou information. I would suggest leave the log items as it is.
Patch attached.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
pgbench-doc.patch (text/x-patch, 1.8K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index ebdb4b3f46..b65b813ebe 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2398,10 +2398,11 @@ END;
<para>
With the <option>--aggregate-interval</option> option, a different
- format is used for the log files:
+ format is used for the log files (note that the actual log line is not folded).
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
+ <replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>
+ { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-28 02:29 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 03:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-28 02:29 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Sun, 27 Mar 2022 15:28:41 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> > This patch has caused the PDF documentation to fail to build cleanly:
> >
> > [WARN] FOUserAgent - The contents of fo:block line 1 exceed the available area in the inline-progression direction by more than 50 points. (See position 125066:375)
> >
> > It's complaining about this:
> >
> > <synopsis>
> > <replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
> > </synopsis>
> >
> > which runs much too wide in HTML format too, even though that toolchain
> > doesn't tell you so.
>
> Yeah.
>
> > We could silence the warning by inserting an arbitrary line break or two,
> > or refactoring the syntax description into multiple parts. Either way
> > seems to create a risk of confusion.
>
> I think we can fold the line nicely. Here is the rendered image.
>
> Before:
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
>
> After:
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
> { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
>
> Note that before it was like this:
>
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]
>
> So newly added items are "{ failures | serialization_failures deadlock_failures }" and " [ retried retries ]".
>
> > TBH, I think the *real* problem is that the complexity of this log format
> > has blown past "out of hand". Can't we simplify it? Who is really going
> > to use all these numbers? I pity the poor sucker who tries to write a
> > log analysis tool that will handle all the variants.
>
> Well, the extra logging items above only appear when the retry feature
> is enabled. For those who do not use the feature the only new logging
> item is "failures". For those who use the feature, the extra logging
> items are apparently necessary. For example if we write an application
> using repeatable read or serializable transaction isolation mode,
> retrying failed transactions due to srialization error is an essential
> technique. Also the retry rate of transactions will deeply affect the
> performance and in such use cases the newly added items will be
> precisou information. I would suggest leave the log items as it is.
>
> Patch attached.
Even applying this patch, "make postgres-A4.pdf" arises the warning on my
machine. After some investigations, I found that previous document had a break
after 'num_transactions', but it has been removed due to this commit. So,
I would like to get back this as it was. I attached the patch.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments:
pgbench-doc_v2.patch (text/x-diff, 1.6K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index ebdb4b3f46..4437d5ef53 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2401,7 +2401,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 02:29 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-28 03:17 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 03:48 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-28 03:17 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
> Even applying this patch, "make postgres-A4.pdf" arises the warning on my
> machine. After some investigations, I found that previous document had a break
> after 'num_transactions', but it has been removed due to this commit.
Yes, your patch removed "&zwsp;".
> So,
> I would like to get back this as it was. I attached the patch.
This produces errors. Needs ";" postfix?
ref/pgbench.sgml:2404: parser error : EntityRef: expecting ';'
le>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp
^
ref/pgbench.sgml:2781: parser error : chunk is not well balanced
^
reference.sgml:251: parser error : Failure to process entity pgbench
&pgbench;
^
reference.sgml:251: parser error : Entity 'pgbench' not defined
&pgbench;
^
reference.sgml:296: parser error : chunk is not well balanced
^
postgres.sgml:240: parser error : Failure to process entity reference
&reference;
^
postgres.sgml:240: parser error : Entity 'reference' not defined
&reference;
^
make: *** [Makefile:135: html-stamp] $B%(%i!<(B 1
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 02:29 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 03:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-28 03:48 ` Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Yugo NAGATA @ 2022-03-28 03:48 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
On Mon, 28 Mar 2022 12:17:13 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> > Even applying this patch, "make postgres-A4.pdf" arises the warning on my
> > machine. After some investigations, I found that previous document had a break
> > after 'num_transactions', but it has been removed due to this commit.
>
> Yes, your patch removed "&zwsp;".
>
> > So,
> > I would like to get back this as it was. I attached the patch.
>
> This produces errors. Needs ";" postfix?
Oops. Yes, it needs ';'. Also, I found another "&zwsp;" dropped.
I attached the fixed patch.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
Attachments:
pgbench-doc_v3.patch (text/x-diff, 1.7K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index ebdb4b3f46..b16a5b9b7b 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2401,7 +2401,7 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable>&zwsp; <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>&zwsp; { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
</synopsis>
where
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 02:29 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 03:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 03:48 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
@ 2022-03-28 06:50 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 06:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-28 06:50 UTC (permalink / raw)
To: nagata@sraoss.co.jp; +Cc: ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>> > Even applying this patch, "make postgres-A4.pdf" arises the warning on my
>> > machine. After some investigations, I found that previous document had a break
>> > after 'num_transactions', but it has been removed due to this commit.
>>
>> Yes, your patch removed "&zwsp;".
>>
>> > So,
>> > I would like to get back this as it was. I attached the patch.
>>
>> This produces errors. Needs ";" postfix?
>
> Oops. Yes, it needs ';'. Also, I found another "&zwsp;" dropped.
> I attached the fixed patch.
Basic problem with this patch is, this may solve the issue with pdf
generation but this does not solve the issue with HTML generation. The
PDF manual of pgbench has ridiculously long line, which Tom Lane
complained too:
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
Why can't we use just line feeds instead of &zwsp;? Although it's not
a command usage but the SELECT manual already uses line feeds to
nicely break into multiple lines of command usage.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 02:29 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 03:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 03:48 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-28 06:53 ` Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-28 06:53 UTC (permalink / raw)
To: ishii@sraoss.co.jp; +Cc: nagata@sraoss.co.jp; tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
>>> > Even applying this patch, "make postgres-A4.pdf" arises the warning on my
>>> > machine. After some investigations, I found that previous document had a break
>>> > after 'num_transactions', but it has been removed due to this commit.
>>>
>>> Yes, your patch removed "&zwsp;".
>>>
>>> > So,
>>> > I would like to get back this as it was. I attached the patch.
>>>
>>> This produces errors. Needs ";" postfix?
>>
>> Oops. Yes, it needs ';'. Also, I found another "&zwsp;" dropped.
>> I attached the fixed patch.
>
> Basic problem with this patch is, this may solve the issue with pdf
> generation but this does not solve the issue with HTML generation. The
> PDF manual of pgbench has ridiculously long line, which Tom Lane
I meant "HTML manual" here.
> complained too:
>
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
>
> Why can't we use just line feeds instead of &zwsp;? Although it's not
> a command usage but the SELECT manual already uses line feeds to
> nicely break into multiple lines of command usage.
>
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-03-28 07:57 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 00:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 2 replies; 82+ messages in thread
From: Alvaro Herrera @ 2022-03-28 07:57 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Hello,
On 2022-Mar-27, Tatsuo Ishii wrote:
> After:
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
> { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
You're showing an indentation, but looking at the HTML output there is
no such. Is the HTML processor eating leading whitespace or something
like that?
I think that the explanatory paragraph is way too long now, particularly
since it explains --failures-detailed starting in the middle. Also, the
example output doesn't include the failures-detailed mode. I suggest
that this should be broken down even more; first to explain the output
without failures-detailed, including an example, and then the output
with failures-detailed, and an example of that. Something like this,
perhaps:
Aggregated Logging
With the --aggregate-interval option, a different format is used for the log files (note that the actual log line is not folded).
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
failures [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
where interval_start is the start of the interval (as a Unix epoch time stamp), num_transactions is the number of transactions within the interval, sum_latency is the sum of the transaction latencies within the interval, sum_latency_2 is the sum of squares of the transaction latencies within the interval, min_latency is the minimum latency within the interval, and max_latency is the maximum latency within the interval, failures is the number of transactions that ended with a failed SQL command within the interval.
The next fields, sum_lag, sum_lag_2, min_lag, and max_lag, are only present if the --rate option is used. They provide statistics about the time each transaction had to wait for the previous one to finish, i.e., the difference between each transaction's scheduled start time and the time it actually started. The next field, skipped, is only present if the --latency-limit option is used, too. It counts the number of transactions skipped because they would have started too late. The retried and retries fields are present only if the --max-tries option is not equal to 1. They report the number of retried transactions and the sum of all retries after serialization or deadlock errors within the interval. Each transaction is counted in the interval when it was committed.
Notice that while the plain (unaggregated) log file shows which script was used for each transaction, the aggregated log does not. Therefore if you need per-script data, you need to aggregate the data on your own.
Here is some example output:
1345828501 5601 1542744 483552416 61 2573 0
1345828503 7884 1979812 565806736 60 1479 0
1345828505 7208 1979422 567277552 59 1391 0
1345828507 7685 1980268 569784714 60 1398 0
1345828509 7073 1979779 573489941 236 1411 0
If you use option --failures-detailed, instead of the sum of all failed transactions you will get more detailed statistics for the failed transactions:
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
serialization_failures deadlock_failures [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
This is similar to the above, but here the single 'failures' figure is replaced by serialization_failures which is the number of transactions that got a serialization error and were not retried after this, deadlock_failures which is the number of transactions that got a deadlock error and were not retried after this. The other fields are as above. Here is some example output:
[example with detailed failures]
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"If you have nothing to say, maybe you need just the right tool to help you
not say it." (New York Times, about Microsoft PowerPoint)
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2022-03-28 13:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Tom Lane @ 2022-03-28 13:33 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
>> After:
>> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
>> { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
> I think that the explanatory paragraph is way too long now, particularly
> since it explains --failures-detailed starting in the middle. Also, the
> example output doesn't include the failures-detailed mode.
I think the problem is not merely one of documentation, but one of
bad design. Up to now it was possible to tell what was what from
counting the number of columns in the output; but with this design,
that is impossible. That should be fixed. The first thing you have
got to do is drop the alternation { failures | serialization_failures
deadlock_failures }. That doesn't make any sense in the first place:
counting serialization and deadlock failures doesn't make it impossible
for other errors to occur. It'd probably make the most sense to have
three columns always, serialization, deadlock and total. Now maybe
that change alone is sufficient, but I'm not convinced, because the
multiple options at the end of the line mean we will never again be
able to add any more columns without reintroducing ambiguity. I
would be happier if the syntax diagram were such that columns could
only be dropped from right to left.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-04-01 00:57 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-01 05:34 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 2 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-01 00:57 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: alvherre@alvh.no-ip.org; ishii@sraoss.co.jp; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
> Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
>>> After:
>>> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
>>> { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
>
>> I think that the explanatory paragraph is way too long now, particularly
>> since it explains --failures-detailed starting in the middle. Also, the
>> example output doesn't include the failures-detailed mode.
>
> I think the problem is not merely one of documentation, but one of
> bad design. Up to now it was possible to tell what was what from
> counting the number of columns in the output; but with this design,
> that is impossible. That should be fixed. The first thing you have
> got to do is drop the alternation { failures | serialization_failures
> deadlock_failures }. That doesn't make any sense in the first place:
> counting serialization and deadlock failures doesn't make it impossible
> for other errors to occur. It'd probably make the most sense to have
> three columns always, serialization, deadlock and total.
+1.
> Now maybe
> that change alone is sufficient, but I'm not convinced, because the
> multiple options at the end of the line mean we will never again be
> able to add any more columns without reintroducing ambiguity. I
> would be happier if the syntax diagram were such that columns could
> only be dropped from right to left.
Or those three columns always, sum_lag sum_lag_2, min_lag max_lag, skipped, retried retries?
Anyway now that current CF is closing, it will not be possible to
change those logging design soon. Or can we change the logging design
even after CF is closed?
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-01 05:34 ` Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 0 replies; 82+ messages in thread
From: Fabien COELHO @ 2022-04-01 05:34 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
> Or those three columns always, sum_lag sum_lag_2, min_lag max_lag,
> skipped, retried retries?
>
> Anyway now that current CF is closing, it will not be possible to
> change those logging design soon. Or can we change the logging design
> even after CF is closed?
My 0.02€: I'm not sure how the official guidelines are to be interpreted
in that case, but if the design is to be changed, ISTM that it is better
to do it before a release instead of letting the release out with one
format and changing it in the next release?
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-03 06:28 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-03 06:28 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> I think the problem is not merely one of documentation, but one of
>> bad design. Up to now it was possible to tell what was what from
>> counting the number of columns in the output; but with this design,
>> that is impossible. That should be fixed. The first thing you have
>> got to do is drop the alternation { failures | serialization_failures
>> deadlock_failures }. That doesn't make any sense in the first place:
>> counting serialization and deadlock failures doesn't make it impossible
>> for other errors to occur. It'd probably make the most sense to have
>> three columns always, serialization, deadlock and total.
>
> +1.
>
>> Now maybe
>> that change alone is sufficient, but I'm not convinced, because the
>> multiple options at the end of the line mean we will never again be
>> able to add any more columns without reintroducing ambiguity. I
>> would be happier if the syntax diagram were such that columns could
>> only be dropped from right to left.
>
> Or those three columns always, sum_lag sum_lag_2, min_lag max_lag, skipped, retried retries?
What about this? (a log line is not actually folded)
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
failures serialization_failures deadlock_failures retried retries [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]
failures:
always 0 (if --max-tries is 1, the default)
sum of serialization_failures and deadlock_failures (if --max-tries is not 1)
serialization_failures and deadlock_failures:
always 0 (if --max-tries is 1, the default)
0 or more (if --max-tries is not 1)
retried and retries:
always 0 (if --max-tries is 1, the default)
0 or more (if --max-tries is not 1)
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-03 08:20 ` Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 2 replies; 82+ messages in thread
From: Fabien COELHO @ 2022-04-03 08:20 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> Or those three columns always, sum_lag sum_lag_2, min_lag max_lag,
>> skipped, retried retries?
>
> What about this? (a log line is not actually folded)
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
> failures serialization_failures deadlock_failures retried retries [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]
My 0.02€:
I agree that it would be better to have a more deterministic
aggregated log format.
ISTM that it should skip failures and lags if no fancy options has been
selected, i.e.:
[ fails ... retries [ sum_lag ... [ skipped ] ] ?
Alterlatively, as the failure stuff is added to the format, maybe it could
be at the end:
[ sum_lag ... [ skipped [ fails ... retries ] ] ] ?
> failures:
> always 0 (if --max-tries is 1, the default)
> sum of serialization_failures and deadlock_failures (if --max-tries is not 1)
>
> serialization_failures and deadlock_failures:
> always 0 (if --max-tries is 1, the default)
> 0 or more (if --max-tries is not 1)
>
> retried and retries:
> always 0 (if --max-tries is 1, the default)
> 0 or more (if --max-tries is not 1)
>
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
>
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-04-03 11:44 ` Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-03 11:44 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>>> Or those three columns always, sum_lag sum_lag_2, min_lag max_lag,
>>> skipped, retried retries?
>>
>> What about this? (a log line is not actually folded)
>> interval_start num_transactions sum_latency sum_latency_2 min_latency
>> max_latency
>> failures serialization_failures deadlock_failures retried retries [
>> sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]
>
> My 0.02€:
>
> I agree that it would be better to have a more deterministic
> aggregated log format.
>
> ISTM that it should skip failures and lags if no fancy options has
> been selected, i.e.:
>
> [ fails ... retries [ sum_lag ... [ skipped ] ] ?
>
> Alterlatively, as the failure stuff is added to the format, maybe it
> could be at the end:
>
> [ sum_lag ... [ skipped [ fails ... retries ] ] ] ?
I like this one.
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
[sum_lag sum_lag_2 min_lag max_lag [ skipped [
failures serialization_failures deadlock_failures retried retries ] ] ]
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-04-03 11:54 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 15:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 2 replies; 82+ messages in thread
From: Alvaro Herrera @ 2022-04-03 11:54 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
On 2022-Apr-03, Fabien COELHO wrote:
> > What about this? (a log line is not actually folded)
> > interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
> > failures serialization_failures deadlock_failures retried retries [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]
>
> My 0.02€:
>
> I agree that it would be better to have a more deterministic aggregated log
> format.
>
> ISTM that it should skip failures and lags if no fancy options has been
> selected, i.e.:
>
> [ fails ... retries [ sum_lag ... [ skipped ] ] ?
I think it's easier to just say "if feature X is not enabled, then
columns XYZ are always zeroes".
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2022-04-03 12:59 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-03 12:59 UTC (permalink / raw)
To: alvherre@alvh.no-ip.org; +Cc: coelho@cri.ensmp.fr; ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> My 0.02€:
>>
>> I agree that it would be better to have a more deterministic aggregated log
>> format.
>>
>> ISTM that it should skip failures and lags if no fancy options has been
>> selected, i.e.:
>>
>> [ fails ... retries [ sum_lag ... [ skipped ] ] ?
>
> I think it's easier to just say "if feature X is not enabled, then
> columns XYZ are always zeroes".
Ok, I will come up with a patch in this direction.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-04 01:14 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-04 01:14 UTC (permalink / raw)
To: alvherre@alvh.no-ip.org; coelho@cri.ensmp.fr; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> I think it's easier to just say "if feature X is not enabled, then
>> columns XYZ are always zeroes".
>
> Ok, I will come up with a patch in this direction.
Please find attached patch for this.
With the patch, the log line is as follows (actually no line foldings of course):
interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
sum_lag sum_lag_2 min_lag max_lag skipped
failures serialization_failures deadlock_failures retried retries
I updated the doc as well:
- fold the log line using line feed to avoid error in rendering PDF. I
did not use &zwsp; because it does not enhance HTML output.
- split explanation of the log output into multiple paragraphs to
enhance readability.
- replace the example output with full options are specified.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
pgbench-aggregate-log.patch (text/x-patch, 8.1K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index ebdb4b3f46..e1f98ae228 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2401,7 +2401,9 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>
+<replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <replaceable>skipped</replaceable>
+<replaceable>failures</replaceable> <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> <replaceable>retried</replaceable> <replaceable>retries</replaceable>
</synopsis>
where
@@ -2417,41 +2419,55 @@ END;
and
<replaceable>max_latency</replaceable> is the maximum latency within the interval,
<replaceable>failures</replaceable> is the number of transactions that ended
- with a failed SQL command within the interval. If you use option
- <option>--failures-detailed</option>, instead of the sum of all failed
- transactions you will get more detailed statistics for the failed
- transactions grouped by the following types:
- <replaceable>serialization_failures</replaceable> is the number of
- transactions that got a serialization error and were not retried after this,
- <replaceable>deadlock_failures</replaceable> is the number of transactions
- that got a deadlock error and were not retried after this.
+ with a failed SQL command within the interval.
+ </para>
+ <para>
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
- and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
- option is used.
+ and <replaceable>max_lag</replaceable>, only meaningful if the <option>--rate</option>
+ option is used. Otherwise, they are all 0.0.
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
The next field, <replaceable>skipped</replaceable>,
- is only present if the <option>--latency-limit</option> option is used, too.
+ is only meaningful if the <option>--latency-limit</option> option is used, too. Otherwise it is 0.
It counts the number of transactions skipped because they would have
started too late.
- The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
- fields are present only if the <option>--max-tries</option> option is not
- equal to 1. They report the number of retried transactions and the sum of all
- retries after serialization or deadlock errors within the interval.
- Each transaction is counted in the interval when it was committed.
+ </para>
+ <para>
+ <replaceable>failures</replaceable> is the sum of all failed transactions.
+ If <option>--failures-detailed</option> is specified, instead of the sum of
+ all failed transactions you will get more detailed statistics for the
+ failed transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
+ If <option>--failures-detailed</option> is not
+ specified, <replaceable>serialization_failures</replaceable>
+ and <replaceable>deadlock_failures</replaceable> are always 0.
+ </para>
+ <para>
+ The <replaceable>retried</replaceable>
+ and <replaceable>retries</replaceable> fields are only meaningful if
+ the <option>--max-tries</option> option is not equal to 1. Otherwise they
+ are 0. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval. Each
+ transaction is counted in the interval when it was committed.
</para>
<para>
- Here is some example output:
+ Here is some example output with following options:
<screen>
-1345828501 5601 1542744 483552416 61 2573 0
-1345828503 7884 1979812 565806736 60 1479 0
-1345828505 7208 1979422 567277552 59 1391 0
-1345828507 7685 1980268 569784714 60 1398 0
-1345828509 7073 1979779 573489941 236 1411 0
-</screen></para>
+pgbench --aggregate-interval=10 --time=20 --client=10 --log --rate=1000
+--latency-limit=10 --failures-detailed --max-tries=10 test
+</screen>
+
+<screen>
+1649033235 5398 26186948 170797073304 1051 15347 2471626 6290343560 0 8261 0 3925 3925 0 7524 29534
+1649033245 5651 27345519 210270761637 1011 67780 2480555 6835066067 0 9999 496 3839 3839 0 7533 30118
+</screen>
+ </para>
<para>
Notice that while the plain (unaggregated) log file shows which script
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index acf3e56413..aea27b1383 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4494,6 +4494,17 @@ doLog(TState *thread, CState *st,
while ((next = agg->start_time + agg_interval * INT64CONST(1000000)) <= now)
{
+ double lag_sum = 0.0;
+ double lag_sum2 = 0.0;
+ double lag_min = 0.0;
+ double lag_max = 0.0;
+ int64 skipped = 0;
+ int64 serialization_failures = 0;
+ int64 deadlock_failures = 0;
+ int64 serialization_or_deadlock_failures = 0;
+ int64 retried = 0;
+ int64 retries = 0;
+
/* print aggregated report to logfile */
fprintf(logfile, INT64_FORMAT " " INT64_FORMAT " %.0f %.0f %.0f %.0f",
agg->start_time / 1000000, /* seconds since Unix epoch */
@@ -4503,27 +4514,41 @@ doLog(TState *thread, CState *st,
agg->latency.min,
agg->latency.max);
- if (failures_detailed)
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
- agg->serialization_failures,
- agg->deadlock_failures);
- else
- fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
-
if (throttle_delay)
{
- fprintf(logfile, " %.0f %.0f %.0f %.0f",
- agg->lag.sum,
- agg->lag.sum2,
- agg->lag.min,
- agg->lag.max);
- if (latency_limit)
- fprintf(logfile, " " INT64_FORMAT, agg->skipped);
+ lag_sum = agg->lag.sum;
+ lag_sum2 = agg->lag.sum2;
+ lag_min = agg->lag.min;
+ lag_max = agg->lag.max;
+ }
+ fprintf(logfile, " %.0f %.0f %.0f %.0f",
+ lag_sum,
+ lag_sum2,
+ lag_min,
+ lag_max);
+
+ if (latency_limit)
+ skipped = agg->skipped;
+ fprintf(logfile, " " INT64_FORMAT, skipped);
+
+ if (failures_detailed)
+ {
+ serialization_failures = agg->serialization_failures;
+ deadlock_failures = agg->deadlock_failures;
}
+ serialization_or_deadlock_failures = serialization_failures + deadlock_failures;
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT " " INT64_FORMAT,
+ serialization_or_deadlock_failures,
+ serialization_failures,
+ deadlock_failures);
+
if (max_tries != 1)
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
- agg->retried,
- agg->retries);
+ {
+ retried = agg->retried;
+ retries = agg->retries;
+ }
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT, retried, retries);
+
fputc('\n', logfile);
/* reset data and move to next interval */
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-04 13:20 ` Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2022-04-04 13:20 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: alvherre@alvh.no-ip.org; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Hello Tatsuo-san,
> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
> sum_lag sum_lag_2 min_lag max_lag skipped
> failures serialization_failures deadlock_failures retried retries
I would suggest to reorder the last chunk to:
... retried retries failures serfail dlfail
because I intend to add connection failures handling at some point, and it
would make more sense to add the corresponding count at the end with other
fails.
--
Fabien Coelho - CRI, MINES ParisTech
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-04-04 23:38 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-04 23:38 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: ishii@sraoss.co.jp; alvherre@alvh.no-ip.org; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Hi Fabien,
> Hello Tatsuo-san,
>
>> interval_start num_transactions sum_latency sum_latency_2 min_latency
>> max_latency
>> sum_lag sum_lag_2 min_lag max_lag skipped
>> failures serialization_failures deadlock_failures retried retries
>
> I would suggest to reorder the last chunk to:
>
> ... retried retries failures serfail dlfail
>
> because I intend to add connection failures handling at some point,
> and it would make more sense to add the corresponding count at the end
> with other fails.
Ok, I have adjusted the patch. V2 patch attached.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
pgbench-aggregate-log-v2.patch (text/x-patch, 8.1K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index ebdb4b3f46..d1818ff316 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2401,7 +2401,9 @@ END;
format is used for the log files:
<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable> { <replaceable>failures</replaceable> | <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable> } <optional> <replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <optional> <replaceable>skipped</replaceable> </optional> </optional> <optional> <replaceable>retried</replaceable> <replaceable>retries</replaceable> </optional>
+<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>
+<replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <replaceable>skipped</replaceable>
+<replaceable>retried</replaceable> <replaceable>retries</replaceable> <replaceable>failures</replaceable> <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable>
</synopsis>
where
@@ -2417,41 +2419,55 @@ END;
and
<replaceable>max_latency</replaceable> is the maximum latency within the interval,
<replaceable>failures</replaceable> is the number of transactions that ended
- with a failed SQL command within the interval. If you use option
- <option>--failures-detailed</option>, instead of the sum of all failed
- transactions you will get more detailed statistics for the failed
- transactions grouped by the following types:
- <replaceable>serialization_failures</replaceable> is the number of
- transactions that got a serialization error and were not retried after this,
- <replaceable>deadlock_failures</replaceable> is the number of transactions
- that got a deadlock error and were not retried after this.
+ with a failed SQL command within the interval.
+ </para>
+ <para>
The next fields,
<replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
- and <replaceable>max_lag</replaceable>, are only present if the <option>--rate</option>
- option is used.
+ and <replaceable>max_lag</replaceable>, only meaningful if the <option>--rate</option>
+ option is used. Otherwise, they are all 0.0.
They provide statistics about the time each transaction had to wait for the
previous one to finish, i.e., the difference between each transaction's
scheduled start time and the time it actually started.
The next field, <replaceable>skipped</replaceable>,
- is only present if the <option>--latency-limit</option> option is used, too.
+ is only meaningful if the <option>--latency-limit</option> option is used, too. Otherwise it is 0.
It counts the number of transactions skipped because they would have
started too late.
- The <replaceable>retried</replaceable> and <replaceable>retries</replaceable>
- fields are present only if the <option>--max-tries</option> option is not
- equal to 1. They report the number of retried transactions and the sum of all
- retries after serialization or deadlock errors within the interval.
- Each transaction is counted in the interval when it was committed.
+ </para>
+ <para>
+ The <replaceable>retried</replaceable>
+ and <replaceable>retries</replaceable> fields are only meaningful if
+ the <option>--max-tries</option> option is not equal to 1. Otherwise they
+ are 0. They report the number of retried transactions and the sum of all
+ retries after serialization or deadlock errors within the interval. Each
+ transaction is counted in the interval when it was committed.
+ </para>
+ <para>
+ <replaceable>failures</replaceable> is the sum of all failed transactions.
+ If <option>--failures-detailed</option> is specified, instead of the sum of
+ all failed transactions you will get more detailed statistics for the
+ failed transactions grouped by the following types:
+ <replaceable>serialization_failures</replaceable> is the number of
+ transactions that got a serialization error and were not retried after this,
+ <replaceable>deadlock_failures</replaceable> is the number of transactions
+ that got a deadlock error and were not retried after this.
+ If <option>--failures-detailed</option> is not
+ specified, <replaceable>serialization_failures</replaceable>
+ and <replaceable>deadlock_failures</replaceable> are always 0.
</para>
<para>
- Here is some example output:
+ Here is some example output with following options:
<screen>
-1345828501 5601 1542744 483552416 61 2573 0
-1345828503 7884 1979812 565806736 60 1479 0
-1345828505 7208 1979422 567277552 59 1391 0
-1345828507 7685 1980268 569784714 60 1398 0
-1345828509 7073 1979779 573489941 236 1411 0
-</screen></para>
+pgbench --aggregate-interval=10 --time=20 --client=10 --log --rate=1000
+--latency-limit=10 --failures-detailed --max-tries=10 test
+</screen>
+
+<screen>
+1649114136 5815 27552565 177846919143 1078 21716 2756787 7264696105 0 9661 0 7854 31472 4022 4022 0
+1649114146 5958 28460110 182785513108 1083 20391 2539395 6411761497 0 7268 0 8127 32595 4101 4101 0
+</screen>
+ </para>
<para>
Notice that while the plain (unaggregated) log file shows which script
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index acf3e56413..4d4b979e4f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4494,6 +4494,17 @@ doLog(TState *thread, CState *st,
while ((next = agg->start_time + agg_interval * INT64CONST(1000000)) <= now)
{
+ double lag_sum = 0.0;
+ double lag_sum2 = 0.0;
+ double lag_min = 0.0;
+ double lag_max = 0.0;
+ int64 skipped = 0;
+ int64 serialization_failures = 0;
+ int64 deadlock_failures = 0;
+ int64 serialization_or_deadlock_failures = 0;
+ int64 retried = 0;
+ int64 retries = 0;
+
/* print aggregated report to logfile */
fprintf(logfile, INT64_FORMAT " " INT64_FORMAT " %.0f %.0f %.0f %.0f",
agg->start_time / 1000000, /* seconds since Unix epoch */
@@ -4503,27 +4514,41 @@ doLog(TState *thread, CState *st,
agg->latency.min,
agg->latency.max);
- if (failures_detailed)
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
- agg->serialization_failures,
- agg->deadlock_failures);
- else
- fprintf(logfile, " " INT64_FORMAT, getFailures(agg));
-
if (throttle_delay)
{
- fprintf(logfile, " %.0f %.0f %.0f %.0f",
- agg->lag.sum,
- agg->lag.sum2,
- agg->lag.min,
- agg->lag.max);
- if (latency_limit)
- fprintf(logfile, " " INT64_FORMAT, agg->skipped);
+ lag_sum = agg->lag.sum;
+ lag_sum2 = agg->lag.sum2;
+ lag_min = agg->lag.min;
+ lag_max = agg->lag.max;
}
+ fprintf(logfile, " %.0f %.0f %.0f %.0f",
+ lag_sum,
+ lag_sum2,
+ lag_min,
+ lag_max);
+
+ if (latency_limit)
+ skipped = agg->skipped;
+ fprintf(logfile, " " INT64_FORMAT, skipped);
+
if (max_tries != 1)
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
- agg->retried,
- agg->retries);
+ {
+ retried = agg->retried;
+ retries = agg->retries;
+ }
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT, retried, retries);
+
+ if (failures_detailed)
+ {
+ serialization_failures = agg->serialization_failures;
+ deadlock_failures = agg->deadlock_failures;
+ }
+ serialization_or_deadlock_failures = serialization_failures + deadlock_failures;
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT " " INT64_FORMAT,
+ serialization_or_deadlock_failures,
+ serialization_failures,
+ deadlock_failures);
+
fputc('\n', logfile);
/* reset data and move to next interval */
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-06 01:00 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-06 01:00 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; alvherre@alvh.no-ip.org; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> I would suggest to reorder the last chunk to:
>>
>> ... retried retries failures serfail dlfail
>>
>> because I intend to add connection failures handling at some point,
>> and it would make more sense to add the corresponding count at the end
>> with other fails.
>
> Ok, I have adjusted the patch. V2 patch attached.
Patch pushed. Thanks.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-09 20:38 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-04-09 23:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 82+ messages in thread
From: Tom Lane @ 2022-04-09 20:38 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> Patch pushed. Thanks.
The buildfarm is still complaining about the synopsis being too
wide for PDF format. I think what we ought to do is give up on
using a <synopsis> for log lines at all, and instead convert the
documentation into a tabular list of fields. Proposal attached,
which also fixes a couple of outright errors.
One thing that this doesn't fix is that the existing text appears
to suggest that the "failures" column is something different from
the sum of the serialization_failures and deadlock_failures
columns, which it's obvious from the code is not so. If this isn't
a code bug then I think we ought to just drop that column entirely,
because it's redundant.
(BTW, now that I've read this stuff I am quite horrified by how
the non-aggregated log format has been mangled for error retries,
and will be probably be submitting a proposal to change that.
But that's a different issue.)
regards, tom lane
Attachments:
change-log-line-docs-layout.patch (text/x-diff, 13.0K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 9ba26e5e86..d12cbaa8ab 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2289,33 +2289,95 @@ END;
</para>
<para>
- The format of the log is:
-
-<synopsis>
-<replaceable>client_id</replaceable> <replaceable>transaction_no</replaceable> <replaceable>time</replaceable> <replaceable>script_no</replaceable> <replaceable>time_epoch</replaceable> <replaceable>time_us</replaceable> <optional> <replaceable>schedule_lag</replaceable> </optional> <optional> <replaceable>retries</replaceable> </optional>
-</synopsis>
-
- where
- <replaceable>client_id</replaceable> indicates which client session ran the transaction,
- <replaceable>transaction_no</replaceable> counts how many transactions have been
- run by that session,
- <replaceable>time</replaceable> is the total elapsed transaction time in microseconds,
- <replaceable>script_no</replaceable> identifies which script file was used (useful when
- multiple scripts were specified with <option>-f</option> or <option>-b</option>),
- and <replaceable>time_epoch</replaceable>/<replaceable>time_us</replaceable> are a
- Unix-epoch time stamp and an offset
- in microseconds (suitable for creating an ISO 8601
- time stamp with fractional seconds) showing when
- the transaction completed.
- The <replaceable>schedule_lag</replaceable> field is the difference between the
- transaction's scheduled start time, and the time it actually started, in
- microseconds. It is only present when the <option>--rate</option> option is used.
+ Each line in a log file describes one transaction.
+ It contains the following space-separated fields:
+
+ <variablelist>
+ <varlistentry>
+ <term><replaceable>client_id</replaceable></term>
+ <listitem>
+ <para>
+ identifies the client session that ran the transaction
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>transaction_no</replaceable></term>
+ <listitem>
+ <para>
+ counts how many transactions have been run by that session
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>time</replaceable></term>
+ <listitem>
+ <para>
+ transaction's elapsed time, in microseconds
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>script_no</replaceable></term>
+ <listitem>
+ <para>
+ identifies the script file that was used for the transaction
+ (useful when multiple scripts are specified
+ with <option>-f</option> or <option>-b</option>)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>time_epoch</replaceable></term>
+ <listitem>
+ <para>
+ transaction's completion time, as a Unix-epoch time stamp
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>time_us</replaceable></term>
+ <listitem>
+ <para>
+ fractional-second part of transaction's completion time, in
+ microseconds
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>schedule_lag</replaceable></term>
+ <listitem>
+ <para>
+ difference between the transaction's scheduled start time and the
+ time it actually started, in microseconds
+ (present only if <option>--rate</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>retries</replaceable></term>
+ <listitem>
+ <para>
+ count of retries after serialization or deadlock errors during the
+ transaction
+ (present only if <option>--max-tries</option> is not equal to one)
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
When both <option>--rate</option> and <option>--latency-limit</option> are used,
the <replaceable>time</replaceable> for a skipped transaction will be reported as
<literal>skipped</literal>.
- <replaceable>retries</replaceable> is the sum of all retries after the
- serialization or deadlock errors during the current script execution. It is
- present only if the <option>--max-tries</option> option is not equal to 1.
If the transaction ends with a failure, its <replaceable>time</replaceable>
will be reported as <literal>failed</literal>. If you use the
<option>--failures-detailed</option> option, the
@@ -2398,66 +2460,171 @@ END;
<para>
With the <option>--aggregate-interval</option> option, a different
- format is used for the log files:
-
-<synopsis>
-<replaceable>interval_start</replaceable> <replaceable>num_transactions</replaceable> <replaceable>sum_latency</replaceable> <replaceable>sum_latency_2</replaceable> <replaceable>min_latency</replaceable> <replaceable>max_latency</replaceable>
-<replaceable>sum_lag</replaceable> <replaceable>sum_lag_2</replaceable> <replaceable>min_lag</replaceable> <replaceable>max_lag</replaceable> <replaceable>skipped</replaceable>
-<replaceable>retried</replaceable> <replaceable>retries</replaceable> <replaceable>failures</replaceable> <replaceable>serialization_failures</replaceable> <replaceable>deadlock_failures</replaceable>
-</synopsis>
-
- where
- <replaceable>interval_start</replaceable> is the start of the interval (as a Unix
- epoch time stamp),
- <replaceable>num_transactions</replaceable> is the number of transactions
- within the interval,
- <replaceable>sum_latency</replaceable> is the sum of the transaction
- latencies within the interval,
- <replaceable>sum_latency_2</replaceable> is the sum of squares of the
- transaction latencies within the interval,
- <replaceable>min_latency</replaceable> is the minimum latency within the interval,
- and
- <replaceable>max_latency</replaceable> is the maximum latency within the interval,
- <replaceable>failures</replaceable> is the number of transactions that ended
- with a failed SQL command within the interval.
- </para>
- <para>
- The next fields,
- <replaceable>sum_lag</replaceable>, <replaceable>sum_lag_2</replaceable>, <replaceable>min_lag</replaceable>,
- and <replaceable>max_lag</replaceable>, only meaningful if the <option>--rate</option>
- option is used. Otherwise, they are all 0.0.
- They provide statistics about the time each transaction had to wait for the
- previous one to finish, i.e., the difference between each transaction's
- scheduled start time and the time it actually started.
- The next field, <replaceable>skipped</replaceable>,
- is only meaningful if the <option>--latency-limit</option> option is used, too. Otherwise it is 0.
- It counts the number of transactions skipped because they would have
- started too late.
- </para>
- <para>
- The <replaceable>retried</replaceable>
- and <replaceable>retries</replaceable> fields are only meaningful if
- the <option>--max-tries</option> option is not equal to 1. Otherwise they
- are 0. They report the number of retried transactions and the sum of all
- retries after serialization or deadlock errors within the interval. Each
- transaction is counted in the interval when it was committed.
- </para>
- <para>
- <replaceable>failures</replaceable> is the sum of all failed transactions.
- If <option>--failures-detailed</option> is specified, instead of the sum of
- all failed transactions you will get more detailed statistics for the
- failed transactions grouped by the following types:
- <replaceable>serialization_failures</replaceable> is the number of
- transactions that got a serialization error and were not retried after this,
- <replaceable>deadlock_failures</replaceable> is the number of transactions
- that got a deadlock error and were not retried after this.
- If <option>--failures-detailed</option> is not
- specified, <replaceable>serialization_failures</replaceable>
- and <replaceable>deadlock_failures</replaceable> are always 0.
+ format is used for the log files. Each log line describes one
+ aggregation interval. It contains the following space-separated
+ fields:
+
+ <variablelist>
+ <varlistentry>
+ <term><replaceable>interval_start</replaceable></term>
+ <listitem>
+ <para>
+ start time of the interval, as a Unix-epoch time stamp
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>num_transactions</replaceable></term>
+ <listitem>
+ <para>
+ number of transactions within the interval
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>sum_latency</replaceable></term>
+ <listitem>
+ <para>
+ sum of transaction latencies
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>sum_latency_2</replaceable></term>
+ <listitem>
+ <para>
+ sum of squares of transaction latencies
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>min_latency</replaceable></term>
+ <listitem>
+ <para>
+ minimum transaction latency
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>max_latency</replaceable></term>
+ <listitem>
+ <para>
+ maximum transaction latency
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>sum_lag</replaceable></term>
+ <listitem>
+ <para>
+ sum of transaction start delays
+ (zero unless <option>--rate</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>sum_lag_2</replaceable></term>
+ <listitem>
+ <para>
+ sum of squares of transaction start delays
+ (zero unless <option>--rate</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>min_lag</replaceable></term>
+ <listitem>
+ <para>
+ minimum transaction start delay
+ (zero unless <option>--rate</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>max_lag</replaceable></term>
+ <listitem>
+ <para>
+ maximum transaction start delay
+ (zero unless <option>--rate</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>skipped</replaceable></term>
+ <listitem>
+ <para>
+ number of transactions skipped because they would have started too late
+ (zero unless <option>--rate</option>
+ and <option>--latency-limit</option> are specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>retried</replaceable></term>
+ <listitem>
+ <para>
+ number of retried transactions
+ (zero unless <option>--max-tries</option> is not equal to one)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>retries</replaceable></term>
+ <listitem>
+ <para>
+ number of retries after serialization or deadlock errors
+ (zero unless <option>--max-tries</option> is not equal to one)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>failures</replaceable></term>
+ <listitem>
+ <para>
+ number of transactions that ended with a failed SQL command
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>serialization_failures</replaceable></term>
+ <listitem>
+ <para>
+ number of transactions that got a serialization error and were not
+ retried afterwards
+ (zero unless <option>--failures-detailed</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable>deadlock_failures</replaceable></term>
+ <listitem>
+ <para>
+ number of transactions that got a deadlock error and were not
+ retried afterwards
+ (zero unless <option>--failures-detailed</option> is specified)
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
</para>
<para>
- Here is some example output with following options:
+ Here is some example output generated with these options:
<screen>
<userinput>pgbench --aggregate-interval=10 --time=20 --client=10 --log --rate=1000
--latency-limit=10 --failures-detailed --max-tries=10 test</userinput>
@@ -2468,8 +2635,8 @@ END;
</para>
<para>
- Notice that while the plain (unaggregated) log file shows which script
- was used for each transaction, the aggregated log does not. Therefore if
+ Notice that while the plain (unaggregated) log format shows which script
+ was used for each transaction, the aggregated format does not. Therefore if
you need per-script data, you need to aggregate the data on your own.
</para>
=
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-04-09 23:35 ` Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-09 23:35 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: ishii@sraoss.co.jp; coelho@cri.ensmp.fr; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
> The buildfarm is still complaining about the synopsis being too
> wide for PDF format. I think what we ought to do is give up on
> using a <synopsis> for log lines at all, and instead convert the
> documentation into a tabular list of fields. Proposal attached,
> which also fixes a couple of outright errors.
Once I thought about that too. Looks good to me.
> One thing that this doesn't fix is that the existing text appears
> to suggest that the "failures" column is something different from
> the sum of the serialization_failures and deadlock_failures
> columns, which it's obvious from the code is not so. If this isn't
> a code bug then I think we ought to just drop that column entirely,
> because it's redundant.
+1.
> (BTW, now that I've read this stuff I am quite horrified by how
> the non-aggregated log format has been mangled for error retries,
> and will be probably be submitting a proposal to change that.
> But that's a different issue.)
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-04-10 07:37 ` Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-10 19:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-18 05:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 2 replies; 82+ messages in thread
From: Fabien COELHO @ 2022-04-10 07:37 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Hello Tom,
> The buildfarm is still complaining about the synopsis being too
> wide for PDF format. I think what we ought to do is give up on
> using a <synopsis> for log lines at all, and instead convert the
> documentation into a tabular list of fields. Proposal attached,
> which also fixes a couple of outright errors.
Looks ok. Html doc generation ok.
While looking at the html outpout, the "pgbench" command line just below
wraps strangely:
pgbench --aggregate-interval=10 --time=20 --client=10 --log --rate=1000
--latency-limit=10 --failures-detailed --max-tries=10 test
ISTM that there should be no nl in the <textinput>pgbench …</textinput>
section, although maybe it would trigger a complaint in the pdf format.
> One thing that this doesn't fix is that the existing text appears
> to suggest that the "failures" column is something different from
> the sum of the serialization_failures and deadlock_failures
> columns, which it's obvious from the code is not so. If this isn't
> a code bug then I think we ought to just drop that column entirely,
> because it's redundant.
Ok. Fine with me. Possibly at some point there was the idea that there
could be other failures counted, but there are none. Also, there has been
questions about the failures detailed option, or whether the reports
should always be detailed, and the result may be some kind of not
convincing compromise.
> (BTW, now that I've read this stuff I am quite horrified by how
> the non-aggregated log format has been mangled for error retries,
> and will be probably be submitting a proposal to change that.
> But that's a different issue.)
Indeed, any improvement is welcome!
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-04-10 19:32 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 82+ messages in thread
From: Tom Lane @ 2022-04-10 19:32 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Fabien COELHO <coelho@cri.ensmp.fr> writes:
> While looking at the html outpout, the "pgbench" command line just below
> wraps strangely:
> pgbench --aggregate-interval=10 --time=20 --client=10 --log --rate=1000
> --latency-limit=10 --failures-detailed --max-tries=10 test
> ISTM that there should be no nl in the <textinput>pgbench …</textinput>
> section, although maybe it would trigger a complaint in the pdf format.
PDF wraps that text where it wants to anyway, so I removed the newline.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-04-18 05:52 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-19 07:56 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-18 05:52 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: tgl@sss.pgh.pa.us; ishii@sraoss.co.jp; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> One thing that this doesn't fix is that the existing text appears
>> to suggest that the "failures" column is something different from
>> the sum of the serialization_failures and deadlock_failures
>> columns, which it's obvious from the code is not so. If this isn't
>> a code bug then I think we ought to just drop that column entirely,
>> because it's redundant.
>
> Ok. Fine with me. Possibly at some point there was the idea that there
> could be other failures counted, but there are none. Also, there has
> been questions about the failures detailed option, or whether the
> reports should always be detailed, and the result may be some kind of
> not convincing compromise.
Attached is the patch to remove the column.
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
Attachments:
pgbench-remove-failures-column.patch (text/x-patch, 2.1K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 387a836287..dbae4e2321 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -2591,15 +2591,6 @@ END;
</listitem>
</varlistentry>
- <varlistentry>
- <term><replaceable>failures</replaceable></term>
- <listitem>
- <para>
- number of transactions that ended with a failed SQL command
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><replaceable>serialization_failures</replaceable></term>
<listitem>
@@ -2629,8 +2620,8 @@ END;
<screen>
<userinput>pgbench --aggregate-interval=10 --time=20 --client=10 --log --rate=1000 --latency-limit=10 --failures-detailed --max-tries=10 test</userinput>
-1649114136 5815 27552565 177846919143 1078 21716 2756787 7264696105 0 9661 0 7854 31472 4022 4022 0
-1649114146 5958 28460110 182785513108 1083 20391 2539395 6411761497 0 7268 0 8127 32595 4101 4101 0
+1650260552 5178 26171317 177284491527 1136 44462 2647617 7321113867 0 9866 64 7564 28340 4148 0
+1650260562 4808 25573984 220121792172 1171 62083 3037380 9666800914 0 9998 598 7392 26621 4527 0
</screen>
</para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e63cea56a1..f8bcb1ab6d 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4498,7 +4498,6 @@ doLog(TState *thread, CState *st,
int64 skipped = 0;
int64 serialization_failures = 0;
int64 deadlock_failures = 0;
- int64 serialization_or_deadlock_failures = 0;
int64 retried = 0;
int64 retries = 0;
@@ -4540,9 +4539,7 @@ doLog(TState *thread, CState *st,
serialization_failures = agg->serialization_failures;
deadlock_failures = agg->deadlock_failures;
}
- serialization_or_deadlock_failures = serialization_failures + deadlock_failures;
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT " " INT64_FORMAT,
- serialization_or_deadlock_failures,
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
serialization_failures,
deadlock_failures);
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-18 05:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-19 07:56 ` Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-19 08:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 82+ messages in thread
From: Fabien COELHO @ 2022-04-19 07:56 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> Ok. Fine with me. Possibly at some point there was the idea that there
>> could be other failures counted, but there are none. Also, there has
>> been questions about the failures detailed option, or whether the
>> reports should always be detailed, and the result may be some kind of
>> not convincing compromise.
>
> Attached is the patch to remove the column.
Patch applies cleanly. Compilation ok. Global and local "make check" ok.
Doc build ok.
It find it a little annoying that there is no change in tests, it means
that the format is not checked at all:-(
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-18 05:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-19 07:56 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
@ 2022-04-19 08:17 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-19 11:51 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 82+ messages in thread
From: Tatsuo Ishii @ 2022-04-19 08:17 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>>> Ok. Fine with me. Possibly at some point there was the idea that there
>>> could be other failures counted, but there are none. Also, there has
>>> been questions about the failures detailed option, or whether the
>>> reports should always be detailed, and the result may be some kind of
>>> not convincing compromise.
>>
>> Attached is the patch to remove the column.
>
> Patch applies cleanly. Compilation ok. Global and local "make check"
> ok.
> Doc build ok.
Thank you for reviewing. Patch pushed.
> It find it a little annoying that there is no change in tests, it
> means that the format is not checked at all:-(
Yeah. Perhaps it's a little bit hard to perform this kind of tests in
the TAP test?
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-18 05:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-19 07:56 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-19 08:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2022-04-19 11:51 ` Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 0 replies; 82+ messages in thread
From: Fabien COELHO @ 2022-04-19 11:51 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: tgl@sss.pgh.pa.us; alvherre@alvh.no-ip.org; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
>> It find it a little annoying that there is no change in tests, it
>> means that the format is not checked at all:-(
>
> Yeah. Perhaps it's a little bit hard to perform this kind of tests in
> the TAP test?
Not really. I'll look into it.
--
Fabien.
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2022-04-03 15:43 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 82+ messages in thread
From: Tom Lane @ 2022-04-03 15:43 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Tatsuo Ishii <ishii@sraoss.co.jp>; nagata@sraoss.co.jp; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> I think it's easier to just say "if feature X is not enabled, then
> columns XYZ are always zeroes".
+1, that's pretty much what I was thinking.
regards, tom lane
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
@ 2022-03-29 00:19 ` Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 0 replies; 82+ messages in thread
From: Tatsuo Ishii @ 2022-03-29 00:19 UTC (permalink / raw)
To: alvherre@alvh.no-ip.org; +Cc: ishii@sraoss.co.jp; tgl@sss.pgh.pa.us; nagata@sraoss.co.jp; coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; pgsql-hackers; teodor@sigaev.ru
> Hello,
>
> On 2022-Mar-27, Tatsuo Ishii wrote:
>
>> After:
>> interval_start num_transactions sum_latency sum_latency_2 min_latency max_latency
>> { failures | serialization_failures deadlock_failures } [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ] [ retried retries ]
>
> You're showing an indentation, but looking at the HTML output there is
> no such. Is the HTML processor eating leading whitespace or something
> like that?
I just copied from my web browser screen (Firefox 98.0.2 on Ubuntu 20).
Best reagards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
^ permalink raw reply [nested|flat] 82+ messages in thread
* Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 10:15 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
@ 2021-07-05 09:11 ` Yugo NAGATA <nagata@sraoss.co.jp>
1 sibling, 0 replies; 82+ messages in thread
From: Yugo NAGATA @ 2021-07-05 09:11 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; thomas.munro@gmail.com; m.polyakova@postgrespro.ru; alvherre@2ndquadrant.com; pgsql-hackers; teodor@sigaev.ru
Hello Ishii-san,
On Thu, 01 Jul 2021 09:03:42 +0900 (JST)
Tatsuo Ishii <ishii@sraoss.co.jp> wrote:
> > v13 patches gave a compiler warning...
> >
> > $ make >/dev/null
> > pgbench.c: In function $B!F(BcommandError$B!G(B:
> > pgbench.c:3071:17: warning: unused variable $B!F(Bcommand$B!G(B [-Wunused-variable]
> > const Command *command = sql_script[st->use_file].commands[st->command];
> > ^~~~~~~
Hmm, we'll get the warning when --enable-cassert is not specified.
I'll fix it.
> There is a typo in the doc (more over -> moreover).
>
> > of all transaction tries; more over, you cannot use an unlimited number
>
> of all transaction tries; moreover, you cannot use an unlimited number
>
Thanks. I'll fix.
Regards,
Yugo Nagata
--
Yugo NAGATA <nagata@sraoss.co.jp>
^ permalink raw reply [nested|flat] 82+ messages in thread
end of thread, other threads:[~2022-04-19 11:51 UTC | newest]
Thread overview: 82+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-06-22 14:54 Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-22 18:03 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-23 00:31 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-23 08:38 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-25 11:01 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-26 08:20 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-26 10:15 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-06-30 15:02 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-06-30 15:33 ` Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-07 23:35 ` Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 12:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-06-30 22:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-01 00:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-02 00:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-06 07:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 10:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-07 12:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-07 13:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-10 17:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2021-07-13 04:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 04:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-13 05:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-13 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-16 03:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2021-07-19 18:04 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-02-18 14:45 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-02 09:03 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-02 11:41 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-12 14:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-03-19 18:23 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-20 00:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 02:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-20 07:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:10 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 00:46 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-22 00:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 08:27 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-22 09:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 10:08 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-23 18:26 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 02:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-24 04:49 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-24 18:02 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-24 23:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 00:05 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 00:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:07 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-25 01:24 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-25 01:12 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-25 01:25 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-26 16:36 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-27 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 02:29 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 03:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 03:48 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
2022-03-28 06:50 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 06:53 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-03-28 07:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-03-28 13:33 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-01 00:57 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-01 05:34 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 06:28 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 08:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 11:44 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-03 11:54 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Alvaro Herrera <alvherre@alvh.no-ip.org>
2022-04-03 12:59 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 01:14 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-04 13:20 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-04 23:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-06 01:00 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-09 20:38 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-09 23:35 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-10 07:37 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-10 19:32 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-04-18 05:52 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-19 07:56 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-19 08:17 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2022-04-19 11:51 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Fabien COELHO <coelho@cri.ensmp.fr>
2022-04-03 15:43 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tom Lane <tgl@sss.pgh.pa.us>
2022-03-29 00:19 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Tatsuo Ishii <ishii@sraoss.co.jp>
2021-07-05 09:11 ` Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors Yugo NAGATA <nagata@sraoss.co.jp>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox