public inbox for [email protected]
help / color / mirror / Atom feedFrom: Chao Li <[email protected]>
To: Yugo Nagata <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Cc: Fujii Masao <[email protected]>
Cc: Rintaro Ikeda <[email protected]>
Subject: Re: pgbench --continue-on-error: clarify TPS and failure reporting
Date: Thu, 18 Jun 2026 14:25:03 +0800
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
> On Jun 18, 2026, at 13:36, Yugo Nagata <[email protected]> wrote:
>
> On Thu, 18 Jun 2026 11:24:04 +0800
> Chao Li <[email protected]> wrote:
>
>> Hi
>>
>> While testing “[0ab208fa5] pgbench: Add --continue-on-error option”, I got a question about the doc:
>> ```
>> <para>
>> This option is useful when your custom script may raise errors
>> such as unique constraint violations, but you want the benchmark
>> to continue and measure performance including those failures.
>> </para>
>> ```
>>
>> The phrase “measure performance including those failures” gives the impression that failed transactions might be counted in TPS, but they are not.
>
> I agree with your point. The original phrasing "including those failures" is
> indeed ambiguous and could mislead users into thinking failed transactions are
> included in the TPS calculation.
>
>> In my test, all transactions failed, and TPS was reported as 0:
>> ```
>> % pgbench -d evantest -n -t 5 --continue-on-error --failures-detailed -f pgbench-coe.sql
>> pgbench (19beta1)
>> transaction type: pgbench-coe.sql
>> scaling factor: 1
>> query mode: simple
>> number of clients: 1
>> number of threads: 1
>> maximum number of tries: 1
>> number of transactions per client: 5
>> number of transactions actually processed: 0/5
>> number of failed transactions: 5 (100.000%)
>> number of serialization failures: 0 (0.000%)
>> number of deadlock failures: 0 (0.000%)
>> number of other failures: 5 (100.000%)
>> latency average = 0.389 ms (including failures)
>> initial connection time = 4.172 ms
>> tps = 0.000000 (without initial connection time)
>> ```
>>
>> If this behavior is intended, I wonder if we should clarify the TPS counting logic more explicitly. For example:
>> ```
>> <para>
>> This option is useful when your custom script may raise errors
>> such as unique constraint violations, but you want the benchmark
>> to continue despite individual statement failures. Failed transactions
>> are reported separately and included in latency statistics, but are not
>> counted as transactions actually processed for TPS.
>> </para>
>> ```
>
> Your proposed documentation change is much clearer and accurately reflects
> the actual behavior. I prefer this version as well.
>
> Regards,
> Yugo Nagata
>
> --
> Yugo Nagata <[email protected]>
Thanks for confirming the doc issue. The here comes a small doc patch.
I also noticed there is a later paragraph about “latency” that also needs an update, it’s included in the patch as well.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
Attachments:
[application/octet-stream] v1-0001-doc-Clarify-pgbench-reporting-with-continue-on-er.patch (2.3K, ../[email protected]/2-v1-0001-doc-Clarify-pgbench-reporting-with-continue-on-er.patch)
download | inline diff:
From abf245787746ffa7f696298d22101b4e0226f711 Mon Sep 17 00:00:00 2001
From: "Chao Li (Evan)" <[email protected]>
Date: Thu, 18 Jun 2026 14:21:13 +0800
Subject: [PATCH v1] doc: Clarify pgbench reporting with --continue-on-error
Clarify that transactions failed under --continue-on-error are reported
separately and are not counted as transactions actually processed or in
TPS. Also adjust the latency description to avoid implying that failed
transactions are generally included in latency measurements.
Author: Chao Li <[email protected]>
Reviewed-by: Yugo Nagata <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/pgbench.sgml | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2e401d1ceb8..abdb1df6c89 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -770,10 +770,13 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
the connection to fail.
See <xref linkend="failures-and-retries"/> for more information.
</para>
+
<para>
This option is useful when your custom script may raise errors
such as unique constraint violations, but you want the benchmark
- to continue and measure performance including those failures.
+ to continue despite individual statement failures. Failed
+ transactions are reported separately, but are not counted as
+ transactions actually processed, nor are they included in TPS.
</para>
</listitem>
</varlistentry>
@@ -2933,9 +2936,10 @@ statement latencies in milliseconds, failures and retries:
<para>
The latency of a successful transaction includes the entire time of
- transaction execution with rollbacks and retries. The latency is measured
- only for successful transactions and commands but not for failed transactions
- or commands.
+ transaction execution with rollbacks and retries. The latency is measured
+ only for successful transactions and commands, except that failed
+ transactions are included in the simple average latency when the
+ <option>--continue-on-error</option> option is used.
</para>
<para>
--
2.50.1 (Apple Git-155)
view thread (6+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: pgbench --continue-on-error: clarify TPS and failure reporting
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox