From: Chao Li <li.evan.chao@gmail.com>
Message-Id: <A476B863-DF6E-4815-96C7-B6288669B05E@gmail.com>
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_AE157C43-EC34-4777-9D04-86C524786153"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.600.51.1.1\))
Subject: Re: pgbench --continue-on-error: clarify TPS and failure reporting
Date: Mon, 22 Jun 2026 16:10:22 +0800
In-Reply-To: <09361838-9213-4961-95A5-A0CD4AABF2DD@gmail.com>
Cc: Yugo Nagata <nagata@sraoss.co.jp>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>,
 Rintaro Ikeda <ikedarintarof@oss.nttdata.com>
To: Fujii Masao <masao.fujii@gmail.com>
References: <597C9755-4386-488D-A289-16D50AD81FBF@gmail.com>
 <20260618143631.25c2ed34b1d7527ee2d92a17@sraoss.co.jp>
 <958C5B66-50B0-47C8-BBA5-7D32F7A09566@gmail.com>
 <CAHGQGwFX6Wqdp=ZEjs6WSu3GDSpgM9qZTixBHXKB9xaVzQ+0-A@mail.gmail.com>
 <09361838-9213-4961-95A5-A0CD4AABF2DD@gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/A476B863-DF6E-4815-96C7-B6288669B05E%40gmail.com>
Precedence: bulk


--Apple-Mail=_AE157C43-EC34-4777-9D04-86C524786153
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jun 18, 2026, at 20:45, Chao Li <li.evan.chao@gmail.com> wrote:
>=20
>=20
>=20
>> On Jun 18, 2026, at 17:40, Fujii Masao <masao.fujii@gmail.com> wrote:
>>=20
>> On Thu, Jun 18, 2026 at 3:25=E2=80=AFPM Chao Li =
<li.evan.chao@gmail.com> wrote:
>>> Thanks for confirming the doc issue. The here comes a small doc =
patch.
>>>=20
>>> I also noticed there is a later paragraph about =E2=80=9Clatency=E2=80=
=9D that also needs an update, it=E2=80=99s included in the patch as =
well.
>>=20
>> Thanks for the patch! The TPS-related changes look good to me.
>>=20
>>   The latency of a successful transaction includes the entire time of
>> -   transaction execution with rollbacks and retries. The latency is =
measured
>> -   only for successful transactions and commands but not for failed
>> transactions
>> -   or commands.
>> +   transaction execution with rollbacks and retries.  The latency is =
measured
>> +   only for successful transactions and commands, except that failed
>> +   transactions are included in the simple average latency when the
>> +   <option>--continue-on-error</option> option is used.
>>=20
>> It doesn't seem that the behavior of the latency average in the main
>> report depends on --continue-on-error itself. Instead, when neither
>> --progress, --rate, nor --latency-limit is specified, the main
>> report shows a simple latency average computed as the total benchmark
>> duration divided by the number of successful and failed transactions,
>> so failed transactions are included.
>>=20
>> On the other hand, when any of those options is specified, the
>> latency average and latency stddev in the main report are based
>> only on successful transactions. Likewise, the per-script and
>> per-command latency statistics are measured only for successful
>> transactions and commands.
>>=20
>> So how about something like this instead?
>>=20
>> -------------------------------
>> <para>
>>   The latency of a successful transaction includes the entire time of
>> -   transaction execution with rollbacks and retries. The simple
>> -   <literal>latency average</literal> shown in the main report is =
computed from
>> -   the total benchmark duration divided by both successful and =
failed
>> -   transactions, and therefore includes failed transactions. In =
contrast,
>> -   detailed latency statistics, including the per-script and =
per-command
>> -   reports, are measured only for successful transactions and =
commands, not
>> -   for failed transactions or commands.
>> +   transaction execution with rollbacks and retries. In the main =
report, when
>> +   neither <option>--progress</option>, <option>--rate</option>, nor
>> +   <option>--latency-limit</option> is specified, the
>> +   <literal>latency average</literal> is computed from the total =
benchmark
>> +   duration divided by both successful and failed transactions, and =
therefore
>> +   includes failed transactions. Otherwise, the <literal>latency
>> average</literal>
>> +   and <literal>latency stddev</literal> shown in the main report =
are measured
>> +   only for successful transactions. Detailed latency statistics, =
including
>> +   the per-script and per-command reports, are also measured only =
for
>> +   successful transactions and commands, not for failed transactions =
or
>> +   commands.
>>  </para>
>> -------------------------------
>>=20
>=20
> I feel that might be too detailed. Listing these three option names =
here could make the doc harder to read and maintain. When failures are =
included, the output line already says so explicitly:
> ```
> latency average =3D 0.387 ms (including failures)
> ```
>=20
> Can we simply say something like this?
> ```
>  <para>
>   The latency of a successful transaction includes the entire time of
>   transaction execution with rollbacks and retries.  The latency is =
measured
>   only for successful transactions and commands, not for failed =
transactions
>   or commands, unless the report explicitly says that it includes =
failures.
>  </para>
> ```
>=20
> Best regards,
> --
> Chao Li (Evan)
> HighGo Software Co., Ltd.
> https://www.highgo.com/

Thinking it over, I came up with a version that combines Fujii-san=E2=80=99=
s suggestion and my previous version:
```
  <para>
   The latency of a successful transaction includes the entire time of
   transaction execution with rollbacks and retries.  In the main =
report,
   when none of <option>--progress</option>, <option>--rate</option>, =
and
   <option>--latency-limit</option> is specified, <literal>latency =
average</literal>
   is computed from both successful and failed transactions.  In such =
cases,
   this is explicitly indicated by <literal>(including =
failures)</literal>.
   Other latency statistics, including the per-script and per-command =
reports,
   are measured only for successful transactions and commands, not for =
failed
   transactions or commands.
  </para>
```

PFA v2.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/


--Apple-Mail=_AE157C43-EC34-4777-9D04-86C524786153
Content-Disposition: attachment;
	filename=v2-0001-doc-Clarify-pgbench-reporting-with-continue-on-er.patch
Content-Type: application/octet-stream;
	x-unix-mode=0644;
	name="v2-0001-doc-Clarify-pgbench-reporting-with-continue-on-er.patch"
Content-Transfer-Encoding: quoted-printable

=46rom=200a944fb519d8eeb74037dccbf3cf5f40ce613180=20Mon=20Sep=2017=20=
00:00:00=202001=0AFrom:=20"Chao=20Li=20(Evan)"=20<lic@highgo.com>=0A=
Date:=20Thu,=2018=20Jun=202026=2014:21:13=20+0800=0ASubject:=20[PATCH=20=
v2]=20doc:=20Clarify=20pgbench=20reporting=20with=20--continue-on-error=0A=
=0AClarify=20that=20transactions=20failed=20under=20--continue-on-error=20=
are=20reported=0Aseparately=20and=20are=20not=20counted=20as=20=
transactions=20actually=20processed=20or=20in=0ATPS.=20Also=20adjust=20=
the=20latency=20description=20to=20avoid=20implying=20that=20failed=0A=
transactions=20are=20generally=20included=20in=20latency=20measurements.=0A=
=0AAuthor:=20Chao=20Li=20<lic@highgo.com>=0AReviewed-by:=20Yugo=20Nagata=20=
<nagata@sraoss.co.jp>=0AReviewed-by:=20Fujii=20Masao=20=
<masao.fujii@gmail.com>=0ADiscussion:=20=
https://postgr.es/m/597C9755-4386-488D-A289-16D50AD81FBF@gmail.com=0A---=0A=
=20doc/src/sgml/ref/pgbench.sgml=20|=2016=20++++++++++++----=0A=201=20=
file=20changed,=2012=20insertions(+),=204=20deletions(-)=0A=0Adiff=20=
--git=20a/doc/src/sgml/ref/pgbench.sgml=20=
b/doc/src/sgml/ref/pgbench.sgml=0Aindex=202e401d1ceb8..7619b8ad743=20=
100644=0A---=20a/doc/src/sgml/ref/pgbench.sgml=0A+++=20=
b/doc/src/sgml/ref/pgbench.sgml=0A@@=20-770,10=20+770,13=20@@=20pgbench=20=
<optional>=20<replaceable>options</replaceable>=20</optional>=20=
<replaceable>d=0A=20=20=20=20=20=20=20=20=20the=20connection=20to=20=
fail.=0A=20=20=20=20=20=20=20=20=20See=20<xref=20=
linkend=3D"failures-and-retries"/>=20for=20more=20information.=0A=20=20=20=
=20=20=20=20=20</para>=0A+=0A=20=20=20=20=20=20=20=20<para>=0A=20=20=20=20=
=20=20=20=20=20This=20option=20is=20useful=20when=20your=20custom=20=
script=20may=20raise=20errors=0A=20=20=20=20=20=20=20=20=20such=20as=20=
unique=20constraint=20violations,=20but=20you=20want=20the=20benchmark=0A=
-=20=20=20=20=20=20=20=20to=20continue=20and=20measure=20performance=20=
including=20those=20failures.=0A+=20=20=20=20=20=20=20=20to=20continue=20=
despite=20individual=20statement=20failures.=20=20Failed=0A+=20=20=20=20=20=
=20=20=20transactions=20are=20reported=20separately,=20but=20are=20not=20=
counted=20as=0A+=20=20=20=20=20=20=20=20transactions=20actually=20=
processed,=20nor=20are=20they=20included=20in=20TPS.=0A=20=20=20=20=20=20=
=20=20</para>=0A=20=20=20=20=20=20=20</listitem>=0A=20=20=20=20=20=20=
</varlistentry>=0A@@=20-2933,9=20+2936,14=20@@=20statement=20latencies=20=
in=20milliseconds,=20failures=20and=20retries:=0A=20=0A=20=20=20<para>=0A=
=20=20=20=20The=20latency=20of=20a=20successful=20transaction=20includes=20=
the=20entire=20time=20of=0A-=20=20=20transaction=20execution=20with=20=
rollbacks=20and=20retries.=20The=20latency=20is=20measured=0A-=20=20=20=
only=20for=20successful=20transactions=20and=20commands=20but=20not=20=
for=20failed=20transactions=0A-=20=20=20or=20commands.=0A+=20=20=20=
transaction=20execution=20with=20rollbacks=20and=20retries.=20=20In=20=
the=20main=20report,=0A+=20=20=20when=20none=20of=20=
<option>--progress</option>,=20<option>--rate</option>,=20and=0A+=20=20=20=
<option>--latency-limit</option>=20is=20specified,=20<literal>latency=20=
average</literal>=0A+=20=20=20is=20computed=20from=20both=20successful=20=
and=20failed=20transactions.=20=20In=20such=20cases,=0A+=20=20=20this=20=
is=20explicitly=20indicated=20by=20<literal>(including=20=
failures)</literal>.=0A+=20=20=20Other=20latency=20statistics,=20=
including=20the=20per-script=20and=20per-command=20reports,=0A+=20=20=20=
are=20measured=20only=20for=20successful=20transactions=20and=20=
commands,=20not=20for=20failed=0A+=20=20=20transactions=20or=20commands.=0A=
=20=20=20</para>=0A=20=0A=20=20=20<para>=0A--=20=0A2.50.1=20(Apple=20=
Git-155)=0A=0A=

--Apple-Mail=_AE157C43-EC34-4777-9D04-86C524786153--