From: Chao Li <li.evan.chao@gmail.com>
Message-Id: <958C5B66-50B0-47C8-BBA5-7D32F7A09566@gmail.com>
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_71BBEBC3-E186-4597-86EB-03BF8B21D0D3"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.600.51.1.1\))
Subject: Re: pgbench --continue-on-error: clarify TPS and failure reporting
Date: Thu, 18 Jun 2026 14:25:03 +0800
In-Reply-To: <20260618143631.25c2ed34b1d7527ee2d92a17@sraoss.co.jp>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>,
 Fujii Masao <masao.fujii@gmail.com>,
 Rintaro Ikeda <ikedarintarof@oss.nttdata.com>
To: Yugo Nagata <nagata@sraoss.co.jp>
References: <597C9755-4386-488D-A289-16D50AD81FBF@gmail.com>
 <20260618143631.25c2ed34b1d7527ee2d92a17@sraoss.co.jp>
Archived-At: 
 <https://www.postgresql.org/message-id/958C5B66-50B0-47C8-BBA5-7D32F7A09566%40gmail.com>
Precedence: bulk


--Apple-Mail=_71BBEBC3-E186-4597-86EB-03BF8B21D0D3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jun 18, 2026, at 13:36, Yugo Nagata <nagata@sraoss.co.jp> wrote:
>=20
> On Thu, 18 Jun 2026 11:24:04 +0800
> Chao Li <li.evan.chao@gmail.com> wrote:
>=20
>> Hi
>>=20
>> While testing =E2=80=9C[0ab208fa5] pgbench: Add --continue-on-error =
option=E2=80=9D, I got a question about the doc:
>> ```
>>       <para>
>>        This option is useful when your custom script may raise errors
>>        such as unique constraint violations, but you want the =
benchmark
>>        to continue and measure performance including those failures.
>>       </para>
>> ```
>>=20
>> The phrase =E2=80=9Cmeasure performance including those failures=E2=80=9D=
 gives the impression that failed transactions might be counted in TPS, =
but they are not.
>=20
> I agree with your point. The original phrasing "including those =
failures" is
> indeed=E3=80=80ambiguous and could mislead users into thinking failed =
transactions are
> included in the TPS calculation.
>=20
>> In my test, all transactions failed, and TPS was reported as 0:
>> ```
>> % pgbench  -d evantest -n -t 5 --continue-on-error =
--failures-detailed -f pgbench-coe.sql
>> pgbench (19beta1)
>> transaction type: pgbench-coe.sql
>> scaling factor: 1
>> query mode: simple
>> number of clients: 1
>> number of threads: 1
>> maximum number of tries: 1
>> number of transactions per client: 5
>> number of transactions actually processed: 0/5
>> number of failed transactions: 5 (100.000%)
>> number of serialization failures: 0 (0.000%)
>> number of deadlock failures: 0 (0.000%)
>> number of other failures: 5 (100.000%)
>> latency average =3D 0.389 ms (including failures)
>> initial connection time =3D 4.172 ms
>> tps =3D 0.000000 (without initial connection time)
>> ```
>>=20
>> If this behavior is intended, I wonder if we should clarify the TPS =
counting logic more explicitly. For example:
>> ```
>>       <para>
>>        This option is useful when your custom script may raise errors
>>        such as unique constraint violations, but you want the =
benchmark
>>        to continue despite individual statement failures. Failed =
transactions
>>        are reported separately and included in latency statistics, =
but are not
>>        counted as transactions actually processed for TPS.
>>       </para>
>> ```
>=20
> Your proposed documentation change is much clearer and accurately =
reflects
> the actual behavior. I prefer this version as well.
>=20
> Regards,
> Yugo Nagata
>=20
> --=20
> Yugo Nagata <nagata@sraoss.co.jp>

Thanks for confirming the doc issue. The here comes a small doc patch.

I also noticed there is a later paragraph about =E2=80=9Clatency=E2=80=9D =
that also needs an update, it=E2=80=99s included in the patch as well.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/


--Apple-Mail=_71BBEBC3-E186-4597-86EB-03BF8B21D0D3
Content-Disposition: attachment;
	filename=v1-0001-doc-Clarify-pgbench-reporting-with-continue-on-er.patch
Content-Type: application/octet-stream;
	x-unix-mode=0644;
	name="v1-0001-doc-Clarify-pgbench-reporting-with-continue-on-er.patch"
Content-Transfer-Encoding: quoted-printable

=46rom=20abf245787746ffa7f696298d22101b4e0226f711=20Mon=20Sep=2017=20=
00:00:00=202001=0AFrom:=20"Chao=20Li=20(Evan)"=20<lic@highgo.com>=0A=
Date:=20Thu,=2018=20Jun=202026=2014:21:13=20+0800=0ASubject:=20[PATCH=20=
v1]=20doc:=20Clarify=20pgbench=20reporting=20with=20--continue-on-error=0A=
=0AClarify=20that=20transactions=20failed=20under=20--continue-on-error=20=
are=20reported=0Aseparately=20and=20are=20not=20counted=20as=20=
transactions=20actually=20processed=20or=20in=0ATPS.=20Also=20adjust=20=
the=20latency=20description=20to=20avoid=20implying=20that=20failed=0A=
transactions=20are=20generally=20included=20in=20latency=20measurements.=0A=
=0AAuthor:=20Chao=20Li=20<lic@highgo.com>=0AReviewed-by:=20Yugo=20Nagata=20=
<nagata@sraoss.co.jp>=0ADiscussion:=20=
https://postgr.es/m/597C9755-4386-488D-A289-16D50AD81FBF@gmail.com=0A---=0A=
=20doc/src/sgml/ref/pgbench.sgml=20|=2012=20++++++++----=0A=201=20file=20=
changed,=208=20insertions(+),=204=20deletions(-)=0A=0Adiff=20--git=20=
a/doc/src/sgml/ref/pgbench.sgml=20b/doc/src/sgml/ref/pgbench.sgml=0A=
index=202e401d1ceb8..abdb1df6c89=20100644=0A---=20=
a/doc/src/sgml/ref/pgbench.sgml=0A+++=20b/doc/src/sgml/ref/pgbench.sgml=0A=
@@=20-770,10=20+770,13=20@@=20pgbench=20<optional>=20=
<replaceable>options</replaceable>=20</optional>=20<replaceable>d=0A=20=20=
=20=20=20=20=20=20=20the=20connection=20to=20fail.=0A=20=20=20=20=20=20=20=
=20=20See=20<xref=20linkend=3D"failures-and-retries"/>=20for=20more=20=
information.=0A=20=20=20=20=20=20=20=20</para>=0A+=0A=20=20=20=20=20=20=20=
=20<para>=0A=20=20=20=20=20=20=20=20=20This=20option=20is=20useful=20=
when=20your=20custom=20script=20may=20raise=20errors=0A=20=20=20=20=20=20=
=20=20=20such=20as=20unique=20constraint=20violations,=20but=20you=20=
want=20the=20benchmark=0A-=20=20=20=20=20=20=20=20to=20continue=20and=20=
measure=20performance=20including=20those=20failures.=0A+=20=20=20=20=20=20=
=20=20to=20continue=20despite=20individual=20statement=20failures.=20=20=
Failed=0A+=20=20=20=20=20=20=20=20transactions=20are=20reported=20=
separately,=20but=20are=20not=20counted=20as=0A+=20=20=20=20=20=20=20=20=
transactions=20actually=20processed,=20nor=20are=20they=20included=20in=20=
TPS.=0A=20=20=20=20=20=20=20=20</para>=0A=20=20=20=20=20=20=20=
</listitem>=0A=20=20=20=20=20=20</varlistentry>=0A@@=20-2933,9=20=
+2936,10=20@@=20statement=20latencies=20in=20milliseconds,=20failures=20=
and=20retries:=0A=20=0A=20=20=20<para>=0A=20=20=20=20The=20latency=20of=20=
a=20successful=20transaction=20includes=20the=20entire=20time=20of=0A-=20=
=20=20transaction=20execution=20with=20rollbacks=20and=20retries.=20The=20=
latency=20is=20measured=0A-=20=20=20only=20for=20successful=20=
transactions=20and=20commands=20but=20not=20for=20failed=20transactions=0A=
-=20=20=20or=20commands.=0A+=20=20=20transaction=20execution=20with=20=
rollbacks=20and=20retries.=20=20The=20latency=20is=20measured=0A+=20=20=20=
only=20for=20successful=20transactions=20and=20commands,=20except=20that=20=
failed=0A+=20=20=20transactions=20are=20included=20in=20the=20simple=20=
average=20latency=20when=20the=0A+=20=20=20=
<option>--continue-on-error</option>=20option=20is=20used.=0A=20=20=20=
</para>=0A=20=0A=20=20=20<para>=0A--=20=0A2.50.1=20(Apple=20Git-155)=0A=0A=

--Apple-Mail=_71BBEBC3-E186-4597-86EB-03BF8B21D0D3--