MIME-Version: 1.0
References: <597C9755-4386-488D-A289-16D50AD81FBF@gmail.com>
 <20260618143631.25c2ed34b1d7527ee2d92a17@sraoss.co.jp>
 <958C5B66-50B0-47C8-BBA5-7D32F7A09566@gmail.com>
 <CAHGQGwFX6Wqdp=ZEjs6WSu3GDSpgM9qZTixBHXKB9xaVzQ+0-A@mail.gmail.com>
 <09361838-9213-4961-95A5-A0CD4AABF2DD@gmail.com>
 <A476B863-DF6E-4815-96C7-B6288669B05E@gmail.com>
In-Reply-To: <A476B863-DF6E-4815-96C7-B6288669B05E@gmail.com>
From: Xuneng Zhou <xunengzhou@gmail.com>
Date: Fri, 10 Jul 2026 16:36:33 +0800
Message-ID: 
 <CABPTF7V3QJDFa+rHYrChOOdiC2Dr1ydAtSUQGp3gA1p-fNXHaw@mail.gmail.com>
Subject: Re: pgbench --continue-on-error: clarify TPS and failure reporting
To: Chao Li <li.evan.chao@gmail.com>
Cc: Fujii Masao <masao.fujii@gmail.com>, Yugo Nagata <nagata@sraoss.co.jp>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>,
	Rintaro Ikeda <ikedarintarof@oss.nttdata.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CABPTF7V3QJDFa%2BrHYrChOOdiC2Dr1ydAtSUQGp3gA1p-fNXHaw%40mail.gmail.com>
Precedence: bulk

Hi Evan,

On Mon, Jun 22, 2026 at 4:11=E2=80=AFPM Chao Li <li.evan.chao@gmail.com> wr=
ote:
>
>
>
> > On Jun 18, 2026, at 20:45, Chao Li <li.evan.chao@gmail.com> wrote:
> >
> >
> >
> >> On Jun 18, 2026, at 17:40, Fujii Masao <masao.fujii@gmail.com> wrote:
> >>
> >> On Thu, Jun 18, 2026 at 3:25=E2=80=AFPM Chao Li <li.evan.chao@gmail.co=
m> wrote:
> >>> Thanks for confirming the doc issue. The here comes a small doc patch=
.
> >>>
> >>> I also noticed there is a later paragraph about =E2=80=9Clatency=E2=
=80=9D that also needs an update, it=E2=80=99s included in the patch as wel=
l.
> >>
> >> Thanks for the patch! The TPS-related changes look good to me.
> >>
> >>   The latency of a successful transaction includes the entire time of
> >> -   transaction execution with rollbacks and retries. The latency is m=
easured
> >> -   only for successful transactions and commands but not for failed
> >> transactions
> >> -   or commands.
> >> +   transaction execution with rollbacks and retries.  The latency is =
measured
> >> +   only for successful transactions and commands, except that failed
> >> +   transactions are included in the simple average latency when the
> >> +   <option>--continue-on-error</option> option is used.
> >>
> >> It doesn't seem that the behavior of the latency average in the main
> >> report depends on --continue-on-error itself. Instead, when neither
> >> --progress, --rate, nor --latency-limit is specified, the main
> >> report shows a simple latency average computed as the total benchmark
> >> duration divided by the number of successful and failed transactions,
> >> so failed transactions are included.
> >>
> >> On the other hand, when any of those options is specified, the
> >> latency average and latency stddev in the main report are based
> >> only on successful transactions. Likewise, the per-script and
> >> per-command latency statistics are measured only for successful
> >> transactions and commands.
> >>
> >> So how about something like this instead?
> >>
> >> -------------------------------
> >> <para>
> >>   The latency of a successful transaction includes the entire time of
> >> -   transaction execution with rollbacks and retries. The simple
> >> -   <literal>latency average</literal> shown in the main report is com=
puted from
> >> -   the total benchmark duration divided by both successful and failed
> >> -   transactions, and therefore includes failed transactions. In contr=
ast,
> >> -   detailed latency statistics, including the per-script and per-comm=
and
> >> -   reports, are measured only for successful transactions and command=
s, not
> >> -   for failed transactions or commands.
> >> +   transaction execution with rollbacks and retries. In the main repo=
rt, when
> >> +   neither <option>--progress</option>, <option>--rate</option>, nor
> >> +   <option>--latency-limit</option> is specified, the
> >> +   <literal>latency average</literal> is computed from the total benc=
hmark
> >> +   duration divided by both successful and failed transactions, and t=
herefore
> >> +   includes failed transactions. Otherwise, the <literal>latency
> >> average</literal>
> >> +   and <literal>latency stddev</literal> shown in the main report are=
 measured
> >> +   only for successful transactions. Detailed latency statistics, inc=
luding
> >> +   the per-script and per-command reports, are also measured only for
> >> +   successful transactions and commands, not for failed transactions =
or
> >> +   commands.
> >>  </para>
> >> -------------------------------
> >>
> >
> > I feel that might be too detailed. Listing these three option names her=
e could make the doc harder to read and maintain. When failures are include=
d, the output line already says so explicitly:
> > ```
> > latency average =3D 0.387 ms (including failures)
> > ```
> >
> > Can we simply say something like this?
> > ```
> >  <para>
> >   The latency of a successful transaction includes the entire time of
> >   transaction execution with rollbacks and retries.  The latency is mea=
sured
> >   only for successful transactions and commands, not for failed transac=
tions
> >   or commands, unless the report explicitly says that it includes failu=
res.
> >  </para>
> > ```
> >
> > Best regards,
> > --
> > Chao Li (Evan)
> > HighGo Software Co., Ltd.
> > https://www.highgo.com/
>
> Thinking it over, I came up with a version that combines Fujii-san=E2=80=
=99s suggestion and my previous version:
> ```
>   <para>
>    The latency of a successful transaction includes the entire time of
>    transaction execution with rollbacks and retries.  In the main report,
>    when none of <option>--progress</option>, <option>--rate</option>, and
>    <option>--latency-limit</option> is specified, <literal>latency averag=
e</literal>
>    is computed from both successful and failed transactions.  In such cas=
es,
>    this is explicitly indicated by <literal>(including failures)</literal=
>.
>    Other latency statistics, including the per-script and per-command rep=
orts,
>    are measured only for successful transactions and commands, not for fa=
iled
>    transactions or commands.
>   </para>
> ```

        <para>
         This option is useful when your custom script may raise errors
         such as unique constraint violations, but you want the benchmark
-        to continue and measure performance including those failures.
+        to continue despite individual statement failures.  Failed
+        transactions are reported separately, but are not counted as
+        transactions actually processed, nor are they included in TPS.

"Nor are they included in TPS" seems misleading to me. Failed
transactions do not enter the TPS accumulator, but their execution
time remains in bench_duration, so they can reduce the overall TPS.

tps =3D total->cnt / bench_duration;

--=20
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.