Date: Thu, 18 Jun 2026 14:36:31 +0900
From: Yugo Nagata <nagata@sraoss.co.jp>
To: Chao Li <li.evan.chao@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>, Fujii Masao
 <masao.fujii@gmail.com>, Rintaro Ikeda <ikedarintarof@oss.nttdata.com>
Subject: Re: pgbench --continue-on-error: clarify TPS and failure reporting
Message-Id: <20260618143631.25c2ed34b1d7527ee2d92a17@sraoss.co.jp>
In-Reply-To: <597C9755-4386-488D-A289-16D50AD81FBF@gmail.com>
References: <597C9755-4386-488D-A289-16D50AD81FBF@gmail.com>
Content-Type: text/plain; charset=ISO-2022-JP
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?iso-2022-jp?B?NHV4eGwrRUJ3MWVNNEdoWFhwVEhNUUNJN0trR2tVQjFFYUhHRVNQZ1dU?=
 =?iso-2022-jp?B?aTRueTJNaG1LYnlNZzlIc2V0ZXFlckZMdG4zQ0lWYVdGVHgrdUFXNDdH?=
 =?iso-2022-jp?B?UnFaRWJjMGNBNTRpakJpUVdBU1JvOFJJdFZYS1dsOGViNHpJRGxVcFRG?=
 =?iso-2022-jp?B?bE5xT25Cb2ZaU2xmSmw3ZHo1dXcwQjBGbmRkZDBock5WaUJ4T2FXNnpa?=
 =?iso-2022-jp?B?YzNlUTVvNjB0OVp2cHI0ZU90NGZkeklnUXpvYmdMblA0M1dHZmRRZ09v?=
 =?iso-2022-jp?B?eGRuWVdhSFlnR0FXdzkrQkxjSVZqaFhEelJzWWIrYUQ1c2hSR2tINlYy?=
 =?iso-2022-jp?B?MU1PYjhoNWlubFlhS25xNXRGWFdkRUVMN2N3RTZXSW44bVRmdXc5U2to?=
 =?iso-2022-jp?B?Q2RGMFF2cW0vNmlVM05pNkZLUjZKbWEreXh1WlNHU25rSXJzSGw1aHB3?=
 =?iso-2022-jp?B?alVDMTNWMnZCdi83ZWNzMFluSnh0Y2wwclpSbHZRSGM1dHZ3YnZXTFM5?=
 =?iso-2022-jp?B?a2RxL0dyb1JPZkV1bkhKSDJMTnZQd0JGVnVWNW1qQ25NNjR0M0FROGtS?=
 =?iso-2022-jp?B?OTVIdXhHWER0dEFTeG9nSGhYRFYxazBEb3dzbDNHcXpDenNBWnpYNXhH?=
 =?iso-2022-jp?B?V010WC82VmVxZEdwbEdOWThlc280NlEzcDVDYjZTZmY1VVBLamhiRkQw?=
 =?iso-2022-jp?B?R3A3Vm4zdnplT29Kbis5djg5Q0o1MzRQbmpjRFYzTTYrd2xtcFlCTlRq?=
 =?iso-2022-jp?B?ckQ0eDhDRXd3RnJFcVU5Q1ExMVBLSnROV3MvQWJaNm1seThKT0VFclAr?=
 =?iso-2022-jp?B?eURxaFJGVWtWNlJtQlViMkpCb3lZK1dBbHVZdG1nQ0FqVm5mUTFROVlv?=
 =?iso-2022-jp?B?WFUzdTR4aVFWYWc3RUo2elgxcHZ1TEdzZkZoMzFyU2lpUHVvMHl1eUQ5?=
 =?iso-2022-jp?B?MlhOSVdKbHZXNEk2NktDWVdQZXNWT2FCZVpicnRGVVRmdnZURnltbU9a?=
 =?iso-2022-jp?B?aUN5WVQzYndwVURtVDRNVUpwS3N0dXhLME5abkpiRTZKTmZXbTlZOVFj?=
 =?iso-2022-jp?B?ckt2MS81dHc5eGpMSEh2aG1oRE0wUUlETEN4VG1KbnRtNWRERXZWblpz?=
 =?iso-2022-jp?B?NzVNbFI1cHVSb0wwTUE4a3VkVlFMRkdrR0NBcC80VmhDUy9HZGVvVVN2?=
 =?iso-2022-jp?B?c1hvSmZ3N2dQSmcrR2hTZGlBVk9ERTlrbVhjNzk0d0dIcnhjM1lSMWlI?=
 =?iso-2022-jp?B?Z2JHVkpnTnZBS1E4RkhkbjFBdTVqNXQvcUlEb0RJZkllampaaVpNQ0hx?=
 =?iso-2022-jp?B?enRDYmtTb21zSTNrcXExUTRrcm5pcGVldXVKeWREb08xcTc5dlNvb04w?=
 =?iso-2022-jp?B?OUoxTFdQczNCR3JvbTUzNWZ1WmJPNDFXSmlQdkF5NVRwNTVobmk4OUtK?=
 =?iso-2022-jp?B?Ui9mZ2JDS1RtOW12WVV3aTBGSThGUUduV0J0LzIzWHJoTlFkclFwdnBw?=
 =?iso-2022-jp?B?bUxVd1piV09RUzM3eHo3c0JsdUZXRjY2ejlWalFXK2NNRE56WGdTRkN6?=
 =?iso-2022-jp?B?UndHVjhxem1oQ2QyWTVLa1NRaFNhSVUwdXBFZ1FpZUJVZWh2SGFEak5p?=
 =?iso-2022-jp?B?WElibkFXZnhEdzYzWll2QWkyVVd3UW95bWszRnNVY0tJZm9BOHo2Q0w3?=
 =?iso-2022-jp?B?VTNCMFA5YmliU25vTnJWaE1RMmRaVFJlalZuTkRjNTROaTYrWUszQmQr?=
 =?iso-2022-jp?B?b3luMkNmeGtWdDVzcmFWSmJCT2x6UHBja2FEb1owWTNiZlRPS2xmdUxT?=
 =?iso-2022-jp?B?ZUkxVkhzR21hZlNRc3djcWlzdnB3V293ZmNqY1U3VHJFQUIyZUFyM2p4?=
 =?iso-2022-jp?B?cjZ4enZ5K29IcVpZYjdKOW5BOWh2YjZnM2RWREVQUEtscVNZR1E5VTQ3?=
 =?iso-2022-jp?B?VnBDc2NXK2FMbzhBOXZFVUJzbXpOb3hWV3dudFlCeVRtbnJNMSs4eWtZ?=
 =?iso-2022-jp?B?OUNMVE9kNWJJMGJ6bFg0bGk1bjJLM2tEZVAvQk40SlorNmhkbDBkWE53?=
 =?iso-2022-jp?B?bmt5UXlibkVDSExsOURhSlMvaE5UeEdIaTJTWklWOFN0cFNTaWlmRURi?=
 =?iso-2022-jp?B?bnBDSFRlZHR4TlMvaUlSZkhjQkN4emNneUZIakxuYXhpbmFsMm1QQ3E5?=
 =?iso-2022-jp?B?T25mUVZkVk9zY0d1bjlIaTRGZ2t6THFCL1lGeG50TkJmSWcxRGZmR0Z0?=
 =?iso-2022-jp?B?N2h0MWJZcElDd2R1UFFXUjhVUUttTWNMbnJ3SmY0b045b0pNUHhxU2s0?=
 =?iso-2022-jp?B?RkJlMllXcmhocjFzbEx5K3JhbkZjRmp2R0tMa3V0K2dMUWlkYWorVk5x?=
 =?iso-2022-jp?B?Vmlmekd5TDlQM0RNck82QUtBc2cvQlFHSVhqdVBIMXNoYUphZVNYUEJQ?=
 =?iso-2022-jp?B?UGl5ZWZYaGkzMmlPZEpVYWtYVWkvTGc3MGpEbnJ0QWVTUWZnZlBzcjFT?=
 =?iso-2022-jp?B?VUc5U1dwbkhreXBKajVaQmVEbjZZOFZrSTVOb1hkMk9ScHA0Z1JndHJr?=
 =?iso-2022-jp?B?QVd0ajlnZ2p6TGtYQWNhWkpvR2Uzc1RMQi9yaFFoNkNRQkVx?=
X-MS-Exchange-AntiSpam-MessageData-1: bUpfnRpg07pVxQ==
X-OriginatorOrg: sraoss.co.jp
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 6422d639-1b97-4136-6981-08deccfb8ddf
X-MS-Exchange-CrossTenant-AuthSource: OSZP286MB2160.JPNP286.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2026 05:36:32.3152
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 75db6c32-352e-4063-ae79-cafb8623f6db
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 
 cTPXv9XLv14wK0ak0P3JhCh4DU5tFTzx4jjtjaTsxX+q/fxJNrZO65wzWd+7ZkqCfU83wqZgQmaXKxNbO1V7cA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYXP286MB5654
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: 
 <https://www.postgresql.org/message-id/20260618143631.25c2ed34b1d7527ee2d92a17%40sraoss.co.jp>
Precedence: bulk

On Thu, 18 Jun 2026 11:24:04 +0800
Chao Li <li.evan.chao@gmail.com> wrote:

> Hi
> 
> While testing “[0ab208fa5] pgbench: Add --continue-on-error option”, I got a question about the doc:
> ```
>        <para>
>         This option is useful when your custom script may raise errors
>         such as unique constraint violations, but you want the benchmark
>         to continue and measure performance including those failures.
>        </para>
> ```
> 
> The phrase “measure performance including those failures” gives the impression that failed transactions might be counted in TPS, but they are not.

I agree with your point. The original phrasing "including those failures" is
indeed　ambiguous and could mislead users into thinking failed transactions are
included in the TPS calculation.

> In my test, all transactions failed, and TPS was reported as 0:
> ```
> % pgbench  -d evantest -n -t 5 --continue-on-error --failures-detailed -f pgbench-coe.sql
> pgbench (19beta1)
> transaction type: pgbench-coe.sql
> scaling factor: 1
> query mode: simple
> number of clients: 1
> number of threads: 1
> maximum number of tries: 1
> number of transactions per client: 5
> number of transactions actually processed: 0/5
> number of failed transactions: 5 (100.000%)
> number of serialization failures: 0 (0.000%)
> number of deadlock failures: 0 (0.000%)
> number of other failures: 5 (100.000%)
> latency average = 0.389 ms (including failures)
> initial connection time = 4.172 ms
> tps = 0.000000 (without initial connection time)
> ```
> 
> If this behavior is intended, I wonder if we should clarify the TPS counting logic more explicitly. For example:
> ```
>        <para>
>         This option is useful when your custom script may raise errors
>         such as unique constraint violations, but you want the benchmark
>         to continue despite individual statement failures. Failed transactions
>         are reported separately and included in latency statistics, but are not
>         counted as transactions actually processed for TPS.
>        </para>
> ```

Your proposed documentation change is much clearer and accurately reflects
the actual behavior. I prefer this version as well.

Regards,
Yugo Nagata

-- 
Yugo Nagata <nagata@sraoss.co.jp>