Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vK1rF-00Cd5I-2W for pgsql-hackers@arkaria.postgresql.org; Fri, 14 Nov 2025 22:01:28 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1vK1rC-008K5K-2O for pgsql-hackers@arkaria.postgresql.org; Fri, 14 Nov 2025 22:01:26 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vJvbz-006yvT-1D for pgsql-hackers@lists.postgresql.org; Fri, 14 Nov 2025 15:21:19 +0000 Received: from mail-westus3azolkn19011066.outbound.protection.outlook.com ([52.103.23.66] helo=PH0PR06CU001.outbound.protection.outlook.com) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vJvbx-007jwu-0m for pgsql-hackers@postgresql.org; Fri, 14 Nov 2025 15:21:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bEF6aFDAPQ0X9C4P8c95yb755GhYvkuMuOLMAtzETtXEIAyKpUUlDKWnmhRY7rGRU0sYty3TQO0ibmNu5ofnjnzut5qe77KGxO7pd8cocNjipF0ptQXPQX0mlHHoEb1HRzaaAOEeyFgEKyQmPERp4WYwSo76N265FApiNaQBlL9BpUk/Hn6AmLvP/bX0IJdsYHPyO7GpLtZhv8B6zaDt6VA4G8grPLdDT3xZDz3H2khEZ4N1GHvxKpoMNA5UnA6QBO2McNijpa9B1F5vK7lx+SC3ppjtUlcIzYhsi6Q6RBDVZ6mRXvUOFTQ7FyLCLTGG8OQTx3UkuQUGD9xm58QxiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vg06F3Vt7GLpqDzRmERQMbE3qy3Wy/fBt1eIWRZ0V9I=; b=bZsCkGyvNRSfAfwNLuzAL/nnDSUruK8rOK3WLD2mHauI5E2yx4aDf2/vCezVmkdI0a+EjHjNLosC6v+13JO6SibWVUvhtW03Gd+xdyFSaPxXO/8hZxYBNyz11x0nZZpriT5EZO2dHnk7P14EK6PNGXYnfworw4NzwfdIRqb+Di8D/t8HKYVmkICIIYD4BJrzeUSWN1Q+iO5vDYhCs12naHo99fzwNR3hzoVDlORZQEIaOjamzimrSGFRe7yeFNzjg6D0lEDO73oSUEAWCF8czE4euooiT2zmoFON/2Rr0AXwwSFGhCcrNvqrb2736QC8OB36BKcCf6ch4vEwlVRd0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vg06F3Vt7GLpqDzRmERQMbE3qy3Wy/fBt1eIWRZ0V9I=; b=ItaKZ8E58Qu/P9uGyfWFd6LZfD0R3HJtF+n4yxsjFKJjhmjJfDzwKlkww6T0iB7bOZ1yLHVideOzxZ2DK/6sxV2/I83aUqhSy6yz4JarvTzoVVW/fY+Kp9KrYwVHw9dq0S1i/Kb+y6qwRrqRQ0C8hzIpYdJLPBWHURQNwn8XWDrA6HcxM45z0g6cbqGpQIn/PjqoJc7IimrRkffWuCM4njWq4LDmd55eDDHY7M5OqhlMB7GLjGVfT1jYfL81qQe9bA3G3bi9PgsweKU5UFmO5JPUcs3g4tqA7b/ZYAs/TrdefBFCQcuHp2yDUhmjQIBoCsw5+o0ZNfluEgPZUUNcTw== Received: from DS0PR08MB9565.namprd08.prod.outlook.com (2603:10b6:8:1ba::17) by DS2PR08MB10731.namprd08.prod.outlook.com (2603:10b6:8:27a::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9320.17; Fri, 14 Nov 2025 15:21:12 +0000 Received: from DS0PR08MB9565.namprd08.prod.outlook.com ([fe80::835a:699e:47ee:c088]) by DS0PR08MB9565.namprd08.prod.outlook.com ([fe80::835a:699e:47ee:c088%4]) with mapi id 15.20.9320.013; Fri, 14 Nov 2025 15:21:12 +0000 From: Boris Mironov To: Ashutosh Bapat , "pgsql-hackers@postgresql.org" Subject: Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Thread-Topic: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Thread-Index: AQHcUwuRPlzLCtdZw06GAdoL15fmHbTwZ0aAgAGpzjGAACRkgIAAEcrG Date: Fri, 14 Nov 2025 15:21:12 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-CA, en-US Content-Language: en-CA X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DS0PR08MB9565:EE_|DS2PR08MB10731:EE_ x-ms-office365-filtering-correlation-id: 5a0b334b-6c41-4f30-2e4b-08de239171ea x-microsoft-antispam: BCL:0;ARA:14566002|461199028|15030799006|15080799012|8062599012|8060799015|51005399006|19110799012|12121999013|31061999003|3412199025|440099028|40105399003|102099032; x-microsoft-antispam-message-info: =?iso-8859-1?Q?P59zk1AR7v2H4UlozmVOYblwp3WLwBFoL+IdQ6BINEUurmtVMI2NePjnQq?= =?iso-8859-1?Q?nUFEGMaGGzvMBbOA+froBIVCa+RsLIduRjdwU7jaYTCH0MsAIpxlQmRE2L?= =?iso-8859-1?Q?lwrai6N6e6FsWv9cHojx7iaH3CdUEBKUa8XoW3gEGtBwaBDIbYIi6ED1Jw?= =?iso-8859-1?Q?DDZiFlTVMcKrjzFB2tsCpzTSgs737cS/lNO/RIDFGAqjvIu4h7PY4zhWqh?= =?iso-8859-1?Q?wLH4kWtm/glDk2qIIWmX4kCPbc6+2aJ2VmbG7v3Qc5CvF3HRqMB1JZ0AxO?= =?iso-8859-1?Q?lOwTEaExDuLhwEnUgbZFroFpxhWDIomoxx85FPEdyxPWJS2gIOa5W5VDbc?= =?iso-8859-1?Q?qqQ89Zp9Teadh/gTw2mkEpLnlHHl++cQu3kgmCaBnvHRgGG0SCQuxrq1Ys?= =?iso-8859-1?Q?gHFwQSreoB+JucVjkucGJ8AiDl8HVRKYm0p+qfrosbkZC26uGmL3wnySej?= =?iso-8859-1?Q?laMwFu6rPy1m0Igk588DzCWpsg1Ca9ONqjpVI8zbBcmEzgF4t+jDgmqGWR?= =?iso-8859-1?Q?q5A+EfdfCyUdlKQ00TYlMN15CKRXb5bIpTTOMSVhkPhHsljeyfSDfG8VV2?= =?iso-8859-1?Q?frutNhy9ragaqe5n9JEVEcACnycRiM4aMV8Cdrs876hLdC/38CAl/ZkN5M?= =?iso-8859-1?Q?pE8PHbyqFPNT0UWXDtE+UQLldDJFCRiYs7MwcCGXJoBsyxhLO1MBHe9LDQ?= =?iso-8859-1?Q?bsxBPdrOa4Th4uW/7OmNBSjScLLOShiMVOV51+aBYK8A1wKmiGNJ1d4Iu+?= =?iso-8859-1?Q?rJydNzevwag4eFlV7AtCNwv1pRg/HmEt0t8cPU7VwGl4hWQLcu0POVBewx?= =?iso-8859-1?Q?2auiH1I/mQyq0jinsEWttZJvhwVgKUrOC+uKmxP9gQ6W/LBZF+k6Zq4mLr?= =?iso-8859-1?Q?yl11WdNR0uduIdJK/XMi3Zywyll3xMuXuFZi1tpcTN7Qel1mQodRju4I6W?= =?iso-8859-1?Q?3dPqMaJuDNEWZ2IeSCZzyXyi0l8uw109xbrquhM+aUPj+BmyIX9Uy0uJCK?= =?iso-8859-1?Q?eCeg5K41x1DLbwCEejUpLA9LeM9l0tnxFucYagmQRWEAPJrj8HB8lmEHrm?= =?iso-8859-1?Q?OuMSyL9R2WLcbkkhPcXswUCsKbzbfAGbOCfWHfHsNyeDJ/74x1Dst/g6qg?= =?iso-8859-1?Q?/YghIilasf47DQtp0S06WrLK4eg7LBMWVESyHkX2sPrRP6GKhj3iZ32jj9?= =?iso-8859-1?Q?3yB8dmOi97nNjx+5OQbXER6HEI/tWNsNk2ua5I3PH7bh01vHzK2CkF4GmT?= =?iso-8859-1?Q?tvqNJAyvorsd9YIB3saib0DWFnQqtv1dDwcidB2M2EQiYfGwjPX5FjeJlW?= =?iso-8859-1?Q?CE94?= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?2t4Hggm+dIFo0VyyloKFq+zJ874irMi3LC0Bpt59qafoFBtCVsde98SMUJ?= =?iso-8859-1?Q?wrr0q1X64+uDAj2R89xJ3JrOz3tM+U8+Q878441jtTn3ObEYEt/yxUMJvn?= =?iso-8859-1?Q?otNvl8Yjr6O0TSYN7lAuiWFA4QJZ/dvEMFc4MbipnGAawinxqZf1DmkeVl?= =?iso-8859-1?Q?G2rlFUW1xKJAlnHhAxjJOIrL+PVjfEjFwTlRUO2G2/1bmeSeG0KWTW4z44?= =?iso-8859-1?Q?283mKuRIFEE8t6AT7hQp5dWStJJLMP36BWrtTDRgL2YwYO2OryW8NNEeUP?= =?iso-8859-1?Q?OEVAFWFYWu0pcIY3r8S7L+N6RKxhavfkZeiMHDRl5gLZI79zpBHVruUjug?= =?iso-8859-1?Q?WZ8RnJW5XXqU1IZTxprlYFmdR+/zfwUIIB6jlYDykijJFzX2b8AHkqT9Gh?= =?iso-8859-1?Q?2toj8nggb7eA0kup40VRV7CZq6/xvkGIHRc+Zp8TB9jRfaEwL1MshXMHP7?= =?iso-8859-1?Q?RgxM9houT4xb2mZOmigAdVSaJUERp0iB4JZrhepoGw+BfyO848EKA5hSd/?= =?iso-8859-1?Q?rVCCxlhvmz1qU+ONYndeOWilXRoi3B73bwP+xgRgIwdyjnWQigB2DLIgBb?= =?iso-8859-1?Q?3pojRgoFnZgMbcxHi80B886FKIrmHOdazk7vTL8JwU33lsMZ8GLb0v9SyS?= =?iso-8859-1?Q?ANGgFWv3R/vGl06cN+inPAsRfA5fGnsKNbQHo0wvCBTaji3B6aw3hegHEV?= =?iso-8859-1?Q?6WztXBR/XLtQG8hVRkQS6PJI5Qoj6V4jRAPoEczXDh264f8CNotJ0kybI2?= =?iso-8859-1?Q?oWRTU4FKzNxphBC1C7EMCuxIV9ziRNzMsEi8AxpAIcuo31wzb1HbsQqPWi?= =?iso-8859-1?Q?T5eEQoUbMLen61lztfUTt0tBG+yliJMFBLg4YjQL0D1dHRzY+C+IOqr4w2?= =?iso-8859-1?Q?jzyU06wZCHBplwfVu/F0RTmbEJJhBhvro5qen0tfQtGUOgXDrPyuba1hal?= =?iso-8859-1?Q?chC1vJt+ACh2mc1CvZ7emrRDqhhs0LUCYtSiqHWZR5P9MF/3H2FTZYI9tV?= =?iso-8859-1?Q?rcSOHpUVE5PZhqfqU8caAGiD6DryXWpyWAeivxxGVlrlkwOJdctp0+ViFt?= =?iso-8859-1?Q?52AcV6u6eamhuSyzPdsxKtVRN9gboDyz/6JLAltl4C4Y59qPQMIXWj+ftX?= =?iso-8859-1?Q?H24Mdg9kf4VQpNUsZ4FHojjVSU7CPBQ8DvgUuOao2v1+so9l5vQPfn1BTT?= =?iso-8859-1?Q?eq+mUdUBOv0etDnfB8jA0DO2Z/sWGo7LkTXRXqmjFukm6V+XKQS4tBGADu?= =?iso-8859-1?Q?czniV4qkfRdSyDoxUnwS/Pu9fKIOD0EgqYkcAf0wQ=3D?= Content-Type: multipart/alternative; boundary="_000_DS0PR08MB956560D79EA051E98688F78088CAADS0PR08MB9565namp_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DS0PR08MB9565.namprd08.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 5a0b334b-6c41-4f30-2e4b-08de239171ea X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Nov 2025 15:21:12.1020 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS2PR08MB10731 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --_000_DS0PR08MB956560D79EA051E98688F78088CAADS0PR08MB9565namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Ashutosh, > If there is one method that is better than all others, community will > be more willing to accept implementation of that one method than > multiple implementations so as to reduce maintenance burden. Ok then. I'll leave "COPY FROM STDIN BINARY" implementation out of 3 only. Would you prefer to replace original COPY FROM STDIN TEXT by this code or add it as new "init-step" (e.g., with code "c")? I also have noted that current code doesn't prevent pgbench parameter like "--init-steps=3DdtgG". It allows to run data generation step twice. Each of these "g" and "G" will present own timing in status line. Is this an oversight or intentional? > The code in the patch does not have enough comments. It's hard to > understand the methods just from the code. Each of the generateData* > functions could use a prologue explaining the data generation method > it uses. To add comments is not a problem at all. So far, it was just "code for myse= lf" and I was checking if there is any interest in community to include it. >> I'm sure that much more testing is required to run this code under diffe= rent >> conditions and hardware to get a better picture. So far it looks very pr= omising. > Sure. Cheers, Boris --_000_DS0PR08MB956560D79EA051E98688F78088CAADS0PR08MB9565namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi Ashutosh,

> If there is one method that is better= than all others, community will
> be more willing to accept implementation of that one method than
> multiple implementations so as to reduce maintenance burden.

Ok then. I'll leave "COPY FROM STDIN BINARY" implementation out o= f 3 only.
Would you prefer to replace original COPY FROM STDIN TEXT by this
code or add it as new "init-step" (e.g., with code "c")= ?

I also have noted that current code doesn't prevent pgbench parameter
like "--init-steps=3DdtgG". It allows to run data generation step= twice.
Each of these "g" and "G" will present own timing in st= atus line. Is this
an oversight or intentional?

> The code in the patch does not have enough comments. It's hard to
> understand the methods just from the code. Each of the generateData* > functions could use a prologue explaining the data generation method > it uses.

To add comments is not a problem at all. So far, it was just "code for= myself"
and I was checking if there is any interest in community to include it.

>> I'm sure that much more testing is required to run this code under= different
>> conditions and hardware to get a better picture. So far it looks v= ery promising.

> Sure.

Cheers,
Boris
--_000_DS0PR08MB956560D79EA051E98688F78088CAADS0PR08MB9565namp_--