public inbox for [email protected]
help / color / mirror / Atom feedIdea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
21+ messages / 5 participants
[nested] [flat]
* Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
@ 2025-11-11 13:33 Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2025-11-11 13:33 UTC (permalink / raw)
To: pgsql-hackers
Hello hackers,
For some of my specific hardware tests I needed to generate big databases well beyond RAM size. Hence I turned to pgbench tool and its default 2 modes for client- and server-side generation for TPC-B tests. When I use "scale" factor in range of few thousands (eg, 3000 - 5000) data generation phase takes quite some time. I looked at it as opportunity to prove/disprove 2 hypothesises:
*
will INSERT mode work faster if we commit once every "scale" and turn single INSERT into "for" loop with commits for 3 tables in the end of each loop
*
will "INSERT .. SELECT FROM unnest" be faster than "INSERT .. SELECT FROM generate_series"
*
will BINARY mode work faster than TEXT even though we send much more data
*
and so on
As a result of my experiments I produced significant patch for pgbench utility and though that it might be of interest not just for me. Therefore I'm sending draft version of it in diff format for current development tree on GitHub. As of November 11, 2025 I can merge with main branch of the project on GitHub.
Spoiler alert: "COPY FROM BINARY" is significantly faster than current "COPY FROM TEXT"
Would be happy to polish it if there is interest to such change.
Cheers
Attachments:
[application/octet-stream] pgbench.c.diff (29.0K, 3-pgbench.c.diff)
download | inline diff:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..71aa1d9479f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,14 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +189,18 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * mode of data generation to use
+ */
+static char data_generation_type = '?';
+
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -402,14 +422,15 @@ typedef struct StatsData
* directly successful transactions (they were successfully completed on
* the first try).
*
- * A failed transaction is defined as unsuccessfully retried transactions.
- * It can be one of two types:
- *
- * failed (the number of failed transactions) =
+ * 'failed' (the number of failed transactions) =
* 'serialization_failures' (they got a serialization error and were not
- * successfully retried) +
+ * successfully retried) +
* 'deadlock_failures' (they got a deadlock error and were not
- * successfully retried).
+ * successfully retried) +
+ * 'other_sql_failures' (they failed on the first try or after retries
+ * due to a SQL error other than serialization or
+ * deadlock; they are counted as a failed transaction
+ * only when --continue-on-error is specified).
*
* If the transaction was retried after a serialization or a deadlock
* error this does not guarantee that this retry was successful. Thus
@@ -421,7 +442,7 @@ typedef struct StatsData
*
* 'retried' (number of all retried transactions) =
* successfully retried transactions +
- * failed transactions.
+ * unsuccessful retried transactions.
*----------
*/
int64 cnt; /* number of successful transactions, not
@@ -440,6 +461,11 @@ typedef struct StatsData
int64 deadlock_failures; /* number of transactions that were not
* successfully retried after a deadlock
* error */
+ int64 other_sql_failures; /* number of failed transactions for
+ * reasons other than
+ * serialization/deadlock failure, which
+ * is counted if --continue-on-error is
+ * specified */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -457,6 +483,7 @@ typedef enum EStatus
{
ESTATUS_NO_ERROR = 0,
ESTATUS_META_COMMAND_ERROR,
+ ESTATUS_CONN_ERROR,
/* SQL errors */
ESTATUS_SERIALIZATION_ERROR,
@@ -770,6 +797,7 @@ static int64 total_weight = 0;
static bool verbose_errors = false; /* print verbose messages of all errors */
static bool exit_on_abort = false; /* exit when any client is aborted */
+static bool continue_on_error = false; /* continue after errors */
/* Builtin test scripts */
typedef struct BuiltinScript
@@ -842,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -906,7 +935,10 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -949,6 +981,7 @@ usage(void)
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --continue-on-error continue running after an SQL error\n"
" --exit-on-abort exit when any client is aborted\n"
" --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
@@ -1467,6 +1500,7 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->retried = 0;
sd->serialization_failures = 0;
sd->deadlock_failures = 0;
+ sd->other_sql_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1516,6 +1550,9 @@ accumStats(StatsData *stats, bool skipped, double lat, double lag,
case ESTATUS_DEADLOCK_ERROR:
stats->deadlock_failures++;
break;
+ case ESTATUS_OTHER_SQL_ERROR:
+ stats->other_sql_failures++;
+ break;
default:
/* internal error which should never occur */
pg_fatal("unexpected error status: %d", estatus);
@@ -3231,11 +3268,43 @@ sendCommand(CState *st, Command *command)
}
/*
- * Get the error status from the error code.
+ * Read and discard all available results from the connection.
+ */
+static void
+discardAvailableResults(CState *st)
+{
+ PGresult *res = NULL;
+
+ for (;;)
+ {
+ res = PQgetResult(st->con);
+
+ /*
+ * Read and discard results until PQgetResult() returns NULL (no more
+ * results) or a connection failure is detected. If the pipeline
+ * status is PQ_PIPELINE_ABORTED, more results may still be available
+ * even after PQgetResult() returns NULL, so continue reading in that
+ * case.
+ */
+ if ((res == NULL && PQpipelineStatus(st->con) != PQ_PIPELINE_ABORTED) ||
+ PQstatus(st->con) == CONNECTION_BAD)
+ break;
+
+ PQclear(res);
+ }
+ PQclear(res);
+}
+
+/*
+ * Determine the error status based on the connection status and error code.
*/
static EStatus
-getSQLErrorStatus(const char *sqlState)
+getSQLErrorStatus(CState *st, const char *sqlState)
{
+ discardAvailableResults(st);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ return ESTATUS_CONN_ERROR;
+
if (sqlState != NULL)
{
if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
@@ -3257,6 +3326,17 @@ canRetryError(EStatus estatus)
estatus == ESTATUS_DEADLOCK_ERROR);
}
+/*
+ * Returns true if --continue-on-error is specified and this error allows
+ * processing to continue.
+ */
+static bool
+canContinueOnError(EStatus estatus)
+{
+ return (continue_on_error &&
+ estatus == ESTATUS_OTHER_SQL_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -3375,9 +3455,9 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
- st->estatus = getSQLErrorStatus(PQresultErrorField(res,
- PG_DIAG_SQLSTATE));
- if (canRetryError(st->estatus))
+ st->estatus = getSQLErrorStatus(st, PQresultErrorField(res,
+ PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus) || canContinueOnError(st->estatus))
{
if (verbose_errors)
commandError(st, PQresultErrorMessage(res));
@@ -3409,11 +3489,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
error:
PQclear(res);
PQclear(next_res);
- do
- {
- res = PQgetResult(st->con);
- PQclear(res);
- } while (res);
+ discardAvailableResults(st);
return false;
}
@@ -4041,7 +4117,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
- else if (canRetryError(st->estatus))
+ else if (canRetryError(st->estatus) || canContinueOnError(st->estatus))
st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
@@ -4562,7 +4638,8 @@ static int64
getFailures(const StatsData *stats)
{
return (stats->serialization_failures +
- stats->deadlock_failures);
+ stats->deadlock_failures +
+ stats->other_sql_failures);
}
/*
@@ -4582,6 +4659,8 @@ getResultString(bool skipped, EStatus estatus)
return "serialization";
case ESTATUS_DEADLOCK_ERROR:
return "deadlock";
+ case ESTATUS_OTHER_SQL_ERROR:
+ return "other";
default:
/* internal error which should never occur */
pg_fatal("unexpected error status: %d", estatus);
@@ -4637,6 +4716,7 @@ doLog(TState *thread, CState *st,
int64 skipped = 0;
int64 serialization_failures = 0;
int64 deadlock_failures = 0;
+ int64 other_sql_failures = 0;
int64 retried = 0;
int64 retries = 0;
@@ -4677,10 +4757,12 @@ doLog(TState *thread, CState *st,
{
serialization_failures = agg->serialization_failures;
deadlock_failures = agg->deadlock_failures;
+ other_sql_failures = agg->other_sql_failures;
}
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT " " INT64_FORMAT,
serialization_failures,
- deadlock_failures);
+ deadlock_failures,
+ other_sql_failures);
fputc('\n', logfile);
@@ -5120,9 +5202,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5138,25 +5220,384 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Dumps binary buffer to file (purely for debugging)
*/
static void
-initGenerateDataServerSide(PGconn *con)
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t total = base * scale;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = 0; i < total; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
+ */
+static void
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5170,27 +5611,166 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
+ "insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
executeStatement(con, sql.data);
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
}
/*
@@ -5276,6 +5856,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5287,7 +5869,22 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'g':
+ case 'C':
+ case 'G':
+ case 'i':
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5326,10 +5923,13 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
@@ -6319,6 +6919,7 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
cur.serialization_failures +=
threads[i].stats.serialization_failures;
cur.deadlock_failures += threads[i].stats.deadlock_failures;
+ cur.other_sql_failures += threads[i].stats.other_sql_failures;
}
/* we count only actually executed transactions */
@@ -6461,7 +7062,8 @@ printResults(StatsData *total,
/*
* Remaining stats are nonsensical if we failed to execute any xacts due
- * to others than serialization or deadlock errors
+ * to other than serialization or deadlock errors and --continue-on-error
+ * is not set.
*/
if (total_cnt <= 0)
return;
@@ -6477,6 +7079,9 @@ printResults(StatsData *total,
printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
total->deadlock_failures,
100.0 * total->deadlock_failures / total_cnt);
+ printf("number of other failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->other_sql_failures,
+ 100.0 * total->other_sql_failures / total_cnt);
}
/* it can be non-zero only if max_tries is not equal to one */
@@ -6580,6 +7185,10 @@ printResults(StatsData *total,
sstats->deadlock_failures,
(100.0 * sstats->deadlock_failures /
script_total_cnt));
+ printf(" - number of other failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->other_sql_failures,
+ (100.0 * sstats->other_sql_failures /
+ script_total_cnt));
}
/*
@@ -6739,6 +7348,7 @@ main(int argc, char **argv)
{"verbose-errors", no_argument, NULL, 15},
{"exit-on-abort", no_argument, NULL, 16},
{"debug", no_argument, NULL, 17},
+ {"continue-on-error", no_argument, NULL, 18},
{NULL, 0, NULL, 0}
};
@@ -7092,6 +7702,10 @@ main(int argc, char **argv)
case 17: /* debug */
pg_logging_increase_verbosity();
break;
+ case 18: /* continue-on-error */
+ benchmarking_option_set = true;
+ continue_on_error = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -7447,6 +8061,7 @@ main(int argc, char **argv)
stats.retried += thread->stats.retried;
stats.serialization_failures += thread->stats.serialization_failures;
stats.deadlock_failures += thread->stats.deadlock_failures;
+ stats.other_sql_failures += thread->stats.other_sql_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2025-11-13 10:17 ` Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Ashutosh Bapat @ 2025-11-13 10:17 UTC (permalink / raw)
To: Boris Mironov <[email protected]>; +Cc: pgsql-hackers
Hi Boris,
On Wed, Nov 12, 2025 at 3:25 AM Boris Mironov <[email protected]> wrote:
>
> Hello hackers,
>
> For some of my specific hardware tests I needed to generate big databases well beyond RAM size. Hence I turned to pgbench tool and its default 2 modes for client- and server-side generation for TPC-B tests. When I use "scale" factor in range of few thousands (eg, 3000 - 5000) data generation phase takes quite some time. I looked at it as opportunity to prove/disprove 2 hypothesises:
>
> will INSERT mode work faster if we commit once every "scale" and turn single INSERT into "for" loop with commits for 3 tables in the end of each loop
> will "INSERT .. SELECT FROM unnest" be faster than "INSERT .. SELECT FROM generate_series"
> will BINARY mode work faster than TEXT even though we send much more data
> and so on
>
> As a result of my experiments I produced significant patch for pgbench utility and though that it might be of interest not just for me. Therefore I'm sending draft version of it in diff format for current development tree on GitHub. As of November 11, 2025 I can merge with main branch of the project on GitHub.
>
> Spoiler alert: "COPY FROM BINARY" is significantly faster than current "COPY FROM TEXT"
>
> Would be happy to polish it if there is interest to such change.
Making pgbench data initialization faster at a higher scale is
desirable and the community might be willing to accept such a change.
Running very large benchmarks is becoming common these days. However,
it's not clear what you are proposing and what's the performance
improvement. Answering following question may help: Your patch
implements all the above methods? Do all of them provide performance
improvement? If each of them performs better under certain conditions,
what are those conditions? Is there one method which performs better
than all others, which is it and why not implement just that method?
What performance numbers are we looking at? Can the methods which use
batch commits, also run those batches in parallel?
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
@ 2025-11-14 15:21 ` Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2025-11-14 15:21 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; pgsql-hackers
Hi Ashutosh,
> If there is one method that is better than all others, community will
> be more willing to accept implementation of that one method than
> multiple implementations so as to reduce maintenance burden.
Ok then. I'll leave "COPY FROM STDIN BINARY" implementation out of 3 only.
Would you prefer to replace original COPY FROM STDIN TEXT by this
code or add it as new "init-step" (e.g., with code "c")?
I also have noted that current code doesn't prevent pgbench parameter
like "--init-steps=dtgG". It allows to run data generation step twice.
Each of these "g" and "G" will present own timing in status line. Is this
an oversight or intentional?
> The code in the patch does not have enough comments. It's hard to
> understand the methods just from the code. Each of the generateData*
> functions could use a prologue explaining the data generation method
> it uses.
To add comments is not a problem at all. So far, it was just "code for myself"
and I was checking if there is any interest in community to include it.
>> I'm sure that much more testing is required to run this code under different
>> conditions and hardware to get a better picture. So far it looks very promising.
> Sure.
Cheers,
Boris
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2025-11-17 04:58 ` Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Ashutosh Bapat @ 2025-11-17 04:58 UTC (permalink / raw)
To: Boris Mironov <[email protected]>; +Cc: pgsql-hackers
On Fri, Nov 14, 2025 at 8:51 PM Boris Mironov <[email protected]> wrote:
>
> Hi Ashutosh,
>
> > If there is one method that is better than all others, community will
> > be more willing to accept implementation of that one method than
> > multiple implementations so as to reduce maintenance burden.
>
> Ok then. I'll leave "COPY FROM STDIN BINARY" implementation out of 3 only.
> Would you prefer to replace original COPY FROM STDIN TEXT by this
> code or add it as new "init-step" (e.g., with code "c")?
>
TEXT copy may be useful for cross platform client side data
generation. BINARY might be useful for same platform client side
generation or server side generation. Just a thought, use TEXT or
BINARY automatically based on where it's cross-platform or same
platform setup.
> I also have noted that current code doesn't prevent pgbench parameter
> like "--init-steps=dtgG". It allows to run data generation step twice.
> Each of these "g" and "G" will present own timing in status line. Is this
> an oversight or intentional?
>
I would review the commit a386942bd29b0ef0c9df061392659880d22cdf43 and
the discussion thread
https://postgr.es/m/alpine.DEB.2.21.1904061826420.3678@lancre
mentioned in the commit message to find that out. At first glance it
looks like an oversight, but I haven't reviewed the commit and thread
myself. That thread might reveal why generate_series() was used
instead of BINARY COPY for server side data generation. If it needs to
change it's better to start a separate thread and separate patch for
that discussion.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
@ 2025-11-17 12:43 ` Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2025-11-17 12:43 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers
Hi Ashutosh,
> TEXT copy may be useful for cross platform client side data
> generation. BINARY might be useful for same platform client side
> generation or server side generation. Just a thought, use TEXT or
> BINARY automatically based on where it's cross-platform or same
> platform setup.
It is true that BINARY format is not as flexible as TEXT. Postgres expects
data in wire to arrive in "network byte order". AFAIK only Solaris can
send its data without byte reordering. I support such exception via
#ifdef __sparc__
I don't see an easy way to make decision within pgbench on which
COPY mode to use TEXT or BINARY except specifically asking for one
via command line parameter. This is why I left flag "g" for TEXT mode
and added BINARY as "C" (for "Copy" and upper case as faster). I guess,
we can add alias "c" for old client-side generation as "copy", but slower
version of it.
While we are on topic of client- vs server-side generation. IMHO these are
quite misleading terms. Both of them are executed by RDBMS on server side,
but "server" one gets quite short query (and quite slow in execution) and
"client" one gets quite big network transfer (and quite fast in execution). The
reason is difference in data path that needs to reflected in documentation.
On top of it server-side thrashes DB cache, while client-side works via ring buffer
that doesn't allocate more than 1/8 of shared_buffers,
> I would review the commit a386942bd29b0ef0c9df061392659880d22cdf43 and
> the discussion thread
> https://postgr.es/m/alpine.DEB.2.21.1904061826420.3678@lancre
> mentioned in the commit message to find that out. At first glance it
> looks like an oversight, but I haven't reviewed the commit and thread
> myself. That thread might reveal why generate_series() was used
> instead of BINARY COPY for server side data generation. If it needs to
> change it's better to start a separate thread and separate patch for
> that discussion.
Thank you for this hint. I went through whole thread and there they discuss
how to reflect certain behavior of init-steps and nothing about COPY BINARY.
Major point of generate_series() introduction is to send short query to DB
and not to worry about network performance. It is quite true that COPY
sends tons of data over network and it might be an issue for slow network.
They also touched on topic of "one huge transaction" for whole generated
dataset or few smaller transaction.
Allow me to repost my benchmarks here (as it was lost for pgsql-hasckers
because I just used Reply instead of Reply-To-All)
Tests:
Test | Binary | Init mode | Query and details
-----|----------|-----------|-------------------------------------------------------
1 | original | G | INSERT FROM generate_series in single huge transaction
2 | enhanced | G | INSERT FROM generate_series in single huge transaction
3 | enhanced | i | INSERT FROM generate_series in one transaction per scale
4 | enhanced | I | INSERT FROM unnest in one transaction per scale
5 | original | g | COPY FROM STDIN TEXT in single transaction
6 | enhanced | g | COPY FROM STDIN TEXT in single transaction
7 | enhanced | C | COPY FROM STDIN BINARY in single transaction
Test | Scale and seconds to complete data generation
| 1 | 2 | 10 | 20 | 100 | 200 | 1000 | 2000
-----|------|------|------|------|-------|-------|--------|--------
1 | 0.19 | 0.37 | 2.01 | 4.34 | 22.58 | 46.64 | 245.98 | 525.99
2 | 0.30 | 0.47 | 2.18 | 4.37 | 25.38 | 56.66 | 240.89 | 482.63
3 | 0.18 | 0.39 | 2.14 | 4.19 | 23.78 | 47.63 | 240.91 | 483.19
4 | 0.18 | 0.38 | 2.17 | 4.39 | 23.68 | 47.93 | 242.63 | 487.33
5 | 0.11 | 0.22 | 1.46 | 2.95 | 15.69 | 32.86 | 154.16 | 311.00
6 | 0.11 | 0.22 | 1.43 | 2.89 | 16.01 | 29.41 | 158.10 | 307.54
7 | 0.14 | 0.12 | 0.56 | 1.16 | 6.22 | 12.70 | 64.70 | 135.58
"Original" binary is pgbench v17.6.
"Enhanced" binary is pgbench 19-devel with proposed patch.
As we can see another point of discussion in mentioned earlier
thread on pgsql-hackers said that multi transactions for init-step
do NOT bring any benefit. My numbers show some increase in
performance by simply INSERT-ing data in loop with one COMMIT
per "scale" on lower scales. On higher scales benefit dissapears. My
guess here is quite active process WAL archiver.
COPY TEXT is 36% faster than INSERT with multiple transactions.
COPY BINARY is ~72% faster than INSERT with multiple transactions.
At this point I'm torn between keeping old modes and logic for
backward compatibility and introduction of new modes for INSERT & COPY
versus simply replacing old less efficient logic with new one.
Sorry for quite long response.
Best regards,
Boris
________________________________
From: Ashutosh Bapat <[email protected]>
Sent: November 16, 2025 11:58 PM
To: Boris Mironov <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
On Fri, Nov 14, 2025 at 8:51 PM Boris Mironov <[email protected]> wrote:
>
> Hi Ashutosh,
>
> > If there is one method that is better than all others, community will
> > be more willing to accept implementation of that one method than
> > multiple implementations so as to reduce maintenance burden.
>
> Ok then. I'll leave "COPY FROM STDIN BINARY" implementation out of 3 only.
> Would you prefer to replace original COPY FROM STDIN TEXT by this
> code or add it as new "init-step" (e.g., with code "c")?
>
TEXT copy may be useful for cross platform client side data
generation. BINARY might be useful for same platform client side
generation or server side generation. Just a thought, use TEXT or
BINARY automatically based on where it's cross-platform or same
platform setup.
> I also have noted that current code doesn't prevent pgbench parameter
> like "--init-steps=dtgG". It allows to run data generation step twice.
> Each of these "g" and "G" will present own timing in status line. Is this
> an oversight or intentional?
>
I would review the commit a386942bd29b0ef0c9df061392659880d22cdf43 and
the discussion thread
https://postgr.es/m/alpine.DEB.2.21.1904061826420.3678@lancre
mentioned in the commit message to find that out. At first glance it
looks like an oversight, but I haven't reviewed the commit and thread
myself. That thread might reveal why generate_series() was used
instead of BINARY COPY for server side data generation. If it needs to
change it's better to start a separate thread and separate patch for
that discussion.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2025-11-21 13:26 ` Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2025-11-21 13:26 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers
Hi Ashutosh,
Just wanted to let you know that I've submitted this patch
to CommitFest (see https://commitfest.postgresql.org/patch/6245/)
Interestingly enough there is one more patch from Mircea Cadariu in the same
CommitFest about pgbench (https://commitfest.postgresql.org/patch/6242/)
That patch has been submitted few days ago and is proposing to run
data generation phase in parallel threads. It shows significant
improvements over performance of original single-thread code.
Hopefully sooner or later pgbench will get significant performance
gains in data generation from these two patches.
Original version of my patch failed in GitHub tests. Therefore I have
to start posting updated versions here.
Attached is updated version that sets default value for filler columns.
This trick allows significantly shrink network traffic for COPY FROM BINARY.
Absence of filler column in dataflow has failed my original patch in GitHub
pipeline.
I also switched from one huge transaction for COPY FROM BINARY to
"one per scale". This will simplify merge with multi-threaded data load
proposed by Mircea. Unfortunately, it killed possibility to freeze data right
away, which was possible when table truncation and data load was done
in the same transaction.
I think it would be fair to leave all original modes of data generation
until official review in CommitFest. Hence "INSERT SELECT FROM UNNEST"
is staying so far as there might be interest in community for benchmarking
of columnar tables (eg, for OLAP loads or Timescale DB).
Best regards,
Boris
Attachments:
[application/octet-stream] pgbench.c.diff (32.3K, 3-pgbench.c.diff)
download | inline diff:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1515ed405ba..6b89007a63b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,14 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +189,18 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * mode of data generation to use
+ */
+static char data_generation_type = '?';
+
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -402,14 +422,15 @@ typedef struct StatsData
* directly successful transactions (they were successfully completed on
* the first try).
*
- * A failed transaction is defined as unsuccessfully retried transactions.
- * It can be one of two types:
- *
- * failed (the number of failed transactions) =
+ * 'failed' (the number of failed transactions) =
* 'serialization_failures' (they got a serialization error and were not
- * successfully retried) +
+ * successfully retried) +
* 'deadlock_failures' (they got a deadlock error and were not
- * successfully retried).
+ * successfully retried) +
+ * 'other_sql_failures' (they failed on the first try or after retries
+ * due to a SQL error other than serialization or
+ * deadlock; they are counted as a failed transaction
+ * only when --continue-on-error is specified).
*
* If the transaction was retried after a serialization or a deadlock
* error this does not guarantee that this retry was successful. Thus
@@ -421,7 +442,7 @@ typedef struct StatsData
*
* 'retried' (number of all retried transactions) =
* successfully retried transactions +
- * failed transactions.
+ * unsuccessful retried transactions.
*----------
*/
int64 cnt; /* number of successful transactions, not
@@ -440,6 +461,11 @@ typedef struct StatsData
int64 deadlock_failures; /* number of transactions that were not
* successfully retried after a deadlock
* error */
+ int64 other_sql_failures; /* number of failed transactions for
+ * reasons other than
+ * serialization/deadlock failure, which
+ * is counted if --continue-on-error is
+ * specified */
SimpleStats latency;
SimpleStats lag;
} StatsData;
@@ -457,6 +483,7 @@ typedef enum EStatus
{
ESTATUS_NO_ERROR = 0,
ESTATUS_META_COMMAND_ERROR,
+ ESTATUS_CONN_ERROR,
/* SQL errors */
ESTATUS_SERIALIZATION_ERROR,
@@ -770,6 +797,7 @@ static int64 total_weight = 0;
static bool verbose_errors = false; /* print verbose messages of all errors */
static bool exit_on_abort = false; /* exit when any client is aborted */
+static bool continue_on_error = false; /* continue after errors */
/* Builtin test scripts */
typedef struct BuiltinScript
@@ -842,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -906,7 +935,10 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -949,6 +981,7 @@ usage(void)
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
+ " --continue-on-error continue running after an SQL error\n"
" --exit-on-abort exit when any client is aborted\n"
" --failures-detailed report the failures grouped by basic types\n"
" --log-prefix=PREFIX prefix for transaction time log file\n"
@@ -1467,6 +1500,7 @@ initStats(StatsData *sd, pg_time_usec_t start)
sd->retried = 0;
sd->serialization_failures = 0;
sd->deadlock_failures = 0;
+ sd->other_sql_failures = 0;
initSimpleStats(&sd->latency);
initSimpleStats(&sd->lag);
}
@@ -1516,6 +1550,9 @@ accumStats(StatsData *stats, bool skipped, double lat, double lag,
case ESTATUS_DEADLOCK_ERROR:
stats->deadlock_failures++;
break;
+ case ESTATUS_OTHER_SQL_ERROR:
+ stats->other_sql_failures++;
+ break;
default:
/* internal error which should never occur */
pg_fatal("unexpected error status: %d", estatus);
@@ -3231,11 +3268,43 @@ sendCommand(CState *st, Command *command)
}
/*
- * Get the error status from the error code.
+ * Read and discard all available results from the connection.
+ */
+static void
+discardAvailableResults(CState *st)
+{
+ PGresult *res = NULL;
+
+ for (;;)
+ {
+ res = PQgetResult(st->con);
+
+ /*
+ * Read and discard results until PQgetResult() returns NULL (no more
+ * results) or a connection failure is detected. If the pipeline
+ * status is PQ_PIPELINE_ABORTED, more results may still be available
+ * even after PQgetResult() returns NULL, so continue reading in that
+ * case.
+ */
+ if ((res == NULL && PQpipelineStatus(st->con) != PQ_PIPELINE_ABORTED) ||
+ PQstatus(st->con) == CONNECTION_BAD)
+ break;
+
+ PQclear(res);
+ }
+ PQclear(res);
+}
+
+/*
+ * Determine the error status based on the connection status and error code.
*/
static EStatus
-getSQLErrorStatus(const char *sqlState)
+getSQLErrorStatus(CState *st, const char *sqlState)
{
+ discardAvailableResults(st);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ return ESTATUS_CONN_ERROR;
+
if (sqlState != NULL)
{
if (strcmp(sqlState, ERRCODE_T_R_SERIALIZATION_FAILURE) == 0)
@@ -3257,6 +3326,17 @@ canRetryError(EStatus estatus)
estatus == ESTATUS_DEADLOCK_ERROR);
}
+/*
+ * Returns true if --continue-on-error is specified and this error allows
+ * processing to continue.
+ */
+static bool
+canContinueOnError(EStatus estatus)
+{
+ return (continue_on_error &&
+ estatus == ESTATUS_OTHER_SQL_ERROR);
+}
+
/*
* Process query response from the backend.
*
@@ -3375,9 +3455,9 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
case PGRES_NONFATAL_ERROR:
case PGRES_FATAL_ERROR:
- st->estatus = getSQLErrorStatus(PQresultErrorField(res,
- PG_DIAG_SQLSTATE));
- if (canRetryError(st->estatus))
+ st->estatus = getSQLErrorStatus(st, PQresultErrorField(res,
+ PG_DIAG_SQLSTATE));
+ if (canRetryError(st->estatus) || canContinueOnError(st->estatus))
{
if (verbose_errors)
commandError(st, PQresultErrorMessage(res));
@@ -3409,11 +3489,7 @@ readCommandResponse(CState *st, MetaCommand meta, char *varprefix)
error:
PQclear(res);
PQclear(next_res);
- do
- {
- res = PQgetResult(st->con);
- PQclear(res);
- } while (res);
+ discardAvailableResults(st);
return false;
}
@@ -3511,14 +3587,18 @@ doRetry(CState *st, pg_time_usec_t *now)
}
/*
- * Read results and discard it until a sync point.
+ * Read and discard results until the last sync point.
*/
static int
discardUntilSync(CState *st)
{
bool received_sync = false;
- /* send a sync */
+ /*
+ * Send a Sync message to ensure at least one PGRES_PIPELINE_SYNC is
+ * received and to avoid an infinite loop, since all earlier ones may have
+ * already been received.
+ */
if (!PQpipelineSync(st->con))
{
pg_log_error("client %d aborted: failed to send a pipeline sync",
@@ -3526,29 +3606,42 @@ discardUntilSync(CState *st)
return 0;
}
- /* receive PGRES_PIPELINE_SYNC and null following it */
+ /*
+ * Continue reading results until the last sync point, i.e., until
+ * reaching null just after PGRES_PIPELINE_SYNC.
+ */
for (;;)
{
PGresult *res = PQgetResult(st->con);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ PQclear(res);
+ return 0;
+ }
+
if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
received_sync = true;
- else if (received_sync)
+ else if (received_sync && res == NULL)
{
- /*
- * PGRES_PIPELINE_SYNC must be followed by another
- * PGRES_PIPELINE_SYNC or NULL; otherwise, assert failure.
- */
- Assert(res == NULL);
-
/*
* Reset ongoing sync count to 0 since all PGRES_PIPELINE_SYNC
* results have been discarded.
*/
st->num_syncs = 0;
- PQclear(res);
break;
}
+ else
+ {
+ /*
+ * If a PGRES_PIPELINE_SYNC is followed by something other than
+ * PGRES_PIPELINE_SYNC or NULL, another PGRES_PIPELINE_SYNC will
+ * appear later. Reset received_sync to false to wait for it.
+ */
+ received_sync = false;
+ }
PQclear(res);
}
@@ -4041,7 +4134,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQpipelineStatus(st->con) != PQ_PIPELINE_ON)
st->state = CSTATE_END_COMMAND;
}
- else if (canRetryError(st->estatus))
+ else if (canRetryError(st->estatus) || canContinueOnError(st->estatus))
st->state = CSTATE_ERROR;
else
st->state = CSTATE_ABORTED;
@@ -4562,7 +4655,8 @@ static int64
getFailures(const StatsData *stats)
{
return (stats->serialization_failures +
- stats->deadlock_failures);
+ stats->deadlock_failures +
+ stats->other_sql_failures);
}
/*
@@ -4582,6 +4676,8 @@ getResultString(bool skipped, EStatus estatus)
return "serialization";
case ESTATUS_DEADLOCK_ERROR:
return "deadlock";
+ case ESTATUS_OTHER_SQL_ERROR:
+ return "other";
default:
/* internal error which should never occur */
pg_fatal("unexpected error status: %d", estatus);
@@ -4637,6 +4733,7 @@ doLog(TState *thread, CState *st,
int64 skipped = 0;
int64 serialization_failures = 0;
int64 deadlock_failures = 0;
+ int64 other_sql_failures = 0;
int64 retried = 0;
int64 retries = 0;
@@ -4677,10 +4774,12 @@ doLog(TState *thread, CState *st,
{
serialization_failures = agg->serialization_failures;
deadlock_failures = agg->deadlock_failures;
+ other_sql_failures = agg->other_sql_failures;
}
- fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT,
+ fprintf(logfile, " " INT64_FORMAT " " INT64_FORMAT " " INT64_FORMAT,
serialization_failures,
- deadlock_failures);
+ deadlock_failures,
+ other_sql_failures);
fputc('\n', logfile);
@@ -4886,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84)",
- "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88)",
- "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88) default ''",
1
}
};
@@ -5120,9 +5219,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5138,25 +5237,389 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Dumps binary buffer to file (purely for debugging)
*/
static void
-initGenerateDataServerSide(PGconn *con)
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = start; i < start + base; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+ }
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
+ */
+static void
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5170,27 +5633,166 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
+ "insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
executeStatement(con, sql.data);
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
}
/*
@@ -5276,6 +5878,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5287,7 +5891,22 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'g':
+ case 'C':
+ case 'G':
+ case 'i':
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5326,10 +5945,13 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
@@ -6319,6 +6941,7 @@ printProgressReport(TState *threads, int64 test_start, pg_time_usec_t now,
cur.serialization_failures +=
threads[i].stats.serialization_failures;
cur.deadlock_failures += threads[i].stats.deadlock_failures;
+ cur.other_sql_failures += threads[i].stats.other_sql_failures;
}
/* we count only actually executed transactions */
@@ -6461,7 +7084,8 @@ printResults(StatsData *total,
/*
* Remaining stats are nonsensical if we failed to execute any xacts due
- * to others than serialization or deadlock errors
+ * to other than serialization or deadlock errors and --continue-on-error
+ * is not set.
*/
if (total_cnt <= 0)
return;
@@ -6477,6 +7101,9 @@ printResults(StatsData *total,
printf("number of deadlock failures: " INT64_FORMAT " (%.3f%%)\n",
total->deadlock_failures,
100.0 * total->deadlock_failures / total_cnt);
+ printf("number of other failures: " INT64_FORMAT " (%.3f%%)\n",
+ total->other_sql_failures,
+ 100.0 * total->other_sql_failures / total_cnt);
}
/* it can be non-zero only if max_tries is not equal to one */
@@ -6580,6 +7207,10 @@ printResults(StatsData *total,
sstats->deadlock_failures,
(100.0 * sstats->deadlock_failures /
script_total_cnt));
+ printf(" - number of other failures: " INT64_FORMAT " (%.3f%%)\n",
+ sstats->other_sql_failures,
+ (100.0 * sstats->other_sql_failures /
+ script_total_cnt));
}
/*
@@ -6739,6 +7370,7 @@ main(int argc, char **argv)
{"verbose-errors", no_argument, NULL, 15},
{"exit-on-abort", no_argument, NULL, 16},
{"debug", no_argument, NULL, 17},
+ {"continue-on-error", no_argument, NULL, 18},
{NULL, 0, NULL, 0}
};
@@ -7092,6 +7724,10 @@ main(int argc, char **argv)
case 17: /* debug */
pg_logging_increase_verbosity();
break;
+ case 18: /* continue-on-error */
+ benchmarking_option_set = true;
+ continue_on_error = true;
+ break;
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -7447,6 +8083,7 @@ main(int argc, char **argv)
stats.retried += thread->stats.retried;
stats.serialization_failures += thread->stats.serialization_failures;
stats.deadlock_failures += thread->stats.deadlock_failures;
+ stats.other_sql_failures += thread->stats.other_sql_failures;
latency_late += thread->latency_late;
conn_total_duration += thread->conn_duration;
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2025-11-22 11:02 ` Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2025-11-22 11:02 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers
Hello,
Adding tests for new modes into Perl testing framework for pgbench.
The goal is to pass GitHub checks for the patch in green to simplify reviewer's life.
Cheers,
Boris
Attachments:
[application/octet-stream] v3-pgbench-faster-modes.patch (44.9K, 3-v3-pgbench-faster-modes.patch)
download | inline diff:
From c768f399c556295de7d53895410e686d86b4b960 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 19:34:58 +0700
Subject: [PATCH 01/10] Converting one huge transaction into series of one per
'scale'
---
src/bin/pgbench/pgbench.c | 61 ++++++++++++++++++++++++++-------------
1 file changed, 41 insertions(+), 20 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..284a7c860f1 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,6 +181,12 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * scaling factor after which we switch to multiple transactions during
+ * data population phase on server side
+ */
+static int64 single_txn_scale_limit = 1;
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5213,6 +5219,7 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
+ int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5225,30 +5232,44 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ executeStatement(con, "commit");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ for (int i = 0; i < scale; i += chunk) {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "select bid + 1, 0 "
+ "from generate_series(%d, %d) as bid", i, i + chunk);
+ //"select bid, 0 "
+ //"from generate_series(1, %d) as bid", nbranches * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid,bid,tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ //"select tid, (tid - 1) / %d + 1, 0 "
+ //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ //"select aid, (aid - 1) / %d + 1, 0, '' "
+ //"from generate_series(1, " INT64_FORMAT ") as aid",
+ //naccounts, (int64) naccounts * scale);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
termPQExpBuffer(&sql);
-
- executeStatement(con, "commit");
}
/*
--
2.43.0
From 0eddb156c187d829c4381bc928c5314705928852 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:13:23 +0700
Subject: [PATCH 02/10] Getting rid off limit for single transaction size
during data generation
---
src/bin/pgbench/pgbench.c | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 284a7c860f1..28b72e4cf1f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,12 +181,6 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
-/*
- * scaling factor after which we switch to multiple transactions during
- * data population phase on server side
- */
-static int64 single_txn_scale_limit = 1;
-
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5219,7 +5213,6 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
- int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5236,13 +5229,13 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i += chunk) {
+ for (int i = 0; i < scale; i++) {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + chunk);
+ "from generate_series(%d, %d) as bid", i, i + 1);
//"select bid, 0 "
//"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
@@ -5251,7 +5244,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_tellers(tid,bid,tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
- ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
//"select tid, (tid - 1) / %d + 1, 0 "
//"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
@@ -5260,7 +5253,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_accounts(aid,bid,abalance,filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
//"select aid, (aid - 1) / %d + 1, 0, '' "
//"from generate_series(1, " INT64_FORMAT ") as aid",
//naccounts, (int64) naccounts * scale);
--
2.43.0
From c5659cf474ec273c057668f30a4f435fd02f2da7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:38:36 +0700
Subject: [PATCH 03/10] No need to keep old code in comments
---
src/bin/pgbench/pgbench.c | 7 -------
1 file changed, 7 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 28b72e4cf1f..97895aa9edf 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5236,8 +5236,6 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
"from generate_series(%d, %d) as bid", i, i + 1);
- //"select bid, 0 "
- //"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5245,8 +5243,6 @@ initGenerateDataServerSide(PGconn *con)
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
- //"select tid, (tid - 1) / %d + 1, 0 "
- //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5254,9 +5250,6 @@ initGenerateDataServerSide(PGconn *con)
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
- //"select aid, (aid - 1) / %d + 1, 0, '' "
- //"from generate_series(1, " INT64_FORMAT ") as aid",
- //naccounts, (int64) naccounts * scale);
executeStatement(con, sql.data);
executeStatement(con, "commit");
--
2.43.0
From e47b52ddf23593dad9375ef5356fd41d0621ede3 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 19:06:48 +0700
Subject: [PATCH 04/10] Adding server-side data generation via unnest
---
src/bin/pgbench/pgbench.c | 199 ++++++++++++++++++++++++++++++++++----
1 file changed, 182 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97895aa9edf..65d77cdefea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,12 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +187,11 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ *
+ */
+static char data_generation_type = '?';
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -914,7 +925,9 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5203,18 +5216,16 @@ initGenerateDataClientSide(PGconn *con)
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
*/
static void
-initGenerateDataServerSide(PGconn *con)
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5225,31 +5236,136 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ initPQExpBuffer(&sql);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "select bid, 0 "
+ "from generate_series(1, %d)", scale * nbranches);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
+ executeStatement(con, sql.data);
+
executeStatement(con, "commit");
+ termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i++) {
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + 1);
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
+ "insert into pgbench_tellers(tid, bid, tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
executeStatement(con, "commit");
@@ -5258,6 +5374,32 @@ initGenerateDataServerSide(PGconn *con)
termPQExpBuffer(&sql);
}
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
+}
+
/*
* Invoke vacuum on the standard tables
*/
@@ -5341,6 +5483,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5352,7 +5496,26 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'G':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'i':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5395,6 +5558,8 @@ runInitSteps(const char *initialize_steps)
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
--
2.43.0
From 5e1827b889b283f50299ce6ab1a73f9f55a4a84f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 20:00:56 +0700
Subject: [PATCH 05/10] Fixing typo in query
---
src/bin/pgbench/pgbench.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 65d77cdefea..03e37df4434 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5241,7 +5241,8 @@ generateDataInsertSingleTXN(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d)", scale * nbranches);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
--
2.43.0
From 7ca86521fda6929b8e0de3fc77dcbb8984009c88 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 11 Nov 2025 19:39:45 +0700
Subject: [PATCH 06/10] Adding support for COPY BINARY mode
---
src/bin/pgbench/pgbench.c | 393 ++++++++++++++++++++++++++++++++++++--
1 file changed, 381 insertions(+), 12 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03e37df4434..71aa1d9479f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -176,6 +176,8 @@ typedef struct socket_set
/* 'one transaction per scale' server-side methods */
#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
@@ -188,10 +190,17 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
static int scale = 1;
/*
- *
+ * mode of data generation to use
*/
static char data_generation_type = '?';
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -861,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -925,6 +935,7 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
" G: generate data, server-side in single transaction\n"
" i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
" I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
@@ -5191,9 +5202,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5209,12 +5220,373 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
+/*
+ * Dumps binary buffer to file (purely for debugging)
+ */
+static void
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t total = base * scale;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = 0; i < total; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
* whole dataset in single transaction
@@ -5500,14 +5872,10 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
+ case 'g':
+ case 'C':
case 'G':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'i':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'I':
data_init_type++;
data_generation_type = *step;
@@ -5555,6 +5923,7 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
--
2.43.0
From 4aa0ac05765edf6b5f0c13e18ac677287ce78206 Mon Sep 17 00:00:00 2001
From: Fujii Masao <[email protected]>
Date: Fri, 14 Nov 2025 22:40:39 +0900
Subject: [PATCH 07/10] pgbench: Fix assertion failure with multiple
\syncpipeline in pipeline mode.
Previously, when pgbench ran a custom script that triggered retriable errors
(e.g., deadlocks) followed by multiple \syncpipeline commands in pipeline mode,
the following assertion failure could occur:
Assertion failed: (res == ((void*)0)), function discardUntilSync, file pgbench.c, line 3594.
The issue was that discardUntilSync() assumed a pipeline sync result
(PGRES_PIPELINE_SYNC) would always be followed by either another sync result
or NULL. This assumption was incorrect: when multiple sync requests were sent,
a sync result could instead be followed by another result type. In such cases,
discardUntilSync() mishandled the results, leading to the assertion failure.
This commit fixes the issue by making discardUntilSync() correctly handle cases
where a pipeline sync result is followed by other result types. It now continues
discarding results until another pipeline sync followed by NULL is reached.
Backpatched to v17, where support for \syncpipeline command in pgbench was
introduced.
Author: Yugo Nagata <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 17
---
src/bin/pgbench/pgbench.c | 39 ++++++++++++++++++++++++++++-----------
1 file changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..a425176ecdc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -3563,14 +3563,18 @@ doRetry(CState *st, pg_time_usec_t *now)
}
/*
- * Read results and discard it until a sync point.
+ * Read and discard results until the last sync point.
*/
static int
discardUntilSync(CState *st)
{
bool received_sync = false;
- /* send a sync */
+ /*
+ * Send a Sync message to ensure at least one PGRES_PIPELINE_SYNC is
+ * received and to avoid an infinite loop, since all earlier ones may have
+ * already been received.
+ */
if (!PQpipelineSync(st->con))
{
pg_log_error("client %d aborted: failed to send a pipeline sync",
@@ -3578,29 +3582,42 @@ discardUntilSync(CState *st)
return 0;
}
- /* receive PGRES_PIPELINE_SYNC and null following it */
+ /*
+ * Continue reading results until the last sync point, i.e., until
+ * reaching null just after PGRES_PIPELINE_SYNC.
+ */
for (;;)
{
PGresult *res = PQgetResult(st->con);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ PQclear(res);
+ return 0;
+ }
+
if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
received_sync = true;
- else if (received_sync)
+ else if (received_sync && res == NULL)
{
- /*
- * PGRES_PIPELINE_SYNC must be followed by another
- * PGRES_PIPELINE_SYNC or NULL; otherwise, assert failure.
- */
- Assert(res == NULL);
-
/*
* Reset ongoing sync count to 0 since all PGRES_PIPELINE_SYNC
* results have been discarded.
*/
st->num_syncs = 0;
- PQclear(res);
break;
}
+ else
+ {
+ /*
+ * If a PGRES_PIPELINE_SYNC is followed by something other than
+ * PGRES_PIPELINE_SYNC or NULL, another PGRES_PIPELINE_SYNC will
+ * appear later. Reset received_sync to false to wait for it.
+ */
+ received_sync = false;
+ }
PQclear(res);
}
--
2.43.0
From 9c4f19055597e9adb25e65c2aa8bedf20a09e13d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 19:05:58 +0700
Subject: [PATCH 08/10] Setting empty string as default value in filler column
---
src/bin/pgbench/pgbench.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 967f6ce6984..03b5e5c28f0 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84)",
- "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88)",
- "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88) default ''",
1
}
};
--
2.43.0
From dcb85d26f8132eaaf9d096e814b9bda49db7d478 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 20:06:24 +0700
Subject: [PATCH 09/10] Switching COPY FROM BINARY ti run in multiple
transactions
---
src/bin/pgbench/pgbench.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03b5e5c28f0..6b89007a63b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5496,20 +5496,20 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
static void
initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int64_t base, initRowMethodBin init_row)
+ int counter, int64_t base, initRowMethodBin init_row)
{
int n;
PGresult *res;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t total = base * scale;
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5526,7 +5526,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyHeader(con);
- for (int64_t i = 0; i < total; i++)
+ for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
}
@@ -5573,15 +5573,20 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- naccounts, initAccountBinary);
-
executeStatement(con, "commit");
+ for (int i = 0; i < scale; i++)
+ {
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+ }
+
pg_free(bin_copy_buffer);
}
--
2.43.0
From b8e28881225234fd00b55235bc60fad2dc60b544 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sat, 22 Nov 2025 17:06:00 +0700
Subject: [PATCH 10/10] Adding tests for new modes of data generation
---
src/bin/pgbench/pgbench.c | 21 ++++----
src/bin/pgbench/t/001_pgbench_with_server.pl | 52 +++++++++++++++++---
2 files changed, 56 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6b89007a63b..dd4e5d5e056 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -164,7 +164,7 @@ typedef struct socket_set
#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
@@ -192,7 +192,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = '?';
+static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
/*
* COPY FROM BINARY execution buffer
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84)",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default ''",
- "aid bigint not null,bid int,abalance int,filler char(84) default ''",
+ "aid int not null,bid int,abalance int,filler char(84) default '?'",
+ "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88) default ''",
- "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88)",
1
}
};
@@ -7837,6 +7837,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 581e9af7907..a377048ead1 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -125,7 +130,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +148,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -164,6 +170,38 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtI',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtC',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2025-11-23 07:51 ` Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2025-11-23 07:51 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: pgsql-hackers
Hello,
Updating code to satisfy compiler warnings about unused code.
Cheers,
Boris
Attachments:
[application/octet-stream] v4-pgbench-faster-modes.patch (51.6K, 3-v4-pgbench-faster-modes.patch)
download | inline diff:
From c768f399c556295de7d53895410e686d86b4b960 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 19:34:58 +0700
Subject: [PATCH 01/12] Converting one huge transaction into series of one per
'scale'
---
src/bin/pgbench/pgbench.c | 61 ++++++++++++++++++++++++++-------------
1 file changed, 41 insertions(+), 20 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..284a7c860f1 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,6 +181,12 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * scaling factor after which we switch to multiple transactions during
+ * data population phase on server side
+ */
+static int64 single_txn_scale_limit = 1;
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5213,6 +5219,7 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
+ int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5225,30 +5232,44 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ executeStatement(con, "commit");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ for (int i = 0; i < scale; i += chunk) {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "select bid + 1, 0 "
+ "from generate_series(%d, %d) as bid", i, i + chunk);
+ //"select bid, 0 "
+ //"from generate_series(1, %d) as bid", nbranches * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid,bid,tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ //"select tid, (tid - 1) / %d + 1, 0 "
+ //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ //"select aid, (aid - 1) / %d + 1, 0, '' "
+ //"from generate_series(1, " INT64_FORMAT ") as aid",
+ //naccounts, (int64) naccounts * scale);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
termPQExpBuffer(&sql);
-
- executeStatement(con, "commit");
}
/*
--
2.43.0
From 0eddb156c187d829c4381bc928c5314705928852 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:13:23 +0700
Subject: [PATCH 02/12] Getting rid off limit for single transaction size
during data generation
---
src/bin/pgbench/pgbench.c | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 284a7c860f1..28b72e4cf1f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,12 +181,6 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
-/*
- * scaling factor after which we switch to multiple transactions during
- * data population phase on server side
- */
-static int64 single_txn_scale_limit = 1;
-
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5219,7 +5213,6 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
- int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5236,13 +5229,13 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i += chunk) {
+ for (int i = 0; i < scale; i++) {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + chunk);
+ "from generate_series(%d, %d) as bid", i, i + 1);
//"select bid, 0 "
//"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
@@ -5251,7 +5244,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_tellers(tid,bid,tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
- ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
//"select tid, (tid - 1) / %d + 1, 0 "
//"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
@@ -5260,7 +5253,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_accounts(aid,bid,abalance,filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
//"select aid, (aid - 1) / %d + 1, 0, '' "
//"from generate_series(1, " INT64_FORMAT ") as aid",
//naccounts, (int64) naccounts * scale);
--
2.43.0
From c5659cf474ec273c057668f30a4f435fd02f2da7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:38:36 +0700
Subject: [PATCH 03/12] No need to keep old code in comments
---
src/bin/pgbench/pgbench.c | 7 -------
1 file changed, 7 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 28b72e4cf1f..97895aa9edf 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5236,8 +5236,6 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
"from generate_series(%d, %d) as bid", i, i + 1);
- //"select bid, 0 "
- //"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5245,8 +5243,6 @@ initGenerateDataServerSide(PGconn *con)
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
- //"select tid, (tid - 1) / %d + 1, 0 "
- //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5254,9 +5250,6 @@ initGenerateDataServerSide(PGconn *con)
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
- //"select aid, (aid - 1) / %d + 1, 0, '' "
- //"from generate_series(1, " INT64_FORMAT ") as aid",
- //naccounts, (int64) naccounts * scale);
executeStatement(con, sql.data);
executeStatement(con, "commit");
--
2.43.0
From e47b52ddf23593dad9375ef5356fd41d0621ede3 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 19:06:48 +0700
Subject: [PATCH 04/12] Adding server-side data generation via unnest
---
src/bin/pgbench/pgbench.c | 199 ++++++++++++++++++++++++++++++++++----
1 file changed, 182 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97895aa9edf..65d77cdefea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,12 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +187,11 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ *
+ */
+static char data_generation_type = '?';
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -914,7 +925,9 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5203,18 +5216,16 @@ initGenerateDataClientSide(PGconn *con)
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
*/
static void
-initGenerateDataServerSide(PGconn *con)
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5225,31 +5236,136 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ initPQExpBuffer(&sql);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "select bid, 0 "
+ "from generate_series(1, %d)", scale * nbranches);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
+ executeStatement(con, sql.data);
+
executeStatement(con, "commit");
+ termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i++) {
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + 1);
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
+ "insert into pgbench_tellers(tid, bid, tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
executeStatement(con, "commit");
@@ -5258,6 +5374,32 @@ initGenerateDataServerSide(PGconn *con)
termPQExpBuffer(&sql);
}
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
+}
+
/*
* Invoke vacuum on the standard tables
*/
@@ -5341,6 +5483,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5352,7 +5496,26 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'G':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'i':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5395,6 +5558,8 @@ runInitSteps(const char *initialize_steps)
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
--
2.43.0
From 5e1827b889b283f50299ce6ab1a73f9f55a4a84f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 20:00:56 +0700
Subject: [PATCH 05/12] Fixing typo in query
---
src/bin/pgbench/pgbench.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 65d77cdefea..03e37df4434 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5241,7 +5241,8 @@ generateDataInsertSingleTXN(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d)", scale * nbranches);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
--
2.43.0
From 7ca86521fda6929b8e0de3fc77dcbb8984009c88 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 11 Nov 2025 19:39:45 +0700
Subject: [PATCH 06/12] Adding support for COPY BINARY mode
---
src/bin/pgbench/pgbench.c | 393 ++++++++++++++++++++++++++++++++++++--
1 file changed, 381 insertions(+), 12 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03e37df4434..71aa1d9479f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -176,6 +176,8 @@ typedef struct socket_set
/* 'one transaction per scale' server-side methods */
#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
@@ -188,10 +190,17 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
static int scale = 1;
/*
- *
+ * mode of data generation to use
*/
static char data_generation_type = '?';
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -861,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -925,6 +935,7 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
" G: generate data, server-side in single transaction\n"
" i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
" I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
@@ -5191,9 +5202,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5209,12 +5220,373 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
+/*
+ * Dumps binary buffer to file (purely for debugging)
+ */
+static void
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t total = base * scale;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = 0; i < total; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
* whole dataset in single transaction
@@ -5500,14 +5872,10 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
+ case 'g':
+ case 'C':
case 'G':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'i':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'I':
data_init_type++;
data_generation_type = *step;
@@ -5555,6 +5923,7 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
--
2.43.0
From 4aa0ac05765edf6b5f0c13e18ac677287ce78206 Mon Sep 17 00:00:00 2001
From: Fujii Masao <[email protected]>
Date: Fri, 14 Nov 2025 22:40:39 +0900
Subject: [PATCH 07/12] pgbench: Fix assertion failure with multiple
\syncpipeline in pipeline mode.
Previously, when pgbench ran a custom script that triggered retriable errors
(e.g., deadlocks) followed by multiple \syncpipeline commands in pipeline mode,
the following assertion failure could occur:
Assertion failed: (res == ((void*)0)), function discardUntilSync, file pgbench.c, line 3594.
The issue was that discardUntilSync() assumed a pipeline sync result
(PGRES_PIPELINE_SYNC) would always be followed by either another sync result
or NULL. This assumption was incorrect: when multiple sync requests were sent,
a sync result could instead be followed by another result type. In such cases,
discardUntilSync() mishandled the results, leading to the assertion failure.
This commit fixes the issue by making discardUntilSync() correctly handle cases
where a pipeline sync result is followed by other result types. It now continues
discarding results until another pipeline sync followed by NULL is reached.
Backpatched to v17, where support for \syncpipeline command in pgbench was
introduced.
Author: Yugo Nagata <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 17
---
src/bin/pgbench/pgbench.c | 39 ++++++++++++++++++++++++++++-----------
1 file changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..a425176ecdc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -3563,14 +3563,18 @@ doRetry(CState *st, pg_time_usec_t *now)
}
/*
- * Read results and discard it until a sync point.
+ * Read and discard results until the last sync point.
*/
static int
discardUntilSync(CState *st)
{
bool received_sync = false;
- /* send a sync */
+ /*
+ * Send a Sync message to ensure at least one PGRES_PIPELINE_SYNC is
+ * received and to avoid an infinite loop, since all earlier ones may have
+ * already been received.
+ */
if (!PQpipelineSync(st->con))
{
pg_log_error("client %d aborted: failed to send a pipeline sync",
@@ -3578,29 +3582,42 @@ discardUntilSync(CState *st)
return 0;
}
- /* receive PGRES_PIPELINE_SYNC and null following it */
+ /*
+ * Continue reading results until the last sync point, i.e., until
+ * reaching null just after PGRES_PIPELINE_SYNC.
+ */
for (;;)
{
PGresult *res = PQgetResult(st->con);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ PQclear(res);
+ return 0;
+ }
+
if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
received_sync = true;
- else if (received_sync)
+ else if (received_sync && res == NULL)
{
- /*
- * PGRES_PIPELINE_SYNC must be followed by another
- * PGRES_PIPELINE_SYNC or NULL; otherwise, assert failure.
- */
- Assert(res == NULL);
-
/*
* Reset ongoing sync count to 0 since all PGRES_PIPELINE_SYNC
* results have been discarded.
*/
st->num_syncs = 0;
- PQclear(res);
break;
}
+ else
+ {
+ /*
+ * If a PGRES_PIPELINE_SYNC is followed by something other than
+ * PGRES_PIPELINE_SYNC or NULL, another PGRES_PIPELINE_SYNC will
+ * appear later. Reset received_sync to false to wait for it.
+ */
+ received_sync = false;
+ }
PQclear(res);
}
--
2.43.0
From 9c4f19055597e9adb25e65c2aa8bedf20a09e13d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 19:05:58 +0700
Subject: [PATCH 08/12] Setting empty string as default value in filler column
---
src/bin/pgbench/pgbench.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 967f6ce6984..03b5e5c28f0 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84)",
- "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88)",
- "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88) default ''",
1
}
};
--
2.43.0
From 2aabaa52dffdb78fbefaef95163881c15e18ef29 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 21 Nov 2025 15:03:11 +0200
Subject: [PATCH 09/12] Use strtoi64() in pgbench, replacing its open-coded
implementation
Makes the code a little simpler.
The old implementation accepted trailing whitespace, but that was
unnecessary. Firstly, its sibling function for parsing decimals,
strtodouble(), does not accept trailing whitespace. Secondly, none of
the callers can pass a string with trailing whitespace to it.
In the passing, check specifically for ERANGE before printing the "out
of range" error. On some systems, strtoul() and strtod() return EINVAL
on an empty or all-spaces string, and "invalid input syntax" is more
appropriate for that than "out of range". For the existing
strtodouble() function this is purely academical because it's never
called with errorOK==false, but let's be tidy. (Perhaps we should
remove the dead codepaths altogether, but I'll leave that for another
day.)
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Yuefei Shi <[email protected]>
Reviewed-by: Neil Chen <[email protected]>
Discussion: https://www.postgresql.org/message-id/[email protected]
---
src/bin/pgbench/pgbench.c | 83 +++++++++------------------------------
1 file changed, 19 insertions(+), 64 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a425176ecdc..68774a59efd 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -982,13 +982,17 @@ usage(void)
progname, progname, PACKAGE_BUGREPORT, PACKAGE_NAME, PACKAGE_URL);
}
-/* return whether str matches "^\s*[-+]?[0-9]+$" */
+/*
+ * Return whether str matches "^\s*[-+]?[0-9]+$"
+ *
+ * This should agree with strtoint64() on what's accepted, ignoring overflows.
+ */
static bool
is_an_int(const char *str)
{
const char *ptr = str;
- /* skip leading spaces; cast is consistent with strtoint64 */
+ /* skip leading spaces */
while (*ptr && isspace((unsigned char) *ptr))
ptr++;
@@ -1012,9 +1016,6 @@ is_an_int(const char *str)
/*
* strtoint64 -- convert a string to 64-bit integer
*
- * This function is a slightly modified version of pg_strtoint64() from
- * src/backend/utils/adt/numutils.c.
- *
* The function returns whether the conversion worked, and if so
* "*result" is set to the result.
*
@@ -1023,71 +1024,25 @@ is_an_int(const char *str)
bool
strtoint64(const char *str, bool errorOK, int64 *result)
{
- const char *ptr = str;
- int64 tmp = 0;
- bool neg = false;
-
- /*
- * Do our own scan, rather than relying on sscanf which might be broken
- * for long long.
- *
- * As INT64_MIN can't be stored as a positive 64 bit integer, accumulate
- * value as a negative number.
- */
-
- /* skip leading spaces */
- while (*ptr && isspace((unsigned char) *ptr))
- ptr++;
-
- /* handle sign */
- if (*ptr == '-')
- {
- ptr++;
- neg = true;
- }
- else if (*ptr == '+')
- ptr++;
+ char *end;
- /* require at least one digit */
- if (unlikely(!isdigit((unsigned char) *ptr)))
- goto invalid_syntax;
+ errno = 0;
+ *result = strtoi64(str, &end, 10);
- /* process digits */
- while (*ptr && isdigit((unsigned char) *ptr))
+ if (unlikely(errno == ERANGE))
{
- int8 digit = (*ptr++ - '0');
-
- if (unlikely(pg_mul_s64_overflow(tmp, 10, &tmp)) ||
- unlikely(pg_sub_s64_overflow(tmp, digit, &tmp)))
- goto out_of_range;
+ if (!errorOK)
+ pg_log_error("value \"%s\" is out of range for type bigint", str);
+ return false;
}
- /* allow trailing whitespace, but not other trailing chars */
- while (*ptr != '\0' && isspace((unsigned char) *ptr))
- ptr++;
-
- if (unlikely(*ptr != '\0'))
- goto invalid_syntax;
-
- if (!neg)
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
- if (unlikely(tmp == PG_INT64_MIN))
- goto out_of_range;
- tmp = -tmp;
+ if (!errorOK)
+ pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
+ return false;
}
-
- *result = tmp;
return true;
-
-out_of_range:
- if (!errorOK)
- pg_log_error("value \"%s\" is out of range for type bigint", str);
- return false;
-
-invalid_syntax:
- if (!errorOK)
- pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
- return false;
}
/* convert string to double, detecting overflows/underflows */
@@ -1099,14 +1054,14 @@ strtodouble(const char *str, bool errorOK, double *dv)
errno = 0;
*dv = strtod(str, &end);
- if (unlikely(errno != 0))
+ if (unlikely(errno == ERANGE))
{
if (!errorOK)
pg_log_error("value \"%s\" is out of range for type double", str);
return false;
}
- if (unlikely(end == str || *end != '\0'))
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
if (!errorOK)
pg_log_error("invalid input syntax for type double: \"%s\"", str);
--
2.43.0
From dcb85d26f8132eaaf9d096e814b9bda49db7d478 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 20:06:24 +0700
Subject: [PATCH 10/12] Switching COPY FROM BINARY ti run in multiple
transactions
---
src/bin/pgbench/pgbench.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03b5e5c28f0..6b89007a63b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5496,20 +5496,20 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
static void
initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int64_t base, initRowMethodBin init_row)
+ int counter, int64_t base, initRowMethodBin init_row)
{
int n;
PGresult *res;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t total = base * scale;
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5526,7 +5526,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyHeader(con);
- for (int64_t i = 0; i < total; i++)
+ for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
}
@@ -5573,15 +5573,20 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- naccounts, initAccountBinary);
-
executeStatement(con, "commit");
+ for (int i = 0; i < scale; i++)
+ {
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+ }
+
pg_free(bin_copy_buffer);
}
--
2.43.0
From b8e28881225234fd00b55235bc60fad2dc60b544 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sat, 22 Nov 2025 17:06:00 +0700
Subject: [PATCH 11/12] Adding tests for new modes of data generation
---
src/bin/pgbench/pgbench.c | 21 ++++----
src/bin/pgbench/t/001_pgbench_with_server.pl | 52 +++++++++++++++++---
2 files changed, 56 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6b89007a63b..dd4e5d5e056 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -164,7 +164,7 @@ typedef struct socket_set
#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
@@ -192,7 +192,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = '?';
+static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
/*
* COPY FROM BINARY execution buffer
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84)",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default ''",
- "aid bigint not null,bid int,abalance int,filler char(84) default ''",
+ "aid int not null,bid int,abalance int,filler char(84) default '?'",
+ "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88) default ''",
- "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88)",
1
}
};
@@ -7837,6 +7837,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 581e9af7907..a377048ead1 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -125,7 +130,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +148,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -164,6 +170,38 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtI',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtC',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From b3f2bce34232d299abc7644a8579f0ce49c8c9d6 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 23 Nov 2025 14:05:59 +0700
Subject: [PATCH 12/12] Fixing compiler warnings about unused procedures by
removing or commenting them out as they might be needed a bit later
---
src/bin/pgbench/pgbench.c | 31 +++++--------------------------
1 file changed, 5 insertions(+), 26 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 0a3ba21dcc9..682db61ff61 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5201,7 +5201,7 @@ initGenerateDataClientSideText(PGconn *con)
/*
* Dumps binary buffer to file (purely for debugging)
- */
+ *
static void
dumpBufferToFile(char *filename)
{
@@ -5226,6 +5226,7 @@ dumpBufferToFile(char *filename)
fclose(file_ptr);
}
+ */
/*
* Save char data to buffer
@@ -5271,37 +5272,14 @@ addColumnCounter(int16_t n)
/*
* adds column with NULL value
- */
+ *
static void
addNullColumn()
{
int32_t null = -1;
bufferData((void *) &null, sizeof(null));
}
-
-/*
- * adds column with int8 value
*/
-static void
-addInt8Column(int8_t value)
-{
- int8_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
-
-/*
- * adds column with int16 value
- */
-static void
-addInt16Column(int16_t value)
-{
- int16_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
/*
* adds column with inti32 value
@@ -5329,7 +5307,7 @@ addInt64Column(int64_t value)
/*
* adds column with char value
- */
+ *
static void
addCharColumn(char *value)
{
@@ -5337,6 +5315,7 @@ addCharColumn(char *value)
bufferData((void *) &size, sizeof(size));
bufferCharData(value, size);
}
+ */
/*
* Starts communication with server for COPY FROM BINARY statement
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-01 12:18 ` Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-01 12:18 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; pgsql-hackers
Hi Egor,
Quick update.
Sending slightly updated version of the patch as couple
of small bugs have been fixed. Also more tests have been
added to cover specifically all possible init modes when
they called with "multi-" of "single-" transaction switches
as well as combination of multiple init modes in one call
to pgbench (eg, -IdtMccGc).
Cheers,
Boris
Attachments:
[application/octet-stream] v9-pgbench-faster-modes.patch (123.0K, 3-v9-pgbench-faster-modes.patch)
download | inline diff:
From c768f399c556295de7d53895410e686d86b4b960 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 19:34:58 +0700
Subject: [PATCH v9 01/26] Converting one huge transaction into series of one
per 'scale'
---
src/bin/pgbench/pgbench.c | 61 ++++++++++++++++++++++++++-------------
1 file changed, 41 insertions(+), 20 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..284a7c860f1 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,6 +181,12 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * scaling factor after which we switch to multiple transactions during
+ * data population phase on server side
+ */
+static int64 single_txn_scale_limit = 1;
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5213,6 +5219,7 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
+ int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5225,30 +5232,44 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ executeStatement(con, "commit");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ for (int i = 0; i < scale; i += chunk) {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "select bid + 1, 0 "
+ "from generate_series(%d, %d) as bid", i, i + chunk);
+ //"select bid, 0 "
+ //"from generate_series(1, %d) as bid", nbranches * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid,bid,tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ //"select tid, (tid - 1) / %d + 1, 0 "
+ //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ //"select aid, (aid - 1) / %d + 1, 0, '' "
+ //"from generate_series(1, " INT64_FORMAT ") as aid",
+ //naccounts, (int64) naccounts * scale);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
termPQExpBuffer(&sql);
-
- executeStatement(con, "commit");
}
/*
--
2.43.0
From 0eddb156c187d829c4381bc928c5314705928852 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:13:23 +0700
Subject: [PATCH v9 02/26] Getting rid off limit for single transaction size
during data generation
---
src/bin/pgbench/pgbench.c | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 284a7c860f1..28b72e4cf1f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,12 +181,6 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
-/*
- * scaling factor after which we switch to multiple transactions during
- * data population phase on server side
- */
-static int64 single_txn_scale_limit = 1;
-
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5219,7 +5213,6 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
- int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5236,13 +5229,13 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i += chunk) {
+ for (int i = 0; i < scale; i++) {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + chunk);
+ "from generate_series(%d, %d) as bid", i, i + 1);
//"select bid, 0 "
//"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
@@ -5251,7 +5244,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_tellers(tid,bid,tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
- ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
//"select tid, (tid - 1) / %d + 1, 0 "
//"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
@@ -5260,7 +5253,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_accounts(aid,bid,abalance,filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
//"select aid, (aid - 1) / %d + 1, 0, '' "
//"from generate_series(1, " INT64_FORMAT ") as aid",
//naccounts, (int64) naccounts * scale);
--
2.43.0
From c5659cf474ec273c057668f30a4f435fd02f2da7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:38:36 +0700
Subject: [PATCH v9 03/26] No need to keep old code in comments
---
src/bin/pgbench/pgbench.c | 7 -------
1 file changed, 7 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 28b72e4cf1f..97895aa9edf 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5236,8 +5236,6 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
"from generate_series(%d, %d) as bid", i, i + 1);
- //"select bid, 0 "
- //"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5245,8 +5243,6 @@ initGenerateDataServerSide(PGconn *con)
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
- //"select tid, (tid - 1) / %d + 1, 0 "
- //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5254,9 +5250,6 @@ initGenerateDataServerSide(PGconn *con)
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
- //"select aid, (aid - 1) / %d + 1, 0, '' "
- //"from generate_series(1, " INT64_FORMAT ") as aid",
- //naccounts, (int64) naccounts * scale);
executeStatement(con, sql.data);
executeStatement(con, "commit");
--
2.43.0
From e47b52ddf23593dad9375ef5356fd41d0621ede3 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 19:06:48 +0700
Subject: [PATCH v9 04/26] Adding server-side data generation via unnest
---
src/bin/pgbench/pgbench.c | 199 ++++++++++++++++++++++++++++++++++----
1 file changed, 182 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97895aa9edf..65d77cdefea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,12 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +187,11 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ *
+ */
+static char data_generation_type = '?';
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -914,7 +925,9 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5203,18 +5216,16 @@ initGenerateDataClientSide(PGconn *con)
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
*/
static void
-initGenerateDataServerSide(PGconn *con)
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5225,31 +5236,136 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ initPQExpBuffer(&sql);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "select bid, 0 "
+ "from generate_series(1, %d)", scale * nbranches);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
+ executeStatement(con, sql.data);
+
executeStatement(con, "commit");
+ termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i++) {
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + 1);
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
+ "insert into pgbench_tellers(tid, bid, tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
executeStatement(con, "commit");
@@ -5258,6 +5374,32 @@ initGenerateDataServerSide(PGconn *con)
termPQExpBuffer(&sql);
}
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
+}
+
/*
* Invoke vacuum on the standard tables
*/
@@ -5341,6 +5483,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5352,7 +5496,26 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'G':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'i':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5395,6 +5558,8 @@ runInitSteps(const char *initialize_steps)
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
--
2.43.0
From 5e1827b889b283f50299ce6ab1a73f9f55a4a84f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 20:00:56 +0700
Subject: [PATCH v9 05/26] Fixing typo in query
---
src/bin/pgbench/pgbench.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 65d77cdefea..03e37df4434 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5241,7 +5241,8 @@ generateDataInsertSingleTXN(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d)", scale * nbranches);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
--
2.43.0
From 7ca86521fda6929b8e0de3fc77dcbb8984009c88 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 11 Nov 2025 19:39:45 +0700
Subject: [PATCH v9 06/26] Adding support for COPY BINARY mode
---
src/bin/pgbench/pgbench.c | 393 ++++++++++++++++++++++++++++++++++++--
1 file changed, 381 insertions(+), 12 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03e37df4434..71aa1d9479f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -176,6 +176,8 @@ typedef struct socket_set
/* 'one transaction per scale' server-side methods */
#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
@@ -188,10 +190,17 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
static int scale = 1;
/*
- *
+ * mode of data generation to use
*/
static char data_generation_type = '?';
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -861,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -925,6 +935,7 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
" G: generate data, server-side in single transaction\n"
" i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
" I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
@@ -5191,9 +5202,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5209,12 +5220,373 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
+/*
+ * Dumps binary buffer to file (purely for debugging)
+ */
+static void
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t total = base * scale;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = 0; i < total; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
* whole dataset in single transaction
@@ -5500,14 +5872,10 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
+ case 'g':
+ case 'C':
case 'G':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'i':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'I':
data_init_type++;
data_generation_type = *step;
@@ -5555,6 +5923,7 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
--
2.43.0
From 4aa0ac05765edf6b5f0c13e18ac677287ce78206 Mon Sep 17 00:00:00 2001
From: Fujii Masao <[email protected]>
Date: Fri, 14 Nov 2025 22:40:39 +0900
Subject: [PATCH v9 07/26] pgbench: Fix assertion failure with multiple
\syncpipeline in pipeline mode.
Previously, when pgbench ran a custom script that triggered retriable errors
(e.g., deadlocks) followed by multiple \syncpipeline commands in pipeline mode,
the following assertion failure could occur:
Assertion failed: (res == ((void*)0)), function discardUntilSync, file pgbench.c, line 3594.
The issue was that discardUntilSync() assumed a pipeline sync result
(PGRES_PIPELINE_SYNC) would always be followed by either another sync result
or NULL. This assumption was incorrect: when multiple sync requests were sent,
a sync result could instead be followed by another result type. In such cases,
discardUntilSync() mishandled the results, leading to the assertion failure.
This commit fixes the issue by making discardUntilSync() correctly handle cases
where a pipeline sync result is followed by other result types. It now continues
discarding results until another pipeline sync followed by NULL is reached.
Backpatched to v17, where support for \syncpipeline command in pgbench was
introduced.
Author: Yugo Nagata <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 17
---
src/bin/pgbench/pgbench.c | 39 ++++++++++++++++++++++++++++-----------
1 file changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..a425176ecdc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -3563,14 +3563,18 @@ doRetry(CState *st, pg_time_usec_t *now)
}
/*
- * Read results and discard it until a sync point.
+ * Read and discard results until the last sync point.
*/
static int
discardUntilSync(CState *st)
{
bool received_sync = false;
- /* send a sync */
+ /*
+ * Send a Sync message to ensure at least one PGRES_PIPELINE_SYNC is
+ * received and to avoid an infinite loop, since all earlier ones may have
+ * already been received.
+ */
if (!PQpipelineSync(st->con))
{
pg_log_error("client %d aborted: failed to send a pipeline sync",
@@ -3578,29 +3582,42 @@ discardUntilSync(CState *st)
return 0;
}
- /* receive PGRES_PIPELINE_SYNC and null following it */
+ /*
+ * Continue reading results until the last sync point, i.e., until
+ * reaching null just after PGRES_PIPELINE_SYNC.
+ */
for (;;)
{
PGresult *res = PQgetResult(st->con);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ PQclear(res);
+ return 0;
+ }
+
if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
received_sync = true;
- else if (received_sync)
+ else if (received_sync && res == NULL)
{
- /*
- * PGRES_PIPELINE_SYNC must be followed by another
- * PGRES_PIPELINE_SYNC or NULL; otherwise, assert failure.
- */
- Assert(res == NULL);
-
/*
* Reset ongoing sync count to 0 since all PGRES_PIPELINE_SYNC
* results have been discarded.
*/
st->num_syncs = 0;
- PQclear(res);
break;
}
+ else
+ {
+ /*
+ * If a PGRES_PIPELINE_SYNC is followed by something other than
+ * PGRES_PIPELINE_SYNC or NULL, another PGRES_PIPELINE_SYNC will
+ * appear later. Reset received_sync to false to wait for it.
+ */
+ received_sync = false;
+ }
PQclear(res);
}
--
2.43.0
From 9c4f19055597e9adb25e65c2aa8bedf20a09e13d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 19:05:58 +0700
Subject: [PATCH v9 08/26] Setting empty string as default value in filler
column
---
src/bin/pgbench/pgbench.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 967f6ce6984..03b5e5c28f0 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84)",
- "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88)",
- "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88) default ''",
1
}
};
--
2.43.0
From 2aabaa52dffdb78fbefaef95163881c15e18ef29 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 21 Nov 2025 15:03:11 +0200
Subject: [PATCH v9 09/26] Use strtoi64() in pgbench, replacing its open-coded
implementation
Makes the code a little simpler.
The old implementation accepted trailing whitespace, but that was
unnecessary. Firstly, its sibling function for parsing decimals,
strtodouble(), does not accept trailing whitespace. Secondly, none of
the callers can pass a string with trailing whitespace to it.
In the passing, check specifically for ERANGE before printing the "out
of range" error. On some systems, strtoul() and strtod() return EINVAL
on an empty or all-spaces string, and "invalid input syntax" is more
appropriate for that than "out of range". For the existing
strtodouble() function this is purely academical because it's never
called with errorOK==false, but let's be tidy. (Perhaps we should
remove the dead codepaths altogether, but I'll leave that for another
day.)
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Yuefei Shi <[email protected]>
Reviewed-by: Neil Chen <[email protected]>
Discussion: https://www.postgresql.org/message-id/[email protected]
---
src/bin/pgbench/pgbench.c | 83 +++++++++------------------------------
1 file changed, 19 insertions(+), 64 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a425176ecdc..68774a59efd 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -982,13 +982,17 @@ usage(void)
progname, progname, PACKAGE_BUGREPORT, PACKAGE_NAME, PACKAGE_URL);
}
-/* return whether str matches "^\s*[-+]?[0-9]+$" */
+/*
+ * Return whether str matches "^\s*[-+]?[0-9]+$"
+ *
+ * This should agree with strtoint64() on what's accepted, ignoring overflows.
+ */
static bool
is_an_int(const char *str)
{
const char *ptr = str;
- /* skip leading spaces; cast is consistent with strtoint64 */
+ /* skip leading spaces */
while (*ptr && isspace((unsigned char) *ptr))
ptr++;
@@ -1012,9 +1016,6 @@ is_an_int(const char *str)
/*
* strtoint64 -- convert a string to 64-bit integer
*
- * This function is a slightly modified version of pg_strtoint64() from
- * src/backend/utils/adt/numutils.c.
- *
* The function returns whether the conversion worked, and if so
* "*result" is set to the result.
*
@@ -1023,71 +1024,25 @@ is_an_int(const char *str)
bool
strtoint64(const char *str, bool errorOK, int64 *result)
{
- const char *ptr = str;
- int64 tmp = 0;
- bool neg = false;
-
- /*
- * Do our own scan, rather than relying on sscanf which might be broken
- * for long long.
- *
- * As INT64_MIN can't be stored as a positive 64 bit integer, accumulate
- * value as a negative number.
- */
-
- /* skip leading spaces */
- while (*ptr && isspace((unsigned char) *ptr))
- ptr++;
-
- /* handle sign */
- if (*ptr == '-')
- {
- ptr++;
- neg = true;
- }
- else if (*ptr == '+')
- ptr++;
+ char *end;
- /* require at least one digit */
- if (unlikely(!isdigit((unsigned char) *ptr)))
- goto invalid_syntax;
+ errno = 0;
+ *result = strtoi64(str, &end, 10);
- /* process digits */
- while (*ptr && isdigit((unsigned char) *ptr))
+ if (unlikely(errno == ERANGE))
{
- int8 digit = (*ptr++ - '0');
-
- if (unlikely(pg_mul_s64_overflow(tmp, 10, &tmp)) ||
- unlikely(pg_sub_s64_overflow(tmp, digit, &tmp)))
- goto out_of_range;
+ if (!errorOK)
+ pg_log_error("value \"%s\" is out of range for type bigint", str);
+ return false;
}
- /* allow trailing whitespace, but not other trailing chars */
- while (*ptr != '\0' && isspace((unsigned char) *ptr))
- ptr++;
-
- if (unlikely(*ptr != '\0'))
- goto invalid_syntax;
-
- if (!neg)
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
- if (unlikely(tmp == PG_INT64_MIN))
- goto out_of_range;
- tmp = -tmp;
+ if (!errorOK)
+ pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
+ return false;
}
-
- *result = tmp;
return true;
-
-out_of_range:
- if (!errorOK)
- pg_log_error("value \"%s\" is out of range for type bigint", str);
- return false;
-
-invalid_syntax:
- if (!errorOK)
- pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
- return false;
}
/* convert string to double, detecting overflows/underflows */
@@ -1099,14 +1054,14 @@ strtodouble(const char *str, bool errorOK, double *dv)
errno = 0;
*dv = strtod(str, &end);
- if (unlikely(errno != 0))
+ if (unlikely(errno == ERANGE))
{
if (!errorOK)
pg_log_error("value \"%s\" is out of range for type double", str);
return false;
}
- if (unlikely(end == str || *end != '\0'))
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
if (!errorOK)
pg_log_error("invalid input syntax for type double: \"%s\"", str);
--
2.43.0
From dcb85d26f8132eaaf9d096e814b9bda49db7d478 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 20:06:24 +0700
Subject: [PATCH v9 10/26] Switching COPY FROM BINARY ti run in multiple
transactions
---
src/bin/pgbench/pgbench.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03b5e5c28f0..6b89007a63b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5496,20 +5496,20 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
static void
initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int64_t base, initRowMethodBin init_row)
+ int counter, int64_t base, initRowMethodBin init_row)
{
int n;
PGresult *res;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t total = base * scale;
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5526,7 +5526,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyHeader(con);
- for (int64_t i = 0; i < total; i++)
+ for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
}
@@ -5573,15 +5573,20 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- naccounts, initAccountBinary);
-
executeStatement(con, "commit");
+ for (int i = 0; i < scale; i++)
+ {
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+ }
+
pg_free(bin_copy_buffer);
}
--
2.43.0
From b8e28881225234fd00b55235bc60fad2dc60b544 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sat, 22 Nov 2025 17:06:00 +0700
Subject: [PATCH v9 11/26] Adding tests for new modes of data generation
---
src/bin/pgbench/pgbench.c | 21 ++++----
src/bin/pgbench/t/001_pgbench_with_server.pl | 52 +++++++++++++++++---
2 files changed, 56 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6b89007a63b..dd4e5d5e056 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -164,7 +164,7 @@ typedef struct socket_set
#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
@@ -192,7 +192,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = '?';
+static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
/*
* COPY FROM BINARY execution buffer
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84)",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default ''",
- "aid bigint not null,bid int,abalance int,filler char(84) default ''",
+ "aid int not null,bid int,abalance int,filler char(84) default '?'",
+ "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88) default ''",
- "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88)",
1
}
};
@@ -7837,6 +7837,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 581e9af7907..a377048ead1 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -125,7 +130,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +148,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -164,6 +170,38 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtI',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtC',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From b3f2bce34232d299abc7644a8579f0ce49c8c9d6 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 23 Nov 2025 14:05:59 +0700
Subject: [PATCH v9 12/26] Fixing compiler warnings about unused procedures by
removing or commenting them out as they might be needed a bit later
---
src/bin/pgbench/pgbench.c | 31 +++++--------------------------
1 file changed, 5 insertions(+), 26 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 0a3ba21dcc9..682db61ff61 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5201,7 +5201,7 @@ initGenerateDataClientSideText(PGconn *con)
/*
* Dumps binary buffer to file (purely for debugging)
- */
+ *
static void
dumpBufferToFile(char *filename)
{
@@ -5226,6 +5226,7 @@ dumpBufferToFile(char *filename)
fclose(file_ptr);
}
+ */
/*
* Save char data to buffer
@@ -5271,37 +5272,14 @@ addColumnCounter(int16_t n)
/*
* adds column with NULL value
- */
+ *
static void
addNullColumn()
{
int32_t null = -1;
bufferData((void *) &null, sizeof(null));
}
-
-/*
- * adds column with int8 value
*/
-static void
-addInt8Column(int8_t value)
-{
- int8_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
-
-/*
- * adds column with int16 value
- */
-static void
-addInt16Column(int16_t value)
-{
- int16_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
/*
* adds column with inti32 value
@@ -5329,7 +5307,7 @@ addInt64Column(int64_t value)
/*
* adds column with char value
- */
+ *
static void
addCharColumn(char *value)
{
@@ -5337,6 +5315,7 @@ addCharColumn(char *value)
bufferData((void *) &size, sizeof(size));
bufferCharData(value, size);
}
+ */
/*
* Starts communication with server for COPY FROM BINARY statement
--
2.43.0
From 9ab7fe302ba7de40593e7cba8ea1ca3b876c1ea5 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 19:22:16 +0700
Subject: [PATCH v9 13/26] Moving PQclear call to the end of procedure to avoid
access to previously released memory
---
src/bin/pgbench/pgbench.c | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 682db61ff61..3687b65871e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5363,11 +5363,11 @@ static void
initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of branch table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 3 columns
*/
- int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
flushBuffer(con, res, max_row_len);
@@ -5384,9 +5384,9 @@ static void
initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of tellers table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 4 columns
*/
int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
@@ -5406,9 +5406,9 @@ static void
initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of accounts table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 4 columns
*/
int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
@@ -5455,11 +5455,11 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
if (PQresultStatus(res) != PGRES_COPY_IN)
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
- PQclear(res);
sendBinaryCopyHeader(con);
+
for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
@@ -5470,8 +5470,10 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
sendBinaryCopyTrailer(con);
+
if (PQresultStatus(res) == PGRES_COPY_IN)
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
@@ -5484,6 +5486,8 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
}
+
+ PQclear(res);
}
/*
@@ -5499,8 +5503,9 @@ initGenerateDataClientSideBinary(PGconn *con)
bin_copy_buffer_length = 0;
/*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
+ * we do all of this in multiple transactions
+ * to minimize load on DB server and perhaps
+ * in future allow load in parallel sessions
*/
executeStatement(con, "begin");
@@ -5511,6 +5516,8 @@ initGenerateDataClientSideBinary(PGconn *con)
for (int i = 0; i < scale; i++)
{
+ executeStatement(con, "begin");
+
initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary);
initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
--
2.43.0
From 3d7b2c53b3fa45b8848369bfbc49a0c7e5854304 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 19:47:26 +0700
Subject: [PATCH v9 14/26] Fixing error about freeing memory twice
---
src/bin/pgbench/pgbench.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3687b65871e..6f42438d970 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5462,6 +5462,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
+ res = PQgetResult(con);
init_row(con, res, i, base);
}
@@ -5473,7 +5474,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyTrailer(con);
-
+ res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
@@ -5481,7 +5482,6 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
res = PQgetResult(con);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
- PQclear(res);
}
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
--
2.43.0
From a71c90f556ba143335d9631ebbdf96b7120c6d1d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 20:24:54 +0700
Subject: [PATCH v9 15/26] Fixing memory leak during data flush in COPY BINARY
test
---
src/bin/pgbench/pgbench.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6f42438d970..8eca537b718 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -871,7 +871,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
+typedef void (*initRowMethodBin) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5345,13 +5345,18 @@ sendBinaryCopyTrailer(PGconn *con)
* Flashes current buffer over network if needed
*/
static void
-flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+flushBuffer(PGconn *con, int16_t row_len)
{
+ PGresult *res;
+
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
+ res = PQgetResult(con);
/* flush current buffer */
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ pg_fatal("It is NOT a COPY command that is currently running");
bin_copy_buffer_length = 0;
}
}
@@ -5360,7 +5365,7 @@ flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of branch table has following extra bytes:
@@ -5369,7 +5374,7 @@ initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(2);
@@ -5381,7 +5386,7 @@ initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of tellers table has following extra bytes:
@@ -5390,7 +5395,7 @@ initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(3);
@@ -5403,7 +5408,7 @@ initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of accounts table has following extra bytes:
@@ -5412,7 +5417,7 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(3);
@@ -5462,10 +5467,10 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- res = PQgetResult(con);
- init_row(con, res, i, base);
+ init_row(con, i, base);
}
+ res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
@@ -5474,6 +5479,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyTrailer(con);
+
res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
--
2.43.0
From a5933f40be0f3d408bb920f296fb5119c3ec28f4 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 30 Jan 2026 13:59:36 +0700
Subject: [PATCH v9 16/26] Fixing memory leak shown by valgrind
---
src/bin/pgbench/pgbench.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8eca537b718..466d9023d72 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5357,6 +5357,8 @@ flushBuffer(PGconn *con, int16_t row_len)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
pg_fatal("It is NOT a COPY command that is currently running");
+
+ PQclear(res);
bin_copy_buffer_length = 0;
}
}
@@ -5460,6 +5462,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
if (PQresultStatus(res) != PGRES_COPY_IN)
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
sendBinaryCopyHeader(con);
@@ -5475,6 +5478,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+ PQclear(res);
sendBinaryCopyTrailer(con);
@@ -5485,6 +5489,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
{
+ PQclear(res);
res = PQgetResult(con);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
@@ -5492,7 +5497,6 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
}
-
PQclear(res);
}
--
2.43.0
From 6a3036f898b050d5e71e70eaceccd63003aa6444 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 6 Feb 2026 13:53:07 +0700
Subject: [PATCH v9 17/26] Adding ability to switch data init between single
and multiple transactions
---
src/bin/pgbench/pgbench.c | 192 +++++++++----------
src/bin/pgbench/t/001_pgbench_with_server.pl | 93 ++++++++-
2 files changed, 173 insertions(+), 112 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 466d9023d72..af71e358e71 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,9 +159,8 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-
-#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
+#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
+#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,14 +170,17 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
-/* original single transaction server-side method */
-#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
-/* 'one transaction per scale' server-side methods */
-#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
-#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
-#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
-#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
-
+/* server-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT generate_series to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest to generate data */
+/* client-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN .. BINARY to generate data */
+/* data init pseudo steps */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as multiple transactions */
+
+static bool multi_xact = false; /* init data type (as single or multiple transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -192,7 +194,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
/*
* COPY FROM BINARY execution buffer
@@ -934,11 +936,13 @@ usage(void)
" run selected initialization steps, in the specified order\n"
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
- " g: generate data, client-side\n"
- " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
- " G: generate data, server-side in single transaction\n"
- " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
- " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
+ " to generate data, client-side:\n"
+ " g: COPY .. FROM STDIN .. TEXT\n"
+ " c: COPY .. FROM STDIN .. BINARY\n"
+ " to generate data, server-side:\n"
+ " G: INSERT .. SELECT generate_series\n"
+ " U: INSERT .. SELECT unnest\n"
+ " M: use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5049,8 +5053,8 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTable(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+initPopulateTableText(PGconn *con, const char *table, int64 base,
+ initRowMethod init_row)
{
int n;
int64 k;
@@ -5178,6 +5182,9 @@ initGenerateDataClientSideText(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
+ if (multi_xact)
+ fprintf(stderr, "WARNING! Multiple transactions are not supported in this mode\n");
+
/*
* we do all of this in one transaction to enable the backend's
* data-loading optimizations
@@ -5191,9 +5198,9 @@ initGenerateDataClientSideText(PGconn *con)
* fill branches, tellers, accounts in that order in case foreign keys
* already exist
*/
- initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
+ initPopulateTableText(con, "pgbench_branches", nbranches, initBranch);
+ initPopulateTableText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
@@ -5450,7 +5457,8 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
+ if (!multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5522,11 +5530,13 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int i = 0; i < scale; i++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary);
@@ -5535,9 +5545,13 @@ initGenerateDataClientSideBinary(PGconn *con)
initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
i, naccounts, initAccountBinary);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
pg_free(bin_copy_buffer);
}
@@ -5547,14 +5561,15 @@ initGenerateDataClientSideBinary(PGconn *con)
static void
initGenerateDataClientSide(PGconn *con)
{
- fprintf(stderr, "generating data (client-side) in ");
+ fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
switch (data_generation_type)
{
- case GEN_TYPE_COPY_ORIGINAL:
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
initGenerateDataClientSideText(con);
break;
- case GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
initGenerateDataClientSideBinary(con);
break;
}
@@ -5562,58 +5577,7 @@ initGenerateDataClientSide(PGconn *con)
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
- * whole dataset in single transaction
- */
-static void
-generateDataInsertSingleTXN(PGconn *con)
-{
- PQExpBufferData sql;
-
- fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
-
-
- /*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
- */
- executeStatement(con, "begin");
-
- /* truncate away any old data */
- initTruncateTables(con);
-
- initPQExpBuffer(&sql);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid, bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid",
- scale * nbranches);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid, bid, tbalance) "
- "select tid + 1, tid / %d + 1, 0 "
- "from generate_series(0, %d) as tid",
- ntellers, (scale * ntellers) - 1);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid, bid, abalance, "
- "filler) "
- "select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(0, " INT64_FORMAT ") as aid",
- naccounts, (int64) (scale * naccounts) - 1);
- executeStatement(con, sql.data);
-
- executeStatement(con, "commit");
-
- termPQExpBuffer(&sql);
-}
-
-
-/*
- * Generating data via INSERT .. SELECT .. FROM generate_series
- * One transaction per 'scale'
+ * Possibly as "One transaction per scale" in multi-transaction mode
*/
static void
generateDataInsertSeries(PGconn *con)
@@ -5629,11 +5593,13 @@ generateDataInsertSeries(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int i = 0; i < scale; i++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
@@ -5657,15 +5623,19 @@ generateDataInsertSeries(PGconn *con)
(int64) (i + 1) * naccounts - 1);
executeStatement(con, sql.data);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
}
/*
* Generating data via INSERT .. SELECT .. FROM unnest
- * One transaction per 'scale'
+ * Possibly as "One transaction per scale" in multi-tansaction mode
*/
static void
generateDataInsertUnnest(PGconn *con)
@@ -5681,11 +5651,13 @@ generateDataInsertUnnest(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int s = 0; s < scale; s++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
@@ -5714,14 +5686,18 @@ generateDataInsertUnnest(PGconn *con)
(int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
}
/*
- * Fill the standard tables with some data generated on the server
+ * Fill the standard tables with some data generated on the server side
*
* As already the case with the client-side data generation, the filler
* column defaults to NULL in pgbench_branches and pgbench_tellers,
@@ -5730,17 +5706,15 @@ generateDataInsertUnnest(PGconn *con)
static void
initGenerateDataServerSide(PGconn *con)
{
- fprintf(stderr, "generating data (server-side) ");
+ fprintf(stderr, "generating data (server-side as %s transaction%s) ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
switch (data_generation_type)
{
- case GEN_TYPE_INSERT_ORIGINAL:
- generateDataInsertSingleTXN(con);
- break;
- case GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
generateDataInsertSeries(con);
break;
- case GEN_TYPE_INSERT_UNNEST:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
generateDataInsertUnnest(con);
break;
}
@@ -5845,19 +5819,20 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
- case 'g':
- case 'C':
- case 'G':
- case 'i':
- case 'I':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
data_init_type++;
data_generation_type = *step;
break;
}
}
+ if (data_init_type == 0)
+ pg_log_error("WARNING! No data generation type is provided");
if (data_init_type > 1)
- pg_log_error("WARNING! More than one type of server-side data generation is requested");
+ pg_log_error("WARNING! More than one type of data initialization is requested");
}
/*
@@ -5895,17 +5870,22 @@ runInitSteps(const char *initialize_steps)
op = "create tables";
initCreateTables(con);
break;
- case 'g':
- case 'C':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
initGenerateDataClientSide(con);
break;
- case 'G':
- case 'i':
- case 'I':
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
initGenerateDataServerSide(con);
break;
+ case INIT_STEP_GEN_TYPE_SINGLE_XACT:
+ multi_xact = false;
+ break;
+ case INIT_STEP_GEN_TYPE_MULTI_XACT:
+ multi_xact = true;
+ break;
case 'v':
op = "vacuum";
initVacuum(con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index a377048ead1..acb1e31d3e8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -117,6 +117,7 @@ $node->pgbench(
[qr{Perhaps you need to do initialization}],
'run without init');
+
# Initialize pgbench tables scale 1
$node->pgbench(
'-i', 0,
@@ -160,7 +161,7 @@ $node->pgbench(
qr{creating tables},
qr{creating 3 partitions},
qr{creating primary keys},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{creating foreign keys},
qr{(?!vacuuming)}, # no vacuum
qr{done in \d+\.\d\d s }
@@ -170,15 +171,16 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+
# Test server-side generation with UNNEST
$node->pgbench(
- '--initialize --init-steps=dtI',
+ '--initialize --init-steps=dtU',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{done in \d+\.\d\d s }
],
'pgbench --init-steps server-side UNNEST');
@@ -186,15 +188,48 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side (unnest)');
-# Test server-side generation with UNNEST
+
+# Test server-side generation with COPY TEXT
+$node->pgbench(
+ '--initialize --init-steps=dtg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (text)');
+
+
+# Test server-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtC',
+ '--initialize --init-steps=dtc',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side\)},
+ qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
'pgbench --init-steps client-side BINARY');
@@ -202,6 +237,52 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'client-side (binary)');
+$node->pgbench(
+ '--initialize --init-steps=dtSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From 2471bb47bf715402ece61b084b69afe3bb53742e Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 6 Feb 2026 13:55:47 +0700
Subject: [PATCH v9 18/26] Removing commented out procedures
---
src/bin/pgbench/pgbench.c | 52 ---------------------------------------
1 file changed, 52 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index af71e358e71..34464b7037c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5206,35 +5206,6 @@ initGenerateDataClientSideText(PGconn *con)
}
-/*
- * Dumps binary buffer to file (purely for debugging)
- *
-static void
-dumpBufferToFile(char *filename)
-{
- FILE *file_ptr;
- size_t bytes_written;
-
- file_ptr = fopen(filename, "wb");
- if (file_ptr == NULL)
- {
- fprintf(stderr, "Error opening file %s\n", filename);
- return; // EXIT_FAILURE;
- }
-
- bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
-
- if (bytes_written != bin_copy_buffer_length)
- {
- fprintf(stderr, "Error writing to file or incomplete write\n");
- fclose(file_ptr);
- return; // EXIT_FAILURE;
- }
-
- fclose(file_ptr);
-}
- */
-
/*
* Save char data to buffer
*/
@@ -5277,17 +5248,6 @@ addColumnCounter(int16_t n)
bufferData((void *) &n, sizeof(n));
}
-/*
- * adds column with NULL value
- *
-static void
-addNullColumn()
-{
- int32_t null = -1;
- bufferData((void *) &null, sizeof(null));
-}
- */
-
/*
* adds column with inti32 value
*/
@@ -5312,18 +5272,6 @@ addInt64Column(int64_t value)
bufferData((void *) &data, sizeof(data));
}
-/*
- * adds column with char value
- *
-static void
-addCharColumn(char *value)
-{
- int32_t size = strlen(value);
- bufferData((void *) &size, sizeof(size));
- bufferCharData(value, size);
-}
- */
-
/*
* Starts communication with server for COPY FROM BINARY statement
*/
--
2.43.0
From 10dfa622b09c420d79da8774c9321f91540c8600 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 09:08:17 +0700
Subject: [PATCH v9 19/26] Small changes due to big code review
---
src/bin/pgbench/pgbench.c | 71 +++++++++++++++++++++------------------
1 file changed, 39 insertions(+), 32 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 34464b7037c..9f3d7f99a25 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -873,7 +873,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBin) (PGconn *con, int64_t curr, int32_t parent);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -942,7 +942,8 @@ usage(void)
" to generate data, server-side:\n"
" G: INSERT .. SELECT generate_series\n"
" U: INSERT .. SELECT unnest\n"
- " M: use multiple transactions to initialize data\n"
+ " S: flag to use single transaction to initialize data\n"
+ " M: flag to use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5053,8 +5054,8 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTableText(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
+ initRowMethod init_row)
{
int n;
int64 k;
@@ -5178,7 +5179,7 @@ initPopulateTableText(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSideText(PGconn *con)
+initGenerateDataClientSideTextFrmt(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
@@ -5198,9 +5199,9 @@ initGenerateDataClientSideText(PGconn *con)
* fill branches, tellers, accounts in that order in case foreign keys
* already exist
*/
- initPopulateTableText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableText(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTableText(con, "pgbench_accounts", naccounts, initAccount);
+ initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
@@ -5212,6 +5213,8 @@ initGenerateDataClientSideText(PGconn *con)
static void
bufferCharData(char *src, int32_t len)
{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
bin_copy_buffer_length += len;
}
@@ -5227,13 +5230,16 @@ bufferData(void *src, int32_t len)
bufferCharData(src, len);
#else
if (len == 1)
+ {
bufferCharData(src, len);
- else
- for (int32_t i = 0; i < len; i++)
- {
- ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
- ((char *) src)[len - i - 1];
- }
+ return;
+ }
+
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
bin_copy_buffer_length += len;
#endif
@@ -5322,7 +5328,7 @@ flushBuffer(PGconn *con, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of branch table has following extra bytes:
@@ -5333,7 +5339,7 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(2);
+ addColumnCounter(columnCounter);
addInt32Column(curr + 1);
addInt32Column(0);
@@ -5343,7 +5349,7 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of tellers table has following extra bytes:
@@ -5354,7 +5360,7 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(3);
+ addColumnCounter(columnCounter);
addInt32Column(curr + 1);
addInt32Column(curr / parent + 1);
@@ -5365,7 +5371,7 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of accounts table has following extra bytes:
@@ -5376,7 +5382,7 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(3);
+ addColumnCounter(columnCounter);
if (scale <= SCALE_32BIT_THRESHOLD)
addInt32Column(curr + 1);
@@ -5391,8 +5397,9 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
* Universal wrapper for sending data in binary format
*/
static void
-initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int counter, int64_t base, initRowMethodBin init_row)
+initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBinary init_row,
+ int columnCounter)
{
int n;
PGresult *res;
@@ -5426,7 +5433,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- init_row(con, i, base);
+ init_row(con, i, base, columnCounter);
}
res = PQgetResult(con);
@@ -5460,7 +5467,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
* Wrapper for binary data load
*/
static void
-initGenerateDataClientSideBinary(PGconn *con)
+initGenerateDataClientSideBinaryFrmt(PGconn *con)
{
fprintf(stderr, "BINARY mode...\n");
@@ -5486,12 +5493,12 @@ initGenerateDataClientSideBinary(PGconn *con)
if (multi_xact)
executeStatement(con, "begin");
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- i, nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- i, naccounts, initAccountBinary);
+ initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary, 2);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary, 3);
+ initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary, 3);
if (multi_xact)
executeStatement(con, "commit");
@@ -5515,10 +5522,10 @@ initGenerateDataClientSide(PGconn *con)
switch (data_generation_type)
{
case INIT_STEP_GEN_TYPE_COPY_TEXT:
- initGenerateDataClientSideText(con);
+ initGenerateDataClientSideTextFrmt(con);
break;
case INIT_STEP_GEN_TYPE_COPY_BINARY:
- initGenerateDataClientSideBinary(con);
+ initGenerateDataClientSideBinaryFrmt(con);
break;
}
}
--
2.43.0
From d95c53d412ff0e94e162eaf3990233fae007ae3f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 10:07:13 +0700
Subject: [PATCH v9 20/26] Running indent to fix code formatting
---
src/bin/pgbench/pgbench.c | 111 +++++++++++++++++++++-----------------
1 file changed, 62 insertions(+), 49 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 9f3d7f99a25..8198f1d4c9a 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,7 +159,7 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
+#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
@@ -171,16 +171,24 @@ typedef struct socket_set
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
/* server-side methods to generate data */
-#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT generate_series to generate data */
-#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT
+ * generate_series to generate
+ * data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest
+ * to generate data */
/* client-side methods to generate data */
-#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
-#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN .. BINARY to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN ..
+ * TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN ..
+ * BINARY to generate data */
/* data init pseudo steps */
-#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as single transaction */
-#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as multiple transactions */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as
+ * single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as
+ * multiple transactions */
-static bool multi_xact = false; /* init data type (as single or multiple transactions) */
+static bool multi_xact = false; /* init data type (as single or multiple
+ * transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -194,14 +202,15 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
/*
* COPY FROM BINARY execution buffer
*/
-#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
-static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
-static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM
+ * BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
@@ -872,8 +881,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5200,7 +5209,7 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
* already exist
*/
initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
@@ -5260,8 +5269,9 @@ addColumnCounter(int16_t n)
static void
addInt32Column(int32_t value)
{
- int32_t data = value;
- int32_t size = sizeof(data);
+ int32_t data = value;
+ int32_t size = sizeof(data);
+
bufferData((void *) &size, sizeof(size));
bufferData((void *) &data, sizeof(data));
}
@@ -5272,8 +5282,9 @@ addInt32Column(int32_t value)
static void
addInt64Column(int64_t value)
{
- int64_t data = value;
- int32_t size = sizeof(data);
+ int64_t data = value;
+ int32_t size = sizeof(data);
+
bufferData((void *) &size, sizeof(size));
bufferData((void *) &data, sizeof(data));
}
@@ -5284,9 +5295,9 @@ addInt64Column(int64_t value)
static void
sendBinaryCopyHeader(PGconn *con)
{
- char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
- '\0','\0','\0','\0',
- '\0','\0','\0','\0' };
+ char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ '\0', '\0', '\0', '\0',
+ '\0', '\0', '\0', '\0'};
PQputCopyData(con, header, 19);
}
@@ -5297,7 +5308,7 @@ sendBinaryCopyHeader(PGconn *con)
static void
sendBinaryCopyTrailer(PGconn *con)
{
- static char trailer[] = { 0xFF, 0xFF };
+ static char trailer[] = {0xFF, 0xFF};
PQputCopyData(con, trailer, 2);
}
@@ -5308,7 +5319,7 @@ sendBinaryCopyTrailer(PGconn *con)
static void
flushBuffer(PGconn *con, int16_t row_len)
{
- PGresult *res;
+ PGresult *res;
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
@@ -5330,12 +5341,13 @@ flushBuffer(PGconn *con, int16_t row_len)
static void
initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of branch table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 3 columns
+ *---
*/
- int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
+ int16_t max_row_len = 35 + 2 + 4 * 3; /* max row size is 35 */
flushBuffer(con, max_row_len);
@@ -5351,12 +5363,13 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter
static void
initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of tellers table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 4 columns
+ *---
*/
- int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+ int16_t max_row_len = 40 + 2 + 4 * 4; /* max row size is 40 */
flushBuffer(con, max_row_len);
@@ -5373,12 +5386,13 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter
static void
initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of accounts table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 4 columns
*/
- int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+ int16_t max_row_len = 250 + 2 + 4 * 4; /* max row size is 250 for
+ * int64 */
flushBuffer(con, max_row_len);
@@ -5401,11 +5415,11 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
int counter, int64_t base, initRowMethodBinary init_row,
int columnCounter)
{
- int n;
- PGresult *res;
- char copy_statement[256];
- const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t start = base * counter;
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
@@ -5450,7 +5464,7 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
- if (PQputCopyEnd(con, NULL) == 1) /* success */
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
{
PQclear(res);
res = PQgetResult(con);
@@ -5476,9 +5490,8 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
bin_copy_buffer_length = 0;
/*
- * we do all of this in multiple transactions
- * to minimize load on DB server and perhaps
- * in future allow load in parallel sessions
+ * we do all of this in multiple transactions to minimize load on DB
+ * server and perhaps in future allow load in parallel sessions
*/
executeStatement(con, "begin");
@@ -5495,8 +5508,8 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary, 2);
- initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary, 3);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary, 3);
initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
i, naccounts, initAccountBinary, 3);
@@ -5570,10 +5583,10 @@ generateDataInsertSeries(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_accounts(aid, bid, abalance, "
- "filler) "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", "
- INT64_FORMAT ") as aid",
+ INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts,
(int64) (i + 1) * naccounts - 1);
executeStatement(con, sql.data);
@@ -5622,7 +5635,7 @@ generateDataInsertUnnest(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_tellers(tid, bid, tbalance) "
"select unnest(array_agg(s.i order by s.i)) as tid, "
- "%d as bid, 0 as tbalance "
+ "%d as bid, 0 as tbalance "
"from generate_series(%d, %d) as s(i)",
s + 1, s * ntellers + 1, (s + 1) * ntellers);
executeStatement(con, sql.data);
@@ -5630,13 +5643,13 @@ generateDataInsertUnnest(PGconn *con)
printfPQExpBuffer(&sql,
"with data as ("
" select generate_series(" INT64_FORMAT ", "
- INT64_FORMAT ") as i) "
+ INT64_FORMAT ") as i) "
"insert into pgbench_accounts(aid, bid, "
- "abalance, filler) "
+ "abalance, filler) "
"select unnest(aid), unnest(bid), 0 as abalance, "
- "'' as filler "
+ "'' as filler "
"from (select array_agg(i+1) aid, "
- "array_agg(i/%d + 1) bid from data)",
+ "array_agg(i/%d + 1) bid from data)",
(int64) s * naccounts + 1,
(int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
@@ -5758,7 +5771,7 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
- char data_init_type = 0;
+ char data_init_type = 0;
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
--
2.43.0
From 270d2bf14b6873ff66fc37ca0a5dccc055dab1f7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 14:57:18 +0700
Subject: [PATCH v9 21/26] Removing debugging filler value
---
src/bin/pgbench/pgbench.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8198f1d4c9a..710b2a5b70d 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4954,8 +4954,8 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
@@ -4966,8 +4966,8 @@ initCreateTables(PGconn *con)
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default '?'",
- "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
--
2.43.0
From 86a111c63e7006d4801ce7bb594a5b77d87b42b0 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 23 Feb 2026 16:15:06 +0700
Subject: [PATCH v9 22/26] Moving out logic showing data load progress into own
procedure
---
src/bin/pgbench/pgbench.c | 142 ++++++++++++++++++++------------------
1 file changed, 75 insertions(+), 67 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 710b2a5b70d..e3e0e073792 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -487,6 +487,9 @@ typedef struct StatsData
*/
static pg_time_usec_t epoch_shift;
+/* used to track elapsed time and estimate of the remaining time of data load */
+static pg_time_usec_t data_load_start;
+
/*
* Error status for errors during script execution.
*/
@@ -5062,27 +5065,82 @@ initAccount(PQExpBufferData *sql, int64 curr)
curr + 1, curr / naccounts + 1);
}
+static void
+showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
+{
+ static int chars = 0;
+ static int prev_chars = 0;
+ static int log_interval = 1;
+
+ /* Stay on the same line if reporting to a terminal */
+ char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+
+ double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - data_load_start);
+ double remaining_sec = ((double) total - current) * elapsed_sec / current;
+
+ /*
+ * If we want to stick with the original logging, print a message each
+ * 100k inserted rows.
+ */
+ if ((!use_quiet) && (current % 100000 == 0))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current one,
+ * add spaces to the current line to fully overwrite any remaining
+ * characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+ }
+ /* let's not call the timing for each row, but only each 100 rows */
+ else if (use_quiet && (current % 100 == 0))
+ {
+ /* have we reached the next interval (or end)? */
+ if ((current == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current
+ * one, add spaces to the current line to fully overwrite any
+ * remaining characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+
+ /* skip to the next interval */
+ log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
+ }
+ }
+
+ if (current == total && chars != 0 && eol != '\n')
+ fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
+}
+
static void
initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
initRowMethod init_row)
{
int n;
int64 k;
- int chars = 0;
- int prev_chars = 0;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
int64 total = base * scale;
- /* used to track elapsed time and estimate of the remaining time */
- pg_time_usec_t start;
- int log_interval = 1;
-
- /* Stay on the same line if reporting to a terminal */
- char eol = isatty(fileno(stderr)) ? '\r' : '\n';
-
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
@@ -5103,12 +5161,8 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- start = pg_time_now();
-
for (k = 0; k < total; k++)
{
- int64 j = k + 1;
-
init_row(&sql, k);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
@@ -5116,62 +5170,9 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
if (CancelRequested)
break;
- /*
- * If we want to stick with the original logging, print a message each
- * 100k inserted rows.
- */
- if ((!use_quiet) && (j % 100000 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
- }
- /* let's not call the timing for each row, but only each 100 rows */
- else if (use_quiet && (j % 100 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- /* have we reached the next interval (or end)? */
- if ((j == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
- {
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
-
- /* skip to the next interval */
- log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
- }
- }
+ showPopulateTableCopyProgress(table, k + 1, total);
}
- if (chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
if (PQputline(con, "\\.\n"))
pg_fatal("very last PQputline failed");
@@ -5448,6 +5449,11 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
init_row(con, i, base, columnCounter);
+
+ if (CancelRequested)
+ break;
+
+ showPopulateTableCopyProgress(table, i, base * scale);
}
res = PQgetResult(con);
@@ -5532,6 +5538,8 @@ initGenerateDataClientSide(PGconn *con)
fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+ data_load_start = pg_time_now();
+
switch (data_generation_type)
{
case INIT_STEP_GEN_TYPE_COPY_TEXT:
--
2.43.0
From afe3f69b23190f375b57d5cf0a3eae2cd8e40caa Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 23 Feb 2026 17:07:16 +0700
Subject: [PATCH v9 23/26] Adding multi-transaction mode to client-side data
generation
---
src/bin/pgbench/pgbench.c | 52 ++++++++++++--------
src/bin/pgbench/t/001_pgbench_with_server.pl | 47 +++++++++---------
2 files changed, 56 insertions(+), 43 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e3e0e073792..1b73da904ad 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5130,22 +5130,22 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
}
static void
-initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
+initPopulateTableCopyText(PGconn *con, const char *table, int counter, int64 base,
initRowMethod init_row)
{
int n;
- int64 k;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
- int64 total = base * scale;
+ int64 start = base * counter;
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
copy_statement_fmt = "copy %s from stdin with (freeze on)";
@@ -5161,16 +5161,16 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- for (k = 0; k < total; k++)
+ for (int64_t i = start; i < start + base; i++)
{
- init_row(&sql, k);
+ init_row(&sql, i);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
if (CancelRequested)
break;
- showPopulateTableCopyProgress(table, k + 1, total);
+ showPopulateTableCopyProgress(table, i, base * scale);
}
@@ -5193,9 +5193,6 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
- if (multi_xact)
- fprintf(stderr, "WARNING! Multiple transactions are not supported in this mode\n");
-
/*
* we do all of this in one transaction to enable the backend's
* data-loading optimizations
@@ -5205,15 +5202,28 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- /*
- * fill branches, tellers, accounts in that order in case foreign keys
- * already exist
- */
- initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ /*
+ * fill branches, tellers, accounts in that order in case foreign keys
+ * already exist
+ */
+ initPopulateTableCopyText(con, "pgbench_branches", i, nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", i, ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", i, naccounts, initAccount);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
}
@@ -5426,9 +5436,9 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
- if (!multi_xact)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index acb1e31d3e8..46569b0eb37 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -189,7 +189,7 @@ $node->pgbench(
check_data_state($node, 'server-side (unnest)');
-# Test server-side generation with COPY TEXT
+# Test client-side generation with COPY TEXT
$node->pgbench(
'--initialize --init-steps=dtg',
0,
@@ -200,45 +200,48 @@ $node->pgbench(
qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side TEXT');
+ 'pgbench --init-steps client-side TEXT (single XACT #1)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (text)');
$node->pgbench(
- '--initialize --init-steps=dtMg',
+ '--initialize --init-steps=dtSg',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_branches done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_tellers done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_accounts done},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side TEXT');
+ 'pgbench --init-steps client-side TEXT (single XACT #2)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (text)');
-
-# Test server-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtc',
+ '--initialize --init-steps=dtMg',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as single transaction},
+ qr{generating data \(client-side as multiple transactions},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side TEXT (multiple XACTs)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
-# Check data state, after server-side data generation.
-check_data_state($node, 'client-side (binary)');
+# Test client-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtSc',
+ '--initialize --init-steps=dtc',
0,
[qr{^$}],
[
@@ -247,24 +250,24 @@ $node->pgbench(
qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side BINARY (single XACT #1)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
$node->pgbench(
- '--initialize --init-steps=dtMc',
+ '--initialize --init-steps=dtSc',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side BINARY (single XACT #2)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
$node->pgbench(
@@ -279,7 +282,7 @@ $node->pgbench(
],
'pgbench --init-steps client-side BINARY');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
--
2.43.0
From d9c61efdd853c999b92061ad4f88dd012265dbba Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 18:24:18 +0700
Subject: [PATCH v9 24/26] Removing double call to checkInitSteps
---
src/bin/pgbench/pgbench.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1b73da904ad..7d3a1583152 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5816,9 +5816,9 @@ checkInitSteps(const char *initialize_steps)
}
if (data_init_type == 0)
- pg_log_error("WARNING! No data generation type is provided");
+ pg_log_warning("No data generation type is provided");
if (data_init_type > 1)
- pg_log_error("WARNING! More than one type of data initialization is requested");
+ pg_log_warning("More than one type of data initialization is requested");
}
/*
@@ -7449,7 +7449,6 @@ main(int argc, char **argv)
case 'I':
pg_free(initialize_steps);
initialize_steps = pg_strdup(optarg);
- checkInitSteps(initialize_steps);
initialization_option_set = true;
break;
case 'j': /* jobs */
--
2.43.0
From d593a2063a997c86de4b9350b97d4ad6527bddef Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 18:45:25 +0700
Subject: [PATCH v9 25/26] Fixed 2 small bugs
1. Overwrite of init mode by last mode in checkInitSteps
2. Bledding of init progress from previous mode under new one (eg, '-IggGG')
---
src/bin/pgbench/pgbench.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 7d3a1583152..daf33d73814 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5125,8 +5125,11 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
}
}
- if (current == total && chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
+ if (current + 1 == total && chars != 0)
+ {
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fputc(eol, stderr);
+ }
}
static void
@@ -5810,7 +5813,6 @@ checkInitSteps(const char *initialize_steps)
case INIT_STEP_GEN_TYPE_INSERT_SERIES:
case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
data_init_type++;
- data_generation_type = *step;
break;
}
}
@@ -5859,11 +5861,13 @@ runInitSteps(const char *initialize_steps)
case INIT_STEP_GEN_TYPE_COPY_TEXT:
case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
+ data_generation_type = *step;
initGenerateDataClientSide(con);
break;
case INIT_STEP_GEN_TYPE_INSERT_SERIES:
case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
+ data_generation_type = *step;
initGenerateDataServerSide(con);
break;
case INIT_STEP_GEN_TYPE_SINGLE_XACT:
--
2.43.0
From 265d4b017529c9bdc21ecdbf7fe4794789260545 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 19:06:45 +0700
Subject: [PATCH v9 26/26] Added more tests
---
src/bin/pgbench/t/001_pgbench_with_server.pl | 98 ++++++++++++++++++++
1 file changed, 98 insertions(+)
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 46569b0eb37..73d2d754273 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -172,6 +172,53 @@ $node->pgbench(
check_data_state($node, 'server-side');
+# Test server-side generation with generate_series
+$node->pgbench(
+ '--initialize --init-steps=dtG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series single XACT)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series multiple XACTs)');
+
+
# Test server-side generation with UNNEST
$node->pgbench(
'--initialize --init-steps=dtU',
@@ -188,6 +235,36 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side (unnest)');
+$node->pgbench(
+ '--initialize --init-steps=dtSU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
# Test client-side generation with COPY TEXT
$node->pgbench(
@@ -286,6 +363,27 @@ $node->pgbench(
check_data_state($node, 'client-side (binary)');
+# Check data state, after different modes of client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMccSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (multiple XACT modes)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary different XACT modes in list of --init-steps)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-02 14:12 ` Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Madyshev Egor @ 2026-03-02 14:12 UTC (permalink / raw)
To: Boris Mironov <[email protected]>; pgsql-hackers
Hi Boris,
I have reviewed the new patch. Overall, it looks correct,
but I have a few minor questions.
1. What do you think about moving characters in 'detail: Allowed step
characters are: "dtgMScGUvpf"' so that generation modes and
transactions count modes are not mixed? For example "dtMSgcGUvpf".
2. In the initCreateTables function, default values are set as empty
strings '' in the pgbench_history and pgbench_accounts tables. Was
this done intentionally, and if so, what is the reason? In the
pgbench_tellers and pgbench_branches tables, the implicit default
would be NULL - why was this logic changed?
3. In showPopulateTableCopyProgress, I think it would be better to
calculate elapsed_sec and remaining_sec inside the condition blocks,
as is done in the original code.
4. Do the changes and bug fixes in the patch affect performance? Are
the existing performance measurements still valid?
Best regards,
Egor
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
@ 2026-03-03 04:06 ` Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-03 04:06 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; pgsql-hackers
Hi Egor,
Thank you for your time reviewing this patch and all of its versions!
> 1. What do you think about moving characters in 'detail: Allowed step
> characters are: "dtgMScGUvpf"' so that generation modes and
> transactions count modes are not mixed? For example "dtMSgcGUvpf".
Done
> 2. In the initCreateTables function, default values are set as empty
> strings '' in the pgbench_history and pgbench_accounts tables. Was
> this done intentionally, and if so, what is the reason? In the
> pgbench_tellers and pgbench_branches tables, the implicit default
> would be NULL - why was this logic changed?
I personally would set default filler for every table since "TPC-B-like"
means that filler is not used as expected by actual requirement of
TPC-B for row length. There are 2 specific tests for not-NULL values
in those 2 tables that do not have "default values". I would prefer
to use similar approach everywhere and bring on requirement of TPC-B
for row length. Since TPC-B is kind of archaic now this isn't best
tactic to fight another battle against "well established test set".
> 3. In showPopulateTableCopyProgress, I think it would be better to
> calculate elapsed_sec and remaining_sec inside the condition blocks,
> as is done in the original code.
I believe original code with complete copy of both variables initialization
twice in the same proc doesn't bring any benefits. Hence shortened it.
> 4. Do the changes and bug fixes in the patch affect performance? Are
> the existing performance measurements still valid?
We didn't significantly alter logic since last benchmark. I'll try to find some
time to rerun all tests again and publish updated results.
Best regards,
Egor
Attachments:
[application/octet-stream] v10-pgbench-faster-modes.patch (125.2K, 3-v10-pgbench-faster-modes.patch)
download | inline diff:
From c768f399c556295de7d53895410e686d86b4b960 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 19:34:58 +0700
Subject: [PATCH v10 01/27] Converting one huge transaction into series of one
per 'scale'
---
src/bin/pgbench/pgbench.c | 61 ++++++++++++++++++++++++++-------------
1 file changed, 41 insertions(+), 20 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..284a7c860f1 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,6 +181,12 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * scaling factor after which we switch to multiple transactions during
+ * data population phase on server side
+ */
+static int64 single_txn_scale_limit = 1;
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5213,6 +5219,7 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
+ int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5225,30 +5232,44 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ executeStatement(con, "commit");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ for (int i = 0; i < scale; i += chunk) {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "select bid + 1, 0 "
+ "from generate_series(%d, %d) as bid", i, i + chunk);
+ //"select bid, 0 "
+ //"from generate_series(1, %d) as bid", nbranches * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid,bid,tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ //"select tid, (tid - 1) / %d + 1, 0 "
+ //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ //"select aid, (aid - 1) / %d + 1, 0, '' "
+ //"from generate_series(1, " INT64_FORMAT ") as aid",
+ //naccounts, (int64) naccounts * scale);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
termPQExpBuffer(&sql);
-
- executeStatement(con, "commit");
}
/*
--
2.43.0
From 0eddb156c187d829c4381bc928c5314705928852 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:13:23 +0700
Subject: [PATCH v10 02/27] Getting rid off limit for single transaction size
during data generation
---
src/bin/pgbench/pgbench.c | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 284a7c860f1..28b72e4cf1f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,12 +181,6 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
-/*
- * scaling factor after which we switch to multiple transactions during
- * data population phase on server side
- */
-static int64 single_txn_scale_limit = 1;
-
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5219,7 +5213,6 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
- int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5236,13 +5229,13 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i += chunk) {
+ for (int i = 0; i < scale; i++) {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + chunk);
+ "from generate_series(%d, %d) as bid", i, i + 1);
//"select bid, 0 "
//"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
@@ -5251,7 +5244,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_tellers(tid,bid,tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
- ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
//"select tid, (tid - 1) / %d + 1, 0 "
//"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
@@ -5260,7 +5253,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_accounts(aid,bid,abalance,filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
//"select aid, (aid - 1) / %d + 1, 0, '' "
//"from generate_series(1, " INT64_FORMAT ") as aid",
//naccounts, (int64) naccounts * scale);
--
2.43.0
From c5659cf474ec273c057668f30a4f435fd02f2da7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:38:36 +0700
Subject: [PATCH v10 03/27] No need to keep old code in comments
---
src/bin/pgbench/pgbench.c | 7 -------
1 file changed, 7 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 28b72e4cf1f..97895aa9edf 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5236,8 +5236,6 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
"from generate_series(%d, %d) as bid", i, i + 1);
- //"select bid, 0 "
- //"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5245,8 +5243,6 @@ initGenerateDataServerSide(PGconn *con)
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
- //"select tid, (tid - 1) / %d + 1, 0 "
- //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5254,9 +5250,6 @@ initGenerateDataServerSide(PGconn *con)
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
- //"select aid, (aid - 1) / %d + 1, 0, '' "
- //"from generate_series(1, " INT64_FORMAT ") as aid",
- //naccounts, (int64) naccounts * scale);
executeStatement(con, sql.data);
executeStatement(con, "commit");
--
2.43.0
From e47b52ddf23593dad9375ef5356fd41d0621ede3 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 19:06:48 +0700
Subject: [PATCH v10 04/27] Adding server-side data generation via unnest
---
src/bin/pgbench/pgbench.c | 199 ++++++++++++++++++++++++++++++++++----
1 file changed, 182 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97895aa9edf..65d77cdefea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,12 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +187,11 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ *
+ */
+static char data_generation_type = '?';
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -914,7 +925,9 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5203,18 +5216,16 @@ initGenerateDataClientSide(PGconn *con)
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
*/
static void
-initGenerateDataServerSide(PGconn *con)
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5225,31 +5236,136 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ initPQExpBuffer(&sql);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "select bid, 0 "
+ "from generate_series(1, %d)", scale * nbranches);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
+ executeStatement(con, sql.data);
+
executeStatement(con, "commit");
+ termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i++) {
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + 1);
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
+ "insert into pgbench_tellers(tid, bid, tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
executeStatement(con, "commit");
@@ -5258,6 +5374,32 @@ initGenerateDataServerSide(PGconn *con)
termPQExpBuffer(&sql);
}
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
+}
+
/*
* Invoke vacuum on the standard tables
*/
@@ -5341,6 +5483,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5352,7 +5496,26 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'G':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'i':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5395,6 +5558,8 @@ runInitSteps(const char *initialize_steps)
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
--
2.43.0
From 5e1827b889b283f50299ce6ab1a73f9f55a4a84f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 20:00:56 +0700
Subject: [PATCH v10 05/27] Fixing typo in query
---
src/bin/pgbench/pgbench.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 65d77cdefea..03e37df4434 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5241,7 +5241,8 @@ generateDataInsertSingleTXN(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d)", scale * nbranches);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
--
2.43.0
From 7ca86521fda6929b8e0de3fc77dcbb8984009c88 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 11 Nov 2025 19:39:45 +0700
Subject: [PATCH v10 06/27] Adding support for COPY BINARY mode
---
src/bin/pgbench/pgbench.c | 393 ++++++++++++++++++++++++++++++++++++--
1 file changed, 381 insertions(+), 12 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03e37df4434..71aa1d9479f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -176,6 +176,8 @@ typedef struct socket_set
/* 'one transaction per scale' server-side methods */
#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
@@ -188,10 +190,17 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
static int scale = 1;
/*
- *
+ * mode of data generation to use
*/
static char data_generation_type = '?';
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -861,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -925,6 +935,7 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
" G: generate data, server-side in single transaction\n"
" i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
" I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
@@ -5191,9 +5202,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5209,12 +5220,373 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
+/*
+ * Dumps binary buffer to file (purely for debugging)
+ */
+static void
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t total = base * scale;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = 0; i < total; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
* whole dataset in single transaction
@@ -5500,14 +5872,10 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
+ case 'g':
+ case 'C':
case 'G':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'i':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'I':
data_init_type++;
data_generation_type = *step;
@@ -5555,6 +5923,7 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
--
2.43.0
From 4aa0ac05765edf6b5f0c13e18ac677287ce78206 Mon Sep 17 00:00:00 2001
From: Fujii Masao <[email protected]>
Date: Fri, 14 Nov 2025 22:40:39 +0900
Subject: [PATCH v10 07/27] pgbench: Fix assertion failure with multiple
\syncpipeline in pipeline mode.
Previously, when pgbench ran a custom script that triggered retriable errors
(e.g., deadlocks) followed by multiple \syncpipeline commands in pipeline mode,
the following assertion failure could occur:
Assertion failed: (res == ((void*)0)), function discardUntilSync, file pgbench.c, line 3594.
The issue was that discardUntilSync() assumed a pipeline sync result
(PGRES_PIPELINE_SYNC) would always be followed by either another sync result
or NULL. This assumption was incorrect: when multiple sync requests were sent,
a sync result could instead be followed by another result type. In such cases,
discardUntilSync() mishandled the results, leading to the assertion failure.
This commit fixes the issue by making discardUntilSync() correctly handle cases
where a pipeline sync result is followed by other result types. It now continues
discarding results until another pipeline sync followed by NULL is reached.
Backpatched to v17, where support for \syncpipeline command in pgbench was
introduced.
Author: Yugo Nagata <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 17
---
src/bin/pgbench/pgbench.c | 39 ++++++++++++++++++++++++++++-----------
1 file changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..a425176ecdc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -3563,14 +3563,18 @@ doRetry(CState *st, pg_time_usec_t *now)
}
/*
- * Read results and discard it until a sync point.
+ * Read and discard results until the last sync point.
*/
static int
discardUntilSync(CState *st)
{
bool received_sync = false;
- /* send a sync */
+ /*
+ * Send a Sync message to ensure at least one PGRES_PIPELINE_SYNC is
+ * received and to avoid an infinite loop, since all earlier ones may have
+ * already been received.
+ */
if (!PQpipelineSync(st->con))
{
pg_log_error("client %d aborted: failed to send a pipeline sync",
@@ -3578,29 +3582,42 @@ discardUntilSync(CState *st)
return 0;
}
- /* receive PGRES_PIPELINE_SYNC and null following it */
+ /*
+ * Continue reading results until the last sync point, i.e., until
+ * reaching null just after PGRES_PIPELINE_SYNC.
+ */
for (;;)
{
PGresult *res = PQgetResult(st->con);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ PQclear(res);
+ return 0;
+ }
+
if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
received_sync = true;
- else if (received_sync)
+ else if (received_sync && res == NULL)
{
- /*
- * PGRES_PIPELINE_SYNC must be followed by another
- * PGRES_PIPELINE_SYNC or NULL; otherwise, assert failure.
- */
- Assert(res == NULL);
-
/*
* Reset ongoing sync count to 0 since all PGRES_PIPELINE_SYNC
* results have been discarded.
*/
st->num_syncs = 0;
- PQclear(res);
break;
}
+ else
+ {
+ /*
+ * If a PGRES_PIPELINE_SYNC is followed by something other than
+ * PGRES_PIPELINE_SYNC or NULL, another PGRES_PIPELINE_SYNC will
+ * appear later. Reset received_sync to false to wait for it.
+ */
+ received_sync = false;
+ }
PQclear(res);
}
--
2.43.0
From 9c4f19055597e9adb25e65c2aa8bedf20a09e13d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 19:05:58 +0700
Subject: [PATCH v10 08/27] Setting empty string as default value in filler
column
---
src/bin/pgbench/pgbench.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 967f6ce6984..03b5e5c28f0 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84)",
- "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88)",
- "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88) default ''",
1
}
};
--
2.43.0
From 2aabaa52dffdb78fbefaef95163881c15e18ef29 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 21 Nov 2025 15:03:11 +0200
Subject: [PATCH v10 09/27] Use strtoi64() in pgbench, replacing its open-coded
implementation
Makes the code a little simpler.
The old implementation accepted trailing whitespace, but that was
unnecessary. Firstly, its sibling function for parsing decimals,
strtodouble(), does not accept trailing whitespace. Secondly, none of
the callers can pass a string with trailing whitespace to it.
In the passing, check specifically for ERANGE before printing the "out
of range" error. On some systems, strtoul() and strtod() return EINVAL
on an empty or all-spaces string, and "invalid input syntax" is more
appropriate for that than "out of range". For the existing
strtodouble() function this is purely academical because it's never
called with errorOK==false, but let's be tidy. (Perhaps we should
remove the dead codepaths altogether, but I'll leave that for another
day.)
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Yuefei Shi <[email protected]>
Reviewed-by: Neil Chen <[email protected]>
Discussion: https://www.postgresql.org/message-id/[email protected]
---
src/bin/pgbench/pgbench.c | 83 +++++++++------------------------------
1 file changed, 19 insertions(+), 64 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a425176ecdc..68774a59efd 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -982,13 +982,17 @@ usage(void)
progname, progname, PACKAGE_BUGREPORT, PACKAGE_NAME, PACKAGE_URL);
}
-/* return whether str matches "^\s*[-+]?[0-9]+$" */
+/*
+ * Return whether str matches "^\s*[-+]?[0-9]+$"
+ *
+ * This should agree with strtoint64() on what's accepted, ignoring overflows.
+ */
static bool
is_an_int(const char *str)
{
const char *ptr = str;
- /* skip leading spaces; cast is consistent with strtoint64 */
+ /* skip leading spaces */
while (*ptr && isspace((unsigned char) *ptr))
ptr++;
@@ -1012,9 +1016,6 @@ is_an_int(const char *str)
/*
* strtoint64 -- convert a string to 64-bit integer
*
- * This function is a slightly modified version of pg_strtoint64() from
- * src/backend/utils/adt/numutils.c.
- *
* The function returns whether the conversion worked, and if so
* "*result" is set to the result.
*
@@ -1023,71 +1024,25 @@ is_an_int(const char *str)
bool
strtoint64(const char *str, bool errorOK, int64 *result)
{
- const char *ptr = str;
- int64 tmp = 0;
- bool neg = false;
-
- /*
- * Do our own scan, rather than relying on sscanf which might be broken
- * for long long.
- *
- * As INT64_MIN can't be stored as a positive 64 bit integer, accumulate
- * value as a negative number.
- */
-
- /* skip leading spaces */
- while (*ptr && isspace((unsigned char) *ptr))
- ptr++;
-
- /* handle sign */
- if (*ptr == '-')
- {
- ptr++;
- neg = true;
- }
- else if (*ptr == '+')
- ptr++;
+ char *end;
- /* require at least one digit */
- if (unlikely(!isdigit((unsigned char) *ptr)))
- goto invalid_syntax;
+ errno = 0;
+ *result = strtoi64(str, &end, 10);
- /* process digits */
- while (*ptr && isdigit((unsigned char) *ptr))
+ if (unlikely(errno == ERANGE))
{
- int8 digit = (*ptr++ - '0');
-
- if (unlikely(pg_mul_s64_overflow(tmp, 10, &tmp)) ||
- unlikely(pg_sub_s64_overflow(tmp, digit, &tmp)))
- goto out_of_range;
+ if (!errorOK)
+ pg_log_error("value \"%s\" is out of range for type bigint", str);
+ return false;
}
- /* allow trailing whitespace, but not other trailing chars */
- while (*ptr != '\0' && isspace((unsigned char) *ptr))
- ptr++;
-
- if (unlikely(*ptr != '\0'))
- goto invalid_syntax;
-
- if (!neg)
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
- if (unlikely(tmp == PG_INT64_MIN))
- goto out_of_range;
- tmp = -tmp;
+ if (!errorOK)
+ pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
+ return false;
}
-
- *result = tmp;
return true;
-
-out_of_range:
- if (!errorOK)
- pg_log_error("value \"%s\" is out of range for type bigint", str);
- return false;
-
-invalid_syntax:
- if (!errorOK)
- pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
- return false;
}
/* convert string to double, detecting overflows/underflows */
@@ -1099,14 +1054,14 @@ strtodouble(const char *str, bool errorOK, double *dv)
errno = 0;
*dv = strtod(str, &end);
- if (unlikely(errno != 0))
+ if (unlikely(errno == ERANGE))
{
if (!errorOK)
pg_log_error("value \"%s\" is out of range for type double", str);
return false;
}
- if (unlikely(end == str || *end != '\0'))
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
if (!errorOK)
pg_log_error("invalid input syntax for type double: \"%s\"", str);
--
2.43.0
From dcb85d26f8132eaaf9d096e814b9bda49db7d478 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 20:06:24 +0700
Subject: [PATCH v10 10/27] Switching COPY FROM BINARY ti run in multiple
transactions
---
src/bin/pgbench/pgbench.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03b5e5c28f0..6b89007a63b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5496,20 +5496,20 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
static void
initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int64_t base, initRowMethodBin init_row)
+ int counter, int64_t base, initRowMethodBin init_row)
{
int n;
PGresult *res;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t total = base * scale;
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5526,7 +5526,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyHeader(con);
- for (int64_t i = 0; i < total; i++)
+ for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
}
@@ -5573,15 +5573,20 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- naccounts, initAccountBinary);
-
executeStatement(con, "commit");
+ for (int i = 0; i < scale; i++)
+ {
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+ }
+
pg_free(bin_copy_buffer);
}
--
2.43.0
From b8e28881225234fd00b55235bc60fad2dc60b544 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sat, 22 Nov 2025 17:06:00 +0700
Subject: [PATCH v10 11/27] Adding tests for new modes of data generation
---
src/bin/pgbench/pgbench.c | 21 ++++----
src/bin/pgbench/t/001_pgbench_with_server.pl | 52 +++++++++++++++++---
2 files changed, 56 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6b89007a63b..dd4e5d5e056 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -164,7 +164,7 @@ typedef struct socket_set
#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
@@ -192,7 +192,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = '?';
+static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
/*
* COPY FROM BINARY execution buffer
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84)",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default ''",
- "aid bigint not null,bid int,abalance int,filler char(84) default ''",
+ "aid int not null,bid int,abalance int,filler char(84) default '?'",
+ "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88) default ''",
- "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88)",
1
}
};
@@ -7837,6 +7837,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 581e9af7907..a377048ead1 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -125,7 +130,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +148,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -164,6 +170,38 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtI',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtC',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From b3f2bce34232d299abc7644a8579f0ce49c8c9d6 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 23 Nov 2025 14:05:59 +0700
Subject: [PATCH v10 12/27] Fixing compiler warnings about unused procedures by
removing or commenting them out as they might be needed a bit later
---
src/bin/pgbench/pgbench.c | 31 +++++--------------------------
1 file changed, 5 insertions(+), 26 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 0a3ba21dcc9..682db61ff61 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5201,7 +5201,7 @@ initGenerateDataClientSideText(PGconn *con)
/*
* Dumps binary buffer to file (purely for debugging)
- */
+ *
static void
dumpBufferToFile(char *filename)
{
@@ -5226,6 +5226,7 @@ dumpBufferToFile(char *filename)
fclose(file_ptr);
}
+ */
/*
* Save char data to buffer
@@ -5271,37 +5272,14 @@ addColumnCounter(int16_t n)
/*
* adds column with NULL value
- */
+ *
static void
addNullColumn()
{
int32_t null = -1;
bufferData((void *) &null, sizeof(null));
}
-
-/*
- * adds column with int8 value
*/
-static void
-addInt8Column(int8_t value)
-{
- int8_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
-
-/*
- * adds column with int16 value
- */
-static void
-addInt16Column(int16_t value)
-{
- int16_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
/*
* adds column with inti32 value
@@ -5329,7 +5307,7 @@ addInt64Column(int64_t value)
/*
* adds column with char value
- */
+ *
static void
addCharColumn(char *value)
{
@@ -5337,6 +5315,7 @@ addCharColumn(char *value)
bufferData((void *) &size, sizeof(size));
bufferCharData(value, size);
}
+ */
/*
* Starts communication with server for COPY FROM BINARY statement
--
2.43.0
From 9ab7fe302ba7de40593e7cba8ea1ca3b876c1ea5 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 19:22:16 +0700
Subject: [PATCH v10 13/27] Moving PQclear call to the end of procedure to
avoid access to previously released memory
---
src/bin/pgbench/pgbench.c | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 682db61ff61..3687b65871e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5363,11 +5363,11 @@ static void
initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of branch table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 3 columns
*/
- int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
flushBuffer(con, res, max_row_len);
@@ -5384,9 +5384,9 @@ static void
initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of tellers table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 4 columns
*/
int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
@@ -5406,9 +5406,9 @@ static void
initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of accounts table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 4 columns
*/
int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
@@ -5455,11 +5455,11 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
if (PQresultStatus(res) != PGRES_COPY_IN)
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
- PQclear(res);
sendBinaryCopyHeader(con);
+
for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
@@ -5470,8 +5470,10 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
sendBinaryCopyTrailer(con);
+
if (PQresultStatus(res) == PGRES_COPY_IN)
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
@@ -5484,6 +5486,8 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
}
+
+ PQclear(res);
}
/*
@@ -5499,8 +5503,9 @@ initGenerateDataClientSideBinary(PGconn *con)
bin_copy_buffer_length = 0;
/*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
+ * we do all of this in multiple transactions
+ * to minimize load on DB server and perhaps
+ * in future allow load in parallel sessions
*/
executeStatement(con, "begin");
@@ -5511,6 +5516,8 @@ initGenerateDataClientSideBinary(PGconn *con)
for (int i = 0; i < scale; i++)
{
+ executeStatement(con, "begin");
+
initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary);
initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
--
2.43.0
From 3d7b2c53b3fa45b8848369bfbc49a0c7e5854304 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 19:47:26 +0700
Subject: [PATCH v10 14/27] Fixing error about freeing memory twice
---
src/bin/pgbench/pgbench.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3687b65871e..6f42438d970 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5462,6 +5462,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
+ res = PQgetResult(con);
init_row(con, res, i, base);
}
@@ -5473,7 +5474,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyTrailer(con);
-
+ res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
@@ -5481,7 +5482,6 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
res = PQgetResult(con);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
- PQclear(res);
}
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
--
2.43.0
From a71c90f556ba143335d9631ebbdf96b7120c6d1d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 20:24:54 +0700
Subject: [PATCH v10 15/27] Fixing memory leak during data flush in COPY BINARY
test
---
src/bin/pgbench/pgbench.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6f42438d970..8eca537b718 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -871,7 +871,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
+typedef void (*initRowMethodBin) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5345,13 +5345,18 @@ sendBinaryCopyTrailer(PGconn *con)
* Flashes current buffer over network if needed
*/
static void
-flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+flushBuffer(PGconn *con, int16_t row_len)
{
+ PGresult *res;
+
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
+ res = PQgetResult(con);
/* flush current buffer */
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ pg_fatal("It is NOT a COPY command that is currently running");
bin_copy_buffer_length = 0;
}
}
@@ -5360,7 +5365,7 @@ flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of branch table has following extra bytes:
@@ -5369,7 +5374,7 @@ initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(2);
@@ -5381,7 +5386,7 @@ initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of tellers table has following extra bytes:
@@ -5390,7 +5395,7 @@ initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(3);
@@ -5403,7 +5408,7 @@ initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of accounts table has following extra bytes:
@@ -5412,7 +5417,7 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(3);
@@ -5462,10 +5467,10 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- res = PQgetResult(con);
- init_row(con, res, i, base);
+ init_row(con, i, base);
}
+ res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
@@ -5474,6 +5479,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyTrailer(con);
+
res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
--
2.43.0
From a5933f40be0f3d408bb920f296fb5119c3ec28f4 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 30 Jan 2026 13:59:36 +0700
Subject: [PATCH v10 16/27] Fixing memory leak shown by valgrind
---
src/bin/pgbench/pgbench.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8eca537b718..466d9023d72 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5357,6 +5357,8 @@ flushBuffer(PGconn *con, int16_t row_len)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
pg_fatal("It is NOT a COPY command that is currently running");
+
+ PQclear(res);
bin_copy_buffer_length = 0;
}
}
@@ -5460,6 +5462,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
if (PQresultStatus(res) != PGRES_COPY_IN)
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
sendBinaryCopyHeader(con);
@@ -5475,6 +5478,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+ PQclear(res);
sendBinaryCopyTrailer(con);
@@ -5485,6 +5489,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
{
+ PQclear(res);
res = PQgetResult(con);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
@@ -5492,7 +5497,6 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
}
-
PQclear(res);
}
--
2.43.0
From 6a3036f898b050d5e71e70eaceccd63003aa6444 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 6 Feb 2026 13:53:07 +0700
Subject: [PATCH v10 17/27] Adding ability to switch data init between single
and multiple transactions
---
src/bin/pgbench/pgbench.c | 192 +++++++++----------
src/bin/pgbench/t/001_pgbench_with_server.pl | 93 ++++++++-
2 files changed, 173 insertions(+), 112 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 466d9023d72..af71e358e71 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,9 +159,8 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-
-#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
+#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
+#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,14 +170,17 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
-/* original single transaction server-side method */
-#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
-/* 'one transaction per scale' server-side methods */
-#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
-#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
-#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
-#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
-
+/* server-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT generate_series to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest to generate data */
+/* client-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN .. BINARY to generate data */
+/* data init pseudo steps */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as multiple transactions */
+
+static bool multi_xact = false; /* init data type (as single or multiple transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -192,7 +194,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
/*
* COPY FROM BINARY execution buffer
@@ -934,11 +936,13 @@ usage(void)
" run selected initialization steps, in the specified order\n"
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
- " g: generate data, client-side\n"
- " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
- " G: generate data, server-side in single transaction\n"
- " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
- " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
+ " to generate data, client-side:\n"
+ " g: COPY .. FROM STDIN .. TEXT\n"
+ " c: COPY .. FROM STDIN .. BINARY\n"
+ " to generate data, server-side:\n"
+ " G: INSERT .. SELECT generate_series\n"
+ " U: INSERT .. SELECT unnest\n"
+ " M: use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5049,8 +5053,8 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTable(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+initPopulateTableText(PGconn *con, const char *table, int64 base,
+ initRowMethod init_row)
{
int n;
int64 k;
@@ -5178,6 +5182,9 @@ initGenerateDataClientSideText(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
+ if (multi_xact)
+ fprintf(stderr, "WARNING! Multiple transactions are not supported in this mode\n");
+
/*
* we do all of this in one transaction to enable the backend's
* data-loading optimizations
@@ -5191,9 +5198,9 @@ initGenerateDataClientSideText(PGconn *con)
* fill branches, tellers, accounts in that order in case foreign keys
* already exist
*/
- initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
+ initPopulateTableText(con, "pgbench_branches", nbranches, initBranch);
+ initPopulateTableText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
@@ -5450,7 +5457,8 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
+ if (!multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5522,11 +5530,13 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int i = 0; i < scale; i++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary);
@@ -5535,9 +5545,13 @@ initGenerateDataClientSideBinary(PGconn *con)
initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
i, naccounts, initAccountBinary);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
pg_free(bin_copy_buffer);
}
@@ -5547,14 +5561,15 @@ initGenerateDataClientSideBinary(PGconn *con)
static void
initGenerateDataClientSide(PGconn *con)
{
- fprintf(stderr, "generating data (client-side) in ");
+ fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
switch (data_generation_type)
{
- case GEN_TYPE_COPY_ORIGINAL:
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
initGenerateDataClientSideText(con);
break;
- case GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
initGenerateDataClientSideBinary(con);
break;
}
@@ -5562,58 +5577,7 @@ initGenerateDataClientSide(PGconn *con)
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
- * whole dataset in single transaction
- */
-static void
-generateDataInsertSingleTXN(PGconn *con)
-{
- PQExpBufferData sql;
-
- fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
-
-
- /*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
- */
- executeStatement(con, "begin");
-
- /* truncate away any old data */
- initTruncateTables(con);
-
- initPQExpBuffer(&sql);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid, bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid",
- scale * nbranches);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid, bid, tbalance) "
- "select tid + 1, tid / %d + 1, 0 "
- "from generate_series(0, %d) as tid",
- ntellers, (scale * ntellers) - 1);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid, bid, abalance, "
- "filler) "
- "select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(0, " INT64_FORMAT ") as aid",
- naccounts, (int64) (scale * naccounts) - 1);
- executeStatement(con, sql.data);
-
- executeStatement(con, "commit");
-
- termPQExpBuffer(&sql);
-}
-
-
-/*
- * Generating data via INSERT .. SELECT .. FROM generate_series
- * One transaction per 'scale'
+ * Possibly as "One transaction per scale" in multi-transaction mode
*/
static void
generateDataInsertSeries(PGconn *con)
@@ -5629,11 +5593,13 @@ generateDataInsertSeries(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int i = 0; i < scale; i++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
@@ -5657,15 +5623,19 @@ generateDataInsertSeries(PGconn *con)
(int64) (i + 1) * naccounts - 1);
executeStatement(con, sql.data);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
}
/*
* Generating data via INSERT .. SELECT .. FROM unnest
- * One transaction per 'scale'
+ * Possibly as "One transaction per scale" in multi-tansaction mode
*/
static void
generateDataInsertUnnest(PGconn *con)
@@ -5681,11 +5651,13 @@ generateDataInsertUnnest(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int s = 0; s < scale; s++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
@@ -5714,14 +5686,18 @@ generateDataInsertUnnest(PGconn *con)
(int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
}
/*
- * Fill the standard tables with some data generated on the server
+ * Fill the standard tables with some data generated on the server side
*
* As already the case with the client-side data generation, the filler
* column defaults to NULL in pgbench_branches and pgbench_tellers,
@@ -5730,17 +5706,15 @@ generateDataInsertUnnest(PGconn *con)
static void
initGenerateDataServerSide(PGconn *con)
{
- fprintf(stderr, "generating data (server-side) ");
+ fprintf(stderr, "generating data (server-side as %s transaction%s) ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
switch (data_generation_type)
{
- case GEN_TYPE_INSERT_ORIGINAL:
- generateDataInsertSingleTXN(con);
- break;
- case GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
generateDataInsertSeries(con);
break;
- case GEN_TYPE_INSERT_UNNEST:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
generateDataInsertUnnest(con);
break;
}
@@ -5845,19 +5819,20 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
- case 'g':
- case 'C':
- case 'G':
- case 'i':
- case 'I':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
data_init_type++;
data_generation_type = *step;
break;
}
}
+ if (data_init_type == 0)
+ pg_log_error("WARNING! No data generation type is provided");
if (data_init_type > 1)
- pg_log_error("WARNING! More than one type of server-side data generation is requested");
+ pg_log_error("WARNING! More than one type of data initialization is requested");
}
/*
@@ -5895,17 +5870,22 @@ runInitSteps(const char *initialize_steps)
op = "create tables";
initCreateTables(con);
break;
- case 'g':
- case 'C':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
initGenerateDataClientSide(con);
break;
- case 'G':
- case 'i':
- case 'I':
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
initGenerateDataServerSide(con);
break;
+ case INIT_STEP_GEN_TYPE_SINGLE_XACT:
+ multi_xact = false;
+ break;
+ case INIT_STEP_GEN_TYPE_MULTI_XACT:
+ multi_xact = true;
+ break;
case 'v':
op = "vacuum";
initVacuum(con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index a377048ead1..acb1e31d3e8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -117,6 +117,7 @@ $node->pgbench(
[qr{Perhaps you need to do initialization}],
'run without init');
+
# Initialize pgbench tables scale 1
$node->pgbench(
'-i', 0,
@@ -160,7 +161,7 @@ $node->pgbench(
qr{creating tables},
qr{creating 3 partitions},
qr{creating primary keys},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{creating foreign keys},
qr{(?!vacuuming)}, # no vacuum
qr{done in \d+\.\d\d s }
@@ -170,15 +171,16 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+
# Test server-side generation with UNNEST
$node->pgbench(
- '--initialize --init-steps=dtI',
+ '--initialize --init-steps=dtU',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{done in \d+\.\d\d s }
],
'pgbench --init-steps server-side UNNEST');
@@ -186,15 +188,48 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side (unnest)');
-# Test server-side generation with UNNEST
+
+# Test server-side generation with COPY TEXT
+$node->pgbench(
+ '--initialize --init-steps=dtg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (text)');
+
+
+# Test server-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtC',
+ '--initialize --init-steps=dtc',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side\)},
+ qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
'pgbench --init-steps client-side BINARY');
@@ -202,6 +237,52 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'client-side (binary)');
+$node->pgbench(
+ '--initialize --init-steps=dtSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From 2471bb47bf715402ece61b084b69afe3bb53742e Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 6 Feb 2026 13:55:47 +0700
Subject: [PATCH v10 18/27] Removing commented out procedures
---
src/bin/pgbench/pgbench.c | 52 ---------------------------------------
1 file changed, 52 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index af71e358e71..34464b7037c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5206,35 +5206,6 @@ initGenerateDataClientSideText(PGconn *con)
}
-/*
- * Dumps binary buffer to file (purely for debugging)
- *
-static void
-dumpBufferToFile(char *filename)
-{
- FILE *file_ptr;
- size_t bytes_written;
-
- file_ptr = fopen(filename, "wb");
- if (file_ptr == NULL)
- {
- fprintf(stderr, "Error opening file %s\n", filename);
- return; // EXIT_FAILURE;
- }
-
- bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
-
- if (bytes_written != bin_copy_buffer_length)
- {
- fprintf(stderr, "Error writing to file or incomplete write\n");
- fclose(file_ptr);
- return; // EXIT_FAILURE;
- }
-
- fclose(file_ptr);
-}
- */
-
/*
* Save char data to buffer
*/
@@ -5277,17 +5248,6 @@ addColumnCounter(int16_t n)
bufferData((void *) &n, sizeof(n));
}
-/*
- * adds column with NULL value
- *
-static void
-addNullColumn()
-{
- int32_t null = -1;
- bufferData((void *) &null, sizeof(null));
-}
- */
-
/*
* adds column with inti32 value
*/
@@ -5312,18 +5272,6 @@ addInt64Column(int64_t value)
bufferData((void *) &data, sizeof(data));
}
-/*
- * adds column with char value
- *
-static void
-addCharColumn(char *value)
-{
- int32_t size = strlen(value);
- bufferData((void *) &size, sizeof(size));
- bufferCharData(value, size);
-}
- */
-
/*
* Starts communication with server for COPY FROM BINARY statement
*/
--
2.43.0
From 10dfa622b09c420d79da8774c9321f91540c8600 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 09:08:17 +0700
Subject: [PATCH v10 19/27] Small changes due to big code review
---
src/bin/pgbench/pgbench.c | 71 +++++++++++++++++++++------------------
1 file changed, 39 insertions(+), 32 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 34464b7037c..9f3d7f99a25 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -873,7 +873,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBin) (PGconn *con, int64_t curr, int32_t parent);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -942,7 +942,8 @@ usage(void)
" to generate data, server-side:\n"
" G: INSERT .. SELECT generate_series\n"
" U: INSERT .. SELECT unnest\n"
- " M: use multiple transactions to initialize data\n"
+ " S: flag to use single transaction to initialize data\n"
+ " M: flag to use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5053,8 +5054,8 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTableText(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
+ initRowMethod init_row)
{
int n;
int64 k;
@@ -5178,7 +5179,7 @@ initPopulateTableText(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSideText(PGconn *con)
+initGenerateDataClientSideTextFrmt(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
@@ -5198,9 +5199,9 @@ initGenerateDataClientSideText(PGconn *con)
* fill branches, tellers, accounts in that order in case foreign keys
* already exist
*/
- initPopulateTableText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableText(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTableText(con, "pgbench_accounts", naccounts, initAccount);
+ initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
@@ -5212,6 +5213,8 @@ initGenerateDataClientSideText(PGconn *con)
static void
bufferCharData(char *src, int32_t len)
{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
bin_copy_buffer_length += len;
}
@@ -5227,13 +5230,16 @@ bufferData(void *src, int32_t len)
bufferCharData(src, len);
#else
if (len == 1)
+ {
bufferCharData(src, len);
- else
- for (int32_t i = 0; i < len; i++)
- {
- ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
- ((char *) src)[len - i - 1];
- }
+ return;
+ }
+
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
bin_copy_buffer_length += len;
#endif
@@ -5322,7 +5328,7 @@ flushBuffer(PGconn *con, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of branch table has following extra bytes:
@@ -5333,7 +5339,7 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(2);
+ addColumnCounter(columnCounter);
addInt32Column(curr + 1);
addInt32Column(0);
@@ -5343,7 +5349,7 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of tellers table has following extra bytes:
@@ -5354,7 +5360,7 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(3);
+ addColumnCounter(columnCounter);
addInt32Column(curr + 1);
addInt32Column(curr / parent + 1);
@@ -5365,7 +5371,7 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of accounts table has following extra bytes:
@@ -5376,7 +5382,7 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(3);
+ addColumnCounter(columnCounter);
if (scale <= SCALE_32BIT_THRESHOLD)
addInt32Column(curr + 1);
@@ -5391,8 +5397,9 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
* Universal wrapper for sending data in binary format
*/
static void
-initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int counter, int64_t base, initRowMethodBin init_row)
+initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBinary init_row,
+ int columnCounter)
{
int n;
PGresult *res;
@@ -5426,7 +5433,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- init_row(con, i, base);
+ init_row(con, i, base, columnCounter);
}
res = PQgetResult(con);
@@ -5460,7 +5467,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
* Wrapper for binary data load
*/
static void
-initGenerateDataClientSideBinary(PGconn *con)
+initGenerateDataClientSideBinaryFrmt(PGconn *con)
{
fprintf(stderr, "BINARY mode...\n");
@@ -5486,12 +5493,12 @@ initGenerateDataClientSideBinary(PGconn *con)
if (multi_xact)
executeStatement(con, "begin");
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- i, nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- i, naccounts, initAccountBinary);
+ initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary, 2);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary, 3);
+ initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary, 3);
if (multi_xact)
executeStatement(con, "commit");
@@ -5515,10 +5522,10 @@ initGenerateDataClientSide(PGconn *con)
switch (data_generation_type)
{
case INIT_STEP_GEN_TYPE_COPY_TEXT:
- initGenerateDataClientSideText(con);
+ initGenerateDataClientSideTextFrmt(con);
break;
case INIT_STEP_GEN_TYPE_COPY_BINARY:
- initGenerateDataClientSideBinary(con);
+ initGenerateDataClientSideBinaryFrmt(con);
break;
}
}
--
2.43.0
From d95c53d412ff0e94e162eaf3990233fae007ae3f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 10:07:13 +0700
Subject: [PATCH v10 20/27] Running indent to fix code formatting
---
src/bin/pgbench/pgbench.c | 111 +++++++++++++++++++++-----------------
1 file changed, 62 insertions(+), 49 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 9f3d7f99a25..8198f1d4c9a 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,7 +159,7 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
+#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
@@ -171,16 +171,24 @@ typedef struct socket_set
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
/* server-side methods to generate data */
-#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT generate_series to generate data */
-#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT
+ * generate_series to generate
+ * data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest
+ * to generate data */
/* client-side methods to generate data */
-#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
-#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN .. BINARY to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN ..
+ * TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN ..
+ * BINARY to generate data */
/* data init pseudo steps */
-#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as single transaction */
-#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as multiple transactions */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as
+ * single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as
+ * multiple transactions */
-static bool multi_xact = false; /* init data type (as single or multiple transactions) */
+static bool multi_xact = false; /* init data type (as single or multiple
+ * transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -194,14 +202,15 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
/*
* COPY FROM BINARY execution buffer
*/
-#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
-static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
-static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM
+ * BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
@@ -872,8 +881,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5200,7 +5209,7 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
* already exist
*/
initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
@@ -5260,8 +5269,9 @@ addColumnCounter(int16_t n)
static void
addInt32Column(int32_t value)
{
- int32_t data = value;
- int32_t size = sizeof(data);
+ int32_t data = value;
+ int32_t size = sizeof(data);
+
bufferData((void *) &size, sizeof(size));
bufferData((void *) &data, sizeof(data));
}
@@ -5272,8 +5282,9 @@ addInt32Column(int32_t value)
static void
addInt64Column(int64_t value)
{
- int64_t data = value;
- int32_t size = sizeof(data);
+ int64_t data = value;
+ int32_t size = sizeof(data);
+
bufferData((void *) &size, sizeof(size));
bufferData((void *) &data, sizeof(data));
}
@@ -5284,9 +5295,9 @@ addInt64Column(int64_t value)
static void
sendBinaryCopyHeader(PGconn *con)
{
- char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
- '\0','\0','\0','\0',
- '\0','\0','\0','\0' };
+ char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ '\0', '\0', '\0', '\0',
+ '\0', '\0', '\0', '\0'};
PQputCopyData(con, header, 19);
}
@@ -5297,7 +5308,7 @@ sendBinaryCopyHeader(PGconn *con)
static void
sendBinaryCopyTrailer(PGconn *con)
{
- static char trailer[] = { 0xFF, 0xFF };
+ static char trailer[] = {0xFF, 0xFF};
PQputCopyData(con, trailer, 2);
}
@@ -5308,7 +5319,7 @@ sendBinaryCopyTrailer(PGconn *con)
static void
flushBuffer(PGconn *con, int16_t row_len)
{
- PGresult *res;
+ PGresult *res;
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
@@ -5330,12 +5341,13 @@ flushBuffer(PGconn *con, int16_t row_len)
static void
initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of branch table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 3 columns
+ *---
*/
- int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
+ int16_t max_row_len = 35 + 2 + 4 * 3; /* max row size is 35 */
flushBuffer(con, max_row_len);
@@ -5351,12 +5363,13 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter
static void
initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of tellers table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 4 columns
+ *---
*/
- int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+ int16_t max_row_len = 40 + 2 + 4 * 4; /* max row size is 40 */
flushBuffer(con, max_row_len);
@@ -5373,12 +5386,13 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter
static void
initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of accounts table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 4 columns
*/
- int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+ int16_t max_row_len = 250 + 2 + 4 * 4; /* max row size is 250 for
+ * int64 */
flushBuffer(con, max_row_len);
@@ -5401,11 +5415,11 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
int counter, int64_t base, initRowMethodBinary init_row,
int columnCounter)
{
- int n;
- PGresult *res;
- char copy_statement[256];
- const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t start = base * counter;
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
@@ -5450,7 +5464,7 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
- if (PQputCopyEnd(con, NULL) == 1) /* success */
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
{
PQclear(res);
res = PQgetResult(con);
@@ -5476,9 +5490,8 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
bin_copy_buffer_length = 0;
/*
- * we do all of this in multiple transactions
- * to minimize load on DB server and perhaps
- * in future allow load in parallel sessions
+ * we do all of this in multiple transactions to minimize load on DB
+ * server and perhaps in future allow load in parallel sessions
*/
executeStatement(con, "begin");
@@ -5495,8 +5508,8 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary, 2);
- initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary, 3);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary, 3);
initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
i, naccounts, initAccountBinary, 3);
@@ -5570,10 +5583,10 @@ generateDataInsertSeries(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_accounts(aid, bid, abalance, "
- "filler) "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", "
- INT64_FORMAT ") as aid",
+ INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts,
(int64) (i + 1) * naccounts - 1);
executeStatement(con, sql.data);
@@ -5622,7 +5635,7 @@ generateDataInsertUnnest(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_tellers(tid, bid, tbalance) "
"select unnest(array_agg(s.i order by s.i)) as tid, "
- "%d as bid, 0 as tbalance "
+ "%d as bid, 0 as tbalance "
"from generate_series(%d, %d) as s(i)",
s + 1, s * ntellers + 1, (s + 1) * ntellers);
executeStatement(con, sql.data);
@@ -5630,13 +5643,13 @@ generateDataInsertUnnest(PGconn *con)
printfPQExpBuffer(&sql,
"with data as ("
" select generate_series(" INT64_FORMAT ", "
- INT64_FORMAT ") as i) "
+ INT64_FORMAT ") as i) "
"insert into pgbench_accounts(aid, bid, "
- "abalance, filler) "
+ "abalance, filler) "
"select unnest(aid), unnest(bid), 0 as abalance, "
- "'' as filler "
+ "'' as filler "
"from (select array_agg(i+1) aid, "
- "array_agg(i/%d + 1) bid from data)",
+ "array_agg(i/%d + 1) bid from data)",
(int64) s * naccounts + 1,
(int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
@@ -5758,7 +5771,7 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
- char data_init_type = 0;
+ char data_init_type = 0;
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
--
2.43.0
From 270d2bf14b6873ff66fc37ca0a5dccc055dab1f7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 14:57:18 +0700
Subject: [PATCH v10 21/27] Removing debugging filler value
---
src/bin/pgbench/pgbench.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8198f1d4c9a..710b2a5b70d 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4954,8 +4954,8 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
@@ -4966,8 +4966,8 @@ initCreateTables(PGconn *con)
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default '?'",
- "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
--
2.43.0
From 86a111c63e7006d4801ce7bb594a5b77d87b42b0 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 23 Feb 2026 16:15:06 +0700
Subject: [PATCH v10 22/27] Moving out logic showing data load progress into
own procedure
---
src/bin/pgbench/pgbench.c | 142 ++++++++++++++++++++------------------
1 file changed, 75 insertions(+), 67 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 710b2a5b70d..e3e0e073792 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -487,6 +487,9 @@ typedef struct StatsData
*/
static pg_time_usec_t epoch_shift;
+/* used to track elapsed time and estimate of the remaining time of data load */
+static pg_time_usec_t data_load_start;
+
/*
* Error status for errors during script execution.
*/
@@ -5062,27 +5065,82 @@ initAccount(PQExpBufferData *sql, int64 curr)
curr + 1, curr / naccounts + 1);
}
+static void
+showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
+{
+ static int chars = 0;
+ static int prev_chars = 0;
+ static int log_interval = 1;
+
+ /* Stay on the same line if reporting to a terminal */
+ char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+
+ double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - data_load_start);
+ double remaining_sec = ((double) total - current) * elapsed_sec / current;
+
+ /*
+ * If we want to stick with the original logging, print a message each
+ * 100k inserted rows.
+ */
+ if ((!use_quiet) && (current % 100000 == 0))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current one,
+ * add spaces to the current line to fully overwrite any remaining
+ * characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+ }
+ /* let's not call the timing for each row, but only each 100 rows */
+ else if (use_quiet && (current % 100 == 0))
+ {
+ /* have we reached the next interval (or end)? */
+ if ((current == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current
+ * one, add spaces to the current line to fully overwrite any
+ * remaining characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+
+ /* skip to the next interval */
+ log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
+ }
+ }
+
+ if (current == total && chars != 0 && eol != '\n')
+ fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
+}
+
static void
initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
initRowMethod init_row)
{
int n;
int64 k;
- int chars = 0;
- int prev_chars = 0;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
int64 total = base * scale;
- /* used to track elapsed time and estimate of the remaining time */
- pg_time_usec_t start;
- int log_interval = 1;
-
- /* Stay on the same line if reporting to a terminal */
- char eol = isatty(fileno(stderr)) ? '\r' : '\n';
-
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
@@ -5103,12 +5161,8 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- start = pg_time_now();
-
for (k = 0; k < total; k++)
{
- int64 j = k + 1;
-
init_row(&sql, k);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
@@ -5116,62 +5170,9 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
if (CancelRequested)
break;
- /*
- * If we want to stick with the original logging, print a message each
- * 100k inserted rows.
- */
- if ((!use_quiet) && (j % 100000 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
- }
- /* let's not call the timing for each row, but only each 100 rows */
- else if (use_quiet && (j % 100 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- /* have we reached the next interval (or end)? */
- if ((j == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
- {
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
-
- /* skip to the next interval */
- log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
- }
- }
+ showPopulateTableCopyProgress(table, k + 1, total);
}
- if (chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
if (PQputline(con, "\\.\n"))
pg_fatal("very last PQputline failed");
@@ -5448,6 +5449,11 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
init_row(con, i, base, columnCounter);
+
+ if (CancelRequested)
+ break;
+
+ showPopulateTableCopyProgress(table, i, base * scale);
}
res = PQgetResult(con);
@@ -5532,6 +5538,8 @@ initGenerateDataClientSide(PGconn *con)
fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+ data_load_start = pg_time_now();
+
switch (data_generation_type)
{
case INIT_STEP_GEN_TYPE_COPY_TEXT:
--
2.43.0
From afe3f69b23190f375b57d5cf0a3eae2cd8e40caa Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 23 Feb 2026 17:07:16 +0700
Subject: [PATCH v10 23/27] Adding multi-transaction mode to client-side data
generation
---
src/bin/pgbench/pgbench.c | 52 ++++++++++++--------
src/bin/pgbench/t/001_pgbench_with_server.pl | 47 +++++++++---------
2 files changed, 56 insertions(+), 43 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e3e0e073792..1b73da904ad 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5130,22 +5130,22 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
}
static void
-initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
+initPopulateTableCopyText(PGconn *con, const char *table, int counter, int64 base,
initRowMethod init_row)
{
int n;
- int64 k;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
- int64 total = base * scale;
+ int64 start = base * counter;
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
copy_statement_fmt = "copy %s from stdin with (freeze on)";
@@ -5161,16 +5161,16 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- for (k = 0; k < total; k++)
+ for (int64_t i = start; i < start + base; i++)
{
- init_row(&sql, k);
+ init_row(&sql, i);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
if (CancelRequested)
break;
- showPopulateTableCopyProgress(table, k + 1, total);
+ showPopulateTableCopyProgress(table, i, base * scale);
}
@@ -5193,9 +5193,6 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
- if (multi_xact)
- fprintf(stderr, "WARNING! Multiple transactions are not supported in this mode\n");
-
/*
* we do all of this in one transaction to enable the backend's
* data-loading optimizations
@@ -5205,15 +5202,28 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- /*
- * fill branches, tellers, accounts in that order in case foreign keys
- * already exist
- */
- initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ /*
+ * fill branches, tellers, accounts in that order in case foreign keys
+ * already exist
+ */
+ initPopulateTableCopyText(con, "pgbench_branches", i, nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", i, ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", i, naccounts, initAccount);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
}
@@ -5426,9 +5436,9 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
- if (!multi_xact)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index acb1e31d3e8..46569b0eb37 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -189,7 +189,7 @@ $node->pgbench(
check_data_state($node, 'server-side (unnest)');
-# Test server-side generation with COPY TEXT
+# Test client-side generation with COPY TEXT
$node->pgbench(
'--initialize --init-steps=dtg',
0,
@@ -200,45 +200,48 @@ $node->pgbench(
qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side TEXT');
+ 'pgbench --init-steps client-side TEXT (single XACT #1)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (text)');
$node->pgbench(
- '--initialize --init-steps=dtMg',
+ '--initialize --init-steps=dtSg',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_branches done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_tellers done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_accounts done},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side TEXT');
+ 'pgbench --init-steps client-side TEXT (single XACT #2)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (text)');
-
-# Test server-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtc',
+ '--initialize --init-steps=dtMg',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as single transaction},
+ qr{generating data \(client-side as multiple transactions},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side TEXT (multiple XACTs)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
-# Check data state, after server-side data generation.
-check_data_state($node, 'client-side (binary)');
+# Test client-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtSc',
+ '--initialize --init-steps=dtc',
0,
[qr{^$}],
[
@@ -247,24 +250,24 @@ $node->pgbench(
qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side BINARY (single XACT #1)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
$node->pgbench(
- '--initialize --init-steps=dtMc',
+ '--initialize --init-steps=dtSc',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side BINARY (single XACT #2)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
$node->pgbench(
@@ -279,7 +282,7 @@ $node->pgbench(
],
'pgbench --init-steps client-side BINARY');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
--
2.43.0
From d9c61efdd853c999b92061ad4f88dd012265dbba Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 18:24:18 +0700
Subject: [PATCH v10 24/27] Removing double call to checkInitSteps
---
src/bin/pgbench/pgbench.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1b73da904ad..7d3a1583152 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5816,9 +5816,9 @@ checkInitSteps(const char *initialize_steps)
}
if (data_init_type == 0)
- pg_log_error("WARNING! No data generation type is provided");
+ pg_log_warning("No data generation type is provided");
if (data_init_type > 1)
- pg_log_error("WARNING! More than one type of data initialization is requested");
+ pg_log_warning("More than one type of data initialization is requested");
}
/*
@@ -7449,7 +7449,6 @@ main(int argc, char **argv)
case 'I':
pg_free(initialize_steps);
initialize_steps = pg_strdup(optarg);
- checkInitSteps(initialize_steps);
initialization_option_set = true;
break;
case 'j': /* jobs */
--
2.43.0
From d593a2063a997c86de4b9350b97d4ad6527bddef Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 18:45:25 +0700
Subject: [PATCH v10 25/27] Fixed 2 small bugs
1. Overwrite of init mode by last mode in checkInitSteps
2. Bledding of init progress from previous mode under new one (eg, '-IggGG')
---
src/bin/pgbench/pgbench.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 7d3a1583152..daf33d73814 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5125,8 +5125,11 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
}
}
- if (current == total && chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
+ if (current + 1 == total && chars != 0)
+ {
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fputc(eol, stderr);
+ }
}
static void
@@ -5810,7 +5813,6 @@ checkInitSteps(const char *initialize_steps)
case INIT_STEP_GEN_TYPE_INSERT_SERIES:
case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
data_init_type++;
- data_generation_type = *step;
break;
}
}
@@ -5859,11 +5861,13 @@ runInitSteps(const char *initialize_steps)
case INIT_STEP_GEN_TYPE_COPY_TEXT:
case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
+ data_generation_type = *step;
initGenerateDataClientSide(con);
break;
case INIT_STEP_GEN_TYPE_INSERT_SERIES:
case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
+ data_generation_type = *step;
initGenerateDataServerSide(con);
break;
case INIT_STEP_GEN_TYPE_SINGLE_XACT:
--
2.43.0
From 265d4b017529c9bdc21ecdbf7fe4794789260545 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 19:06:45 +0700
Subject: [PATCH v10 26/27] Added more tests
---
src/bin/pgbench/t/001_pgbench_with_server.pl | 98 ++++++++++++++++++++
1 file changed, 98 insertions(+)
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 46569b0eb37..73d2d754273 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -172,6 +172,53 @@ $node->pgbench(
check_data_state($node, 'server-side');
+# Test server-side generation with generate_series
+$node->pgbench(
+ '--initialize --init-steps=dtG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series single XACT)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series multiple XACTs)');
+
+
# Test server-side generation with UNNEST
$node->pgbench(
'--initialize --init-steps=dtU',
@@ -188,6 +235,36 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side (unnest)');
+$node->pgbench(
+ '--initialize --init-steps=dtSU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
# Test client-side generation with COPY TEXT
$node->pgbench(
@@ -286,6 +363,27 @@ $node->pgbench(
check_data_state($node, 'client-side (binary)');
+# Check data state, after different modes of client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMccSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (multiple XACT modes)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary different XACT modes in list of --init-steps)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From 3f569dfba37c8722fb5e2b99f3e001a93a959511 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 3 Mar 2026 10:49:54 +0700
Subject: [PATCH v10 27/27] Small changes due to review of patch v9
---
src/bin/pgbench/pgbench.c | 26 +++++++++++++++-----------
1 file changed, 15 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index daf33d73814..81576d8f7ca 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -160,7 +160,7 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtMScgGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -5232,6 +5232,8 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
/*
* Save char data to buffer
+ * Kept as separate proc for possible addition of something
+ * like addCharColumn in future
*/
static void
bufferCharData(char *src, int32_t len)
@@ -5249,22 +5251,24 @@ bufferCharData(char *src, int32_t len)
static void
bufferData(void *src, int32_t len)
{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
#ifdef __sparc__
bufferCharData(src, len);
#else
+
if (len == 1)
- {
bufferCharData(src, len);
- return;
- }
-
- for (int32_t i = 0; i < len; i++)
+ else
{
- ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
- ((char *) src)[len - i - 1];
- }
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
- bin_copy_buffer_length += len;
+ bin_copy_buffer_length += len;
+ }
#endif
}
@@ -5309,7 +5313,7 @@ addInt64Column(int64_t value)
static void
sendBinaryCopyHeader(PGconn *con)
{
- char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
'\0', '\0', '\0', '\0',
'\0', '\0', '\0', '\0'};
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-03 11:13 ` Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-03 11:13 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; pgsql-hackers
Hi Egor,
Sorry for so many extra changes. I promise, this one is good.
I believe it will beneficial to explain how buffer size for each row in COPY BINARY is actually calculated. Instead of hard coded values like 35, 40, 250, I put in comments actual sizes of each field as well as easy to read formulas. IMHO it adds extra value and resolves possible future questions right away.
I greatly appreciate your time spent on this patch.
Best regards,
Boris
Attachments:
[application/octet-stream] v11-pgbench-faster-modes.patch (133.2K, 3-v11-pgbench-faster-modes.patch)
download | inline diff:
From c768f399c556295de7d53895410e686d86b4b960 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 19:34:58 +0700
Subject: [PATCH v11 01/28] Converting one huge transaction into series of one
per 'scale'
---
src/bin/pgbench/pgbench.c | 61 ++++++++++++++++++++++++++-------------
1 file changed, 41 insertions(+), 20 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..284a7c860f1 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,6 +181,12 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * scaling factor after which we switch to multiple transactions during
+ * data population phase on server side
+ */
+static int64 single_txn_scale_limit = 1;
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5213,6 +5219,7 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
+ int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5225,30 +5232,44 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ executeStatement(con, "commit");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ for (int i = 0; i < scale; i += chunk) {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "select bid + 1, 0 "
+ "from generate_series(%d, %d) as bid", i, i + chunk);
+ //"select bid, 0 "
+ //"from generate_series(1, %d) as bid", nbranches * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid,bid,tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ //"select tid, (tid - 1) / %d + 1, 0 "
+ //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ //"select aid, (aid - 1) / %d + 1, 0, '' "
+ //"from generate_series(1, " INT64_FORMAT ") as aid",
+ //naccounts, (int64) naccounts * scale);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
termPQExpBuffer(&sql);
-
- executeStatement(con, "commit");
}
/*
--
2.43.0
From 0eddb156c187d829c4381bc928c5314705928852 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:13:23 +0700
Subject: [PATCH v11 02/28] Getting rid off limit for single transaction size
during data generation
---
src/bin/pgbench/pgbench.c | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 284a7c860f1..28b72e4cf1f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -181,12 +181,6 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
-/*
- * scaling factor after which we switch to multiple transactions during
- * data population phase on server side
- */
-static int64 single_txn_scale_limit = 1;
-
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -5219,7 +5213,6 @@ static void
initGenerateDataServerSide(PGconn *con)
{
PQExpBufferData sql;
- int chunk = (scale >= single_txn_scale_limit) ? 1 : scale;
fprintf(stderr, "generating data (server-side)...\n");
@@ -5236,13 +5229,13 @@ initGenerateDataServerSide(PGconn *con)
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i += chunk) {
+ for (int i = 0; i < scale; i++) {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + chunk);
+ "from generate_series(%d, %d) as bid", i, i + 1);
//"select bid, 0 "
//"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
@@ -5251,7 +5244,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_tellers(tid,bid,tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
- ntellers, i * ntellers, (i + chunk) * ntellers - 1);
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
//"select tid, (tid - 1) / %d + 1, 0 "
//"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
@@ -5260,7 +5253,7 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_accounts(aid,bid,abalance,filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + chunk) * naccounts - 1);
+ naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
//"select aid, (aid - 1) / %d + 1, 0, '' "
//"from generate_series(1, " INT64_FORMAT ") as aid",
//naccounts, (int64) naccounts * scale);
--
2.43.0
From c5659cf474ec273c057668f30a4f435fd02f2da7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 9 Nov 2025 20:38:36 +0700
Subject: [PATCH v11 03/28] No need to keep old code in comments
---
src/bin/pgbench/pgbench.c | 7 -------
1 file changed, 7 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 28b72e4cf1f..97895aa9edf 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5236,8 +5236,6 @@ initGenerateDataServerSide(PGconn *con)
"insert into pgbench_branches(bid,bbalance) "
"select bid + 1, 0 "
"from generate_series(%d, %d) as bid", i, i + 1);
- //"select bid, 0 "
- //"from generate_series(1, %d) as bid", nbranches * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5245,8 +5243,6 @@ initGenerateDataServerSide(PGconn *con)
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
- //"select tid, (tid - 1) / %d + 1, 0 "
- //"from generate_series(1, %d) as tid", ntellers, ntellers * scale);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
@@ -5254,9 +5250,6 @@ initGenerateDataServerSide(PGconn *con)
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
- //"select aid, (aid - 1) / %d + 1, 0, '' "
- //"from generate_series(1, " INT64_FORMAT ") as aid",
- //naccounts, (int64) naccounts * scale);
executeStatement(con, sql.data);
executeStatement(con, "commit");
--
2.43.0
From e47b52ddf23593dad9375ef5356fd41d0621ede3 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 19:06:48 +0700
Subject: [PATCH v11 04/28] Adding server-side data generation via unnest
---
src/bin/pgbench/pgbench.c | 199 ++++++++++++++++++++++++++++++++++----
1 file changed, 182 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 97895aa9edf..65d77cdefea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,6 +171,12 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* original single transaction server-side method */
+#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
+/* 'one transaction per scale' server-side methods */
+#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
+#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +187,11 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ *
+ */
+static char data_generation_type = '?';
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -914,7 +925,9 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " G: generate data, server-side in single transaction\n"
+ " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
+ " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5203,18 +5216,16 @@ initGenerateDataClientSide(PGconn *con)
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * whole dataset in single transaction
*/
static void
-initGenerateDataServerSide(PGconn *con)
+generateDataInsertSingleTXN(PGconn *con)
{
PQExpBufferData sql;
- fprintf(stderr, "generating data (server-side)...\n");
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
+
/*
* we do all of this in one transaction to enable the backend's
@@ -5225,31 +5236,136 @@ initGenerateDataServerSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
+ initPQExpBuffer(&sql);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "select bid, 0 "
+ "from generate_series(1, %d)", scale * nbranches);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(0, %d) as tid",
+ ntellers, (scale * ntellers) - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(0, " INT64_FORMAT ") as aid",
+ naccounts, (int64) (scale * naccounts) - 1);
+ executeStatement(con, sql.data);
+
executeStatement(con, "commit");
+ termPQExpBuffer(&sql);
+}
+
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- for (int i = 0; i < scale; i++) {
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
executeStatement(con, "begin");
printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid + 1, 0 "
- "from generate_series(%d, %d) as bid", i, i + 1);
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
+ "insert into pgbench_tellers(tid, bid, tbalance) "
"select tid + 1, tid / %d + 1, 0 "
"from generate_series(%d, %d) as tid",
ntellers, i * ntellers, (i + 1) * ntellers - 1);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(" INT64_FORMAT ", " INT64_FORMAT ") as aid",
- naccounts, (int64) i * naccounts, (int64) (i + 1) * naccounts - 1);
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ executeStatement(con, "commit");
+ }
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * One transaction per 'scale'
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
executeStatement(con, "commit");
@@ -5258,6 +5374,32 @@ initGenerateDataServerSide(PGconn *con)
termPQExpBuffer(&sql);
}
+/*
+ * Fill the standard tables with some data generated on the server
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side) ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_INSERT_ORIGINAL:
+ generateDataInsertSingleTXN(con);
+ break;
+ case GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
+}
+
/*
* Invoke vacuum on the standard tables
*/
@@ -5341,6 +5483,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5352,7 +5496,26 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case 'G':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'i':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ case 'I':
+ data_init_type++;
+ data_generation_type = *step;
+ break;
+ }
}
+
+ if (data_init_type > 1)
+ pg_log_error("WARNING! More than one type of server-side data generation is requested");
}
/*
@@ -5395,6 +5558,8 @@ runInitSteps(const char *initialize_steps)
initGenerateDataClientSide(con);
break;
case 'G':
+ case 'i':
+ case 'I':
op = "server-side generate";
initGenerateDataServerSide(con);
break;
--
2.43.0
From 5e1827b889b283f50299ce6ab1a73f9f55a4a84f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 10 Nov 2025 20:00:56 +0700
Subject: [PATCH v11 05/28] Fixing typo in query
---
src/bin/pgbench/pgbench.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 65d77cdefea..03e37df4434 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5241,7 +5241,8 @@ generateDataInsertSingleTXN(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
"select bid, 0 "
- "from generate_series(1, %d)", scale * nbranches);
+ "from generate_series(1, %d) as bid",
+ scale * nbranches);
executeStatement(con, sql.data);
printfPQExpBuffer(&sql,
--
2.43.0
From 7ca86521fda6929b8e0de3fc77dcbb8984009c88 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 11 Nov 2025 19:39:45 +0700
Subject: [PATCH v11 06/28] Adding support for COPY BINARY mode
---
src/bin/pgbench/pgbench.c | 393 ++++++++++++++++++++++++++++++++++++--
1 file changed, 381 insertions(+), 12 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03e37df4434..71aa1d9479f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -161,7 +161,7 @@ typedef struct socket_set
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGiIvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -176,6 +176,8 @@ typedef struct socket_set
/* 'one transaction per scale' server-side methods */
#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
+#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
@@ -188,10 +190,17 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
static int scale = 1;
/*
- *
+ * mode of data generation to use
*/
static char data_generation_type = '?';
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -861,7 +870,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -925,6 +935,7 @@ usage(void)
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
" g: generate data, client-side\n"
+ " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
" G: generate data, server-side in single transaction\n"
" i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
" I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
@@ -5191,9 +5202,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideText(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5209,12 +5220,373 @@ initGenerateDataClientSide(PGconn *con)
* already exist
*/
initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
+
+/*
+ * Dumps binary buffer to file (purely for debugging)
+ */
+static void
+dumpBufferToFile(char *filename)
+{
+ FILE *file_ptr;
+ size_t bytes_written;
+
+ file_ptr = fopen(filename, "wb");
+ if (file_ptr == NULL)
+ {
+ fprintf(stderr, "Error opening file %s\n", filename);
+ return; // EXIT_FAILURE;
+ }
+
+ bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
+
+ if (bytes_written != bin_copy_buffer_length)
+ {
+ fprintf(stderr, "Error writing to file or incomplete write\n");
+ fclose(file_ptr);
+ return; // EXIT_FAILURE;
+ }
+
+ fclose(file_ptr);
+}
+
+/*
+ * Save char data to buffer
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with NULL value
+ */
+static void
+addNullColumn()
+{
+ int32_t null = -1;
+ bufferData((void *) &null, sizeof(null));
+}
+
+/*
+ * adds column with int8 value
+ */
+static void
+addInt8Column(int8_t value)
+{
+ int8_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with int16 value
+ */
+static void
+addInt16Column(int16_t value)
+{
+ int16_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with char value
+ */
+static void
+addCharColumn(char *value)
+{
+ int32_t size = strlen(value);
+ bufferData((void *) &size, sizeof(size));
+ bufferCharData(value, size);
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
+ '\0','\0','\0','\0',
+ '\0','\0','\0','\0' };
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = { 0xFF, 0xFF };
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+{
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current teller row to buffer
+ */
+static void
+initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+{
+ /*
+ * Each row has following extra bytes:
+ * - 2 bytes for number of columns
+ * - 4 bytes as length for each column
+ */
+ int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+
+ flushBuffer(con, res, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableBinary(PGconn *con, char *table, char *columns,
+ int64_t base, initRowMethodBin init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t total = base * scale;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+ for (int64_t i = 0; i < total; i++)
+ {
+ init_row(con, res, i, base);
+ }
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
+ sendBinaryCopyTrailer(con);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ PQclear(res);
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinary(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
+
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
+
+ /*
+ * we do all of this in one transaction to enable the backend's
+ * data-loading optimizations
+ */
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side) in ");
+
+ switch (data_generation_type)
+ {
+ case GEN_TYPE_COPY_ORIGINAL:
+ initGenerateDataClientSideText(con);
+ break;
+ case GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinary(con);
+ break;
+ }
+}
+
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
* whole dataset in single transaction
@@ -5500,14 +5872,10 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
+ case 'g':
+ case 'C':
case 'G':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'i':
- data_init_type++;
- data_generation_type = *step;
- break;
case 'I':
data_init_type++;
data_generation_type = *step;
@@ -5555,6 +5923,7 @@ runInitSteps(const char *initialize_steps)
initCreateTables(con);
break;
case 'g':
+ case 'C':
op = "client-side generate";
initGenerateDataClientSide(con);
break;
--
2.43.0
From 4aa0ac05765edf6b5f0c13e18ac677287ce78206 Mon Sep 17 00:00:00 2001
From: Fujii Masao <[email protected]>
Date: Fri, 14 Nov 2025 22:40:39 +0900
Subject: [PATCH v11 07/28] pgbench: Fix assertion failure with multiple
\syncpipeline in pipeline mode.
Previously, when pgbench ran a custom script that triggered retriable errors
(e.g., deadlocks) followed by multiple \syncpipeline commands in pipeline mode,
the following assertion failure could occur:
Assertion failed: (res == ((void*)0)), function discardUntilSync, file pgbench.c, line 3594.
The issue was that discardUntilSync() assumed a pipeline sync result
(PGRES_PIPELINE_SYNC) would always be followed by either another sync result
or NULL. This assumption was incorrect: when multiple sync requests were sent,
a sync result could instead be followed by another result type. In such cases,
discardUntilSync() mishandled the results, leading to the assertion failure.
This commit fixes the issue by making discardUntilSync() correctly handle cases
where a pipeline sync result is followed by other result types. It now continues
discarding results until another pipeline sync followed by NULL is reached.
Backpatched to v17, where support for \syncpipeline command in pgbench was
introduced.
Author: Yugo Nagata <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 17
---
src/bin/pgbench/pgbench.c | 39 ++++++++++++++++++++++++++++-----------
1 file changed, 28 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d8764ba6fe0..a425176ecdc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -3563,14 +3563,18 @@ doRetry(CState *st, pg_time_usec_t *now)
}
/*
- * Read results and discard it until a sync point.
+ * Read and discard results until the last sync point.
*/
static int
discardUntilSync(CState *st)
{
bool received_sync = false;
- /* send a sync */
+ /*
+ * Send a Sync message to ensure at least one PGRES_PIPELINE_SYNC is
+ * received and to avoid an infinite loop, since all earlier ones may have
+ * already been received.
+ */
if (!PQpipelineSync(st->con))
{
pg_log_error("client %d aborted: failed to send a pipeline sync",
@@ -3578,29 +3582,42 @@ discardUntilSync(CState *st)
return 0;
}
- /* receive PGRES_PIPELINE_SYNC and null following it */
+ /*
+ * Continue reading results until the last sync point, i.e., until
+ * reaching null just after PGRES_PIPELINE_SYNC.
+ */
for (;;)
{
PGresult *res = PQgetResult(st->con);
+ if (PQstatus(st->con) == CONNECTION_BAD)
+ {
+ pg_log_error("client %d aborted while rolling back the transaction after an error; perhaps the backend died while processing",
+ st->id);
+ PQclear(res);
+ return 0;
+ }
+
if (PQresultStatus(res) == PGRES_PIPELINE_SYNC)
received_sync = true;
- else if (received_sync)
+ else if (received_sync && res == NULL)
{
- /*
- * PGRES_PIPELINE_SYNC must be followed by another
- * PGRES_PIPELINE_SYNC or NULL; otherwise, assert failure.
- */
- Assert(res == NULL);
-
/*
* Reset ongoing sync count to 0 since all PGRES_PIPELINE_SYNC
* results have been discarded.
*/
st->num_syncs = 0;
- PQclear(res);
break;
}
+ else
+ {
+ /*
+ * If a PGRES_PIPELINE_SYNC is followed by something other than
+ * PGRES_PIPELINE_SYNC or NULL, another PGRES_PIPELINE_SYNC will
+ * appear later. Reset received_sync to false to wait for it.
+ */
+ received_sync = false;
+ }
PQclear(res);
}
--
2.43.0
From 9c4f19055597e9adb25e65c2aa8bedf20a09e13d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 19:05:58 +0700
Subject: [PATCH v11 08/28] Setting empty string as default value in filler
column
---
src/bin/pgbench/pgbench.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 967f6ce6984..03b5e5c28f0 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84)",
- "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84) default ''",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88)",
- "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88) default ''",
1
}
};
--
2.43.0
From 2aabaa52dffdb78fbefaef95163881c15e18ef29 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Fri, 21 Nov 2025 15:03:11 +0200
Subject: [PATCH v11 09/28] Use strtoi64() in pgbench, replacing its open-coded
implementation
Makes the code a little simpler.
The old implementation accepted trailing whitespace, but that was
unnecessary. Firstly, its sibling function for parsing decimals,
strtodouble(), does not accept trailing whitespace. Secondly, none of
the callers can pass a string with trailing whitespace to it.
In the passing, check specifically for ERANGE before printing the "out
of range" error. On some systems, strtoul() and strtod() return EINVAL
on an empty or all-spaces string, and "invalid input syntax" is more
appropriate for that than "out of range". For the existing
strtodouble() function this is purely academical because it's never
called with errorOK==false, but let's be tidy. (Perhaps we should
remove the dead codepaths altogether, but I'll leave that for another
day.)
Reviewed-by: Chao Li <[email protected]>
Reviewed-by: Yuefei Shi <[email protected]>
Reviewed-by: Neil Chen <[email protected]>
Discussion: https://www.postgresql.org/message-id/[email protected]
---
src/bin/pgbench/pgbench.c | 83 +++++++++------------------------------
1 file changed, 19 insertions(+), 64 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a425176ecdc..68774a59efd 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -982,13 +982,17 @@ usage(void)
progname, progname, PACKAGE_BUGREPORT, PACKAGE_NAME, PACKAGE_URL);
}
-/* return whether str matches "^\s*[-+]?[0-9]+$" */
+/*
+ * Return whether str matches "^\s*[-+]?[0-9]+$"
+ *
+ * This should agree with strtoint64() on what's accepted, ignoring overflows.
+ */
static bool
is_an_int(const char *str)
{
const char *ptr = str;
- /* skip leading spaces; cast is consistent with strtoint64 */
+ /* skip leading spaces */
while (*ptr && isspace((unsigned char) *ptr))
ptr++;
@@ -1012,9 +1016,6 @@ is_an_int(const char *str)
/*
* strtoint64 -- convert a string to 64-bit integer
*
- * This function is a slightly modified version of pg_strtoint64() from
- * src/backend/utils/adt/numutils.c.
- *
* The function returns whether the conversion worked, and if so
* "*result" is set to the result.
*
@@ -1023,71 +1024,25 @@ is_an_int(const char *str)
bool
strtoint64(const char *str, bool errorOK, int64 *result)
{
- const char *ptr = str;
- int64 tmp = 0;
- bool neg = false;
-
- /*
- * Do our own scan, rather than relying on sscanf which might be broken
- * for long long.
- *
- * As INT64_MIN can't be stored as a positive 64 bit integer, accumulate
- * value as a negative number.
- */
-
- /* skip leading spaces */
- while (*ptr && isspace((unsigned char) *ptr))
- ptr++;
-
- /* handle sign */
- if (*ptr == '-')
- {
- ptr++;
- neg = true;
- }
- else if (*ptr == '+')
- ptr++;
+ char *end;
- /* require at least one digit */
- if (unlikely(!isdigit((unsigned char) *ptr)))
- goto invalid_syntax;
+ errno = 0;
+ *result = strtoi64(str, &end, 10);
- /* process digits */
- while (*ptr && isdigit((unsigned char) *ptr))
+ if (unlikely(errno == ERANGE))
{
- int8 digit = (*ptr++ - '0');
-
- if (unlikely(pg_mul_s64_overflow(tmp, 10, &tmp)) ||
- unlikely(pg_sub_s64_overflow(tmp, digit, &tmp)))
- goto out_of_range;
+ if (!errorOK)
+ pg_log_error("value \"%s\" is out of range for type bigint", str);
+ return false;
}
- /* allow trailing whitespace, but not other trailing chars */
- while (*ptr != '\0' && isspace((unsigned char) *ptr))
- ptr++;
-
- if (unlikely(*ptr != '\0'))
- goto invalid_syntax;
-
- if (!neg)
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
- if (unlikely(tmp == PG_INT64_MIN))
- goto out_of_range;
- tmp = -tmp;
+ if (!errorOK)
+ pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
+ return false;
}
-
- *result = tmp;
return true;
-
-out_of_range:
- if (!errorOK)
- pg_log_error("value \"%s\" is out of range for type bigint", str);
- return false;
-
-invalid_syntax:
- if (!errorOK)
- pg_log_error("invalid input syntax for type bigint: \"%s\"", str);
- return false;
}
/* convert string to double, detecting overflows/underflows */
@@ -1099,14 +1054,14 @@ strtodouble(const char *str, bool errorOK, double *dv)
errno = 0;
*dv = strtod(str, &end);
- if (unlikely(errno != 0))
+ if (unlikely(errno == ERANGE))
{
if (!errorOK)
pg_log_error("value \"%s\" is out of range for type double", str);
return false;
}
- if (unlikely(end == str || *end != '\0'))
+ if (unlikely(errno != 0 || end == str || *end != '\0'))
{
if (!errorOK)
pg_log_error("invalid input syntax for type double: \"%s\"", str);
--
2.43.0
From dcb85d26f8132eaaf9d096e814b9bda49db7d478 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 21 Nov 2025 20:06:24 +0700
Subject: [PATCH v11 10/28] Switching COPY FROM BINARY ti run in multiple
transactions
---
src/bin/pgbench/pgbench.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 03b5e5c28f0..6b89007a63b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5496,20 +5496,20 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
static void
initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int64_t base, initRowMethodBin init_row)
+ int counter, int64_t base, initRowMethodBin init_row)
{
int n;
PGresult *res;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t total = base * scale;
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5526,7 +5526,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyHeader(con);
- for (int64_t i = 0; i < total; i++)
+ for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
}
@@ -5573,15 +5573,20 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- naccounts, initAccountBinary);
-
executeStatement(con, "commit");
+ for (int i = 0; i < scale; i++)
+ {
+ initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ executeStatement(con, "commit");
+ }
+
pg_free(bin_copy_buffer);
}
--
2.43.0
From b8e28881225234fd00b55235bc60fad2dc60b544 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sat, 22 Nov 2025 17:06:00 +0700
Subject: [PATCH v11 11/28] Adding tests for new modes of data generation
---
src/bin/pgbench/pgbench.c | 21 ++++----
src/bin/pgbench/t/001_pgbench_with_server.pl | 52 +++++++++++++++++---
2 files changed, 56 insertions(+), 17 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6b89007a63b..dd4e5d5e056 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -164,7 +164,7 @@ typedef struct socket_set
#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
@@ -192,7 +192,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = '?';
+static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
/*
* COPY FROM BINARY execution buffer
@@ -4985,26 +4985,26 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
0
},
{
"pgbench_tellers",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
- "tid int not null,bid int,tbalance int,filler char(84) default ''",
+ "tid int not null,bid int,tbalance int,filler char(84)",
+ "tid int not null,bid int,tbalance int,filler char(84)",
1
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default ''",
- "aid bigint not null,bid int,abalance int,filler char(84) default ''",
+ "aid int not null,bid int,abalance int,filler char(84) default '?'",
+ "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
1
},
{
"pgbench_branches",
- "bid int not null,bbalance int,filler char(88) default ''",
- "bid int not null,bbalance int,filler char(88) default ''",
+ "bid int not null,bbalance int,filler char(88)",
+ "bid int not null,bbalance int,filler char(88)",
1
}
};
@@ -7837,6 +7837,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 581e9af7907..a377048ead1 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -125,7 +130,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +148,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -164,6 +170,38 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtI',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtC',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From b3f2bce34232d299abc7644a8579f0ce49c8c9d6 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 23 Nov 2025 14:05:59 +0700
Subject: [PATCH v11 12/28] Fixing compiler warnings about unused procedures by
removing or commenting them out as they might be needed a bit later
---
src/bin/pgbench/pgbench.c | 31 +++++--------------------------
1 file changed, 5 insertions(+), 26 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 0a3ba21dcc9..682db61ff61 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5201,7 +5201,7 @@ initGenerateDataClientSideText(PGconn *con)
/*
* Dumps binary buffer to file (purely for debugging)
- */
+ *
static void
dumpBufferToFile(char *filename)
{
@@ -5226,6 +5226,7 @@ dumpBufferToFile(char *filename)
fclose(file_ptr);
}
+ */
/*
* Save char data to buffer
@@ -5271,37 +5272,14 @@ addColumnCounter(int16_t n)
/*
* adds column with NULL value
- */
+ *
static void
addNullColumn()
{
int32_t null = -1;
bufferData((void *) &null, sizeof(null));
}
-
-/*
- * adds column with int8 value
*/
-static void
-addInt8Column(int8_t value)
-{
- int8_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
-
-/*
- * adds column with int16 value
- */
-static void
-addInt16Column(int16_t value)
-{
- int16_t data = value;
- int32_t size = sizeof(data);
- bufferData((void *) &size, sizeof(size));
- bufferData((void *) &data, sizeof(data));
-}
/*
* adds column with inti32 value
@@ -5329,7 +5307,7 @@ addInt64Column(int64_t value)
/*
* adds column with char value
- */
+ *
static void
addCharColumn(char *value)
{
@@ -5337,6 +5315,7 @@ addCharColumn(char *value)
bufferData((void *) &size, sizeof(size));
bufferCharData(value, size);
}
+ */
/*
* Starts communication with server for COPY FROM BINARY statement
--
2.43.0
From 9ab7fe302ba7de40593e7cba8ea1ca3b876c1ea5 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 19:22:16 +0700
Subject: [PATCH v11 13/28] Moving PQclear call to the end of procedure to
avoid access to previously released memory
---
src/bin/pgbench/pgbench.c | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 682db61ff61..3687b65871e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5363,11 +5363,11 @@ static void
initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of branch table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 3 columns
*/
- int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 32 */
+ int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
flushBuffer(con, res, max_row_len);
@@ -5384,9 +5384,9 @@ static void
initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of tellers table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 4 columns
*/
int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
@@ -5406,9 +5406,9 @@ static void
initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
{
/*
- * Each row has following extra bytes:
+ * Each row of accounts table has following extra bytes:
* - 2 bytes for number of columns
- * - 4 bytes as length for each column
+ * - 4 bytes as length for each of table's 4 columns
*/
int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
@@ -5455,11 +5455,11 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
if (PQresultStatus(res) != PGRES_COPY_IN)
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
- PQclear(res);
sendBinaryCopyHeader(con);
+
for (int64_t i = start; i < start + base; i++)
{
init_row(con, res, i, base);
@@ -5470,8 +5470,10 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+
sendBinaryCopyTrailer(con);
+
if (PQresultStatus(res) == PGRES_COPY_IN)
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
@@ -5484,6 +5486,8 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
}
+
+ PQclear(res);
}
/*
@@ -5499,8 +5503,9 @@ initGenerateDataClientSideBinary(PGconn *con)
bin_copy_buffer_length = 0;
/*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
+ * we do all of this in multiple transactions
+ * to minimize load on DB server and perhaps
+ * in future allow load in parallel sessions
*/
executeStatement(con, "begin");
@@ -5511,6 +5516,8 @@ initGenerateDataClientSideBinary(PGconn *con)
for (int i = 0; i < scale; i++)
{
+ executeStatement(con, "begin");
+
initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary);
initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
--
2.43.0
From 3d7b2c53b3fa45b8848369bfbc49a0c7e5854304 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 19:47:26 +0700
Subject: [PATCH v11 14/28] Fixing error about freeing memory twice
---
src/bin/pgbench/pgbench.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3687b65871e..6f42438d970 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5462,6 +5462,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
+ res = PQgetResult(con);
init_row(con, res, i, base);
}
@@ -5473,7 +5474,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyTrailer(con);
-
+ res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
@@ -5481,7 +5482,6 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
res = PQgetResult(con);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
- PQclear(res);
}
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
--
2.43.0
From a71c90f556ba143335d9631ebbdf96b7120c6d1d Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Thu, 29 Jan 2026 20:24:54 +0700
Subject: [PATCH v11 15/28] Fixing memory leak during data flush in COPY BINARY
test
---
src/bin/pgbench/pgbench.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 6f42438d970..8eca537b718 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -871,7 +871,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBin) (PGconn *con, PGresult *res, int64_t curr, int32_t parent);
+typedef void (*initRowMethodBin) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5345,13 +5345,18 @@ sendBinaryCopyTrailer(PGconn *con)
* Flashes current buffer over network if needed
*/
static void
-flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
+flushBuffer(PGconn *con, int16_t row_len)
{
+ PGresult *res;
+
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
+ res = PQgetResult(con);
/* flush current buffer */
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ pg_fatal("It is NOT a COPY command that is currently running");
bin_copy_buffer_length = 0;
}
}
@@ -5360,7 +5365,7 @@ flushBuffer(PGconn *con, PGresult *res, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of branch table has following extra bytes:
@@ -5369,7 +5374,7 @@ initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(2);
@@ -5381,7 +5386,7 @@ initBranchBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of tellers table has following extra bytes:
@@ -5390,7 +5395,7 @@ initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(3);
@@ -5403,7 +5408,7 @@ initTellerBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*
* Each row of accounts table has following extra bytes:
@@ -5412,7 +5417,7 @@ initAccountBinary(PGconn *con, PGresult *res, int64_t curr, int32_t parent)
*/
int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
- flushBuffer(con, res, max_row_len);
+ flushBuffer(con, max_row_len);
addColumnCounter(3);
@@ -5462,10 +5467,10 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- res = PQgetResult(con);
- init_row(con, res, i, base);
+ init_row(con, i, base);
}
+ res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
@@ -5474,6 +5479,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
sendBinaryCopyTrailer(con);
+
res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
--
2.43.0
From a5933f40be0f3d408bb920f296fb5119c3ec28f4 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 30 Jan 2026 13:59:36 +0700
Subject: [PATCH v11 16/28] Fixing memory leak shown by valgrind
---
src/bin/pgbench/pgbench.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8eca537b718..466d9023d72 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5357,6 +5357,8 @@ flushBuffer(PGconn *con, int16_t row_len)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
pg_fatal("It is NOT a COPY command that is currently running");
+
+ PQclear(res);
bin_copy_buffer_length = 0;
}
}
@@ -5460,6 +5462,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
if (PQresultStatus(res) != PGRES_COPY_IN)
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
sendBinaryCopyHeader(con);
@@ -5475,6 +5478,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+ PQclear(res);
sendBinaryCopyTrailer(con);
@@ -5485,6 +5489,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
{
if (PQputCopyEnd(con, NULL) == 1) /* success */
{
+ PQclear(res);
res = PQgetResult(con);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
@@ -5492,7 +5497,6 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
else
fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
}
-
PQclear(res);
}
--
2.43.0
From 6a3036f898b050d5e71e70eaceccd63003aa6444 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 6 Feb 2026 13:53:07 +0700
Subject: [PATCH v11 17/28] Adding ability to switch data init between single
and multiple transactions
---
src/bin/pgbench/pgbench.c | 192 +++++++++----------
src/bin/pgbench/t/001_pgbench_with_server.pl | 93 ++++++++-
2 files changed, 173 insertions(+), 112 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 466d9023d72..af71e358e71 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,9 +159,8 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-
-#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgCGiIvpf" /* all possible steps */
+#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
+#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -171,14 +170,17 @@ typedef struct socket_set
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
-/* original single transaction server-side method */
-#define GEN_TYPE_INSERT_ORIGINAL 'G' /* use INSERT .. SELECT generate_series to generate data */
-/* 'one transaction per scale' server-side methods */
-#define GEN_TYPE_INSERT_SERIES 'i' /* use INSERT .. SELECT generate_series to generate data */
-#define GEN_TYPE_INSERT_UNNEST 'I' /* use INSERT .. SELECT unnest to generate data */
-#define GEN_TYPE_COPY_ORIGINAL 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
-#define GEN_TYPE_COPY_BINARY 'C' /* use COPY .. FROM STDIN .. BINARY to generate data */
-
+/* server-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT generate_series to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest to generate data */
+/* client-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN .. BINARY to generate data */
+/* data init pseudo steps */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as multiple transactions */
+
+static bool multi_xact = false; /* init data type (as single or multiple transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -192,7 +194,7 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = GEN_TYPE_COPY_ORIGINAL;
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
/*
* COPY FROM BINARY execution buffer
@@ -934,11 +936,13 @@ usage(void)
" run selected initialization steps, in the specified order\n"
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
- " g: generate data, client-side\n"
- " C: client-side (single TNX) COPY .. FROM STDIN .. BINARY\n"
- " G: generate data, server-side in single transaction\n"
- " i: server-side (multiple TXNs) INSERT .. SELECT generate_series\n"
- " I: server-side (multiple TXNs) INSERT .. SELECT unnest\n"
+ " to generate data, client-side:\n"
+ " g: COPY .. FROM STDIN .. TEXT\n"
+ " c: COPY .. FROM STDIN .. BINARY\n"
+ " to generate data, server-side:\n"
+ " G: INSERT .. SELECT generate_series\n"
+ " U: INSERT .. SELECT unnest\n"
+ " M: use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5049,8 +5053,8 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTable(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+initPopulateTableText(PGconn *con, const char *table, int64 base,
+ initRowMethod init_row)
{
int n;
int64 k;
@@ -5178,6 +5182,9 @@ initGenerateDataClientSideText(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
+ if (multi_xact)
+ fprintf(stderr, "WARNING! Multiple transactions are not supported in this mode\n");
+
/*
* we do all of this in one transaction to enable the backend's
* data-loading optimizations
@@ -5191,9 +5198,9 @@ initGenerateDataClientSideText(PGconn *con)
* fill branches, tellers, accounts in that order in case foreign keys
* already exist
*/
- initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
+ initPopulateTableText(con, "pgbench_branches", nbranches, initBranch);
+ initPopulateTableText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
@@ -5450,7 +5457,8 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
get_table_relkind(con, table) == RELKIND_RELATION)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary)";
+ if (!multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
@@ -5522,11 +5530,13 @@ initGenerateDataClientSideBinary(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int i = 0; i < scale; i++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary);
@@ -5535,9 +5545,13 @@ initGenerateDataClientSideBinary(PGconn *con)
initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
i, naccounts, initAccountBinary);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
pg_free(bin_copy_buffer);
}
@@ -5547,14 +5561,15 @@ initGenerateDataClientSideBinary(PGconn *con)
static void
initGenerateDataClientSide(PGconn *con)
{
- fprintf(stderr, "generating data (client-side) in ");
+ fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
switch (data_generation_type)
{
- case GEN_TYPE_COPY_ORIGINAL:
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
initGenerateDataClientSideText(con);
break;
- case GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
initGenerateDataClientSideBinary(con);
break;
}
@@ -5562,58 +5577,7 @@ initGenerateDataClientSide(PGconn *con)
/*
* Generating data via INSERT .. SELECT .. FROM generate_series
- * whole dataset in single transaction
- */
-static void
-generateDataInsertSingleTXN(PGconn *con)
-{
- PQExpBufferData sql;
-
- fprintf(stderr, "via INSERT .. SELECT generate_series... in single TXN\n");
-
-
- /*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
- */
- executeStatement(con, "begin");
-
- /* truncate away any old data */
- initTruncateTables(con);
-
- initPQExpBuffer(&sql);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid, bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid",
- scale * nbranches);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid, bid, tbalance) "
- "select tid + 1, tid / %d + 1, 0 "
- "from generate_series(0, %d) as tid",
- ntellers, (scale * ntellers) - 1);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid, bid, abalance, "
- "filler) "
- "select aid + 1, aid / %d + 1, 0, '' "
- "from generate_series(0, " INT64_FORMAT ") as aid",
- naccounts, (int64) (scale * naccounts) - 1);
- executeStatement(con, sql.data);
-
- executeStatement(con, "commit");
-
- termPQExpBuffer(&sql);
-}
-
-
-/*
- * Generating data via INSERT .. SELECT .. FROM generate_series
- * One transaction per 'scale'
+ * Possibly as "One transaction per scale" in multi-transaction mode
*/
static void
generateDataInsertSeries(PGconn *con)
@@ -5629,11 +5593,13 @@ generateDataInsertSeries(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int i = 0; i < scale; i++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid, bbalance) "
@@ -5657,15 +5623,19 @@ generateDataInsertSeries(PGconn *con)
(int64) (i + 1) * naccounts - 1);
executeStatement(con, sql.data);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
}
/*
* Generating data via INSERT .. SELECT .. FROM unnest
- * One transaction per 'scale'
+ * Possibly as "One transaction per scale" in multi-tansaction mode
*/
static void
generateDataInsertUnnest(PGconn *con)
@@ -5681,11 +5651,13 @@ generateDataInsertUnnest(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
for (int s = 0; s < scale; s++)
{
- executeStatement(con, "begin");
+ if (multi_xact)
+ executeStatement(con, "begin");
printfPQExpBuffer(&sql,
"insert into pgbench_branches(bid,bbalance) "
@@ -5714,14 +5686,18 @@ generateDataInsertUnnest(PGconn *con)
(int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
}
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
termPQExpBuffer(&sql);
}
/*
- * Fill the standard tables with some data generated on the server
+ * Fill the standard tables with some data generated on the server side
*
* As already the case with the client-side data generation, the filler
* column defaults to NULL in pgbench_branches and pgbench_tellers,
@@ -5730,17 +5706,15 @@ generateDataInsertUnnest(PGconn *con)
static void
initGenerateDataServerSide(PGconn *con)
{
- fprintf(stderr, "generating data (server-side) ");
+ fprintf(stderr, "generating data (server-side as %s transaction%s) ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
switch (data_generation_type)
{
- case GEN_TYPE_INSERT_ORIGINAL:
- generateDataInsertSingleTXN(con);
- break;
- case GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
generateDataInsertSeries(con);
break;
- case GEN_TYPE_INSERT_UNNEST:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
generateDataInsertUnnest(con);
break;
}
@@ -5845,19 +5819,20 @@ checkInitSteps(const char *initialize_steps)
switch (*step)
{
- case 'g':
- case 'C':
- case 'G':
- case 'i':
- case 'I':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
data_init_type++;
data_generation_type = *step;
break;
}
}
+ if (data_init_type == 0)
+ pg_log_error("WARNING! No data generation type is provided");
if (data_init_type > 1)
- pg_log_error("WARNING! More than one type of server-side data generation is requested");
+ pg_log_error("WARNING! More than one type of data initialization is requested");
}
/*
@@ -5895,17 +5870,22 @@ runInitSteps(const char *initialize_steps)
op = "create tables";
initCreateTables(con);
break;
- case 'g':
- case 'C':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
initGenerateDataClientSide(con);
break;
- case 'G':
- case 'i':
- case 'I':
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
initGenerateDataServerSide(con);
break;
+ case INIT_STEP_GEN_TYPE_SINGLE_XACT:
+ multi_xact = false;
+ break;
+ case INIT_STEP_GEN_TYPE_MULTI_XACT:
+ multi_xact = true;
+ break;
case 'v':
op = "vacuum";
initVacuum(con);
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index a377048ead1..acb1e31d3e8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -117,6 +117,7 @@ $node->pgbench(
[qr{Perhaps you need to do initialization}],
'run without init');
+
# Initialize pgbench tables scale 1
$node->pgbench(
'-i', 0,
@@ -160,7 +161,7 @@ $node->pgbench(
qr{creating tables},
qr{creating 3 partitions},
qr{creating primary keys},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{creating foreign keys},
qr{(?!vacuuming)}, # no vacuum
qr{done in \d+\.\d\d s }
@@ -170,15 +171,16 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+
# Test server-side generation with UNNEST
$node->pgbench(
- '--initialize --init-steps=dtI',
+ '--initialize --init-steps=dtU',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{done in \d+\.\d\d s }
],
'pgbench --init-steps server-side UNNEST');
@@ -186,15 +188,48 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side (unnest)');
-# Test server-side generation with UNNEST
+
+# Test server-side generation with COPY TEXT
+$node->pgbench(
+ '--initialize --init-steps=dtg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (text)');
+
+
+# Test server-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtC',
+ '--initialize --init-steps=dtc',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side\)},
+ qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
'pgbench --init-steps client-side BINARY');
@@ -202,6 +237,52 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'client-side (binary)');
+$node->pgbench(
+ '--initialize --init-steps=dtSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From 2471bb47bf715402ece61b084b69afe3bb53742e Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Fri, 6 Feb 2026 13:55:47 +0700
Subject: [PATCH v11 18/28] Removing commented out procedures
---
src/bin/pgbench/pgbench.c | 52 ---------------------------------------
1 file changed, 52 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index af71e358e71..34464b7037c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5206,35 +5206,6 @@ initGenerateDataClientSideText(PGconn *con)
}
-/*
- * Dumps binary buffer to file (purely for debugging)
- *
-static void
-dumpBufferToFile(char *filename)
-{
- FILE *file_ptr;
- size_t bytes_written;
-
- file_ptr = fopen(filename, "wb");
- if (file_ptr == NULL)
- {
- fprintf(stderr, "Error opening file %s\n", filename);
- return; // EXIT_FAILURE;
- }
-
- bytes_written = fwrite(bin_copy_buffer, 1, bin_copy_buffer_length, file_ptr);
-
- if (bytes_written != bin_copy_buffer_length)
- {
- fprintf(stderr, "Error writing to file or incomplete write\n");
- fclose(file_ptr);
- return; // EXIT_FAILURE;
- }
-
- fclose(file_ptr);
-}
- */
-
/*
* Save char data to buffer
*/
@@ -5277,17 +5248,6 @@ addColumnCounter(int16_t n)
bufferData((void *) &n, sizeof(n));
}
-/*
- * adds column with NULL value
- *
-static void
-addNullColumn()
-{
- int32_t null = -1;
- bufferData((void *) &null, sizeof(null));
-}
- */
-
/*
* adds column with inti32 value
*/
@@ -5312,18 +5272,6 @@ addInt64Column(int64_t value)
bufferData((void *) &data, sizeof(data));
}
-/*
- * adds column with char value
- *
-static void
-addCharColumn(char *value)
-{
- int32_t size = strlen(value);
- bufferData((void *) &size, sizeof(size));
- bufferCharData(value, size);
-}
- */
-
/*
* Starts communication with server for COPY FROM BINARY statement
*/
--
2.43.0
From 10dfa622b09c420d79da8774c9321f91540c8600 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 09:08:17 +0700
Subject: [PATCH v11 19/28] Small changes due to big code review
---
src/bin/pgbench/pgbench.c | 71 +++++++++++++++++++++------------------
1 file changed, 39 insertions(+), 32 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 34464b7037c..9f3d7f99a25 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -873,7 +873,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBin) (PGconn *con, int64_t curr, int32_t parent);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -942,7 +942,8 @@ usage(void)
" to generate data, server-side:\n"
" G: INSERT .. SELECT generate_series\n"
" U: INSERT .. SELECT unnest\n"
- " M: use multiple transactions to initialize data\n"
+ " S: flag to use single transaction to initialize data\n"
+ " M: flag to use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -5053,8 +5054,8 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTableText(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
+ initRowMethod init_row)
{
int n;
int64 k;
@@ -5178,7 +5179,7 @@ initPopulateTableText(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSideText(PGconn *con)
+initGenerateDataClientSideTextFrmt(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
@@ -5198,9 +5199,9 @@ initGenerateDataClientSideText(PGconn *con)
* fill branches, tellers, accounts in that order in case foreign keys
* already exist
*/
- initPopulateTableText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableText(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTableText(con, "pgbench_accounts", naccounts, initAccount);
+ initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
}
@@ -5212,6 +5213,8 @@ initGenerateDataClientSideText(PGconn *con)
static void
bufferCharData(char *src, int32_t len)
{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
bin_copy_buffer_length += len;
}
@@ -5227,13 +5230,16 @@ bufferData(void *src, int32_t len)
bufferCharData(src, len);
#else
if (len == 1)
+ {
bufferCharData(src, len);
- else
- for (int32_t i = 0; i < len; i++)
- {
- ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
- ((char *) src)[len - i - 1];
- }
+ return;
+ }
+
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
bin_copy_buffer_length += len;
#endif
@@ -5322,7 +5328,7 @@ flushBuffer(PGconn *con, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of branch table has following extra bytes:
@@ -5333,7 +5339,7 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(2);
+ addColumnCounter(columnCounter);
addInt32Column(curr + 1);
addInt32Column(0);
@@ -5343,7 +5349,7 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of tellers table has following extra bytes:
@@ -5354,7 +5360,7 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(3);
+ addColumnCounter(columnCounter);
addInt32Column(curr + 1);
addInt32Column(curr / parent + 1);
@@ -5365,7 +5371,7 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
/*
* Each row of accounts table has following extra bytes:
@@ -5376,7 +5382,7 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
flushBuffer(con, max_row_len);
- addColumnCounter(3);
+ addColumnCounter(columnCounter);
if (scale <= SCALE_32BIT_THRESHOLD)
addInt32Column(curr + 1);
@@ -5391,8 +5397,9 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
* Universal wrapper for sending data in binary format
*/
static void
-initPopulateTableBinary(PGconn *con, char *table, char *columns,
- int counter, int64_t base, initRowMethodBin init_row)
+initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBinary init_row,
+ int columnCounter)
{
int n;
PGresult *res;
@@ -5426,7 +5433,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- init_row(con, i, base);
+ init_row(con, i, base, columnCounter);
}
res = PQgetResult(con);
@@ -5460,7 +5467,7 @@ initPopulateTableBinary(PGconn *con, char *table, char *columns,
* Wrapper for binary data load
*/
static void
-initGenerateDataClientSideBinary(PGconn *con)
+initGenerateDataClientSideBinaryFrmt(PGconn *con)
{
fprintf(stderr, "BINARY mode...\n");
@@ -5486,12 +5493,12 @@ initGenerateDataClientSideBinary(PGconn *con)
if (multi_xact)
executeStatement(con, "begin");
- initPopulateTableBinary(con, "pgbench_branches", "bid, bbalance",
- i, nbranches, initBranchBinary);
- initPopulateTableBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary);
- initPopulateTableBinary(con, "pgbench_accounts", "aid, bid, abalance",
- i, naccounts, initAccountBinary);
+ initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary, 2);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary, 3);
+ initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary, 3);
if (multi_xact)
executeStatement(con, "commit");
@@ -5515,10 +5522,10 @@ initGenerateDataClientSide(PGconn *con)
switch (data_generation_type)
{
case INIT_STEP_GEN_TYPE_COPY_TEXT:
- initGenerateDataClientSideText(con);
+ initGenerateDataClientSideTextFrmt(con);
break;
case INIT_STEP_GEN_TYPE_COPY_BINARY:
- initGenerateDataClientSideBinary(con);
+ initGenerateDataClientSideBinaryFrmt(con);
break;
}
}
--
2.43.0
From d95c53d412ff0e94e162eaf3990233fae007ae3f Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 10:07:13 +0700
Subject: [PATCH v11 20/28] Running indent to fix code formatting
---
src/bin/pgbench/pgbench.c | 111 +++++++++++++++++++++-----------------
1 file changed, 62 insertions(+), 49 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 9f3d7f99a25..8198f1d4c9a 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,7 +159,7 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
+#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
@@ -171,16 +171,24 @@ typedef struct socket_set
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
/* server-side methods to generate data */
-#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT generate_series to generate data */
-#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT
+ * generate_series to generate
+ * data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest
+ * to generate data */
/* client-side methods to generate data */
-#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN .. TEXT to generate data */
-#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN .. BINARY to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN ..
+ * TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN ..
+ * BINARY to generate data */
/* data init pseudo steps */
-#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as single transaction */
-#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as multiple transactions */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as
+ * single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as
+ * multiple transactions */
-static bool multi_xact = false; /* init data type (as single or multiple transactions) */
+static bool multi_xact = false; /* init data type (as single or multiple
+ * transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -194,14 +202,15 @@ static int scale = 1;
/*
* mode of data generation to use
*/
-static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
/*
* COPY FROM BINARY execution buffer
*/
-#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM BINARY */
-static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
-static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM
+ * BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
@@ -872,8 +881,8 @@ static int wait_on_socket_set(socket_set *sa, int64 usecs);
static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
-typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
+typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5200,7 +5209,7 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
* already exist
*/
initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
executeStatement(con, "commit");
@@ -5260,8 +5269,9 @@ addColumnCounter(int16_t n)
static void
addInt32Column(int32_t value)
{
- int32_t data = value;
- int32_t size = sizeof(data);
+ int32_t data = value;
+ int32_t size = sizeof(data);
+
bufferData((void *) &size, sizeof(size));
bufferData((void *) &data, sizeof(data));
}
@@ -5272,8 +5282,9 @@ addInt32Column(int32_t value)
static void
addInt64Column(int64_t value)
{
- int64_t data = value;
- int32_t size = sizeof(data);
+ int64_t data = value;
+ int32_t size = sizeof(data);
+
bufferData((void *) &size, sizeof(size));
bufferData((void *) &data, sizeof(data));
}
@@ -5284,9 +5295,9 @@ addInt64Column(int64_t value)
static void
sendBinaryCopyHeader(PGconn *con)
{
- char header[] = {'P','G','C','O','P','Y','\n','\377','\r','\n','\0',
- '\0','\0','\0','\0',
- '\0','\0','\0','\0' };
+ char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ '\0', '\0', '\0', '\0',
+ '\0', '\0', '\0', '\0'};
PQputCopyData(con, header, 19);
}
@@ -5297,7 +5308,7 @@ sendBinaryCopyHeader(PGconn *con)
static void
sendBinaryCopyTrailer(PGconn *con)
{
- static char trailer[] = { 0xFF, 0xFF };
+ static char trailer[] = {0xFF, 0xFF};
PQputCopyData(con, trailer, 2);
}
@@ -5308,7 +5319,7 @@ sendBinaryCopyTrailer(PGconn *con)
static void
flushBuffer(PGconn *con, int16_t row_len)
{
- PGresult *res;
+ PGresult *res;
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
@@ -5330,12 +5341,13 @@ flushBuffer(PGconn *con, int16_t row_len)
static void
initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of branch table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 3 columns
+ *---
*/
- int16_t max_row_len = 35 + 2 + 4*3; /* max row size is 35 */
+ int16_t max_row_len = 35 + 2 + 4 * 3; /* max row size is 35 */
flushBuffer(con, max_row_len);
@@ -5351,12 +5363,13 @@ initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter
static void
initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of tellers table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 4 columns
+ *---
*/
- int16_t max_row_len = 40 + 2 + 4*4; /* max row size is 40 */
+ int16_t max_row_len = 40 + 2 + 4 * 4; /* max row size is 40 */
flushBuffer(con, max_row_len);
@@ -5373,12 +5386,13 @@ initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter
static void
initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
{
- /*
+ /*---
* Each row of accounts table has following extra bytes:
* - 2 bytes for number of columns
* - 4 bytes as length for each of table's 4 columns
*/
- int16_t max_row_len = 250 + 2 + 4*4; /* max row size is 250 for int64 */
+ int16_t max_row_len = 250 + 2 + 4 * 4; /* max row size is 250 for
+ * int64 */
flushBuffer(con, max_row_len);
@@ -5401,11 +5415,11 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
int counter, int64_t base, initRowMethodBinary init_row,
int columnCounter)
{
- int n;
- PGresult *res;
- char copy_statement[256];
- const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
- int64_t start = base * counter;
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
bin_copy_buffer_length = 0;
@@ -5450,7 +5464,7 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
res = PQgetResult(con);
if (PQresultStatus(res) == PGRES_COPY_IN)
{
- if (PQputCopyEnd(con, NULL) == 1) /* success */
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
{
PQclear(res);
res = PQgetResult(con);
@@ -5476,9 +5490,8 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
bin_copy_buffer_length = 0;
/*
- * we do all of this in multiple transactions
- * to minimize load on DB server and perhaps
- * in future allow load in parallel sessions
+ * we do all of this in multiple transactions to minimize load on DB
+ * server and perhaps in future allow load in parallel sessions
*/
executeStatement(con, "begin");
@@ -5495,8 +5508,8 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
i, nbranches, initBranchBinary, 2);
- initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary, 3);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary, 3);
initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
i, naccounts, initAccountBinary, 3);
@@ -5570,10 +5583,10 @@ generateDataInsertSeries(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_accounts(aid, bid, abalance, "
- "filler) "
+ "filler) "
"select aid + 1, aid / %d + 1, 0, '' "
"from generate_series(" INT64_FORMAT ", "
- INT64_FORMAT ") as aid",
+ INT64_FORMAT ") as aid",
naccounts, (int64) i * naccounts,
(int64) (i + 1) * naccounts - 1);
executeStatement(con, sql.data);
@@ -5622,7 +5635,7 @@ generateDataInsertUnnest(PGconn *con)
printfPQExpBuffer(&sql,
"insert into pgbench_tellers(tid, bid, tbalance) "
"select unnest(array_agg(s.i order by s.i)) as tid, "
- "%d as bid, 0 as tbalance "
+ "%d as bid, 0 as tbalance "
"from generate_series(%d, %d) as s(i)",
s + 1, s * ntellers + 1, (s + 1) * ntellers);
executeStatement(con, sql.data);
@@ -5630,13 +5643,13 @@ generateDataInsertUnnest(PGconn *con)
printfPQExpBuffer(&sql,
"with data as ("
" select generate_series(" INT64_FORMAT ", "
- INT64_FORMAT ") as i) "
+ INT64_FORMAT ") as i) "
"insert into pgbench_accounts(aid, bid, "
- "abalance, filler) "
+ "abalance, filler) "
"select unnest(aid), unnest(bid), 0 as abalance, "
- "'' as filler "
+ "'' as filler "
"from (select array_agg(i+1) aid, "
- "array_agg(i/%d + 1) bid from data)",
+ "array_agg(i/%d + 1) bid from data)",
(int64) s * naccounts + 1,
(int64) (s + 1) * naccounts, naccounts);
executeStatement(con, sql.data);
@@ -5758,7 +5771,7 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
- char data_init_type = 0;
+ char data_init_type = 0;
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
--
2.43.0
From 270d2bf14b6873ff66fc37ca0a5dccc055dab1f7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 16 Feb 2026 14:57:18 +0700
Subject: [PATCH v11 21/28] Removing debugging filler value
---
src/bin/pgbench/pgbench.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 8198f1d4c9a..710b2a5b70d 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -4954,8 +4954,8 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default '?'",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default '?'",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
@@ -4966,8 +4966,8 @@ initCreateTables(PGconn *con)
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84) default '?'",
- "aid bigint not null,bid int,abalance int,filler char(84) default '?'",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
--
2.43.0
From 86a111c63e7006d4801ce7bb594a5b77d87b42b0 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 23 Feb 2026 16:15:06 +0700
Subject: [PATCH v11 22/28] Moving out logic showing data load progress into
own procedure
---
src/bin/pgbench/pgbench.c | 142 ++++++++++++++++++++------------------
1 file changed, 75 insertions(+), 67 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 710b2a5b70d..e3e0e073792 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -487,6 +487,9 @@ typedef struct StatsData
*/
static pg_time_usec_t epoch_shift;
+/* used to track elapsed time and estimate of the remaining time of data load */
+static pg_time_usec_t data_load_start;
+
/*
* Error status for errors during script execution.
*/
@@ -5062,27 +5065,82 @@ initAccount(PQExpBufferData *sql, int64 curr)
curr + 1, curr / naccounts + 1);
}
+static void
+showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
+{
+ static int chars = 0;
+ static int prev_chars = 0;
+ static int log_interval = 1;
+
+ /* Stay on the same line if reporting to a terminal */
+ char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+
+ double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - data_load_start);
+ double remaining_sec = ((double) total - current) * elapsed_sec / current;
+
+ /*
+ * If we want to stick with the original logging, print a message each
+ * 100k inserted rows.
+ */
+ if ((!use_quiet) && (current % 100000 == 0))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current one,
+ * add spaces to the current line to fully overwrite any remaining
+ * characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+ }
+ /* let's not call the timing for each row, but only each 100 rows */
+ else if (use_quiet && (current % 100 == 0))
+ {
+ /* have we reached the next interval (or end)? */
+ if ((current == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current
+ * one, add spaces to the current line to fully overwrite any
+ * remaining characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+
+ /* skip to the next interval */
+ log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
+ }
+ }
+
+ if (current == total && chars != 0 && eol != '\n')
+ fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
+}
+
static void
initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
initRowMethod init_row)
{
int n;
int64 k;
- int chars = 0;
- int prev_chars = 0;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
int64 total = base * scale;
- /* used to track elapsed time and estimate of the remaining time */
- pg_time_usec_t start;
- int log_interval = 1;
-
- /* Stay on the same line if reporting to a terminal */
- char eol = isatty(fileno(stderr)) ? '\r' : '\n';
-
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
@@ -5103,12 +5161,8 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- start = pg_time_now();
-
for (k = 0; k < total; k++)
{
- int64 j = k + 1;
-
init_row(&sql, k);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
@@ -5116,62 +5170,9 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
if (CancelRequested)
break;
- /*
- * If we want to stick with the original logging, print a message each
- * 100k inserted rows.
- */
- if ((!use_quiet) && (j % 100000 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
- }
- /* let's not call the timing for each row, but only each 100 rows */
- else if (use_quiet && (j % 100 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- /* have we reached the next interval (or end)? */
- if ((j == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
- {
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
-
- /* skip to the next interval */
- log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
- }
- }
+ showPopulateTableCopyProgress(table, k + 1, total);
}
- if (chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
if (PQputline(con, "\\.\n"))
pg_fatal("very last PQputline failed");
@@ -5448,6 +5449,11 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
init_row(con, i, base, columnCounter);
+
+ if (CancelRequested)
+ break;
+
+ showPopulateTableCopyProgress(table, i, base * scale);
}
res = PQgetResult(con);
@@ -5532,6 +5538,8 @@ initGenerateDataClientSide(PGconn *con)
fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+ data_load_start = pg_time_now();
+
switch (data_generation_type)
{
case INIT_STEP_GEN_TYPE_COPY_TEXT:
--
2.43.0
From afe3f69b23190f375b57d5cf0a3eae2cd8e40caa Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Mon, 23 Feb 2026 17:07:16 +0700
Subject: [PATCH v11 23/28] Adding multi-transaction mode to client-side data
generation
---
src/bin/pgbench/pgbench.c | 52 ++++++++++++--------
src/bin/pgbench/t/001_pgbench_with_server.pl | 47 +++++++++---------
2 files changed, 56 insertions(+), 43 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e3e0e073792..1b73da904ad 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5130,22 +5130,22 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
}
static void
-initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
+initPopulateTableCopyText(PGconn *con, const char *table, int counter, int64 base,
initRowMethod init_row)
{
int n;
- int64 k;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
- int64 total = base * scale;
+ int64 start = base * counter;
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
copy_statement_fmt = "copy %s from stdin with (freeze on)";
@@ -5161,16 +5161,16 @@ initPopulateTableCopyText(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- for (k = 0; k < total; k++)
+ for (int64_t i = start; i < start + base; i++)
{
- init_row(&sql, k);
+ init_row(&sql, i);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
if (CancelRequested)
break;
- showPopulateTableCopyProgress(table, k + 1, total);
+ showPopulateTableCopyProgress(table, i, base * scale);
}
@@ -5193,9 +5193,6 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
{
fprintf(stderr, "TEXT mode...\n");
- if (multi_xact)
- fprintf(stderr, "WARNING! Multiple transactions are not supported in this mode\n");
-
/*
* we do all of this in one transaction to enable the backend's
* data-loading optimizations
@@ -5205,15 +5202,28 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- /*
- * fill branches, tellers, accounts in that order in case foreign keys
- * already exist
- */
- initPopulateTableCopyText(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTableCopyText(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTableCopyText(con, "pgbench_accounts", naccounts, initAccount);
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ /*
+ * fill branches, tellers, accounts in that order in case foreign keys
+ * already exist
+ */
+ initPopulateTableCopyText(con, "pgbench_branches", i, nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", i, ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", i, naccounts, initAccount);
- executeStatement(con, "commit");
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
}
@@ -5426,9 +5436,9 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
- if (!multi_xact)
- copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
if (n >= sizeof(copy_statement))
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index acb1e31d3e8..46569b0eb37 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -189,7 +189,7 @@ $node->pgbench(
check_data_state($node, 'server-side (unnest)');
-# Test server-side generation with COPY TEXT
+# Test client-side generation with COPY TEXT
$node->pgbench(
'--initialize --init-steps=dtg',
0,
@@ -200,45 +200,48 @@ $node->pgbench(
qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side TEXT');
+ 'pgbench --init-steps client-side TEXT (single XACT #1)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (text)');
$node->pgbench(
- '--initialize --init-steps=dtMg',
+ '--initialize --init-steps=dtSg',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_branches done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_tellers done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_accounts done},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side TEXT');
+ 'pgbench --init-steps client-side TEXT (single XACT #2)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (text)');
-
-# Test server-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtc',
+ '--initialize --init-steps=dtMg',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as single transaction},
+ qr{generating data \(client-side as multiple transactions},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side TEXT (multiple XACTs)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
-# Check data state, after server-side data generation.
-check_data_state($node, 'client-side (binary)');
+# Test client-side generation with COPY BINARY
$node->pgbench(
- '--initialize --init-steps=dtSc',
+ '--initialize --init-steps=dtc',
0,
[qr{^$}],
[
@@ -247,24 +250,24 @@ $node->pgbench(
qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side BINARY (single XACT #1)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
$node->pgbench(
- '--initialize --init-steps=dtMc',
+ '--initialize --init-steps=dtSc',
0,
[qr{^$}],
[
qr{dropping old tables},
qr{creating tables},
- qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
qr{done in \d+\.\d\d s }
],
- 'pgbench --init-steps client-side BINARY');
+ 'pgbench --init-steps client-side BINARY (single XACT #2)');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
$node->pgbench(
@@ -279,7 +282,7 @@ $node->pgbench(
],
'pgbench --init-steps client-side BINARY');
-# Check data state, after server-side data generation.
+# Check data state, after client-side data generation.
check_data_state($node, 'client-side (binary)');
--
2.43.0
From d9c61efdd853c999b92061ad4f88dd012265dbba Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 18:24:18 +0700
Subject: [PATCH v11 24/28] Removing double call to checkInitSteps
---
src/bin/pgbench/pgbench.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1b73da904ad..7d3a1583152 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5816,9 +5816,9 @@ checkInitSteps(const char *initialize_steps)
}
if (data_init_type == 0)
- pg_log_error("WARNING! No data generation type is provided");
+ pg_log_warning("No data generation type is provided");
if (data_init_type > 1)
- pg_log_error("WARNING! More than one type of data initialization is requested");
+ pg_log_warning("More than one type of data initialization is requested");
}
/*
@@ -7449,7 +7449,6 @@ main(int argc, char **argv)
case 'I':
pg_free(initialize_steps);
initialize_steps = pg_strdup(optarg);
- checkInitSteps(initialize_steps);
initialization_option_set = true;
break;
case 'j': /* jobs */
--
2.43.0
From d593a2063a997c86de4b9350b97d4ad6527bddef Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 18:45:25 +0700
Subject: [PATCH v11 25/28] Fixed 2 small bugs
1. Overwrite of init mode by last mode in checkInitSteps
2. Bledding of init progress from previous mode under new one (eg, '-IggGG')
---
src/bin/pgbench/pgbench.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 7d3a1583152..daf33d73814 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -5125,8 +5125,11 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
}
}
- if (current == total && chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
+ if (current + 1 == total && chars != 0)
+ {
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fputc(eol, stderr);
+ }
}
static void
@@ -5810,7 +5813,6 @@ checkInitSteps(const char *initialize_steps)
case INIT_STEP_GEN_TYPE_INSERT_SERIES:
case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
data_init_type++;
- data_generation_type = *step;
break;
}
}
@@ -5859,11 +5861,13 @@ runInitSteps(const char *initialize_steps)
case INIT_STEP_GEN_TYPE_COPY_TEXT:
case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
+ data_generation_type = *step;
initGenerateDataClientSide(con);
break;
case INIT_STEP_GEN_TYPE_INSERT_SERIES:
case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
+ data_generation_type = *step;
initGenerateDataServerSide(con);
break;
case INIT_STEP_GEN_TYPE_SINGLE_XACT:
--
2.43.0
From 265d4b017529c9bdc21ecdbf7fe4794789260545 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Sun, 1 Mar 2026 19:06:45 +0700
Subject: [PATCH v11 26/28] Added more tests
---
src/bin/pgbench/t/001_pgbench_with_server.pl | 98 ++++++++++++++++++++
1 file changed, 98 insertions(+)
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 46569b0eb37..73d2d754273 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -172,6 +172,53 @@ $node->pgbench(
check_data_state($node, 'server-side');
+# Test server-side generation with generate_series
+$node->pgbench(
+ '--initialize --init-steps=dtG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series single XACT)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series multiple XACTs)');
+
+
# Test server-side generation with UNNEST
$node->pgbench(
'--initialize --init-steps=dtU',
@@ -188,6 +235,36 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side (unnest)');
+$node->pgbench(
+ '--initialize --init-steps=dtSU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
# Test client-side generation with COPY TEXT
$node->pgbench(
@@ -286,6 +363,27 @@ $node->pgbench(
check_data_state($node, 'client-side (binary)');
+# Check data state, after different modes of client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMccSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (multiple XACT modes)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary different XACT modes in list of --init-steps)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
From 3f569dfba37c8722fb5e2b99f3e001a93a959511 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 3 Mar 2026 10:49:54 +0700
Subject: [PATCH v11 27/28] Small changes due to review of patch v9
---
src/bin/pgbench/pgbench.c | 26 +++++++++++++++-----------
1 file changed, 15 insertions(+), 11 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index daf33d73814..81576d8f7ca 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -160,7 +160,7 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgMScGUvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtMScgGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
#define DEFAULT_NXACTS 10 /* default nxacts */
@@ -5232,6 +5232,8 @@ initGenerateDataClientSideTextFrmt(PGconn *con)
/*
* Save char data to buffer
+ * Kept as separate proc for possible addition of something
+ * like addCharColumn in future
*/
static void
bufferCharData(char *src, int32_t len)
@@ -5249,22 +5251,24 @@ bufferCharData(char *src, int32_t len)
static void
bufferData(void *src, int32_t len)
{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
#ifdef __sparc__
bufferCharData(src, len);
#else
+
if (len == 1)
- {
bufferCharData(src, len);
- return;
- }
-
- for (int32_t i = 0; i < len; i++)
+ else
{
- ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
- ((char *) src)[len - i - 1];
- }
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
- bin_copy_buffer_length += len;
+ bin_copy_buffer_length += len;
+ }
#endif
}
@@ -5309,7 +5313,7 @@ addInt64Column(int64_t value)
static void
sendBinaryCopyHeader(PGconn *con)
{
- char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
'\0', '\0', '\0', '\0',
'\0', '\0', '\0', '\0'};
--
2.43.0
From 701c438f4adcea84784a1dd478e60902f9083eb7 Mon Sep 17 00:00:00 2001
From: Boris Mironov <[email protected]>
Date: Tue, 3 Mar 2026 18:01:56 +0700
Subject: [PATCH v11 28/28] Rewrote comments and formulas calculating
max_row_len used by COPY BINARY
---
src/bin/pgbench/pgbench.c | 93 +++++++++++++++++++++++++++------------
1 file changed, 65 insertions(+), 28 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 81576d8f7ca..971a1d3a3d8 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -885,7 +885,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
-typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -5127,7 +5127,7 @@ showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
if (current + 1 == total && chars != 0)
{
- fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
fputc(eol, stderr);
}
}
@@ -5313,7 +5313,7 @@ addInt64Column(int64_t value)
static void
sendBinaryCopyHeader(PGconn *con)
{
- static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
'\0', '\0', '\0', '\0',
'\0', '\0', '\0', '\0'};
@@ -5342,6 +5342,9 @@ flushBuffer(PGconn *con, int16_t row_len)
if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
{
res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
/* flush current buffer */
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
@@ -5357,64 +5360,95 @@ flushBuffer(PGconn *con, int16_t row_len)
* Sends current branch row to buffer
*/
static void
-initBranchBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*---
- * Each row of branch table has following extra bytes:
- * - 2 bytes for number of columns
- * - 4 bytes as length for each of table's 3 columns
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of branches table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - branches table has following columns:
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for bbalance column or sizeof(int32_t)
+ * - 88 bytes for filler column (optional since no requirement for row length)
*---
*/
- int16_t max_row_len = 35 + 2 + 4 * 3; /* max row size is 35 */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 88);
flushBuffer(con, max_row_len);
- addColumnCounter(columnCounter);
+ addColumnCounter(2);
addInt32Column(curr + 1);
addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
}
/*
* Sends current teller row to buffer
*/
static void
-initTellerBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*---
- * Each row of tellers table has following extra bytes:
- * - 2 bytes for number of columns
- * - 4 bytes as length for each of table's 4 columns
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of tellers table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - tellers table has following columns:
+ * - 4 bytes for tid column or sizeof(int32_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for tbalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
*---
*/
- int16_t max_row_len = 40 + 2 + 4 * 4; /* max row size is 40 */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 4) + (4 + 84);
flushBuffer(con, max_row_len);
- addColumnCounter(columnCounter);
+ addColumnCounter(3);
addInt32Column(curr + 1);
addInt32Column(curr / parent + 1);
addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
}
/*
* Sends current account row to buffer
*/
static void
-initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounter)
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
{
/*---
- * Each row of accounts table has following extra bytes:
- * - 2 bytes for number of columns
- * - 4 bytes as length for each of table's 4 columns
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of accounts table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - accounts table has following columns (taking into account scale > 20000):
+ * - 8 bytes for aid column or sizeof(int64_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for abalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
*/
- int16_t max_row_len = 250 + 2 + 4 * 4; /* max row size is 250 for
- * int64 */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 8) + (4 + 4) + (4 + 4) + (4 + 84);
flushBuffer(con, max_row_len);
- addColumnCounter(columnCounter);
+ addColumnCounter(3);
if (scale <= SCALE_32BIT_THRESHOLD)
addInt32Column(curr + 1);
@@ -5423,6 +5457,7 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounte
addInt32Column(curr / parent + 1);
addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
}
/*
@@ -5430,8 +5465,7 @@ initAccountBinary(PGconn *con, int64_t curr, int32_t parent, int8_t columnCounte
*/
static void
initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
- int counter, int64_t base, initRowMethodBinary init_row,
- int columnCounter)
+ int counter, int64_t base, initRowMethodBinary init_row)
{
int n;
PGresult *res;
@@ -5465,7 +5499,7 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
for (int64_t i = start; i < start + base; i++)
{
- init_row(con, i, base, columnCounter);
+ init_row(con, i, base);
if (CancelRequested)
break;
@@ -5474,6 +5508,9 @@ initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
}
res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
if (PQresultStatus(res) == PGRES_COPY_IN)
PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
else
@@ -5530,11 +5567,11 @@ initGenerateDataClientSideBinaryFrmt(PGconn *con)
executeStatement(con, "begin");
initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
- i, nbranches, initBranchBinary, 2);
+ i, nbranches, initBranchBinary);
initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
- i, ntellers, initTellerBinary, 3);
+ i, ntellers, initTellerBinary);
initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
- i, naccounts, initAccountBinary, 3);
+ i, naccounts, initAccountBinary);
if (multi_xact)
executeStatement(con, "commit");
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-06 07:26 ` Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Madyshev Egor @ 2026-03-06 07:26 UTC (permalink / raw)
To: Boris Mironov <[email protected]>; pgsql-hackers
Hi Boris,
I'm in the final stage of reviewing your patch. Before completing,
please fix the patch file and rebase it to the current state of
the master branch. The current version contains two commits
that are already in the master.
Because of this, the patch cannot be cleanly applied to the repository.
Best regards,
Egor
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
@ 2026-03-06 14:21 ` Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-06 14:21 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; pgsql-hackers
Hi Egor,
> One more thing I missed in my previous post:
> there is no documentation for the new modes (-c, -U, -M, -S).
> Please add descriptions for them in doc/src/sgml/pgbench.sgml
> (similar to the existing options).
Documentation update has been added
> As soon as the rebase and documentation update are completed,
> the patch will be ready for final review.
Patch has been rebased to current state of the main branch.
Squashing 27 separate commits shrunk patch size almost 4-fold.
Thank you very much. Greatly appreciate your help in this endeavor.
Best regards,
Boris
Attachments:
[application/octet-stream] v12-pgbench-faster-modes.patch (38.0K, 3-v12-pgbench-faster-modes.patch)
download | inline diff:
From 5bf9908cf10918a2ae43ceadd44d6c9089736cff Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email protected]>
Date: Fri, 6 Mar 2026 10:31:35 +0100
Subject: [PATCH v12] Adding new init modes to pgbench including COPY FROM
BINARY as well as populating data in multiple transactions
---
src/bin/pgbench/pgbench.c | 802 ++++++++++++++++---
src/bin/pgbench/t/001_pgbench_with_server.pl | 236 +++++-
2 files changed, 906 insertions(+), 132 deletions(-)
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1dae918cc09..48f2950e245 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,18 +159,36 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtMScgGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* server-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT
+ * generate_series to generate
+ * data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest
+ * to generate data */
+/* client-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN ..
+ * TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN ..
+ * BINARY to generate data */
+/* data init pseudo steps */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as
+ * single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as
+ * multiple transactions */
+
+static bool multi_xact = false; /* init data type (as single or multiple
+ * transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +199,19 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * mode of data generation to use
+ */
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
+
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM
+ * BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -456,6 +487,9 @@ typedef struct StatsData
*/
static pg_time_usec_t epoch_shift;
+/* used to track elapsed time and estimate of the remaining time of data load */
+static pg_time_usec_t data_load_start;
+
/*
* Error status for errors during script execution.
*/
@@ -851,6 +885,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -913,8 +948,14 @@ usage(void)
" run selected initialization steps, in the specified order\n"
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
- " g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " to generate data, client-side:\n"
+ " g: COPY .. FROM STDIN .. TEXT\n"
+ " c: COPY .. FROM STDIN .. BINARY\n"
+ " to generate data, server-side:\n"
+ " G: INSERT .. SELECT generate_series\n"
+ " U: INSERT .. SELECT unnest\n"
+ " S: flag to use single transaction to initialize data\n"
+ " M: flag to use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -4916,8 +4957,8 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
@@ -4928,8 +4969,8 @@ initCreateTables(PGconn *con)
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
@@ -5025,31 +5066,89 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTable(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
+{
+ static int chars = 0;
+ static int prev_chars = 0;
+ static int log_interval = 1;
+
+ /* Stay on the same line if reporting to a terminal */
+ char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+
+ double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - data_load_start);
+ double remaining_sec = ((double) total - current) * elapsed_sec / current;
+
+ /*
+ * If we want to stick with the original logging, print a message each
+ * 100k inserted rows.
+ */
+ if ((!use_quiet) && (current % 100000 == 0))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current one,
+ * add spaces to the current line to fully overwrite any remaining
+ * characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+ }
+ /* let's not call the timing for each row, but only each 100 rows */
+ else if (use_quiet && (current % 100 == 0))
+ {
+ /* have we reached the next interval (or end)? */
+ if ((current == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current
+ * one, add spaces to the current line to fully overwrite any
+ * remaining characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+
+ /* skip to the next interval */
+ log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
+ }
+ }
+
+ if (current + 1 == total && chars != 0)
+ {
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fputc(eol, stderr);
+ }
+}
+
+static void
+initPopulateTableCopyText(PGconn *con, const char *table, int counter, int64 base,
+ initRowMethod init_row)
{
int n;
- int64 k;
- int chars = 0;
- int prev_chars = 0;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
- int64 total = base * scale;
-
- /* used to track elapsed time and estimate of the remaining time */
- pg_time_usec_t start;
- int log_interval = 1;
-
- /* Stay on the same line if reporting to a terminal */
- char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+ int64 start = base * counter;
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
copy_statement_fmt = "copy %s from stdin with (freeze on)";
@@ -5065,75 +5164,18 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- start = pg_time_now();
-
- for (k = 0; k < total; k++)
+ for (int64_t i = start; i < start + base; i++)
{
- int64 j = k + 1;
-
- init_row(&sql, k);
+ init_row(&sql, i);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
if (CancelRequested)
break;
- /*
- * If we want to stick with the original logging, print a message each
- * 100k inserted rows.
- */
- if ((!use_quiet) && (j % 100000 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
- }
- /* let's not call the timing for each row, but only each 100 rows */
- else if (use_quiet && (j % 100 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- /* have we reached the next interval (or end)? */
- if ((j == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
- {
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
-
- /* skip to the next interval */
- log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
- }
- }
+ showPopulateTableCopyProgress(table, i, base * scale);
}
- if (chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
if (PQputline(con, "\\.\n"))
pg_fatal("very last PQputline failed");
@@ -5150,9 +5192,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideTextFrmt(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5163,64 +5205,549 @@ initGenerateDataClientSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- /*
- * fill branches, tellers, accounts in that order in case foreign keys
- * already exist
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ /*
+ * fill branches, tellers, accounts in that order in case foreign keys
+ * already exist
+ */
+ initPopulateTableCopyText(con, "pgbench_branches", i, nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", i, ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", i, naccounts, initAccount);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+}
+
+
+/*
+ * Save char data to buffer
+ * Kept as separate proc for possible addition of something
+ * like addCharColumn in future
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ {
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+ }
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ '\0', '\0', '\0', '\0',
+ '\0', '\0', '\0', '\0'};
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = {0xFF, 0xFF};
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, int16_t row_len)
+{
+ PGresult *res;
+
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ pg_fatal("It is NOT a COPY command that is currently running");
+
+ PQclear(res);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
+{
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of branches table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - branches table has following columns:
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for bbalance column or sizeof(int32_t)
+ * - 88 bytes for filler column (optional since no requirement for row length)
+ *---
*/
- initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 88);
- executeStatement(con, "commit");
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Sends current teller row to buffer
*/
static void
-initGenerateDataServerSide(PGconn *con)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
- PQExpBufferData sql;
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of tellers table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - tellers table has following columns:
+ * - 4 bytes for tid column or sizeof(int32_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for tbalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
+ */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 4) + (4 + 84);
+
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
+{
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of accounts table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - accounts table has following columns (taking into account scale > 20000):
+ * - 8 bytes for aid column or sizeof(int64_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for abalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
+ */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 8) + (4 + 4) + (4 + 4) + (4 + 84);
+
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBinary init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+
+ for (int64_t i = start; i < start + base; i++)
+ {
+ init_row(con, i, base);
+
+ if (CancelRequested)
+ break;
+
+ showPopulateTableCopyProgress(table, i, base * scale);
+ }
+
+ res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+ PQclear(res);
+
+
+ sendBinaryCopyTrailer(con);
+
+
+ res = PQgetResult(con);
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ PQclear(res);
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+ PQclear(res);
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinaryFrmt(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
- fprintf(stderr, "generating data (server-side)...\n");
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
/*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
+ * we do all of this in multiple transactions to minimize load on DB
+ * server and perhaps in future allow load in parallel sessions
*/
executeStatement(con, "begin");
/* truncate away any old data */
initTruncateTables(con);
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+
+ data_load_start = pg_time_now();
+
+ switch (data_generation_type)
+ {
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ initGenerateDataClientSideTextFrmt(con);
+ break;
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinaryFrmt(con);
+ break;
+ }
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * Possibly as "One transaction per scale" in multi-transaction mode
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * Possibly as "One transaction per scale" in multi-tansaction mode
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
- executeStatement(con, "commit");
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
+ executeStatement(con, sql.data);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Fill the standard tables with some data generated on the server side
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side as %s transaction%s) ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+
+ switch (data_generation_type)
+ {
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
}
/*
@@ -5306,6 +5833,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5317,7 +5846,22 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
+ data_init_type++;
+ break;
+ }
}
+
+ if (data_init_type == 0)
+ pg_log_warning("No data generation type is provided");
+ if (data_init_type > 1)
+ pg_log_warning("More than one type of data initialization is requested");
}
/*
@@ -5355,14 +5899,24 @@ runInitSteps(const char *initialize_steps)
op = "create tables";
initCreateTables(con);
break;
- case 'g':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
+ data_generation_type = *step;
initGenerateDataClientSide(con);
break;
- case 'G':
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
+ data_generation_type = *step;
initGenerateDataServerSide(con);
break;
+ case INIT_STEP_GEN_TYPE_SINGLE_XACT:
+ multi_xact = false;
+ break;
+ case INIT_STEP_GEN_TYPE_MULTI_XACT:
+ multi_xact = true;
+ break;
case 'v':
op = "vacuum";
initVacuum(con);
@@ -6940,7 +7494,6 @@ main(int argc, char **argv)
case 'I':
pg_free(initialize_steps);
initialize_steps = pg_strdup(optarg);
- checkInitSteps(initialize_steps);
initialization_option_set = true;
break;
case 'j': /* jobs */
@@ -7245,6 +7798,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index b7685ea5d20..6c7783a77f7 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -112,6 +117,7 @@ $node->pgbench(
[qr{Perhaps you need to do initialization}],
'run without init');
+
# Initialize pgbench tables scale 1
$node->pgbench(
'-i', 0,
@@ -125,7 +131,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +149,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -154,7 +161,7 @@ $node->pgbench(
qr{creating tables},
qr{creating 3 partitions},
qr{creating primary keys},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{creating foreign keys},
qr{(?!vacuuming)}, # no vacuum
qr{done in \d+\.\d\d s }
@@ -164,6 +171,219 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+
+# Test server-side generation with generate_series
+$node->pgbench(
+ '--initialize --init-steps=dtG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series single XACT)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series multiple XACTs)');
+
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+
+# Test client-side generation with COPY TEXT
+$node->pgbench(
+ '--initialize --init-steps=dtg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (single XACT #1)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_branches done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_tellers done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_accounts done},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (single XACT #2)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (multiple XACTs)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+
+# Test client-side generation with COPY BINARY
+$node->pgbench(
+ '--initialize --init-steps=dtc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (single XACT #1)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (single XACT #2)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+
+# Check data state, after different modes of client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMccSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (multiple XACT modes)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary different XACT modes in list of --init-steps)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-09 05:18 ` Boris Mironov <[email protected]>
2026-03-09 07:55 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-09 05:18 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; pgsql-hackers
Hi Egor,
Sorry, but I had to issue new version of the patch to include changes to documentation,
which was missed in previous version.
Now patch consists of 3 files:
doc/src/sgml/ref/pgbench.sgml | 90 ++-
src/bin/pgbench/pgbench.c | 802 ++++++++++++++++---
src/bin/pgbench/t/001_pgbench_with_server.pl | 236 +++++-
3 files changed, 978 insertions(+), 150 deletions(-)
Best regards,
Boris
Attachments:
[application/octet-stream] v14-pgbench-faster-modes.patch (44.8K, 3-v14-pgbench-faster-modes.patch)
download | inline diff:
From d9e4f5096ae9530b629629fb47ec08b922805739 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email protected]>
Date: Fri, 6 Mar 2026 10:31:35 +0100
Subject: [PATCH v12] Adding new init modes to pgbench including COPY FROM
BINARY as well as populating data in multiple transactions
---
doc/src/sgml/ref/pgbench.sgml | 90 ++-
src/bin/pgbench/pgbench.c | 802 ++++++++++++++++---
src/bin/pgbench/t/001_pgbench_with_server.pl | 236 +++++-
3 files changed, 978 insertions(+), 150 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2e401d1ceb8..20f41f332a5 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -217,41 +217,93 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
</listitem>
</varlistentry>
- <varlistentry id="pgbench-option-init-steps-g">
- <term><literal>g</literal> or <literal>G</literal> (Generate data, client-side or server-side)</term>
+
+ <varlistentry id="pgbench-option-init-steps-client">
+ <term><literal>g</literal> or <literal>c</literal> (Generate data, client-side)</term>
<listitem>
<para>
- Generate data and load it into the standard tables,
- replacing any data already present.
- </para>
- <para>
- With <literal>g</literal> (client-side data generation),
- data is generated in <command>pgbench</command> client and then
- sent to the server. This uses the client/server bandwidth
+ Generate data and load it into the standard tables using client-side
+ generation. Data is generated in the <command>pgbench</command> client
+ and then sent to the server. This uses the client/server bandwidth
extensively through a <command>COPY</command>.
<command>pgbench</command> uses the <option>FREEZE</option> option
to load data into ordinary (non-partition) tables with version 14
or later of <productname>PostgreSQL</productname> to speed up
- subsequent <command>VACUUM</command>.
- Using <literal>g</literal> causes logging to
- print one message every 100,000 rows while generating data for all
- tables.
+ subsequent <command>VACUUM</command>. Using <literal>g</literal>
+ or <literal>c</literal> causes logging to print one message
+ every 100,000 rows while generating data for all tables.
+ </para>
+ <para>
+ With <literal>g</literal> (text format), the client sends data in
+ <command>COPY ... FROM STDIN (FORMAT TEXT)</command>.
+ This is default mode for data initialization.
+ </para>
+ <para>
+ With <literal>c</literal> (binary format), the client sends data using
+ <command>COPY ... FROM STDIN (FORMAT BINARY)</command>. Binary transfer
+ can be faster than text, but may be less portable across different
+ versions or platforms.
</para>
<para>
- With <literal>G</literal> (server-side data generation),
+ The default initialization behavior uses client-side data generation
+ with text format (equivalent to <literal>g</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="pgbench-option-init-steps-server">
+ <term><literal>G</literal> or <literal>U</literal> (Generate data, server-side)</term>
+ <listitem>
+ <para>
+ With <literal>G</literal> or <literal>U</literal> (server-side data generation),
only small queries are sent from the <command>pgbench</command>
client and then data is actually generated in the server.
No significant bandwidth is required for this variant, but
the server will do more work.
- Using <literal>G</literal> causes logging not to print any progress
- message while generating data.
+ Using <literal>G</literal> or <literal>U</literal> causes logging not to print
+ any progress message while generating data.
+ </para>
+ <para>
+ With <literal>G</literal>, the server uses <literal>INSERT ...
+ SELECT generate_series(...)</literal> to produce the rows. This method
+ is simple and widely used.
+ </para>
+ <para>
+ With <literal>U</literal>, the server uses <literal>INSERT ...
+ SELECT unnest(...)</literal> to expand arrays of generated values. This can
+ be more efficient when multiple columns are generated together for column-based
+ table, as the work is done in a single scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="pgbench-option-init-steps-transaction-mode">
+ <term><literal>S</literal> or <literal>M</literal> (Transaction mode for initialization)</term>
+ <listitem>
+ <para>
+ Specifies whether the initialization steps are performed in a single
+ transaction or in multiple transactions.
+ </para>
+ <para>
+ With <literal>S</literal> (single transaction), all data generation
+ are executed within one transaction. This is the default
+ behavior. Using a single transaction allows the use of the
+ <option>FREEZE</option> option with client-side <command>COPY</command>.
</para>
<para>
- The default initialization behavior uses client-side data
- generation (equivalent to <literal>g</literal>).
+ With <literal>M</literal> (multiple transactions), data for each scale factor
+ unit (e.g., for each increment of <option>--scale=</option>) is loaded in a
+ separate transaction. For example, if you set <option>--scale=20</option>
+ data will be added to three main tables in 20 separate transactions.
+ In this mode, the <option>FREEZE</option> option
+ cannot be used with client-side <command>COPY</command>.
+ Using multiple transactions can speed up data generation by
+ committing data in smaller batches, which may reduce the overhead of a
+ single huge transaction, but also adds per-transaction commit costs.
</para>
</listitem>
</varlistentry>
+
<varlistentry id="pgbench-option-init-steps-v">
<term><literal>v</literal> (Vacuum)</term>
<listitem>
@@ -260,6 +312,7 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
</listitem>
</varlistentry>
+
<varlistentry id="pgbench-option-init-steps-p">
<term><literal>p</literal> (create Primary keys)</term>
<listitem>
@@ -268,6 +321,7 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
</listitem>
</varlistentry>
+
<varlistentry id="pgbench-option-init-steps-f">
<term><literal>f</literal> (create Foreign keys)</term>
<listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1dae918cc09..48f2950e245 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,18 +159,36 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtMScgGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* server-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT
+ * generate_series to generate
+ * data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest
+ * to generate data */
+/* client-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN ..
+ * TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN ..
+ * BINARY to generate data */
+/* data init pseudo steps */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as
+ * single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as
+ * multiple transactions */
+
+static bool multi_xact = false; /* init data type (as single or multiple
+ * transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +199,19 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * mode of data generation to use
+ */
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
+
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM
+ * BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -456,6 +487,9 @@ typedef struct StatsData
*/
static pg_time_usec_t epoch_shift;
+/* used to track elapsed time and estimate of the remaining time of data load */
+static pg_time_usec_t data_load_start;
+
/*
* Error status for errors during script execution.
*/
@@ -851,6 +885,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -913,8 +948,14 @@ usage(void)
" run selected initialization steps, in the specified order\n"
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
- " g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " to generate data, client-side:\n"
+ " g: COPY .. FROM STDIN .. TEXT\n"
+ " c: COPY .. FROM STDIN .. BINARY\n"
+ " to generate data, server-side:\n"
+ " G: INSERT .. SELECT generate_series\n"
+ " U: INSERT .. SELECT unnest\n"
+ " S: flag to use single transaction to initialize data\n"
+ " M: flag to use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -4916,8 +4957,8 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
@@ -4928,8 +4969,8 @@ initCreateTables(PGconn *con)
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
@@ -5025,31 +5066,89 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTable(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
+{
+ static int chars = 0;
+ static int prev_chars = 0;
+ static int log_interval = 1;
+
+ /* Stay on the same line if reporting to a terminal */
+ char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+
+ double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - data_load_start);
+ double remaining_sec = ((double) total - current) * elapsed_sec / current;
+
+ /*
+ * If we want to stick with the original logging, print a message each
+ * 100k inserted rows.
+ */
+ if ((!use_quiet) && (current % 100000 == 0))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current one,
+ * add spaces to the current line to fully overwrite any remaining
+ * characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+ }
+ /* let's not call the timing for each row, but only each 100 rows */
+ else if (use_quiet && (current % 100 == 0))
+ {
+ /* have we reached the next interval (or end)? */
+ if ((current == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current
+ * one, add spaces to the current line to fully overwrite any
+ * remaining characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+
+ /* skip to the next interval */
+ log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
+ }
+ }
+
+ if (current + 1 == total && chars != 0)
+ {
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fputc(eol, stderr);
+ }
+}
+
+static void
+initPopulateTableCopyText(PGconn *con, const char *table, int counter, int64 base,
+ initRowMethod init_row)
{
int n;
- int64 k;
- int chars = 0;
- int prev_chars = 0;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
- int64 total = base * scale;
-
- /* used to track elapsed time and estimate of the remaining time */
- pg_time_usec_t start;
- int log_interval = 1;
-
- /* Stay on the same line if reporting to a terminal */
- char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+ int64 start = base * counter;
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
copy_statement_fmt = "copy %s from stdin with (freeze on)";
@@ -5065,75 +5164,18 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- start = pg_time_now();
-
- for (k = 0; k < total; k++)
+ for (int64_t i = start; i < start + base; i++)
{
- int64 j = k + 1;
-
- init_row(&sql, k);
+ init_row(&sql, i);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
if (CancelRequested)
break;
- /*
- * If we want to stick with the original logging, print a message each
- * 100k inserted rows.
- */
- if ((!use_quiet) && (j % 100000 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
- }
- /* let's not call the timing for each row, but only each 100 rows */
- else if (use_quiet && (j % 100 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- /* have we reached the next interval (or end)? */
- if ((j == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
- {
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
-
- /* skip to the next interval */
- log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
- }
- }
+ showPopulateTableCopyProgress(table, i, base * scale);
}
- if (chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
if (PQputline(con, "\\.\n"))
pg_fatal("very last PQputline failed");
@@ -5150,9 +5192,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideTextFrmt(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5163,64 +5205,549 @@ initGenerateDataClientSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- /*
- * fill branches, tellers, accounts in that order in case foreign keys
- * already exist
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ /*
+ * fill branches, tellers, accounts in that order in case foreign keys
+ * already exist
+ */
+ initPopulateTableCopyText(con, "pgbench_branches", i, nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", i, ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", i, naccounts, initAccount);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+}
+
+
+/*
+ * Save char data to buffer
+ * Kept as separate proc for possible addition of something
+ * like addCharColumn in future
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ {
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+ }
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ '\0', '\0', '\0', '\0',
+ '\0', '\0', '\0', '\0'};
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = {0xFF, 0xFF};
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, int16_t row_len)
+{
+ PGresult *res;
+
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ pg_fatal("It is NOT a COPY command that is currently running");
+
+ PQclear(res);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
+{
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of branches table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - branches table has following columns:
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for bbalance column or sizeof(int32_t)
+ * - 88 bytes for filler column (optional since no requirement for row length)
+ *---
*/
- initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 88);
- executeStatement(con, "commit");
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Sends current teller row to buffer
*/
static void
-initGenerateDataServerSide(PGconn *con)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
- PQExpBufferData sql;
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of tellers table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - tellers table has following columns:
+ * - 4 bytes for tid column or sizeof(int32_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for tbalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
+ */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 4) + (4 + 84);
+
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
+{
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of accounts table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - accounts table has following columns (taking into account scale > 20000):
+ * - 8 bytes for aid column or sizeof(int64_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for abalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
+ */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 8) + (4 + 4) + (4 + 4) + (4 + 84);
+
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBinary init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+
+ for (int64_t i = start; i < start + base; i++)
+ {
+ init_row(con, i, base);
+
+ if (CancelRequested)
+ break;
+
+ showPopulateTableCopyProgress(table, i, base * scale);
+ }
+
+ res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+ PQclear(res);
+
+
+ sendBinaryCopyTrailer(con);
+
+
+ res = PQgetResult(con);
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ PQclear(res);
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+ PQclear(res);
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinaryFrmt(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
- fprintf(stderr, "generating data (server-side)...\n");
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
/*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
+ * we do all of this in multiple transactions to minimize load on DB
+ * server and perhaps in future allow load in parallel sessions
*/
executeStatement(con, "begin");
/* truncate away any old data */
initTruncateTables(con);
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+
+ data_load_start = pg_time_now();
+
+ switch (data_generation_type)
+ {
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ initGenerateDataClientSideTextFrmt(con);
+ break;
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinaryFrmt(con);
+ break;
+ }
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * Possibly as "One transaction per scale" in multi-transaction mode
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * Possibly as "One transaction per scale" in multi-tansaction mode
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
- executeStatement(con, "commit");
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
+ executeStatement(con, sql.data);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Fill the standard tables with some data generated on the server side
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side as %s transaction%s) ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+
+ switch (data_generation_type)
+ {
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
}
/*
@@ -5306,6 +5833,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5317,7 +5846,22 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
+ data_init_type++;
+ break;
+ }
}
+
+ if (data_init_type == 0)
+ pg_log_warning("No data generation type is provided");
+ if (data_init_type > 1)
+ pg_log_warning("More than one type of data initialization is requested");
}
/*
@@ -5355,14 +5899,24 @@ runInitSteps(const char *initialize_steps)
op = "create tables";
initCreateTables(con);
break;
- case 'g':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
+ data_generation_type = *step;
initGenerateDataClientSide(con);
break;
- case 'G':
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
+ data_generation_type = *step;
initGenerateDataServerSide(con);
break;
+ case INIT_STEP_GEN_TYPE_SINGLE_XACT:
+ multi_xact = false;
+ break;
+ case INIT_STEP_GEN_TYPE_MULTI_XACT:
+ multi_xact = true;
+ break;
case 'v':
op = "vacuum";
initVacuum(con);
@@ -6940,7 +7494,6 @@ main(int argc, char **argv)
case 'I':
pg_free(initialize_steps);
initialize_steps = pg_strdup(optarg);
- checkInitSteps(initialize_steps);
initialization_option_set = true;
break;
case 'j': /* jobs */
@@ -7245,6 +7798,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index b7685ea5d20..6c7783a77f7 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -112,6 +117,7 @@ $node->pgbench(
[qr{Perhaps you need to do initialization}],
'run without init');
+
# Initialize pgbench tables scale 1
$node->pgbench(
'-i', 0,
@@ -125,7 +131,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +149,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -154,7 +161,7 @@ $node->pgbench(
qr{creating tables},
qr{creating 3 partitions},
qr{creating primary keys},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{creating foreign keys},
qr{(?!vacuuming)}, # no vacuum
qr{done in \d+\.\d\d s }
@@ -164,6 +171,219 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+
+# Test server-side generation with generate_series
+$node->pgbench(
+ '--initialize --init-steps=dtG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series single XACT)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series multiple XACTs)');
+
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+
+# Test client-side generation with COPY TEXT
+$node->pgbench(
+ '--initialize --init-steps=dtg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (single XACT #1)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_branches done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_tellers done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_accounts done},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (single XACT #2)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (multiple XACTs)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+
+# Test client-side generation with COPY BINARY
+$node->pgbench(
+ '--initialize --init-steps=dtc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (single XACT #1)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (single XACT #2)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+
+# Check data state, after different modes of client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMccSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (multiple XACT modes)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary different XACT modes in list of --init-steps)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-09 07:55 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-03-09 09:57 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-03-09 07:55 UTC (permalink / raw)
To: 'Boris Mironov' <[email protected]>; Madyshev Egor <[email protected]>; pgsql-hackers
Dear Boris,
There is a similar idea which parallelize the initialization. Not sure, is there
an interactions or comparison between them? Your patch seems to divide a
transaction into several parts but everything is done by the single thread,
whereas proposed in [1] is to dispatch to other threads.
[1]: https://commitfest.postgresql.org/patch/6242/
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 07:55 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Hayato Kuroda (Fujitsu) <[email protected]>
@ 2026-03-09 09:57 ` Boris Mironov <[email protected]>
2026-03-11 11:53 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-09 09:57 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; Madyshev Egor <[email protected]>; pgsql-hackers
Dear Hayato,
> There is a similar idea which parallelize the initialization. Not sure, is there
> an interactions or comparison between them? Your patch seems to divide a
> transaction into several parts but everything is done by the single thread,
> whereas proposed in [1] is to dispatch to other threads.
You're absolutely right. Mircea (author of multithread patch #6242 [1]) contacted
me with a question if our patches could be merged. At that point in time
none of our patches had reviewers and sheer size of each one of them was quite
big (~500 lines of new code). Therefore we decided to leave our patches
as a separate commits and see what community will say.
It's quite clear that there is certain synergy between them and perhaps it would
make perfect sense to introduce them together in same release of PostgreSQL.
I'm not familiar with intricate details of process that takes place after CommitFest,
especially if several reviewed commits in "Ready to go" status touch same
area of the product.
I believe that my patch shouldn't require too many changes in order to be merged
with Mircea's one since core logic to initialize set of rows in three main tables
for one particular "scale" is already in place.
Thank you for your interest in my commit and
Best regards,
Boris
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 07:55 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Hayato Kuroda (Fujitsu) <[email protected]>
2026-03-09 09:57 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-11 11:53 ` Boris Mironov <[email protected]>
2026-03-12 14:28 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-11 11:53 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; pgsql-hackers
Hi Egor,
> The patch is applied with warnings in the documentation.:
> "warning: squelched 1 whitespace error
> warning: 6 lines add whitespace errors."
v15 of the patch fixes 6 trailing spaces in pgbench.sgml
> I've checked everything else is fine for me.
Hopefully there is enough time to get into list of accepted changes
from this CommitFest.
Cheers,
Boris
Attachments:
[application/octet-stream] v15-pgbench-faster-modes.patch (44.8K, 3-v15-pgbench-faster-modes.patch)
download | inline diff:
From d9e4f5096ae9530b629629fb47ec08b922805739 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email protected]>
Date: Fri, 6 Mar 2026 10:31:35 +0100
Subject: [PATCH v12] Adding new init modes to pgbench including COPY FROM
BINARY as well as populating data in multiple transactions
---
doc/src/sgml/ref/pgbench.sgml | 90 ++-
src/bin/pgbench/pgbench.c | 802 ++++++++++++++++---
src/bin/pgbench/t/001_pgbench_with_server.pl | 236 +++++-
3 files changed, 978 insertions(+), 150 deletions(-)
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2e401d1ceb8..20f41f332a5 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -217,41 +217,93 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
</listitem>
</varlistentry>
- <varlistentry id="pgbench-option-init-steps-g">
- <term><literal>g</literal> or <literal>G</literal> (Generate data, client-side or server-side)</term>
+
+ <varlistentry id="pgbench-option-init-steps-client">
+ <term><literal>g</literal> or <literal>c</literal> (Generate data, client-side)</term>
<listitem>
<para>
- Generate data and load it into the standard tables,
- replacing any data already present.
- </para>
- <para>
- With <literal>g</literal> (client-side data generation),
- data is generated in <command>pgbench</command> client and then
- sent to the server. This uses the client/server bandwidth
+ Generate data and load it into the standard tables using client-side
+ generation. Data is generated in the <command>pgbench</command> client
+ and then sent to the server. This uses the client/server bandwidth
extensively through a <command>COPY</command>.
<command>pgbench</command> uses the <option>FREEZE</option> option
to load data into ordinary (non-partition) tables with version 14
or later of <productname>PostgreSQL</productname> to speed up
- subsequent <command>VACUUM</command>.
- Using <literal>g</literal> causes logging to
- print one message every 100,000 rows while generating data for all
- tables.
+ subsequent <command>VACUUM</command>. Using <literal>g</literal>
+ or <literal>c</literal> causes logging to print one message
+ every 100,000 rows while generating data for all tables.
+ </para>
+ <para>
+ With <literal>g</literal> (text format), the client sends data in
+ <command>COPY ... FROM STDIN (FORMAT TEXT)</command>.
+ This is default mode for data initialization.
+ </para>
+ <para>
+ With <literal>c</literal> (binary format), the client sends data using
+ <command>COPY ... FROM STDIN (FORMAT BINARY)</command>. Binary transfer
+ can be faster than text, but may be less portable across different
+ versions or platforms.
</para>
<para>
- With <literal>G</literal> (server-side data generation),
+ The default initialization behavior uses client-side data generation
+ with text format (equivalent to <literal>g</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="pgbench-option-init-steps-server">
+ <term><literal>G</literal> or <literal>U</literal> (Generate data, server-side)</term>
+ <listitem>
+ <para>
+ With <literal>G</literal> or <literal>U</literal> (server-side data generation),
only small queries are sent from the <command>pgbench</command>
client and then data is actually generated in the server.
No significant bandwidth is required for this variant, but
the server will do more work.
- Using <literal>G</literal> causes logging not to print any progress
- message while generating data.
+ Using <literal>G</literal> or <literal>U</literal> causes logging not to print
+ any progress message while generating data.
+ </para>
+ <para>
+ With <literal>G</literal>, the server uses <literal>INSERT ...
+ SELECT generate_series(...)</literal> to produce the rows. This method
+ is simple and widely used.
+ </para>
+ <para>
+ With <literal>U</literal>, the server uses <literal>INSERT ...
+ SELECT unnest(...)</literal> to expand arrays of generated values. This can
+ be more efficient when multiple columns are generated together for column-based
+ table, as the work is done in a single scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="pgbench-option-init-steps-transaction-mode">
+ <term><literal>S</literal> or <literal>M</literal> (Transaction mode for initialization)</term>
+ <listitem>
+ <para>
+ Specifies whether the initialization steps are performed in a single
+ transaction or in multiple transactions.
+ </para>
+ <para>
+ With <literal>S</literal> (single transaction), all data generation
+ are executed within one transaction. This is the default
+ behavior. Using a single transaction allows the use of the
+ <option>FREEZE</option> option with client-side <command>COPY</command>.
</para>
<para>
- The default initialization behavior uses client-side data
- generation (equivalent to <literal>g</literal>).
+ With <literal>M</literal> (multiple transactions), data for each scale factor
+ unit (e.g., for each increment of <option>--scale=</option>) is loaded in a
+ separate transaction. For example, if you set <option>--scale=20</option>
+ data will be added to three main tables in 20 separate transactions.
+ In this mode, the <option>FREEZE</option> option
+ cannot be used with client-side <command>COPY</command>.
+ Using multiple transactions can speed up data generation by
+ committing data in smaller batches, which may reduce the overhead of a
+ single huge transaction, but also adds per-transaction commit costs.
</para>
</listitem>
</varlistentry>
+
<varlistentry id="pgbench-option-init-steps-v">
<term><literal>v</literal> (Vacuum)</term>
<listitem>
@@ -260,6 +312,7 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
</listitem>
</varlistentry>
+
<varlistentry id="pgbench-option-init-steps-p">
<term><literal>p</literal> (create Primary keys)</term>
<listitem>
@@ -268,6 +321,7 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
</listitem>
</varlistentry>
+
<varlistentry id="pgbench-option-init-steps-f">
<term><literal>f</literal> (create Foreign keys)</term>
<listitem>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1dae918cc09..48f2950e245 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -159,18 +159,36 @@ typedef struct socket_set
/********************************************************************
* some configurable parameters */
-
#define DEFAULT_INIT_STEPS "dtgvp" /* default -I setting */
-#define ALL_INIT_STEPS "dtgGvpf" /* all possible steps */
+#define ALL_INIT_STEPS "dtMScgGUvpf" /* all possible steps */
#define LOG_STEP_SECONDS 5 /* seconds between log messages */
-#define DEFAULT_NXACTS 10 /* default nxacts */
+#define DEFAULT_NXACTS 10 /* default nxacts */
#define MIN_GAUSSIAN_PARAM 2.0 /* minimum parameter for gauss */
#define MIN_ZIPFIAN_PARAM 1.001 /* minimum parameter for zipfian */
#define MAX_ZIPFIAN_PARAM 1000.0 /* maximum parameter for zipfian */
+/* server-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_INSERT_SERIES 'G' /* use INSERT .. SELECT
+ * generate_series to generate
+ * data */
+#define INIT_STEP_GEN_TYPE_INSERT_UNNEST 'U' /* use INSERT .. SELECT unnest
+ * to generate data */
+/* client-side methods to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_TEXT 'g' /* use COPY .. FROM STDIN ..
+ * TEXT to generate data */
+#define INIT_STEP_GEN_TYPE_COPY_BINARY 'c' /* use COPY .. FROM STDIN ..
+ * BINARY to generate data */
+/* data init pseudo steps */
+#define INIT_STEP_GEN_TYPE_SINGLE_XACT 'S' /* switch to init data as
+ * single transaction */
+#define INIT_STEP_GEN_TYPE_MULTI_XACT 'M' /* switch to init data as
+ * multiple transactions */
+
+static bool multi_xact = false; /* init data type (as single or multiple
+ * transactions) */
static int nxacts = 0; /* number of transactions per client */
static int duration = 0; /* duration in seconds */
static int64 end_time = 0; /* when to stop in micro seconds, under -T */
@@ -181,6 +199,19 @@ static int64 end_time = 0; /* when to stop in micro seconds, under -T */
*/
static int scale = 1;
+/*
+ * mode of data generation to use
+ */
+static char data_generation_type = INIT_STEP_GEN_TYPE_COPY_TEXT;
+
+/*
+ * COPY FROM BINARY execution buffer
+ */
+#define BIN_COPY_BUF_SIZE 102400 /* maximum buffer size for COPY FROM
+ * BINARY */
+static char *bin_copy_buffer = NULL; /* buffer for COPY FROM BINARY */
+static int32_t bin_copy_buffer_length = 0; /* current buffer size */
+
/*
* fillfactor. for example, fillfactor = 90 will use only 90 percent
* space during inserts and leave 10 percent free.
@@ -456,6 +487,9 @@ typedef struct StatsData
*/
static pg_time_usec_t epoch_shift;
+/* used to track elapsed time and estimate of the remaining time of data load */
+static pg_time_usec_t data_load_start;
+
/*
* Error status for errors during script execution.
*/
@@ -851,6 +885,7 @@ static bool socket_has_input(socket_set *sa, int fd, int idx);
/* callback used to build rows for COPY during data loading */
typedef void (*initRowMethod) (PQExpBufferData *sql, int64 curr);
+typedef void (*initRowMethodBinary) (PGconn *con, int64_t curr, int32_t parent);
/* callback functions for our flex lexer */
static const PsqlScanCallbacks pgbench_callbacks = {
@@ -913,8 +948,14 @@ usage(void)
" run selected initialization steps, in the specified order\n"
" d: drop any existing pgbench tables\n"
" t: create the tables used by the standard pgbench scenario\n"
- " g: generate data, client-side\n"
- " G: generate data, server-side\n"
+ " to generate data, client-side:\n"
+ " g: COPY .. FROM STDIN .. TEXT\n"
+ " c: COPY .. FROM STDIN .. BINARY\n"
+ " to generate data, server-side:\n"
+ " G: INSERT .. SELECT generate_series\n"
+ " U: INSERT .. SELECT unnest\n"
+ " S: flag to use single transaction to initialize data\n"
+ " M: flag to use multiple transactions to initialize data\n"
" v: invoke VACUUM on the standard tables\n"
" p: create primary key indexes on the standard tables\n"
" f: create foreign keys between the standard tables\n"
@@ -4916,8 +4957,8 @@ initCreateTables(PGconn *con)
static const struct ddlinfo DDLs[] = {
{
"pgbench_history",
- "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22)",
- "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22)",
+ "tid int,bid int,aid int,delta int,mtime timestamp,filler char(22) default ''",
+ "tid int,bid int,aid bigint,delta int,mtime timestamp,filler char(22) default ''",
0
},
{
@@ -4928,8 +4969,8 @@ initCreateTables(PGconn *con)
},
{
"pgbench_accounts",
- "aid int not null,bid int,abalance int,filler char(84)",
- "aid bigint not null,bid int,abalance int,filler char(84)",
+ "aid int not null,bid int,abalance int,filler char(84) default ''",
+ "aid bigint not null,bid int,abalance int,filler char(84) default ''",
1
},
{
@@ -5025,31 +5066,89 @@ initAccount(PQExpBufferData *sql, int64 curr)
}
static void
-initPopulateTable(PGconn *con, const char *table, int64 base,
- initRowMethod init_row)
+showPopulateTableCopyProgress(const char *table, int64 current, int64 total)
+{
+ static int chars = 0;
+ static int prev_chars = 0;
+ static int log_interval = 1;
+
+ /* Stay on the same line if reporting to a terminal */
+ char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+
+ double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - data_load_start);
+ double remaining_sec = ((double) total - current) * elapsed_sec / current;
+
+ /*
+ * If we want to stick with the original logging, print a message each
+ * 100k inserted rows.
+ */
+ if ((!use_quiet) && (current % 100000 == 0))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current one,
+ * add spaces to the current line to fully overwrite any remaining
+ * characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+ }
+ /* let's not call the timing for each row, but only each 100 rows */
+ else if (use_quiet && (current % 100 == 0))
+ {
+ /* have we reached the next interval (or end)? */
+ if ((current == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
+ {
+ chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
+ current, total,
+ (int) ((current * 100) / total),
+ table, elapsed_sec, remaining_sec);
+
+ /*
+ * If the previous progress message is longer than the current
+ * one, add spaces to the current line to fully overwrite any
+ * remaining characters from the previous message.
+ */
+ if (prev_chars > chars)
+ fprintf(stderr, "%*c", prev_chars - chars, ' ');
+ fputc(eol, stderr);
+ prev_chars = chars;
+
+ /* skip to the next interval */
+ log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
+ }
+ }
+
+ if (current + 1 == total && chars != 0)
+ {
+ fprintf(stderr, "%*c", chars, ' '); /* Clear the current line */
+ fputc(eol, stderr);
+ }
+}
+
+static void
+initPopulateTableCopyText(PGconn *con, const char *table, int counter, int64 base,
+ initRowMethod init_row)
{
int n;
- int64 k;
- int chars = 0;
- int prev_chars = 0;
PGresult *res;
PQExpBufferData sql;
char copy_statement[256];
const char *copy_statement_fmt = "copy %s from stdin";
- int64 total = base * scale;
-
- /* used to track elapsed time and estimate of the remaining time */
- pg_time_usec_t start;
- int log_interval = 1;
-
- /* Stay on the same line if reporting to a terminal */
- char eol = isatty(fileno(stderr)) ? '\r' : '\n';
+ int64 start = base * counter;
initPQExpBuffer(&sql);
/* Use COPY with FREEZE on v14 and later for all ordinary tables */
if ((PQserverVersion(con) >= 140000) &&
- get_table_relkind(con, table) == RELKIND_RELATION)
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
copy_statement_fmt = "copy %s from stdin with (freeze on)";
@@ -5065,75 +5164,18 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
PQclear(res);
- start = pg_time_now();
-
- for (k = 0; k < total; k++)
+ for (int64_t i = start; i < start + base; i++)
{
- int64 j = k + 1;
-
- init_row(&sql, k);
+ init_row(&sql, i);
if (PQputline(con, sql.data))
pg_fatal("PQputline failed");
if (CancelRequested)
break;
- /*
- * If we want to stick with the original logging, print a message each
- * 100k inserted rows.
- */
- if ((!use_quiet) && (j % 100000 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
- }
- /* let's not call the timing for each row, but only each 100 rows */
- else if (use_quiet && (j % 100 == 0))
- {
- double elapsed_sec = PG_TIME_GET_DOUBLE(pg_time_now() - start);
- double remaining_sec = ((double) total - j) * elapsed_sec / j;
-
- /* have we reached the next interval (or end)? */
- if ((j == total) || (elapsed_sec >= log_interval * LOG_STEP_SECONDS))
- {
- chars = fprintf(stderr, INT64_FORMAT " of " INT64_FORMAT " tuples (%d%%) of %s done (elapsed %.2f s, remaining %.2f s)",
- j, total,
- (int) ((j * 100) / total),
- table, elapsed_sec, remaining_sec);
-
- /*
- * If the previous progress message is longer than the current
- * one, add spaces to the current line to fully overwrite any
- * remaining characters from the previous message.
- */
- if (prev_chars > chars)
- fprintf(stderr, "%*c", prev_chars - chars, ' ');
- fputc(eol, stderr);
- prev_chars = chars;
-
- /* skip to the next interval */
- log_interval = (int) ceil(elapsed_sec / LOG_STEP_SECONDS);
- }
- }
+ showPopulateTableCopyProgress(table, i, base * scale);
}
- if (chars != 0 && eol != '\n')
- fprintf(stderr, "%*c\r", chars, ' '); /* Clear the current line */
if (PQputline(con, "\\.\n"))
pg_fatal("very last PQputline failed");
@@ -5150,9 +5192,9 @@ initPopulateTable(PGconn *con, const char *table, int64 base,
* a blank-padded string in pgbench_accounts.
*/
static void
-initGenerateDataClientSide(PGconn *con)
+initGenerateDataClientSideTextFrmt(PGconn *con)
{
- fprintf(stderr, "generating data (client-side)...\n");
+ fprintf(stderr, "TEXT mode...\n");
/*
* we do all of this in one transaction to enable the backend's
@@ -5163,64 +5205,549 @@ initGenerateDataClientSide(PGconn *con)
/* truncate away any old data */
initTruncateTables(con);
- /*
- * fill branches, tellers, accounts in that order in case foreign keys
- * already exist
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ /*
+ * fill branches, tellers, accounts in that order in case foreign keys
+ * already exist
+ */
+ initPopulateTableCopyText(con, "pgbench_branches", i, nbranches, initBranch);
+ initPopulateTableCopyText(con, "pgbench_tellers", i, ntellers, initTeller);
+ initPopulateTableCopyText(con, "pgbench_accounts", i, naccounts, initAccount);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+}
+
+
+/*
+ * Save char data to buffer
+ * Kept as separate proc for possible addition of something
+ * like addCharColumn in future
+ */
+static void
+bufferCharData(char *src, int32_t len)
+{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
+ memcpy((char *) bin_copy_buffer + bin_copy_buffer_length, (char *) src, len);
+ bin_copy_buffer_length += len;
+}
+
+/*
+ * Converts platform byte order into network byte order
+ * SPARC doesn't reqire that
+ */
+static void
+bufferData(void *src, int32_t len)
+{
+ Assert(bin_copy_buffer_length + len <= BIN_COPY_BUF_SIZE);
+
+#ifdef __sparc__
+ bufferCharData(src, len);
+#else
+
+ if (len == 1)
+ bufferCharData(src, len);
+ else
+ {
+ for (int32_t i = 0; i < len; i++)
+ {
+ ((char *) bin_copy_buffer + bin_copy_buffer_length)[i] =
+ ((char *) src)[len - i - 1];
+ }
+
+ bin_copy_buffer_length += len;
+ }
+#endif
+}
+
+/*
+ * adds column counter
+ */
+static void
+addColumnCounter(int16_t n)
+{
+ bufferData((void *) &n, sizeof(n));
+}
+
+/*
+ * adds column with inti32 value
+ */
+static void
+addInt32Column(int32_t value)
+{
+ int32_t data = value;
+ int32_t size = sizeof(data);
+
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * adds column with inti64 value
+ */
+static void
+addInt64Column(int64_t value)
+{
+ int64_t data = value;
+ int32_t size = sizeof(data);
+
+ bufferData((void *) &size, sizeof(size));
+ bufferData((void *) &data, sizeof(data));
+}
+
+/*
+ * Starts communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyHeader(PGconn *con)
+{
+ static char header[] = {'P', 'G', 'C', 'O', 'P', 'Y', '\n', '\377', '\r', '\n', '\0',
+ '\0', '\0', '\0', '\0',
+ '\0', '\0', '\0', '\0'};
+
+ PQputCopyData(con, header, 19);
+}
+
+/*
+ * Finishes communication with server for COPY FROM BINARY statement
+ */
+static void
+sendBinaryCopyTrailer(PGconn *con)
+{
+ static char trailer[] = {0xFF, 0xFF};
+
+ PQputCopyData(con, trailer, 2);
+}
+
+/*
+ * Flashes current buffer over network if needed
+ */
+static void
+flushBuffer(PGconn *con, int16_t row_len)
+{
+ PGresult *res;
+
+ if (bin_copy_buffer_length + row_len > BIN_COPY_BUF_SIZE)
+ {
+ res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
+ /* flush current buffer */
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ pg_fatal("It is NOT a COPY command that is currently running");
+
+ PQclear(res);
+ bin_copy_buffer_length = 0;
+ }
+}
+
+/*
+ * Sends current branch row to buffer
+ */
+static void
+initBranchBinary(PGconn *con, int64_t curr, int32_t parent)
+{
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of branches table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - branches table has following columns:
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for bbalance column or sizeof(int32_t)
+ * - 88 bytes for filler column (optional since no requirement for row length)
+ *---
*/
- initPopulateTable(con, "pgbench_branches", nbranches, initBranch);
- initPopulateTable(con, "pgbench_tellers", ntellers, initTeller);
- initPopulateTable(con, "pgbench_accounts", naccounts, initAccount);
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 88);
- executeStatement(con, "commit");
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(2);
+
+ addInt32Column(curr + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
}
/*
- * Fill the standard tables with some data generated on the server
- *
- * As already the case with the client-side data generation, the filler
- * column defaults to NULL in pgbench_branches and pgbench_tellers,
- * and is a blank-padded string in pgbench_accounts.
+ * Sends current teller row to buffer
*/
static void
-initGenerateDataServerSide(PGconn *con)
+initTellerBinary(PGconn *con, int64_t curr, int32_t parent)
{
- PQExpBufferData sql;
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of tellers table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - tellers table has following columns:
+ * - 4 bytes for tid column or sizeof(int32_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for tbalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
+ */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 4) + (4 + 4) + (4 + 4) + (4 + 84);
+
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(3);
+
+ addInt32Column(curr + 1);
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
+}
+
+/*
+ * Sends current account row to buffer
+ */
+static void
+initAccountBinary(PGconn *con, int64_t curr, int32_t parent)
+{
+ /*---
+ * Check documentation about COPY command:
+ * https://www.postgresql.org/docs/current/sql-copy.html
+ *
+ * Each row of accounts table is sent as:
+ * - 2 bytes for number of columns in tuple or sizeof(int16_t)
+ * - then 4 bytes or sizeof(int32_t) in front of each field with length of the field
+ *
+ * - accounts table has following columns (taking into account scale > 20000):
+ * - 8 bytes for aid column or sizeof(int64_t)
+ * - 4 bytes for bid column or sizeof(int32_t)
+ * - 4 bytes for abalance column or sizeof(int32_t)
+ * - 84 bytes for filler column (optional since no requirement for row length)
+ *---
+ */
+ /* following is our max intent at the moment */
+ int16_t max_row_len = 2 + (4 + 8) + (4 + 4) + (4 + 4) + (4 + 84);
+
+ flushBuffer(con, max_row_len);
+
+ addColumnCounter(3);
+
+ if (scale <= SCALE_32BIT_THRESHOLD)
+ addInt32Column(curr + 1);
+ else
+ addInt64Column(curr);
+
+ addInt32Column(curr / parent + 1);
+ addInt32Column(0);
+ /* we don't send filler column here to minimize network traffic and WAL */
+}
+
+/*
+ * Universal wrapper for sending data in binary format
+ */
+static void
+initPopulateTableCopyBinary(PGconn *con, char *table, char *columns,
+ int counter, int64_t base, initRowMethodBinary init_row)
+{
+ int n;
+ PGresult *res;
+ char copy_statement[256];
+ const char *copy_statement_fmt = "copy %s (%s) from stdin (format binary)";
+ int64_t start = base * counter;
+
+ bin_copy_buffer_length = 0;
+
+ /* Use COPY with FREEZE on v14 and later for all ordinary tables */
+ if ((PQserverVersion(con) >= 140000) &&
+ get_table_relkind(con, table) == RELKIND_RELATION &&
+ !multi_xact)
+ copy_statement_fmt = "copy %s (%s) from stdin with (format binary, freeze on)";
+
+ n = pg_snprintf(copy_statement, sizeof(copy_statement), copy_statement_fmt, table, columns);
+ if (n >= sizeof(copy_statement))
+ pg_fatal("invalid buffer size: must be at least %d characters long", n);
+ else if (n == -1)
+ pg_fatal("invalid format string");
+
+ res = PQexec(con, copy_statement);
+
+ if (PQresultStatus(res) != PGRES_COPY_IN)
+ pg_fatal("unexpected copy in result: %s", PQerrorMessage(con));
+ PQclear(res);
+
+
+ sendBinaryCopyHeader(con);
+
+
+ for (int64_t i = start; i < start + base; i++)
+ {
+ init_row(con, i, base);
+
+ if (CancelRequested)
+ break;
+
+ showPopulateTableCopyProgress(table, i, base * scale);
+ }
+
+ res = PQgetResult(con);
+
+ Assert(bin_copy_buffer_length <= BIN_COPY_BUF_SIZE);
+
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ PQputCopyData(con, (char *) bin_copy_buffer, bin_copy_buffer_length);
+ else
+ fprintf(stderr, "Unexpected mode %d instead of %d\n", PQresultStatus(res), PGRES_COPY_IN);
+ PQclear(res);
+
+
+ sendBinaryCopyTrailer(con);
+
+
+ res = PQgetResult(con);
+ if (PQresultStatus(res) == PGRES_COPY_IN)
+ {
+ if (PQputCopyEnd(con, NULL) == 1) /* success */
+ {
+ PQclear(res);
+ res = PQgetResult(con);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+ else
+ fprintf(stderr, "Error: %s\n", PQerrorMessage(con));
+ }
+ PQclear(res);
+}
+
+/*
+ * Wrapper for binary data load
+ */
+static void
+initGenerateDataClientSideBinaryFrmt(PGconn *con)
+{
+
+ fprintf(stderr, "BINARY mode...\n");
- fprintf(stderr, "generating data (server-side)...\n");
+ bin_copy_buffer = pg_malloc(BIN_COPY_BUF_SIZE);
+ bin_copy_buffer_length = 0;
/*
- * we do all of this in one transaction to enable the backend's
- * data-loading optimizations
+ * we do all of this in multiple transactions to minimize load on DB
+ * server and perhaps in future allow load in parallel sessions
*/
executeStatement(con, "begin");
/* truncate away any old data */
initTruncateTables(con);
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ initPopulateTableCopyBinary(con, "pgbench_branches", "bid, bbalance",
+ i, nbranches, initBranchBinary);
+ initPopulateTableCopyBinary(con, "pgbench_tellers", "tid, bid, tbalance",
+ i, ntellers, initTellerBinary);
+ initPopulateTableCopyBinary(con, "pgbench_accounts", "aid, bid, abalance",
+ i, naccounts, initAccountBinary);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
+ pg_free(bin_copy_buffer);
+}
+
+/*
+ * Fill the standard tables with some data generated and sent from the client.
+ */
+static void
+initGenerateDataClientSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (client-side as %s transaction%s) in ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+
+ data_load_start = pg_time_now();
+
+ switch (data_generation_type)
+ {
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ initGenerateDataClientSideTextFrmt(con);
+ break;
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ initGenerateDataClientSideBinaryFrmt(con);
+ break;
+ }
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM generate_series
+ * Possibly as "One transaction per scale" in multi-transaction mode
+ */
+static void
+generateDataInsertSeries(PGconn *con)
+{
+ PQExpBufferData sql;
+
+ fprintf(stderr, "via INSERT .. SELECT generate_series... in multiple TXN(s)\n");
+
initPQExpBuffer(&sql);
- printfPQExpBuffer(&sql,
- "insert into pgbench_branches(bid,bbalance) "
- "select bid, 0 "
- "from generate_series(1, %d) as bid", nbranches * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_tellers(tid,bid,tbalance) "
- "select tid, (tid - 1) / %d + 1, 0 "
- "from generate_series(1, %d) as tid", ntellers, ntellers * scale);
- executeStatement(con, sql.data);
-
- printfPQExpBuffer(&sql,
- "insert into pgbench_accounts(aid,bid,abalance,filler) "
- "select aid, (aid - 1) / %d + 1, 0, '' "
- "from generate_series(1, " INT64_FORMAT ") as aid",
- naccounts, (int64) naccounts * scale);
- executeStatement(con, sql.data);
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int i = 0; i < scale; i++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid, bbalance) "
+ "values(%d, 0)", i + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select tid + 1, tid / %d + 1, 0 "
+ "from generate_series(%d, %d) as tid",
+ ntellers, i * ntellers, (i + 1) * ntellers - 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_accounts(aid, bid, abalance, "
+ "filler) "
+ "select aid + 1, aid / %d + 1, 0, '' "
+ "from generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as aid",
+ naccounts, (int64) i * naccounts,
+ (int64) (i + 1) * naccounts - 1);
+ executeStatement(con, sql.data);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
termPQExpBuffer(&sql);
+}
+
+/*
+ * Generating data via INSERT .. SELECT .. FROM unnest
+ * Possibly as "One transaction per scale" in multi-tansaction mode
+ */
+static void
+generateDataInsertUnnest(PGconn *con)
+{
+ PQExpBufferData sql;
- executeStatement(con, "commit");
+ fprintf(stderr, "via INSERT .. SELECT unnest...\n");
+
+ initPQExpBuffer(&sql);
+
+ executeStatement(con, "begin");
+
+ /* truncate away any old data */
+ initTruncateTables(con);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+
+ for (int s = 0; s < scale; s++)
+ {
+ if (multi_xact)
+ executeStatement(con, "begin");
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_branches(bid,bbalance) "
+ "values(%d, 0)", s + 1);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "insert into pgbench_tellers(tid, bid, tbalance) "
+ "select unnest(array_agg(s.i order by s.i)) as tid, "
+ "%d as bid, 0 as tbalance "
+ "from generate_series(%d, %d) as s(i)",
+ s + 1, s * ntellers + 1, (s + 1) * ntellers);
+ executeStatement(con, sql.data);
+
+ printfPQExpBuffer(&sql,
+ "with data as ("
+ " select generate_series(" INT64_FORMAT ", "
+ INT64_FORMAT ") as i) "
+ "insert into pgbench_accounts(aid, bid, "
+ "abalance, filler) "
+ "select unnest(aid), unnest(bid), 0 as abalance, "
+ "'' as filler "
+ "from (select array_agg(i+1) aid, "
+ "array_agg(i/%d + 1) bid from data)",
+ (int64) s * naccounts + 1,
+ (int64) (s + 1) * naccounts, naccounts);
+ executeStatement(con, sql.data);
+
+ if (multi_xact)
+ executeStatement(con, "commit");
+ }
+
+ if (!multi_xact)
+ executeStatement(con, "commit");
+
+ termPQExpBuffer(&sql);
+}
+
+/*
+ * Fill the standard tables with some data generated on the server side
+ *
+ * As already the case with the client-side data generation, the filler
+ * column defaults to NULL in pgbench_branches and pgbench_tellers,
+ * and is a blank-padded string in pgbench_accounts.
+ */
+static void
+initGenerateDataServerSide(PGconn *con)
+{
+ fprintf(stderr, "generating data (server-side as %s transaction%s) ",
+ multi_xact ? "multiple" : "single", multi_xact ? "s" : "");
+
+ switch (data_generation_type)
+ {
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ generateDataInsertSeries(con);
+ break;
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
+ generateDataInsertUnnest(con);
+ break;
+ }
}
/*
@@ -5306,6 +5833,8 @@ initCreateFKeys(PGconn *con)
static void
checkInitSteps(const char *initialize_steps)
{
+ char data_init_type = 0;
+
if (initialize_steps[0] == '\0')
pg_fatal("no initialization steps specified");
@@ -5317,7 +5846,22 @@ checkInitSteps(const char *initialize_steps)
pg_log_error_detail("Allowed step characters are: \"" ALL_INIT_STEPS "\".");
exit(1);
}
+
+ switch (*step)
+ {
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
+ data_init_type++;
+ break;
+ }
}
+
+ if (data_init_type == 0)
+ pg_log_warning("No data generation type is provided");
+ if (data_init_type > 1)
+ pg_log_warning("More than one type of data initialization is requested");
}
/*
@@ -5355,14 +5899,24 @@ runInitSteps(const char *initialize_steps)
op = "create tables";
initCreateTables(con);
break;
- case 'g':
+ case INIT_STEP_GEN_TYPE_COPY_TEXT:
+ case INIT_STEP_GEN_TYPE_COPY_BINARY:
op = "client-side generate";
+ data_generation_type = *step;
initGenerateDataClientSide(con);
break;
- case 'G':
+ case INIT_STEP_GEN_TYPE_INSERT_SERIES:
+ case INIT_STEP_GEN_TYPE_INSERT_UNNEST:
op = "server-side generate";
+ data_generation_type = *step;
initGenerateDataServerSide(con);
break;
+ case INIT_STEP_GEN_TYPE_SINGLE_XACT:
+ multi_xact = false;
+ break;
+ case INIT_STEP_GEN_TYPE_MULTI_XACT:
+ multi_xact = true;
+ break;
case 'v':
op = "vacuum";
initVacuum(con);
@@ -6940,7 +7494,6 @@ main(int argc, char **argv)
case 'I':
pg_free(initialize_steps);
initialize_steps = pg_strdup(optarg);
- checkInitSteps(initialize_steps);
initialization_option_set = true;
break;
case 'j': /* jobs */
@@ -7245,6 +7798,7 @@ main(int argc, char **argv)
}
}
+ checkInitSteps(initialize_steps);
runInitSteps(initialize_steps);
exit(0);
}
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index b7685ea5d20..6c7783a77f7 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -16,25 +16,30 @@ sub check_data_state
local $Test::Builder::Level = $Test::Builder::Level + 1;
my $node = shift;
my $type = shift;
+ my $sql_result;
- my $sql_result = $node->safe_psql('postgres',
- 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
- );
- is($sql_result, '0',
- "$type: filler column of pgbench_accounts has no NULL data");
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_branches WHERE filler IS NULL;'
);
is($sql_result, '1',
"$type: filler column of pgbench_branches has only NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS null_count FROM pgbench_tellers WHERE filler IS NULL;'
);
is($sql_result, '10',
"$type: filler column of pgbench_tellers has only NULL data");
+
+ $sql_result = $node->safe_psql('postgres',
+ 'SELECT count(*) AS null_count FROM pgbench_accounts WHERE filler IS NULL LIMIT 10;'
+ );
+ is($sql_result, '0',
+ "$type: filler column of pgbench_accounts has no NULL data");
+
$sql_result = $node->safe_psql('postgres',
'SELECT count(*) AS data_count FROM pgbench_history;');
- is($sql_result, '0', "$type: pgbench_history has no data");
+ is($sql_result, '0',
+ "$type: pgbench_history has no data");
}
# start a pgbench specific server
@@ -112,6 +117,7 @@ $node->pgbench(
[qr{Perhaps you need to do initialization}],
'run without init');
+
# Initialize pgbench tables scale 1
$node->pgbench(
'-i', 0,
@@ -125,7 +131,7 @@ $node->pgbench(
'pgbench scale 1 initialization',);
# Check data state, after client-side data generation.
-check_data_state($node, 'client-side');
+check_data_state($node, 'client-side (default options)');
# Again, with all possible options
$node->pgbench(
@@ -143,6 +149,7 @@ $node->pgbench(
qr{done in \d+\.\d\d s }
],
'pgbench scale 1 initialization');
+check_data_state($node, 'client-side (all options)');
# Test interaction of --init-steps with legacy step-selection options
$node->pgbench(
@@ -154,7 +161,7 @@ $node->pgbench(
qr{creating tables},
qr{creating 3 partitions},
qr{creating primary keys},
- qr{generating data \(server-side\)},
+ qr{generating data \(server-side as single transaction\)},
qr{creating foreign keys},
qr{(?!vacuuming)}, # no vacuum
qr{done in \d+\.\d\d s }
@@ -164,6 +171,219 @@ $node->pgbench(
# Check data state, after server-side data generation.
check_data_state($node, 'server-side');
+
+# Test server-side generation with generate_series
+$node->pgbench(
+ '--initialize --init-steps=dtG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series single XACT)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMG',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side generate_series');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (generate_series multiple XACTs)');
+
+
+# Test server-side generation with UNNEST
+$node->pgbench(
+ '--initialize --init-steps=dtU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as single transaction\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMU',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(server-side as multiple transactions\)},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps server-side UNNEST');
+
+# Check data state, after server-side data generation.
+check_data_state($node, 'server-side (unnest)');
+
+
+# Test client-side generation with COPY TEXT
+$node->pgbench(
+ '--initialize --init-steps=dtg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (single XACT #1)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_branches done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_tellers done},
+ qr{\d of \d+ tuples \(\d%\) of pgbench_accounts done},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (single XACT #2)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMg',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side TEXT (multiple XACTs)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (text)');
+
+
+# Test client-side generation with COPY BINARY
+$node->pgbench(
+ '--initialize --init-steps=dtc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (single XACT #1)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (single XACT #2)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+
+# Check data state, after different modes of client-side data generation.
+check_data_state($node, 'client-side (binary)');
+
+$node->pgbench(
+ '--initialize --init-steps=dtMccSc',
+ 0,
+ [qr{^$}],
+ [
+ qr{dropping old tables},
+ qr{creating tables},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as multiple transactions},
+ qr{generating data \(client-side as single transaction},
+ qr{done in \d+\.\d\d s }
+ ],
+ 'pgbench --init-steps client-side BINARY (multiple XACT modes)');
+
+# Check data state, after client-side data generation.
+check_data_state($node, 'client-side (binary different XACT modes in list of --init-steps)');
+
+
# Run all builtin scripts, for a few transactions each
$node->pgbench(
'--transactions=5 -Dfoo=bla --client=2 --protocol=simple --builtin=t'
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 07:55 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Hayato Kuroda (Fujitsu) <[email protected]>
2026-03-09 09:57 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-11 11:53 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-12 14:28 ` Boris Mironov <[email protected]>
2026-03-19 15:29 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Fujii Masao <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Boris Mironov @ 2026-03-12 14:28 UTC (permalink / raw)
To: Madyshev Egor <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; pgsql-hackers
Hi Egor,
Thank you very much for your time reviewing this patch
and guiding through some of its inefficiencies.
Yahoo!
Best regards,
Boris
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 07:55 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Hayato Kuroda (Fujitsu) <[email protected]>
2026-03-09 09:57 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-11 11:53 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-12 14:28 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
@ 2026-03-19 15:29 ` Fujii Masao <[email protected]>
2026-03-20 02:59 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Fujii Masao @ 2026-03-19 15:29 UTC (permalink / raw)
To: Boris Mironov <[email protected]>; +Cc: Madyshev Egor <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; pgsql-hackers
On Thu, Mar 12, 2026 at 11:28 PM Boris Mironov
<[email protected]> wrote:
>
> Hi Egor,
>
> Thank you very much for your time reviewing this patch
> and guiding through some of its inefficiencies.
I like the idea of improving the performance of the initial data load
in pgbench. That's definitely useful.
I ran some tests on my MacBook (server and pgbench on the same machine)
using different data loading options. Here are the results:
------------------------------------------------
[ Client / Text mode ]
pgbench -i -Idtg -s100
done in 13.38 s (drop tables 0.00 s, create tables 0.01 s, client-side
generate 13.38 s).
pgbench -i -Idtg -s1000
done in 151.81 s (drop tables 0.00 s, create tables 0.01 s,
client-side generate 151.81 s).
[ Client / Binary mode ]
pgbench -i -Idtc -s100
done in 18.32 s (drop tables 0.00 s, create tables 0.01 s, client-side
generate 18.31 s).
pgbench -i -Idtc -s1000
done in 204.48 s (drop tables 0.00 s, create tables 0.01 s,
client-side generate 204.47 s).
[ Server / generate_series ]
pgbench -i -IdtG -s100
done in 21.30 s (drop tables 0.00 s, create tables 0.00 s, server-side
generate 21.30 s).
pgbench -i -IdtG -s1000
done in 230.94 s (drop tables 0.00 s, create tables 0.01 s,
server-side generate 230.93 s).
[ Server / Unnest ]
pgbench -i -IdtU -s100
done in 23.16 s (drop tables 0.00 s, create tables 0.00 s, server-side
generate 23.16 s).
pgbench -i -IdtU -s1000
done in 249.08 s (drop tables 0.00 s, create tables 0.01 s,
server-side generate 249.07 s).
------------------------------------------------
In my tests, text mode was faster than binary mode. Also, on the server side,
generate_series() was faster than unnest().
So I'm wondering if there are specific conditions where binary mode or unnest()
performs better. If not, it may not be worth supporting these additional data
loading options...
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-17 04:58 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-21 13:26 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-22 11:02 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-23 07:51 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-01 12:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-02 14:12 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-03 11:13 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-06 07:26 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 05:18 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-09 07:55 ` RE: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Hayato Kuroda (Fujitsu) <[email protected]>
2026-03-09 09:57 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-11 11:53 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-12 14:28 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2026-03-19 15:29 ` Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Fujii Masao <[email protected]>
@ 2026-03-20 02:59 ` Boris Mironov <[email protected]>
0 siblings, 0 replies; 21+ messages in thread
From: Boris Mironov @ 2026-03-20 02:59 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Madyshev Egor <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; pgsql-hackers
Dear Fujii Masao,
Thank you for your interest in this patch.
For convenience I duplicate my results from previous post and
reformat your results below:
pgbench (PostgreSQL) 19devel / client-side generation
Mode
Scale | 1 | 2 | 10 | 20 | 100 | 200 | 1000 | 2000
Mc | 0.15 | 0.29 | 1.39 | 2.93 | 14.79 | 30.78 | 161.52 | 330.63
Sc | 0.13 | 0.27 | 1.37 | 2.69 | 14.71 | 29.99 | 152.83 | 298.88
Mg | 0.17 | 0.30 | 1.58 | 3.26 | 15.91 | 31.95 | 160.31 | 326.46
Sg | 0.20 | 0.38 | 1.66 | 3.39 | 18.72 | 36.26 | 176.54 | 351.66
pgbench (PostgreSQL) 19devel / server-side generation
Mode
Scale | 1 | 2 | 10 | 20 | 100 | 200 | 1000 | 2000
MU | 0.22 | 0.47 | 2.35 | 4.71 | 24.80 | 53.19 | 261.44 | 536.05
SU | 0.22 | 0.44 | 2.35 | 4.78 | 24.42 | 49.16 | 246.41 | 495.51
MG | 0.22 | 0.45 | 2.29 | 4.44 | 24.03 | 50.21 | 256.54 | 544.17
SG | 0.23 | 0.43 | 2.35 | 5.04 | 27.74 | 52.72 | 250.27 | 492.77
Fujii's single run results / client-side generation
Mode
Scale | 100 | 1000
Sc | 18.31 | 204.47
Sg | 13.38 | 151.81
Fujii's single run results / server-side generation
Mode
Scale | 100 | 1000
SU | 23.16 | 249.07
SG | 21.30 | 230.93
> I ran some tests on my MacBook (server and pgbench on the same machine)
In my environment I ran pgbench on different virtual machine from DB
server and win of COPY BINARY over COPY TEXT was clear. All results are
average of 5 runs. You can use "-Idtccccc" parameter to run 5 times
same init mode. By default it will be "single transaction for whole data
set". Otherwise "-IdtMccccc" will execute init in "one transaction per
scale" mode 5 times. Could you please rerun your tests with these
tweaks?
> So I'm wondering if there are specific conditions where binary mode or unnest()
> performs better. If not, it may not be worth supporting these additional data
> loading options...
Besides introduction of new data generation modes this patch prepares ground
for multi-threaded data initialization with introduction of "one COMMIT per scale"
mode. By the way, all of your tests were executed in old (default) "single transaction
for whole data set" mode.
For addition of UNNEST init mode, it was added for PostgreSQL forks like TigerDB
that add columnar mode of data storage. This is quite useful for OLAP systems as
much more efficient way of data retrieval of few columns from wide tables. These
are also quite efficient as databases for monitoring systems like Prometheus.
Best regards,
Boris
________________________________
From: Fujii Masao <[email protected]>
Sent: March 19, 2026 11:29 AM
To: Boris Mironov <[email protected]>
Cc: Madyshev Egor <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; [email protected] <[email protected]>
Subject: Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
On Thu, Mar 12, 2026 at 11:28 PM Boris Mironov
<[email protected]> wrote:
>
> Hi Egor,
>
> Thank you very much for your time reviewing this patch
> and guiding through some of its inefficiencies.
I like the idea of improving the performance of the initial data load
in pgbench. That's definitely useful.
I ran some tests on my MacBook (server and pgbench on the same machine)
using different data loading options. Here are the results:
------------------------------------------------
[ Client / Text mode ]
pgbench -i -Idtg -s100
done in 13.38 s (drop tables 0.00 s, create tables 0.01 s, client-side
generate 13.38 s).
pgbench -i -Idtg -s1000
done in 151.81 s (drop tables 0.00 s, create tables 0.01 s,
client-side generate 151.81 s).
[ Client / Binary mode ]
pgbench -i -Idtc -s100
done in 18.32 s (drop tables 0.00 s, create tables 0.01 s, client-side
generate 18.31 s).
pgbench -i -Idtc -s1000
done in 204.48 s (drop tables 0.00 s, create tables 0.01 s,
client-side generate 204.47 s).
[ Server / generate_series ]
pgbench -i -IdtG -s100
done in 21.30 s (drop tables 0.00 s, create tables 0.00 s, server-side
generate 21.30 s).
pgbench -i -IdtG -s1000
done in 230.94 s (drop tables 0.00 s, create tables 0.01 s,
server-side generate 230.93 s).
[ Server / Unnest ]
pgbench -i -IdtU -s100
done in 23.16 s (drop tables 0.00 s, create tables 0.00 s, server-side
generate 23.16 s).
pgbench -i -IdtU -s1000
done in 249.08 s (drop tables 0.00 s, create tables 0.01 s,
server-side generate 249.07 s).
------------------------------------------------
In my tests, text mode was faster than binary mode. Also, on the server side,
generate_series() was faster than unnest().
So I'm wondering if there are specific conditions where binary mode or unnest()
performs better. If not, it may not be worth supporting these additional data
loading options...
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 21+ messages in thread
end of thread, other threads:[~2026-03-20 02:59 UTC | newest]
Thread overview: 21+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-11-11 13:33 Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY) Boris Mironov <[email protected]>
2025-11-13 10:17 ` Ashutosh Bapat <[email protected]>
2025-11-14 15:21 ` Boris Mironov <[email protected]>
2025-11-17 04:58 ` Ashutosh Bapat <[email protected]>
2025-11-17 12:43 ` Boris Mironov <[email protected]>
2025-11-21 13:26 ` Boris Mironov <[email protected]>
2025-11-22 11:02 ` Boris Mironov <[email protected]>
2025-11-23 07:51 ` Boris Mironov <[email protected]>
2026-03-01 12:18 ` Boris Mironov <[email protected]>
2026-03-02 14:12 ` Madyshev Egor <[email protected]>
2026-03-03 04:06 ` Boris Mironov <[email protected]>
2026-03-03 11:13 ` Boris Mironov <[email protected]>
2026-03-06 07:26 ` Madyshev Egor <[email protected]>
2026-03-06 14:21 ` Boris Mironov <[email protected]>
2026-03-09 05:18 ` Boris Mironov <[email protected]>
2026-03-09 07:55 ` Hayato Kuroda (Fujitsu) <[email protected]>
2026-03-09 09:57 ` Boris Mironov <[email protected]>
2026-03-11 11:53 ` Boris Mironov <[email protected]>
2026-03-12 14:28 ` Boris Mironov <[email protected]>
2026-03-19 15:29 ` Fujii Masao <[email protected]>
2026-03-20 02:59 ` Boris Mironov <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox