pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Fabien <coelho@cri.ensmp.fr>
To: PostgreSQL Developers <pgsql-hackers@postgresql.org>
Subject: pgbench stats per script & other stuff
Date: Fri, 17 Jul 2015 15:50:06 +0200 (CEST)
Message-ID: <alpine.DEB.2.10.1507171544450.26125@sto> (raw )
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
This patch adds per-script statistics & other improvements to pgbench
Rationale: Josh asked for the per-script stats:-)
Some restructuring is done so that all stats (-l --aggregate-interval
--progress --per-script-stats, latency & lag...) share the same structures
and functions to accumulate data. This limits a lot the growth of pgbench
from this patch (+17 lines).
In passing, remove the distinction between internal and external scripts.
Pgbench just execute scripts, some of them may be internal...
As a side effect, all scripts can be accumulated "pgbench -B -N -S -f ..."
would execute 4 scripts, 3 of which internal (tpc-b, simple-update,
select-only and another externally supplied one).
Also add a weight option to change the probability of choosing some scripts
when several are available.
Hmmm... Not sure that the --per-script-stats option is really useful. The
stats could always be shown when several scripts are executed?
sh> ./pgbench -T 3 -B -N -w 2 -S -w 7 --per-script-stats
starting vacuum...end.
transaction type: multiple scripts
scaling factor: 1
query mode: simple
number of clients: 1
number of threads: 1
duration: 3 s
number of transactions actually processed: 3192
latency average: 0.940 ms
tps = 1063.756045 (including connections establishing)
tps = 1065.412737 (excluding connections establishing)
SQL script 0: <builtin: TPC-B (sort of)>
- weight is 1
- 297 transactions (tps = 98.977301)
- latency average = 3.001 ms
- latency stddev = 1.320 ms
SQL script 1: <builtin: simple update>
- weight is 2
- 621 transactions (tps = 206.952539)
- latency average = 2.506 ms
- latency stddev = 1.194 ms
SQL script 2: <builtin: select only>
- weight is 7
- 2274 transactions (tps = 757.826205)
- latency average = 0.236 ms
- latency stddev = 0.083 ms
--
Fabien
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
pgbench-script-stats-1.patch (text/x-diff, 48.7K) download
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 2517a3a..eb571a8 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -261,6 +261,18 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
benchmarking arguments:
<variablelist>
+ <varlistentry>
+ <term><option>-B</option></term>
+ <term><option>--tpc-b</option></term>
+ <listitem>
+ <para>
+ Built-in TPC-B like test which updates three tables and performs
+ an insert on a fourth. See below for details.
+ This is the default if no bench is explicitely specified.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-c</option> <replaceable>clients</></term>
@@ -313,8 +325,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<para>
Read transaction script from <replaceable>filename</>.
See below for details.
- <option>-N</option>, <option>-S</option>, and <option>-f</option>
- are mutually exclusive.
+ Multiple <option>-f</> options are allowed.
</para>
</listitem>
</varlistentry>
@@ -404,10 +415,10 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<term><option>--skip-some-updates</option></term>
<listitem>
<para>
- Do not update <structname>pgbench_tellers</> and
- <structname>pgbench_branches</>.
- This will avoid update contention on these tables, but
- it makes the test case even less like TPC-B.
+ Built-in test which updates only one table compared to <option>-B</>.
+ This avoids update contention on <structname>pgbench_tellers</>
+ and <structname>pgbench_branches</>, but it makes the test case
+ even less like TPC-B.
</para>
</listitem>
</varlistentry>
@@ -499,9 +510,9 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
Report the specified scale factor in <application>pgbench</>'s
output. With the built-in tests, this is not necessary; the
correct scale factor will be detected by counting the number of
- rows in the <structname>pgbench_branches</> table. However, when testing
- custom benchmarks (<option>-f</> option), the scale factor
- will be reported as 1 unless this option is used.
+ rows in the <structname>pgbench_branches</> table.
+ However, when testing only custom benchmarks (<option>-f</> option),
+ the scale factor will be reported as 1 unless this option is used.
</para>
</listitem>
</varlistentry>
@@ -511,7 +522,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<term><option>--select-only</option></term>
<listitem>
<para>
- Perform select-only transactions instead of TPC-B-like test.
+ Built-in test with select-only transactions.
</para>
</listitem>
</varlistentry>
@@ -552,6 +563,20 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</varlistentry>
<varlistentry>
+ <term><option>-w</option></term>
+ <term><option>--weight</option></term>
+ <listitem>
+ <para>
+ Set the integer weight for the previous test script
+ (<option>-B</>, <option>-N</>, <option>-S</> or <option>-f</>).
+ When several test scripts are used, the relative probability of
+ drawing each test is proportional to these weights.
+ The default weight is <literal>1</>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>--aggregate-interval=<replaceable>seconds</></option></term>
<listitem>
<para>
@@ -567,6 +592,16 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</varlistentry>
<varlistentry>
+ <term><option>--per-script-stats</option></term>
+ <listitem>
+ <para>
+ Report some statistics per script run by pgbench.
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
<term><option>--sampling-rate=<replaceable>rate</></option></term>
<listitem>
<para>
@@ -661,7 +696,17 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<title>What is the <quote>Transaction</> Actually Performed in pgbench?</title>
<para>
- The default transaction script issues seven commands per transaction:
+ Pgbench executes test scripts chosen randomly from a specified list.
+ They include built-in scripts (<option>-B</>, <option>-N</> and
+ <option>-S</>) and custom scripts provided by the user with <option>-f</>.
+ Each test may be given a relative weight (<option>-w</>)
+ specified <emphasis>after</> the test about which it applies so as
+ to change its draw probability.
+ </para>
+
+ <para>
+ The default transaction script (also invoked with <option>-B</>)
+ issues seven commands per transaction:
</para>
<orderedlist>
@@ -689,10 +734,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
benchmark scenarios by replacing the default transaction script
(described above) with a transaction script read from a file
(<option>-f</option> option). In this case a <quote>transaction</>
- counts as one execution of a script file. You can even specify
- multiple scripts (multiple <option>-f</option> options), in which
- case a random one of the scripts is chosen each time a client session
- starts a new transaction.
+ counts as one execution of a script file.
</para>
<para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e839fa3..c9d3cc6 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -162,11 +162,10 @@ bool use_log; /* log transaction latencies to a file */
bool use_quiet; /* quiet logging onto stderr */
int agg_interval; /* log aggregates instead of individual
* transactions */
+bool per_script_stats = false; /* whether to collect stats per script */
int progress = 0; /* thread progress report every this seconds */
-int progress_nclients = 0; /* number of clients for progress
- * report */
-int progress_nthreads = 0; /* number of threads for progress
- * report */
+int nclients = 1; /* number of clients */
+int nthreads = 1; /* number of threads */
bool is_connect; /* establish connection for each transaction */
bool is_latencies; /* report per-command latencies */
int main_pid; /* main process id used in log filename */
@@ -186,13 +185,39 @@ typedef struct
char *value; /* its value */
} Variable;
-#define MAX_FILES 128 /* max number of SQL script files allowed */
+#define MAX_SCRIPTS 128 /* max number of SQL script files allowed */
#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
/*
- * structures used in custom query mode
+ * simple data structure to keep stats about something.
+ * probably the first value should be kept and used as an offset for
+ * better numerical stability...
+ */
+typedef struct
+{
+ int64_t count; /* how many values where encountered */
+ double min; /* the minimum seen */
+ double max; /* the maximum seen */
+ double sum; /* sum of values */
+ double sum2; /* sum of squared values */
+} SimpleStats;
+
+/* data structure to hold various statistics.
+ * it is used for interval statistics as well as file statistics.
*/
+typedef struct
+{
+ long start_time; /* when the interval starts, for aggregates */
+ int64 cnt; /* number of transactions */
+ int64 skipped; /* number of transactions skipped under --rate
+ * and --latency-limit */
+ SimpleStats latency;
+ SimpleStats lag;
+} StatsData;
+/*
+ * structures used in custom query mode
+ */
typedef struct
{
PGconn *con; /* connection handle to DB */
@@ -208,14 +233,12 @@ typedef struct
instr_time txn_begin; /* used for measuring schedule lag times */
instr_time stmt_begin; /* used for measuring statement latencies */
bool is_throttled; /* whether transaction throttling is done */
- int use_file; /* index in sql_files for this client */
- bool prepared[MAX_FILES];
+ int use_file; /* index in sql_scripts for this client */
+ bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */
/* per client collected stats */
- int cnt; /* xacts count */
+ int64 cnt; /* transaction count */
int ecnt; /* error count */
- int64 txn_latencies; /* cumulated latencies */
- int64 txn_sqlats; /* cumulated square latencies */
} CState;
/*
@@ -227,19 +250,16 @@ typedef struct
pthread_t thread; /* thread handle */
CState *state; /* array of CState */
int nstate; /* length of state[] */
- instr_time start_time; /* thread start time */
- instr_time *exec_elapsed; /* time spent executing cmds (per Command) */
- int *exec_count; /* number of cmd executions (per Command) */
- unsigned short random_state[3]; /* separate randomness for each thread */
- int64 throttle_trigger; /* previous/next throttling (us) */
+ unsigned short random_state[3]; /* separate randomness for each thread */
+ int64 throttle_trigger; /* previous/next throttling (us) */
/* per thread collected stats */
+ instr_time start_time; /* thread start time */
instr_time conn_time;
- int64 throttle_lag; /* total transaction lag behind throttling */
- int64 throttle_lag_max; /* max transaction lag */
- int64 throttle_latency_skipped; /* lagging transactions
- * skipped */
- int64 latency_late; /* late transactions */
+ instr_time *exec_elapsed; /* time spent executing cmds (per Command) */
+ int *exec_count; /* number of cmd executions (per Command) */
+ StatsData stats;
+ int64 latency_late; /* executed but late transactions */
} TState;
#define INVALID_THREAD ((pthread_t) 0)
@@ -273,33 +293,26 @@ typedef struct
PgBenchExpr *expr; /* parsed expression */
} Command;
+/* SQL script to be executed, either a file or an internal script
+ */
typedef struct
{
-
- long start_time; /* when does the interval start */
- int cnt; /* number of transactions */
- int skipped; /* number of transactions skipped under --rate
- * and --latency-limit */
-
- double min_latency; /* min/max latencies */
- double max_latency;
- double sum_latency; /* sum(latency), sum(latency^2) - for
- * estimates */
- double sum2_latency;
-
- double min_lag;
- double max_lag;
- double sum_lag; /* sum(lag) */
- double sum2_lag; /* sum(lag*lag) */
-} AggVals;
-
-static Command **sql_files[MAX_FILES]; /* SQL script files */
-static int num_files; /* number of script files */
+ const char *name;
+ int weight;
+ Command **commands;
+ StatsData stats;
+} SQLScript;
+
+static SQLScript sql_script[MAX_SCRIPTS];
+static int num_scripts; /* number of script in sql_script[] */
static int num_commands = 0; /* total number of Command structs */
+static int total_weight = 0;
+
static int debug = 0; /* debug flag */
-/* default scenario */
-static char *tpc_b = {
+/* -B case, default scenario */
+static const char *b_bench = "<builtin: TPC-B (sort of)>";
+static const char *tpc_b = {
"\\set nbranches " CppAsString2(nbranches) " * :scale\n"
"\\set ntellers " CppAsString2(ntellers) " * :scale\n"
"\\set naccounts " CppAsString2(naccounts) " * :scale\n"
@@ -317,7 +330,8 @@ static char *tpc_b = {
};
/* -N case */
-static char *simple_update = {
+static const char *n_bench = "<builtin: simple update>";
+static const char *simple_update = {
"\\set nbranches " CppAsString2(nbranches) " * :scale\n"
"\\set ntellers " CppAsString2(ntellers) " * :scale\n"
"\\set naccounts " CppAsString2(naccounts) " * :scale\n"
@@ -333,7 +347,8 @@ static char *simple_update = {
};
/* -S case */
-static char *select_only = {
+static const char *s_bench = "<builtin: select only>";
+static const char *select_only = {
"\\set naccounts " CppAsString2(naccounts) " * :scale\n"
"\\setrandom aid 1 :naccounts\n"
"SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n"
@@ -343,8 +358,7 @@ static char *select_only = {
static void setalarm(int seconds);
static void *threadRun(void *arg);
-static void doLog(TState *thread, CState *st, FILE *logfile, instr_time *now,
- AggVals *agg, bool skipped);
+static void doTxStats(TState*, CState*, instr_time*, bool, FILE*, StatsData*);
static void
usage(void)
@@ -364,11 +378,12 @@ usage(void)
" --tablespace=TABLESPACE create tables in the specified tablespace\n"
" --unlogged-tables create tables as unlogged tables\n"
"\nBenchmarking options:\n"
+ " -B, --tpc-b add TPC-B (sort-of) internal bench\n"
" -c, --client=NUM number of concurrent database clients (default: 1)\n"
" -C, --connect establish new connection for each transaction\n"
" -D, --define=VARNAME=VALUE\n"
" define variable for use by custom script\n"
- " -f, --file=FILENAME read transaction script from FILENAME\n"
+ " -f, --file=FILENAME add transaction script from FILENAME\n"
" -j, --jobs=NUM number of threads (default: 1)\n"
" -l, --log write transaction times to log file\n"
" -L, --latency-limit=NUM count transactions lasting more than NUM ms\n"
@@ -376,17 +391,19 @@ usage(void)
" -M, --protocol=simple|extended|prepared\n"
" protocol for submitting queries (default: simple)\n"
" -n, --no-vacuum do not run VACUUM before tests\n"
- " -N, --skip-some-updates skip updates of pgbench_tellers and pgbench_branches\n"
+ " -N, --skip-some-updates add simple updates internal bench\n"
" -P, --progress=NUM show thread progress report every NUM seconds\n"
" -r, --report-latencies report average latency per command\n"
" -R, --rate=NUM target rate in transactions per second\n"
" -s, --scale=NUM report this scale factor in output\n"
- " -S, --select-only perform SELECT-only transactions\n"
+ " -S, --select-only add SELECT-only internal bench\n"
" -t, --transactions=NUM number of transactions each client runs (default: 10)\n"
" -T, --time=NUM duration of benchmark test in seconds\n"
" -v, --vacuum-all vacuum all four standard tables before tests\n"
+ " -w, --weight=NUM set weight for previous script\n"
" --aggregate-interval=NUM aggregate data over NUM seconds\n"
" --sampling-rate=NUM fraction of transactions to log (e.g. 0.01 for 1%%)\n"
+ " --per-file-stats report per-file statistics\n"
"\nCommon options:\n"
" -d, --debug print debugging output\n"
" -h, --host=HOSTNAME database server host or socket directory\n"
@@ -578,6 +595,49 @@ getPoissonRand(TState *thread, int64 center)
return (int64) (-log(uniform) * ((double) center) + 0.5);
}
+static void
+initSimpleStats(SimpleStats * ss)
+{
+ memset(ss, 0, sizeof(SimpleStats));
+}
+
+static void
+doSimpleStats(SimpleStats *ss, double val)
+{
+ if (ss->count == 0 || val < ss->min)
+ ss->min = val;
+ if (ss->count == 0 || val > ss->max)
+ ss->max = val;
+ ss->count ++;
+ ss->sum += val;
+ ss->sum2 += val * val;
+}
+
+static void
+appendSimpleStats(SimpleStats *acc, SimpleStats *ss)
+{
+ if (acc->count == 0 || ss->min < acc->min)
+ acc->min = ss->min;
+ if (acc->count == 0 || ss->max > acc->max)
+ acc->max = ss->max;
+ acc->count += ss->count;
+ acc->sum += ss->sum;
+ acc->sum2 += ss->sum2;
+}
+
+static void
+initStats(StatsData *sd, double start_time)
+{
+ sd->cnt = 0;
+ sd->skipped = 0;
+ initSimpleStats(& sd->latency);
+ initSimpleStats(& sd->lag);
+
+ /* not necessarily overriden? */
+ if (start_time)
+ sd->start_time = start_time;
+}
+
/* call PQexec() and exit() on failure */
static void
executeStatement(PGconn *con, const char *sql)
@@ -1097,33 +1157,21 @@ clientDone(CState *st, bool ok)
return false; /* always false */
}
-static void
-agg_vals_init(AggVals *aggs, instr_time start)
+static int
+chooseScript(TState *thread)
{
- /* basic counters */
- aggs->cnt = 0; /* number of transactions (includes skipped) */
- aggs->skipped = 0; /* xacts skipped under --rate --latency-limit */
-
- aggs->sum_latency = 0; /* SUM(latency) */
- aggs->sum2_latency = 0; /* SUM(latency*latency) */
+ int i = 0, w = 0, wc = (int) getrand(thread, 0, total_weight - 1);
- /* min and max transaction duration */
- aggs->min_latency = 0;
- aggs->max_latency = 0;
+ do {
+ w += sql_script[i++].weight;
+ } while (w <= wc);
- /* schedule lag counters */
- aggs->sum_lag = 0;
- aggs->sum2_lag = 0;
- aggs->min_lag = 0;
- aggs->max_lag = 0;
-
- /* start of the current interval */
- aggs->start_time = INSTR_TIME_GET_DOUBLE(start);
+ return i - 1;
}
/* return false iff client should be disconnected */
static bool
-doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, AggVals *agg)
+doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, StatsData *agg)
{
PGresult *res;
Command **commands;
@@ -1140,7 +1188,7 @@ doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, AggVa
INSTR_TIME_SET_ZERO(now);
top:
- commands = sql_files[st->use_file];
+ commands = sql_script[st->use_file].commands;
/*
* Handle throttling once per transaction by sleeping. It is simpler to
@@ -1176,11 +1224,8 @@ top:
now_us = INSTR_TIME_GET_MICROSEC(now);
while (thread->throttle_trigger < now_us - latency_limit)
{
- thread->throttle_latency_skipped++;
-
- if (logfile)
- doLog(thread, st, logfile, &now, agg, true);
-
+ doTxStats(thread, st, &now, true, logfile, agg);
+ /* next rendez-vous */
wait = getPoissonRand(thread, throttle_delay);
thread->throttle_trigger += wait;
st->txn_scheduled = thread->throttle_trigger;
@@ -1208,11 +1253,8 @@ top:
if (st->throttling)
{
/* Measure lag of throttled transaction relative to target */
- int64 lag = now_us - st->txn_scheduled;
-
- thread->throttle_lag += lag;
- if (lag > thread->throttle_lag_max)
- thread->throttle_lag_max = lag;
+ //doSimpleStats(& thread->lag,
+ // (double) now_us - st->txn_scheduled);
st->throttling = false;
}
}
@@ -1253,35 +1295,10 @@ top:
/* transaction finished: calculate latency and log the transaction */
if (commands[st->state + 1] == NULL)
{
- /* only calculate latency if an option is used that needs it */
- if (progress || throttle_delay || latency_limit)
- {
- int64 latency;
-
- if (INSTR_TIME_IS_ZERO(now))
- INSTR_TIME_SET_CURRENT(now);
-
- latency = INSTR_TIME_GET_MICROSEC(now) - st->txn_scheduled;
-
- st->txn_latencies += latency;
-
- /*
- * XXX In a long benchmark run of high-latency transactions,
- * this int64 addition eventually overflows. For example, 100
- * threads running 10s transactions will overflow it in 2.56
- * hours. With a more-typical OLTP workload of .1s
- * transactions, overflow would take 256 hours.
- */
- st->txn_sqlats += latency * latency;
-
- /* record over the limit transactions if needed. */
- if (latency_limit && latency > latency_limit)
- thread->latency_late++;
- }
-
- /* record the time it took in the log */
- if (logfile)
- doLog(thread, st, logfile, &now, agg, false);
+ if (progress || throttle_delay || latency_limit || per_script_stats || logfile)
+ doTxStats(thread, st, &now, false, logfile, agg);
+ else
+ thread->stats.cnt ++;
}
if (commands[st->state]->type == SQL_COMMAND)
@@ -1324,8 +1341,9 @@ top:
if (commands[st->state] == NULL)
{
st->state = 0;
- st->use_file = (int) getrand(thread, 0, num_files - 1);
- commands = sql_files[st->use_file];
+ st->use_file = num_scripts==1? 0: chooseScript(thread);
+
+ commands = sql_script[st->use_file].commands;
st->is_throttled = false;
/*
@@ -1368,7 +1386,7 @@ top:
}
/* Record transaction start time under logging, progress or throttling */
- if ((logfile || progress || throttle_delay || latency_limit) && st->state == 0)
+ if ((logfile || progress || throttle_delay || latency_limit || per_script_stats) && st->state == 0)
{
INSTR_TIME_SET_CURRENT(st->txn_begin);
@@ -1694,16 +1712,31 @@ top:
return true;
}
+static void
+doStats(StatsData *stats, bool skipped, double lat, double lag)
+{
+ stats->cnt ++;
+
+ if (skipped)
+ /* no latency to record on skipped transactions */
+ stats->skipped ++;
+ else
+ {
+ doSimpleStats(& stats->latency, lat);
+
+ /* and possibly the same for schedule lag */
+ if (throttle_delay)
+ doSimpleStats(& stats->lag, lag);
+ }
+}
+
/*
* print log entry after completing one transaction.
*/
static void
-doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
- bool skipped)
+doLog(TState *thread, CState *st, FILE *logfile, instr_time *now,
+ StatsData *agg, bool skipped, double latency, double lag)
{
- double lag;
- double latency;
-
/*
* Skip the log entry if sampling is enabled and this row doesn't belong
* to the random sample.
@@ -1712,15 +1745,6 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
pg_erand48(thread->random_state) > sample_rate)
return;
- if (INSTR_TIME_IS_ZERO(*now))
- INSTR_TIME_SET_CURRENT(*now);
-
- latency = (double) (INSTR_TIME_GET_MICROSEC(*now) - st->txn_scheduled);
- if (skipped)
- lag = latency;
- else
- lag = (double) (INSTR_TIME_GET_MICROSEC(st->txn_begin) - st->txn_scheduled);
-
/* should we aggregate the results or not? */
if (agg_interval > 0)
{
@@ -1730,39 +1754,7 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
*/
if (agg->start_time + agg_interval >= INSTR_TIME_GET_DOUBLE(*now))
{
- agg->cnt += 1;
- if (skipped)
- {
- /*
- * there is no latency to record if the transaction was
- * skipped
- */
- agg->skipped += 1;
- }
- else
- {
- agg->sum_latency += latency;
- agg->sum2_latency += latency * latency;
-
- /* first in this aggregation interval */
- if ((agg->cnt == 1) || (latency < agg->min_latency))
- agg->min_latency = latency;
-
- if ((agg->cnt == 1) || (latency > agg->max_latency))
- agg->max_latency = latency;
-
- /* and the same for schedule lag */
- if (throttle_delay)
- {
- agg->sum_lag += lag;
- agg->sum2_lag += lag * lag;
-
- if ((agg->cnt == 1) || (lag < agg->min_lag))
- agg->min_lag = lag;
- if ((agg->cnt == 1) || (lag > agg->max_lag))
- agg->max_lag = lag;
- }
- }
+ doStats(agg, skipped, latency, lag);
}
else
{
@@ -1777,52 +1769,34 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
* usage), so we don't need to handle this in a special way
* (see below).
*/
- fprintf(logfile, "%ld %d %.0f %.0f %.0f %.0f",
+ fprintf(logfile, "%ld " INT64_FORMAT " %.0f %.0f %.0f %.0f",
agg->start_time,
agg->cnt,
- agg->sum_latency,
- agg->sum2_latency,
- agg->min_latency,
- agg->max_latency);
+ agg->latency.sum,
+ agg->latency.sum2,
+ agg->latency.min,
+ agg->latency.max);
if (throttle_delay)
{
fprintf(logfile, " %.0f %.0f %.0f %.0f",
- agg->sum_lag,
- agg->sum2_lag,
- agg->min_lag,
- agg->max_lag);
+ agg->lag.sum,
+ agg->lag.sum2,
+ agg->lag.min,
+ agg->lag.max);
if (latency_limit)
- fprintf(logfile, " %d", agg->skipped);
+ fprintf(logfile, " " INT64_FORMAT, agg->skipped);
}
fputc('\n', logfile);
- /* move to the next inteval */
- agg->start_time = agg->start_time + agg_interval;
+ /* move to the next interval */
+ agg->start_time += agg_interval;
/* reset for "no transaction" intervals */
- agg->cnt = 0;
- agg->skipped = 0;
- agg->min_latency = 0;
- agg->max_latency = 0;
- agg->sum_latency = 0;
- agg->sum2_latency = 0;
- agg->min_lag = 0;
- agg->max_lag = 0;
- agg->sum_lag = 0;
- agg->sum2_lag = 0;
+ initStats(agg, 0.0);
}
/* reset the values to include only the current transaction. */
- agg->cnt = 1;
- agg->skipped = skipped ? 1 : 0;
- agg->min_latency = latency;
- agg->max_latency = latency;
- agg->sum_latency = skipped ? 0.0 : latency;
- agg->sum2_latency = skipped ? 0.0 : latency * latency;
- agg->min_lag = lag;
- agg->max_lag = lag;
- agg->sum_lag = lag;
- agg->sum2_lag = lag * lag;
+ doStats(agg, skipped, latency, lag);
}
}
else
@@ -1832,21 +1806,21 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
/* This is more than we really ought to know about instr_time */
if (skipped)
- fprintf(logfile, "%d %d skipped %d %ld %ld",
+ fprintf(logfile, "%d " INT64_FORMAT " skipped %d %ld %ld",
st->id, st->cnt, st->use_file,
(long) now->tv_sec, (long) now->tv_usec);
else
- fprintf(logfile, "%d %d %.0f %d %ld %ld",
+ fprintf(logfile, "%d " INT64_FORMAT " %.0f %d %ld %ld",
st->id, st->cnt, latency, st->use_file,
(long) now->tv_sec, (long) now->tv_usec);
#else
/* On Windows, instr_time doesn't provide a timestamp anyway */
if (skipped)
- fprintf(logfile, "%d %d skipped %d 0 0",
+ fprintf(logfile, "%d "INT64_FORMAT" skipped %d 0 0",
st->id, st->cnt, st->use_file);
else
- fprintf(logfile, "%d %d %.0f %d 0 0",
+ fprintf(logfile, "%d "INT64_FORMAT" %.0f %d 0 0",
st->id, st->cnt, latency, st->use_file);
#endif
if (throttle_delay)
@@ -1855,6 +1829,44 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg,
}
}
+/*
+ * end of transaction statistics
+ */
+static void
+doTxStats(TState *thread, CState *st, instr_time *now,
+ bool skipped, FILE *logfile, StatsData *agg)
+{
+ double latency = 0.0, lag = 0.0;
+
+ if ((!skipped || agg_interval) && INSTR_TIME_IS_ZERO(*now))
+ INSTR_TIME_SET_CURRENT(*now);
+
+ if (!skipped)
+ {
+ /* compute latency & lag if needed */
+ latency = INSTR_TIME_GET_MICROSEC(*now) - st->txn_scheduled;
+ lag = INSTR_TIME_GET_MICROSEC(st->txn_begin) - st->txn_scheduled;
+ }
+
+ if (progress || throttle_delay || latency_limit)
+ {
+ doStats(& thread->stats, skipped, latency, lag);
+
+ /* record over the limit transactions if needed. */
+ if (latency_limit && latency > latency_limit)
+ thread->latency_late++;
+ }
+ else
+ thread->stats.cnt ++;
+
+ if (use_log)
+ doLog(thread, st, logfile, now, agg, skipped, latency, lag);
+
+ if (per_script_stats) /* mutex? hmmm... these are only statistics */
+ doStats(& sql_script[st->use_file].stats, skipped, latency, lag);
+}
+
+
/* discard connections */
static void
disconnect_all(CState *state, int length)
@@ -2475,7 +2487,7 @@ read_line_from_file(FILE *fd)
return NULL;
}
-static int
+static Command **
process_file(char *filename)
{
#define COMMANDS_ALLOC_NUM 128
@@ -2487,12 +2499,6 @@ process_file(char *filename)
char *buf;
int alloc_num;
- if (num_files >= MAX_FILES)
- {
- fprintf(stderr, "at most %d SQL files are allowed\n", MAX_FILES);
- exit(1);
- }
-
alloc_num = COMMANDS_ALLOC_NUM;
my_commands = (Command **) pg_malloc(sizeof(Command *) * alloc_num);
@@ -2503,7 +2509,7 @@ process_file(char *filename)
fprintf(stderr, "could not open file \"%s\": %s\n",
filename, strerror(errno));
pg_free(my_commands);
- return false;
+ return NULL;
}
lineno = 0;
@@ -2535,13 +2541,11 @@ process_file(char *filename)
my_commands[index] = NULL;
- sql_files[num_files++] = my_commands;
-
- return true;
+ return my_commands;
}
static Command **
-process_builtin(char *tb, const char *source)
+process_builtin(const char *tb, const char *source)
{
#define COMMANDS_ALLOC_NUM 128
@@ -2595,35 +2599,54 @@ process_builtin(char *tb, const char *source)
return my_commands;
}
+static void
+addScript(const char *name, Command ** commands)
+{
+ if (commands == NULL)
+ {
+ fprintf(stderr, "empty commands for %s", name);
+ exit(1);
+ }
+
+ if (num_scripts >= MAX_SCRIPTS)
+ {
+ fprintf(stderr, "at most %d SQL scripts are allowed\n", MAX_SCRIPTS);
+ exit(1);
+ }
+
+ sql_script[num_scripts].name = name;
+ sql_script[num_scripts].weight = 1;
+ sql_script[num_scripts].commands = commands;
+ initStats(& sql_script[num_scripts].stats, 0.0);
+ num_scripts++;
+}
+
+static void
+printSimpleStats(char *prefix, SimpleStats *ss)
+{
+ /* print NaN if no transactions where executed */
+ double latency = ss->sum / ss->count;
+ double stddev = sqrt(ss->sum2 / ss->count - latency*latency);
+ printf("%s average = %.3f ms\n", prefix, 0.001 * latency);
+ printf("%s stddev = %.3f ms\n", prefix, 0.001 * stddev);
+}
+
/* print out results */
static void
-printResults(int ttype, int64 normal_xacts, int nclients,
- TState *threads, int nthreads,
- instr_time total_time, instr_time conn_total_time,
- int64 total_latencies, int64 total_sqlats,
- int64 throttle_lag, int64 throttle_lag_max,
- int64 throttle_latency_skipped, int64 latency_late)
+printResults(TState *threads, StatsData *total, instr_time total_time,
+ instr_time conn_total_time, int latency_late)
{
double time_include,
tps_include,
tps_exclude;
- char *s;
time_include = INSTR_TIME_GET_DOUBLE(total_time);
- tps_include = normal_xacts / time_include;
- tps_exclude = normal_xacts / (time_include -
+ tps_include = total->cnt / time_include;
+ tps_exclude = total->cnt / (time_include -
(INSTR_TIME_GET_DOUBLE(conn_total_time) / nthreads));
- if (ttype == 0)
- s = "TPC-B (sort of)";
- else if (ttype == 2)
- s = "Update only pgbench_accounts";
- else if (ttype == 1)
- s = "SELECT only";
- else
- s = "Custom query";
-
- printf("transaction type: %s\n", s);
+ printf("transaction type: %s\n",
+ num_scripts == 1? sql_script[0].name: "multiple scripts");
printf("scaling factor: %d\n", scale);
printf("query mode: %s\n", QUERYMODE[querymode]);
printf("number of clients: %d\n", nclients);
@@ -2631,49 +2654,38 @@ printResults(int ttype, int64 normal_xacts, int nclients,
if (duration <= 0)
{
printf("number of transactions per client: %d\n", nxacts);
- printf("number of transactions actually processed: " INT64_FORMAT "/" INT64_FORMAT "\n",
- normal_xacts, (int64) nxacts * nclients);
+ printf("number of transactions actually processed: "INT64_FORMAT"/%d\n",
+ total->cnt, nxacts * nclients);
}
else
{
printf("duration: %d s\n", duration);
- printf("number of transactions actually processed: " INT64_FORMAT "\n",
- normal_xacts);
+ printf("number of transactions actually processed: "INT64_FORMAT"\n",
+ total->cnt);
}
/* Remaining stats are nonsensical if we failed to execute any xacts */
- if (normal_xacts <= 0)
+ if (total->cnt <= 0)
return;
if (throttle_delay && latency_limit)
- printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n",
- throttle_latency_skipped,
- 100.0 * throttle_latency_skipped / (throttle_latency_skipped + normal_xacts));
+ printf("number of transactions skipped: "INT64_FORMAT" (%.3f %%)\n",
+ total->skipped,
+ 100.0 * total->skipped / (total->skipped + total->cnt));
if (latency_limit)
- printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT " (%.3f %%)\n",
+ printf("number of transactions above the %.1f ms latency limit: %d (%.3f %%)\n",
latency_limit / 1000.0, latency_late,
- 100.0 * latency_late / (throttle_latency_skipped + normal_xacts));
+ 100.0 * latency_late / (total->skipped + total->cnt));
if (throttle_delay || progress || latency_limit)
- {
- /* compute and show latency average and standard deviation */
- double latency = 0.001 * total_latencies / normal_xacts;
- double sqlat = (double) total_sqlats / normal_xacts;
-
- printf("latency average: %.3f ms\n"
- "latency stddev: %.3f ms\n",
- latency, 0.001 * sqrt(sqlat - 1000000.0 * latency * latency));
- }
+ printSimpleStats("latency", & total->latency);
else
- {
/* only an average latency computed from the duration is available */
printf("latency average: %.3f ms\n",
- 1000.0 * duration * nclients / normal_xacts);
- }
+ 1000.0 * duration * nclients / total->cnt);
if (throttle_delay)
- {
/*
* Report average transaction lag under rate limit throttling. This
* is the delay between scheduled and actual start times for the
@@ -2681,27 +2693,50 @@ printResults(int ttype, int64 normal_xacts, int nclients,
* the database load, or the Poisson throttling process.
*/
printf("rate limit schedule lag: avg %.3f (max %.3f) ms\n",
- 0.001 * throttle_lag / normal_xacts, 0.001 * throttle_lag_max);
- }
+ 0.001 * total->lag.sum / total->cnt, 0.001 * total->lag.max);
printf("tps = %f (including connections establishing)\n", tps_include);
printf("tps = %f (excluding connections establishing)\n", tps_exclude);
+ /* Report per-file data */
+ if (per_script_stats)
+ {
+ int i;
+
+ for (i = 0; i < num_scripts; i++)
+ {
+ printf("SQL script %d: %s\n"
+ " - weight is %d\n"
+ " - "INT64_FORMAT" transactions (tps = %f)\n",
+ i, sql_script[i].name,
+ sql_script[i].weight,
+ sql_script[i].stats.cnt, sql_script[i].stats.cnt / time_include);
+
+ if (latency_limit)
+ printf(" - number of transactions skipped: "INT64_FORMAT" (%.3f%%)\n",
+ sql_script[i].stats.skipped,
+ 100.0 * sql_script[i].stats.skipped /
+ (sql_script[i].stats.skipped + sql_script[i].stats.cnt));
+
+ printSimpleStats(" - latency", & sql_script[i].stats.latency);
+ }
+ }
+
/* Report per-command latencies */
if (is_latencies)
{
int i;
- for (i = 0; i < num_files; i++)
+ for (i = 0; i < num_scripts; i++)
{
Command **commands;
- if (num_files > 1)
+ if (num_scripts > 1)
printf("statement latencies in milliseconds, file %d:\n", i + 1);
else
printf("statement latencies in milliseconds:\n");
- for (commands = sql_files[i]; *commands != NULL; commands++)
+ for (commands = sql_script[i].commands; *commands != NULL; commands++)
{
Command *command = *commands;
int cnum = command->command_num;
@@ -2739,6 +2774,7 @@ main(int argc, char **argv)
{
static struct option long_options[] = {
/* systematic long/short named options */
+ {"tpc-b", no_argument, NULL, 'B'},
{"client", required_argument, NULL, 'c'},
{"connect", no_argument, NULL, 'C'},
{"debug", no_argument, NULL, 'd'},
@@ -2749,12 +2785,14 @@ main(int argc, char **argv)
{"initialize", no_argument, NULL, 'i'},
{"jobs", required_argument, NULL, 'j'},
{"log", no_argument, NULL, 'l'},
+ {"latency-limit", required_argument, NULL, 'L'},
{"no-vacuum", no_argument, NULL, 'n'},
{"port", required_argument, NULL, 'p'},
{"progress", required_argument, NULL, 'P'},
{"protocol", required_argument, NULL, 'M'},
{"quiet", no_argument, NULL, 'q'},
{"report-latencies", no_argument, NULL, 'r'},
+ {"rate", required_argument, NULL, 'R'},
{"scale", required_argument, NULL, 's'},
{"select-only", no_argument, NULL, 'S'},
{"skip-some-updates", no_argument, NULL, 'N'},
@@ -2762,6 +2800,7 @@ main(int argc, char **argv)
{"transactions", required_argument, NULL, 't'},
{"username", required_argument, NULL, 'U'},
{"vacuum-all", no_argument, NULL, 'v'},
+ {"weight", required_argument, NULL, 'w'},
/* long-named only options */
{"foreign-keys", no_argument, &foreign_keys, 1},
{"index-tablespace", required_argument, NULL, 3},
@@ -2769,25 +2808,20 @@ main(int argc, char **argv)
{"unlogged-tables", no_argument, &unlogged_tables, 1},
{"sampling-rate", required_argument, NULL, 4},
{"aggregate-interval", required_argument, NULL, 5},
- {"rate", required_argument, NULL, 'R'},
- {"latency-limit", required_argument, NULL, 'L'},
+ {"per-script-stats", no_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
int c;
- int nclients = 1; /* default number of simulated clients */
- int nthreads = 1; /* default number of threads */
int is_init_mode = 0; /* initialize mode? */
int is_no_vacuum = 0; /* no vacuum at all before testing? */
int do_vacuum_accounts = 0; /* do vacuum accounts before testing? */
- int ttype = 0; /* transaction type. 0: TPC-B, 1: SELECT only,
- * 2: skip update of branches and tellers */
int optindex;
- char *filename = NULL;
bool scale_given = false;
bool benchmarking_option_set = false;
bool initialization_option_set = false;
+ bool internal_script_used = false;
CState *state; /* status of clients */
TState *threads; /* array of thread */
@@ -2795,13 +2829,8 @@ main(int argc, char **argv)
instr_time start_time; /* start up time */
instr_time total_time;
instr_time conn_total_time;
- int64 total_xacts = 0;
- int64 total_latencies = 0;
- int64 total_sqlats = 0;
- int64 throttle_lag = 0;
- int64 throttle_lag_max = 0;
- int64 throttle_latency_skipped = 0;
int64 latency_late = 0;
+ StatsData stats;
int i;
int nclients_dealt;
@@ -2847,7 +2876,7 @@ main(int argc, char **argv)
state = (CState *) pg_malloc(sizeof(CState));
memset(state, 0, sizeof(CState));
- while ((c = getopt_long(argc, argv, "ih:nvp:dqSNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "ih:nvp:dqBSNw:c:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -2869,14 +2898,6 @@ main(int argc, char **argv)
case 'd':
debug++;
break;
- case 'S':
- ttype = 1;
- benchmarking_option_set = true;
- break;
- case 'N':
- ttype = 2;
- benchmarking_option_set = true;
- break;
case 'c':
benchmarking_option_set = true;
nclients = atoi(optarg);
@@ -2979,12 +3000,44 @@ main(int argc, char **argv)
initialization_option_set = true;
use_quiet = true;
break;
+ /* what to run*/
case 'f':
+ addScript(optarg, process_file(optarg));
benchmarking_option_set = true;
- ttype = 3;
- filename = pg_strdup(optarg);
- if (process_file(filename) == false || *sql_files[num_files - 1] == NULL)
- exit(1);
+ break;
+ case 'B':
+ addScript(b_bench, process_builtin(tpc_b, b_bench));
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ break;
+ case 'S':
+ addScript(s_bench, process_builtin(select_only, s_bench));
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ break;
+ case 'N':
+ addScript(n_bench, process_builtin(simple_update, n_bench));
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ break;
+ case 'w':
+ {
+ int weight = atoi(optarg);
+ if (weight <= 0)
+ {
+ fprintf(stderr,
+ "invalid weight value: \"%s\"\n", optarg);
+ exit(1);
+ }
+ if (num_scripts == 0)
+ {
+ fprintf(stderr,
+ "weight must come after its script selection"
+ " (-B -N -S or -f ...)\n");
+ exit(1);
+ }
+ sql_script[num_scripts - 1].weight = weight;
+ }
break;
case 'D':
{
@@ -3015,7 +3068,7 @@ main(int argc, char **argv)
break;
case 'M':
benchmarking_option_set = true;
- if (num_files > 0)
+ if (num_scripts > 0)
{
fprintf(stderr, "query mode (-M) should be specified before any transaction scripts (-f)\n");
exit(1);
@@ -3107,6 +3160,9 @@ main(int argc, char **argv)
}
#endif
break;
+ case 6:
+ per_script_stats = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -3114,6 +3170,18 @@ main(int argc, char **argv)
}
}
+ /* set default script if none */
+ if (num_scripts == 0 && !initialization_option_set)
+ {
+ addScript(b_bench, process_builtin(tpc_b, b_bench));
+ benchmarking_option_set = true;
+ internal_script_used = true;
+ }
+
+ /* compute total_weight */
+ for (i = 0; i < num_scripts; i++)
+ total_weight += sql_script[i].weight;
+
/*
* Don't need more threads than there are clients. (This is not merely an
* optimization; throttle_delay is calculated incorrectly below if some
@@ -3198,8 +3266,6 @@ main(int argc, char **argv)
* changed after fork.
*/
main_pid = (int) getpid();
- progress_nclients = nclients;
- progress_nthreads = nthreads;
if (nclients > 1)
{
@@ -3242,7 +3308,7 @@ main(int argc, char **argv)
exit(1);
}
- if (ttype != 3)
+ if (internal_script_used)
{
/*
* get the scaling factor that should be same as count(*) from
@@ -3319,31 +3385,6 @@ main(int argc, char **argv)
INSTR_TIME_SET_CURRENT(start_time);
srandom((unsigned int) INSTR_TIME_GET_MICROSEC(start_time));
- /* process builtin SQL scripts */
- switch (ttype)
- {
- case 0:
- sql_files[0] = process_builtin(tpc_b,
- "<builtin: TPC-B (sort of)>");
- num_files = 1;
- break;
-
- case 1:
- sql_files[0] = process_builtin(select_only,
- "<builtin: select only>");
- num_files = 1;
- break;
-
- case 2:
- sql_files[0] = process_builtin(simple_update,
- "<builtin: simple update>");
- num_files = 1;
- break;
-
- default:
- break;
- }
-
/* set up thread data structures */
threads = (TState *) pg_malloc(sizeof(TState) * nthreads);
nclients_dealt = 0;
@@ -3359,8 +3400,8 @@ main(int argc, char **argv)
thread->random_state[0] = random();
thread->random_state[1] = random();
thread->random_state[2] = random();
- thread->throttle_latency_skipped = 0;
thread->latency_late = 0;
+ initStats(& thread->stats, 0.0);
nclients_dealt += thread->nstate;
@@ -3427,11 +3468,11 @@ main(int argc, char **argv)
#endif /* ENABLE_THREAD_SAFETY */
/* wait for threads and accumulate results */
+ initStats(& stats, 0.0);
INSTR_TIME_SET_ZERO(conn_total_time);
for (i = 0; i < nthreads; i++)
{
TState *thread = &threads[i];
- int j;
#ifdef ENABLE_THREAD_SAFETY
if (threads[i].thread == INVALID_THREAD)
@@ -3444,21 +3485,13 @@ main(int argc, char **argv)
(void) threadRun(thread);
#endif /* ENABLE_THREAD_SAFETY */
- /* thread level stats */
- throttle_lag += thread->throttle_lag;
- throttle_latency_skipped = threads->throttle_latency_skipped;
- latency_late = thread->latency_late;
- if (throttle_lag_max > thread->throttle_lag_max)
- throttle_lag_max = thread->throttle_lag_max;
+ /* aggregate thread level stats */
+ appendSimpleStats(& stats.latency, & thread->stats.latency);
+ appendSimpleStats(& stats.lag, & thread->stats.lag);
+ stats.cnt += thread->stats.cnt;
+ stats.skipped += thread->stats.skipped;
+ latency_late += thread->latency_late;
INSTR_TIME_ADD(conn_total_time, thread->conn_time);
-
- /* client-level stats */
- for (j = 0; j < thread->nstate; j++)
- {
- total_xacts += thread->state[j].cnt;
- total_latencies += thread->state[i].txn_latencies;
- total_sqlats += thread->state[i].txn_sqlats;
- }
}
disconnect_all(state, nclients);
@@ -3474,10 +3507,7 @@ main(int argc, char **argv)
*/
INSTR_TIME_SET_CURRENT(total_time);
INSTR_TIME_SUBTRACT(total_time, start_time);
- printResults(ttype, total_xacts, nclients, threads, nthreads,
- total_time, conn_total_time, total_latencies, total_sqlats,
- throttle_lag, throttle_lag_max, throttle_latency_skipped,
- latency_late);
+ printResults(threads, &stats, total_time, conn_total_time, latency_late);
return 0;
}
@@ -3498,13 +3528,7 @@ threadRun(void *arg)
int64 thread_start = INSTR_TIME_GET_MICROSEC(thread->start_time);
int64 last_report = thread_start;
int64 next_report = last_report + (int64) progress * 1000000;
- int64 last_count = 0,
- last_lats = 0,
- last_sqlats = 0,
- last_lags = 0,
- last_skipped = 0;
-
- AggVals aggs;
+ StatsData last, aggs;
/*
* Initialize throttling rate target for all of the thread's clients. It
@@ -3514,8 +3538,6 @@ threadRun(void *arg)
*/
INSTR_TIME_SET_CURRENT(start);
thread->throttle_trigger = INSTR_TIME_GET_MICROSEC(start);
- thread->throttle_lag = 0;
- thread->throttle_lag_max = 0;
INSTR_TIME_SET_ZERO(thread->conn_time);
@@ -3552,16 +3574,17 @@ threadRun(void *arg)
INSTR_TIME_SET_CURRENT(thread->conn_time);
INSTR_TIME_SUBTRACT(thread->conn_time, thread->start_time);
- agg_vals_init(&aggs, thread->start_time);
+ initStats(&aggs, INSTR_TIME_GET_DOUBLE(thread->start_time));
+ last = aggs;
/* send start up queries in async manner */
for (i = 0; i < nstate; i++)
{
CState *st = &state[i];
- Command **commands = sql_files[st->use_file];
+ Command **commands = sql_script[st->use_file].commands;
int prev_ecnt = st->ecnt;
- st->use_file = getrand(thread, 0, num_files - 1);
+ st->use_file = num_scripts==1? 0: chooseScript(thread);
if (!doCustom(thread, st, &thread->conn_time, logfile, &aggs))
remains--; /* I've aborted */
@@ -3589,7 +3612,7 @@ threadRun(void *arg)
for (i = 0; i < nstate; i++)
{
CState *st = &state[i];
- Command **commands = sql_files[st->use_file];
+ Command **commands = sql_script[st->use_file].commands;
int sock;
if (st->con == NULL)
@@ -3695,7 +3718,7 @@ threadRun(void *arg)
for (i = 0; i < nstate; i++)
{
CState *st = &state[i];
- Command **commands = sql_files[st->use_file];
+ Command **commands = sql_script[st->use_file].commands;
int prev_ecnt = st->ecnt;
if (st->con && (FD_ISSET(PQsocket(st->con), &input_mask)
@@ -3726,11 +3749,7 @@ threadRun(void *arg)
if (now >= next_report)
{
/* generate and show report */
- int64 count = 0,
- lats = 0,
- sqlats = 0,
- lags = 0,
- skipped = 0;
+ StatsData cur;
int64 run = now - last_report;
double tps,
total_run,
@@ -3750,23 +3769,24 @@ threadRun(void *arg)
* (If a read from a 64-bit integer is not atomic, you might
* get a "torn" read and completely bogus latencies though!)
*/
- for (i = 0; i < progress_nclients; i++)
+ initStats(& cur, 0.0);
+ for (i = 0; i < nthreads; i++)
{
- count += state[i].cnt;
- lats += state[i].txn_latencies;
- sqlats += state[i].txn_sqlats;
+ appendSimpleStats(& cur.latency, & thread[i].stats.latency);
+ appendSimpleStats(& cur.lag, & thread[i].stats.lag);
+ cur.cnt += thread[i].stats.cnt;
+ cur.skipped += thread[i].stats.skipped;
}
- for (i = 0; i < progress_nthreads; i++)
- lags += thread[i].throttle_lag;
-
total_run = (now - thread_start) / 1000000.0;
- tps = 1000000.0 * (count - last_count) / run;
- latency = 0.001 * (lats - last_lats) / (count - last_count);
- sqlat = 1.0 * (sqlats - last_sqlats) / (count - last_count);
+ tps = 1000000.0 * (cur.cnt - last.cnt) / run;
+ latency = 0.001 * (cur.latency.sum - last.latency.sum) /
+ (cur.cnt - last.cnt);
+ sqlat = 1.0 * (cur.latency.sum2 - last.latency.sum2)
+ / (cur.cnt - last.cnt);
stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency);
- lag = 0.001 * (lags - last_lags) / (count - last_count);
- skipped = thread->throttle_latency_skipped - last_skipped;
+ lag = 0.001 * (cur.lag.sum - last.lag.sum) /
+ (cur.cnt - last.cnt);
fprintf(stderr,
"progress: %.1f s, %.1f tps, "
@@ -3776,16 +3796,13 @@ threadRun(void *arg)
{
fprintf(stderr, ", lag %.3f ms", lag);
if (latency_limit)
- fprintf(stderr, ", " INT64_FORMAT " skipped", skipped);
+ fprintf(stderr, ", "INT64_FORMAT" skipped",
+ cur.skipped - last.skipped);
}
fprintf(stderr, "\n");
- last_count = count;
- last_lats = lats;
- last_sqlats = sqlats;
- last_lags = lags;
+ last = cur;
last_report = now;
- last_skipped = thread->throttle_latency_skipped;
/*
* Ensure that the next report is in the future, in case
view thread (92+ messages) latest in thread
Message-ID: <alpine.DEB.2.10.1507171544450.26125@sto>
Permalink: ../alpine.DEB.2.10.1507171544450.26125@sto/
Also on: postgresql.org/message-id/alpine.DEB.2.10.1507171544450.26125@sto
copy link · copy postgr.es · copy git am
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: coelho@cri.ensmp.fr
Subject: Re: pgbench stats per script & other stuff
In-Reply-To: <alpine.DEB.2.10.1507171544450.26125@sto>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox