pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgbench - allow to store select results into variables
120+ messages / 17 participants
[nested] [flat]
* pgbench - allow to store select results into variables
@ 2016-07-09 08:20 Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-07-09 08:20 UTC (permalink / raw)
To: pgsql-hackers
Hello devs,
I mentionned my intention to add some features to pgbench back in March:
https://www.postgresql.org/message-id/alpine.DEB.2.10.1603301618570.5677@sto
The attached patch adds an \into meta command to store results of
preceding SELECTs into pgbench variables, so that they can be reused
afterwards.
The feature is useful to make more realistic scripts, currently pgbench
script cannot really interact with the database as results are discarded.
The chosen syntax is easy to understand and the implementation is quite
light, with minimal impact on the code base. I think that this is a
reasonnable compromise.
The SELECTs must yield exactly one row, the number of variables must be
less than the number of columns.
Also attached a set of test scripts, especially to trigger various error
cases.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-1.patch (7.1K, ../../alpine.DEB.2.20.1607091005330.3412@sto/2-pgbench-into-1.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f3afedb..8a7ad3e 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -809,6 +809,29 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-into'>
+ <term>
+ <literal>\into <replaceable>var1</> [<replaceable>var2</> ...]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the preceding
+ <command>SELECT</> commands into these variables.
+ The queries must yield exactly one row and the number of provided
+ variables must be less than the total number of columns of the results.
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+SELECT abalance FROM pgbench_accounts WHERE aid=5432;
+\into balance
+</programlisting>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 87fb006..d9827dc 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -307,6 +307,7 @@ typedef struct
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ char *into[MAX_ARGS]; /* NULL-terminated target variables for \into */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1148,6 +1149,96 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char * into[])
+{
+ PGresult *res;
+ int i = 0;
+ bool at_least_one = false;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ at_least_one = true;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (into != NULL && into[i] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f = 0;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d state %d expecting one row, got %d\n",
+ st->id, st->state, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ while (into[i] != NULL && f < nfields)
+ {
+ /* store result as a string */
+ if (!putVariable(st, "into", into[i], PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d state %d: error storing into var %s\n",
+ st->id, st->state, into[i]);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ i++;
+ f++;
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr, "client %d aborted in state %d: %s",
+ st->id, st->state, PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (!at_least_one)
+ {
+ fprintf(stderr, "client %d state %d: no results\n", st->id, st->state);
+ st->ecnt++;
+ return false;
+ }
+
+ if (into != NULL && into[i] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d: missing results to fill into variable %s\n",
+ st->id, st->state, into[i]);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1764,7 +1855,6 @@ chooseScript(TState *thread)
static bool
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command **commands;
bool trans_needs_throttle = false;
instr_time now;
@@ -1891,22 +1981,11 @@ top:
{
/*
* Read and discard the query result; note this is not included in
- * the statement latency numbers.
+ * the statement latency numbers (above), thus if reading the
+ * response fails the transaction is counted nevertheless.
*/
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- break; /* OK */
- default:
- fprintf(stderr, "client %d aborted in state %d: %s",
- st->id, st->state, PQerrorMessage(st->con));
- PQclear(res);
- return clientDone(st);
- }
- PQclear(res);
- discard_response(st);
+ if (!read_response(st, commands[st->state]->into))
+ return clientDone(st);
}
if (commands[st->state + 1] == NULL)
@@ -2930,6 +3009,24 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "into") == 0)
+ {
+ int i;
+
+ /* at least one variable name must be provided */
+ if (my_command->argc < 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "missing variable name", NULL, -1);
+
+ /* check that all variable names are valid */
+ for (i = 1; i < my_command->argc; i++)
+ {
+ if (!isLegalVariableName(my_command->argv[i]))
+ syntax_error(source, lineno, my_command->line,
+ my_command->argv[0], "invalid variable name",
+ my_command->argv[i], -1);
+ }
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3010,7 +3107,40 @@ ParseScript(const char *script, const char *desc, int weight)
if (sr == PSCAN_BACKSLASH)
{
command = process_backslash_command(sstate, desc);
- if (command)
+
+ /* special case for \into: merge into preceding SQL command */
+ if (command && command->argc >= 1 &&
+ pg_strcasecmp(command->argv[0], "into") == 0)
+ {
+ if (index == 0)
+ {
+ fprintf(stderr,
+ "meta command \\into cannot start a script\n");
+ exit(1);
+ }
+ else if (ps.commands[index-1]->type != SQL_COMMAND)
+ {
+ fprintf(stderr,
+ "meta command \\into must follow a SELECT (%s)\n",
+ ps.commands[index-1]->line);
+ exit(1);
+ }
+ else if (ps.commands[index-1]->into[0] != NULL)
+ {
+ fprintf(stderr,
+ "SQL command %d already has an associated \\into\n",
+ index-1);
+ exit(1);
+ }
+ /* else (command->argc >= 2): already checked by parser */
+
+ /* into is NULL-terminated thanks to pg_malloc0() */
+ memcpy(ps.commands[index-1]->into,
+ &command->argv[1], sizeof(char *) * (MAX_ARGS-1));
+
+ pg_free(command);
+ }
+ else if (command)
{
ps.commands[index] = command;
index++;
[application/x-sql] into.sql (299B, ../../alpine.DEB.2.20.1607091005330.3412@sto/3-into.sql)
download
[application/x-sql] into-err-1.sql (31B, ../../alpine.DEB.2.20.1607091005330.3412@sto/4-into-err-1.sql)
download
[application/x-sql] into-err-2.sql (47B, ../../alpine.DEB.2.20.1607091005330.3412@sto/5-into-err-2.sql)
download
[application/x-sql] into-err-3.sql (76B, ../../alpine.DEB.2.20.1607091005330.3412@sto/6-into-err-3.sql)
download
[application/x-sql] into-err-4.sql (42B, ../../alpine.DEB.2.20.1607091005330.3412@sto/7-into-err-4.sql)
download
[application/x-sql] into-err-5.sql (75B, ../../alpine.DEB.2.20.1607091005330.3412@sto/8-into-err-5.sql)
download
[application/x-sql] into-err-6.sql (88B, ../../alpine.DEB.2.20.1607091005330.3412@sto/9-into-err-6.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-09 08:44 Pavel Stehule <pavel.stehule@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Pavel Stehule @ 2016-07-09 08:44 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: pgsql-hackers
Hi
2016-07-09 10:20 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:
>
> Hello devs,
>
> I mentionned my intention to add some features to pgbench back in March:
>
> https://www.postgresql.org/message-id/alpine.DEB.2.10.1603301618570.5677@sto
>
> The attached patch adds an \into meta command to store results of
> preceding SELECTs into pgbench variables, so that they can be reused
> afterwards.
>
> The feature is useful to make more realistic scripts, currently pgbench
> script cannot really interact with the database as results are discarded.
>
> The chosen syntax is easy to understand and the implementation is quite
> light, with minimal impact on the code base. I think that this is a
> reasonnable compromise.
>
> The SELECTs must yield exactly one row, the number of variables must be
> less than the number of columns.
>
> Also attached a set of test scripts, especially to trigger various error
> cases.
>
>
Why you are introducing \into and not \gset like psql does?
Regards
Pavel
> --
> Fabien.
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
>
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-09 09:19 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Pavel Stehule <pavel.stehule@gmail.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-07-09 09:19 UTC (permalink / raw)
To: Pavel Stehule <pavel.stehule@gmail.com>; +Cc: pgsql-hackers
Hello Pavel,
> Why you are introducing \into and not \gset like psql does?
Good question.
The \into syntax I implemented is more generic, you can send a bunch of
queries together and extract the results, which makes sense from a client
perspective where reducing latency is important:
SELECT 1, 2 \; SELECT 3;
\into one two three
However "gset" only works on the last SELECT and if all columns have a
name. This feature probably makes sense interactively, but for a script it
seems more useful to allow batch processing and collect results
afterwards.
Also a more subjective argument: I do not like the gset automagic naming
feature. I got more inspired by PL/pgSQL and ECPG which both have an
"into" syntax with explicit variable names that let nothing to guessing. I
like things to be simple and explicit, hence the proposed into.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-09 12:19 Pavel Stehule <pavel.stehule@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Pavel Stehule @ 2016-07-09 12:19 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: pgsql-hackers
2016-07-09 11:19 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:
>
> Hello Pavel,
>
> Why you are introducing \into and not \gset like psql does?
>>
>
> Good question.
>
> The \into syntax I implemented is more generic, you can send a bunch of
> queries together and extract the results, which makes sense from a client
> perspective where reducing latency is important:
>
> SELECT 1, 2 \; SELECT 3;
> \into one two three
>
I understand, but it looks little bit scary - but the argument of reducing
latency can be valid
>
> However "gset" only works on the last SELECT and if all columns have a
> name. This feature probably makes sense interactively, but for a script it
> seems more useful to allow batch processing and collect results afterwards.
>
> Also a more subjective argument: I do not like the gset automagic naming
> feature. I got more inspired by PL/pgSQL and ECPG which both have an "into"
> syntax with explicit variable names that let nothing to guessing. I like
> things to be simple and explicit, hence the proposed into.
>
the gset was originally designed differently - but now it is here - and it
is not practical to have two different, but pretty similar statements in
psql and pgbench.
Regards
Pavel
>
> --
> Fabien.
>
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-09 12:52 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Pavel Stehule <pavel.stehule@gmail.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-07-09 12:52 UTC (permalink / raw)
To: Pavel Stehule <pavel.stehule@gmail.com>; +Cc: pgsql-hackers
>> Also a more subjective argument: I do not like the gset automagic naming
>> feature. I got more inspired by PL/pgSQL and ECPG which both have an "into"
>> syntax with explicit variable names that let nothing to guessing. I like
>> things to be simple and explicit, hence the proposed into.
>
> the gset was originally designed differently - but now it is here - and it
> is not practical to have two different, but pretty similar statements in
> psql and pgbench.
In my view they are unrelated: on the one hand "gset" is really an
interactive feature, where typing is costly so "automagic" might make
sense; on the other hand "into" is a scripting feature, where you want to
understand the code and have something as readable as possible, without
surprises.
The commands are named differently and behave differently.
If someone thinks that "gset" is a good idea for pgbench, which I don't,
it could be implemented. I think that an "into" feature, like PL/pgSQL &
ECPG, makes more sense for scripting.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 19:34 Robert Haas <robertmhaas@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 120+ messages in thread
From: Robert Haas @ 2016-07-13 19:34 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Sat, Jul 9, 2016 at 7:52 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
> If someone thinks that "gset" is a good idea for pgbench, which I don't, it
> could be implemented. I think that an "into" feature, like PL/pgSQL & ECPG,
> makes more sense for scripting.
I agree: I like \into.
But:
> SELECT 1, 2 \; SELECT 3;
> \into one two three
I think that's pretty weird.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 20:02 Tom Lane <tgl@sss.pgh.pa.us>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 5 replies; 120+ messages in thread
From: Tom Lane @ 2016-07-13 20:02 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
> On Sat, Jul 9, 2016 at 7:52 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>> If someone thinks that "gset" is a good idea for pgbench, which I don't, it
>> could be implemented. I think that an "into" feature, like PL/pgSQL & ECPG,
>> makes more sense for scripting.
> I agree: I like \into.
> But:
>> SELECT 1, 2 \; SELECT 3;
>> \into one two three
> I think that's pretty weird.
Yeah, that's seriously nasty action-at-a-distance in my view. I'd be okay
with
SELECT 1, 2 \into one two
SELECT 3 \into three
but I do not think that a metacommand on a following line should
retroactively affect the execution of a prior command, much less commands
before the last one. Even if this happens to be easy to do in pgbench's
existing over-contorted logic, it's tremendously confusing to the user;
and it might be much less easy if we try to refactor that logic.
And I'm with Pavel on this: it should work exactly like \gset. Inventing
\into to do almost the same thing in a randomly different way exhibits a
bad case of NIH syndrome. Sure, you can argue about how it's not quite
the same use-case and so you could micro-optimize by doing it differently,
but that's ignoring the cognitive load on users who have to remember two
different commands. Claiming that plpgsql's SELECT INTO is a closer
analogy than psql's \gset is quite bogus, too: the environment is
different (client side vs server side, declared vs undeclared target
variables), and the syntax is different (backslash or not, commas or not,
just for starters). I note also that we were talking a couple months ago
about trying to align psql and pgbench backslash commands more closely.
This would not be a good step in that direction.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 20:07 Robert Haas <robertmhaas@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
4 siblings, 1 reply; 120+ messages in thread
From: Robert Haas @ 2016-07-13 20:07 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Wed, Jul 13, 2016 at 3:02 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Sat, Jul 9, 2016 at 7:52 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>>> If someone thinks that "gset" is a good idea for pgbench, which I don't, it
>>> could be implemented. I think that an "into" feature, like PL/pgSQL & ECPG,
>>> makes more sense for scripting.
>
>> I agree: I like \into.
>
>> But:
>
>>> SELECT 1, 2 \; SELECT 3;
>>> \into one two three
>
>> I think that's pretty weird.
>
> Yeah, that's seriously nasty action-at-a-distance in my view. I'd be okay
> with
>
> SELECT 1, 2 \into one two
> SELECT 3 \into three
>
> but I do not think that a metacommand on a following line should
> retroactively affect the execution of a prior command, much less commands
> before the last one. Even if this happens to be easy to do in pgbench's
> existing over-contorted logic, it's tremendously confusing to the user;
> and it might be much less easy if we try to refactor that logic.
>
> And I'm with Pavel on this: it should work exactly like \gset. Inventing
> \into to do almost the same thing in a randomly different way exhibits a
> bad case of NIH syndrome. Sure, you can argue about how it's not quite
> the same use-case and so you could micro-optimize by doing it differently,
> but that's ignoring the cognitive load on users who have to remember two
> different commands. Claiming that plpgsql's SELECT INTO is a closer
> analogy than psql's \gset is quite bogus, too: the environment is
> different (client side vs server side, declared vs undeclared target
> variables), and the syntax is different (backslash or not, commas or not,
> just for starters). I note also that we were talking a couple months ago
> about trying to align psql and pgbench backslash commands more closely.
> This would not be a good step in that direction.
True, but I'd still argue that \into is a lot more readable than
\gset. Maybe both programs should support both commands.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 20:13 Tom Lane <tgl@sss.pgh.pa.us>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 120+ messages in thread
From: Tom Lane @ 2016-07-13 20:13 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Jul 13, 2016 at 3:02 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I note also that we were talking a couple months ago
>> about trying to align psql and pgbench backslash commands more closely.
>> This would not be a good step in that direction.
> True, but I'd still argue that \into is a lot more readable than
> \gset. Maybe both programs should support both commands.
Meh, personal preference there no doubt. I'd be okay with both programs
supporting both commands, but the \into command as described here would
be quite unreasonable to implement in psql. It needs to act more like
a semicolon-substitute, as \g and the rest of its family do. (I think
I'd also lobby for spelling it \ginto.)
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 20:16 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-07-13 20:16 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Robert,
> I agree: I like \into.
Great!
> But:
>
>> SELECT 1, 2 \; SELECT 3;
>> \into one two three
>
> I think that's pretty weird.
I agree that it is weird, but I do not think that it is bad.
Sending a batch of requests is a feature of libpq which is accessible
through pgbench by using "\;", although the fact is not documented. It
makes sense for a client to send independent queries together so as to
reduce latency.
From pgbench perspective, I would find it pretty weird as well that one
can send several queries together but could only read the answer from...
say the first one, and the others would be lost.
From an implementation perspective doing it is straightforward, and
rejecting it would require some more logic.
An obvious nicer feature would be to allow intermixing \into & \; but ISTM
that it would require to rethink deeply pgbench lexing/parsing which has
just been merged with psql by Tom and others...
If I had not pointed out the fact that it works, maybe no one would have
noticed... so a compromise could be not to advertise the fact that it
works (as the \; feature is not advertised anyway), but letting the
implementation do it because it is simple and may be useful, and rephrase
the documentation so that it is just about the previous select and not
previous select*S*?
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 20:26 Tom Lane <tgl@sss.pgh.pa.us>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Tom Lane @ 2016-07-13 20:26 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Fabien COELHO <coelho@cri.ensmp.fr> writes:
> Sending a batch of requests is a feature of libpq which is accessible
> through pgbench by using "\;", although the fact is not documented. It
> makes sense for a client to send independent queries together so as to
> reduce latency.
You're putting an awful lot of weight on an unsupported assertion about
latency. If a user cares about that, why would she not simply merge the
commands into "SELECT 1, 2, 3 \into one two three" ?
And I still say that what you're proposing might be easy right now, but
it might also be next door to impossible in a refactored implementation.
I don't think we should go there on the basis of a weak argument about
latency. \into should retrieve data only from the last PGresult.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 21:19 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2016-07-13 21:19 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Tom,
>> Sending a batch of requests is a feature of libpq which is accessible
>> through pgbench by using "\;", although the fact is not documented. It
>> makes sense for a client to send independent queries together so as to
>> reduce latency.
>
> You're putting an awful lot of weight on an unsupported assertion about
> latency.
For support, I would submit that many applications today are web/mobile
apps which are quite sensitive to latency. See for instance the Fast 2016
white paper by people at Google which discusses in depth "tail latency" as
a key measure of quality for IO systems used for live services, or the new
HTTP2 protocole (based on Google spdy) which aims at reducing latency
through multiple features (compression, serveur push, pipelining...).
> If a user cares about that, why would she not simply merge the
> commands into "SELECT 1, 2, 3 \into one two three" ?
Because the code would look pretty awful:
SELECT (SELECT first data FROM ... JOIN ... WHERE ... ),
(SELECT second data FROM ... JOIN ... WHERE ...),
(SELECT third data FROM ... JOIN ... WHERE ...);
> And I still say that what you're proposing might be easy right now, but
> it might also be next door to impossible in a refactored implementation.
I do not understand. There is one "multi" sql-command followed by a meta
command, and somehow a refactor implementation would have troubled with
that?
> I don't think we should go there on the basis of a weak argument about
> latency. \into should retrieve data only from the last PGresult.
This looks pretty arbitrary: Why not the first one, as I asked for it
first? Anyway, why allowing to send several queries if you are not allowed
to extract their results.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-13 21:31 David G. Johnston <david.g.johnston@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
4 siblings, 0 replies; 120+ messages in thread
From: David G. Johnston @ 2016-07-13 21:31 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Fabien COELHO <coelho@cri.ensmp.fr>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Wed, Jul 13, 2016 at 4:02 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Sat, Jul 9, 2016 at 7:52 AM, Fabien COELHO <coelho@cri.ensmp.fr>
> wrote:
> >> If someone thinks that "gset" is a good idea for pgbench, which I
> don't, it
> >> could be implemented. I think that an "into" feature, like PL/pgSQL &
> ECPG,
> >> makes more sense for scripting.
>
> > I agree: I like \into.
>
> > But:
>
> >> SELECT 1, 2 \; SELECT 3;
> >> \into one two three
>
> > I think that's pretty weird.
>
> Yeah, that's seriously nasty action-at-a-distance in my view. I'd be okay
> with
>
> SELECT 1, 2 \into one two
> SELECT 3 \into three
>
> but I do not think that a metacommand on a following line should
> retroactively affect the execution of a prior command, much less commands
> before the last one.
You need a test and a definition for:
SELECT 1, 2;
SELECT 3;
\into x, y, z
It should fail - "too many variables" - right?
David J.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-14 08:02 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
4 siblings, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2016-07-14 08:02 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Tom,
>>> SELECT 1, 2 \; SELECT 3;
>>> \into one two three
>
> Yeah, that's seriously nasty action-at-a-distance in my view. I'd be okay
> with
>
> SELECT 1, 2 \into one two
> SELECT 3 \into three
ISTM that is not the same, because then you would have two queries (over
the network) instead of one, so you pay the network latency twice?
> but I do not think that a metacommand on a following line should
> retroactively affect the execution of a prior command, much less commands
> before the last one.
Nope. The meta-command applies to the preceeding SQL command... which
happens to be a \;-compound command. ISTM that all is logically fine.
Some motivation about the feature (not its syntax or implementation), from
a benchmarking perspective:
- clients MUST read the server answers and possibly reuse them, hence a
proposed \into feature. Discarding the answer as pgbench does not really
comply with typical benchmark rules, eg from tpc-b:
"""1.3.2 Each transaction shall return to the driver the Account_Balance
resulting from successful commit of the transaction.
Comment: It is the intent of this clause that the account balance in the
database be returned to the driver, i.e., that the application retrieve
the account balance."""
- latency is important to applications (eg web applications), thus the
ability to compound statements is a good thing. However, if in a bench one
can compound statements but not retrieve their values, it fails the
previous "retrieve the value" requirement.
So basically I wish to avoid splitting compound queries and paying latency
just because of a lack of syntax to do the right thing, hence the proposed
feature which can retrieve data from various parts of a compound
statement.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-15 05:13 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
4 siblings, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2016-07-15 05:13 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Tom,
> Yeah, that's seriously nasty action-at-a-distance in my view. I'd be okay
> with
>
> SELECT 1, 2 \into one two
> SELECT 3 \into three
After giving it some thoughts, it could work on compound commands if \into
does not close the current sql command. Something like:
SELECT 1, 2 ; \into one two
SELECT 3 ; \into three
=> 2 SQL commands
SELECT 1, 2 \; \into one two
SELECT 3 ; \into three
=> 1 compound SQL command
I'd like \; or ; to stay mandatory as separators, though. Or at least to
be allowed.
I'm not quite sure how it could be implemented, though.
> And I'm with Pavel on this: it should work exactly like \gset.
Hmmm. Maybe I'll do that thing in the end, but I really think than gset
only makes sense in interactive context, and is pretty ugly for scripting.
> Inventing \into to do almost the same thing in a randomly different way
> exhibits a bad case of NIH syndrome.
No, it is a question of design suitable to programming:
> SELECT 1, 2 \gset v
could not set variable "?column?"
> Sure, you can argue about how it's not quite the same use-case
Indeed, that is my point.
> and so you could micro-optimize by doing it differently,
No, the underlying implementation is basically the same.
> but that's ignoring the cognitive load on users who have to remember two
> different commands.
I do not buy this argument: It is easier for me to remember that keyword
INTO happens to do the same thing the same way in PL/pgSQL and ECPG,
although with slightly different syntaxes, than to have to remember
psql-specific "gset" which does the same thing but in quite a different
way, because it means both another name and another concept.
> Claiming that plpgsql's SELECT INTO is a closer analogy than psql's
> \gset is quite bogus, too:
I disagree. I mentionned ECPG as well. Both ECPG & PLpgSQL are
"programming", psql is interactive.
> the environment is different (client side vs server side,
ECPG is client side. I think that the side does not matter.
> declared vs undeclared target variables),
Sure, the "gset" hack is only possible for a language without variable
declarations... but that does not make it a good idea.
> and the syntax is different (backslash or not, commas or not, just for
> starters).
Sure, different languages do not have the same syntax.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-07-15 16:33 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
4 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-07-15 16:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello again,
> I'd be okay with
>
> SELECT 1, 2 \into one two
> SELECT 3 \into three
Here is a v2 with more or less this approach, although \into does not end
the query, but applies to the current or last sql command. A query is
still terminated with a ";".
Now it handles things like :
-- standard sql command
SELECT balance FROM bank WHERE id=1;
\into balance
-- compound sql command, three == 3.
SELECT 1, 2 \; SELECT 3 ;
\into three
-- compound query with 2 selects & 3 variables
SELECT i \into one
FROM generate_series(1, 1) AS i \;
SELECT i+1, i+2 \into two three
FROM generate_series(1, 1) AS i ;
I had to add a few lines in psql scanner to count "\;", so the parsing
logic is a little more complicated than before.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-2.patch (16.2K, ../../alpine.DEB.2.20.1607151820160.7291@sto/2-pgbench-into-2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f3afedb..cca2cc2 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -809,6 +809,30 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-into'>
+ <term>
+ <literal>\into <replaceable>var1</> [<replaceable>var2</> ...]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the current or preceding
+ <command>SELECT</> command into these variables.
+ The queries must yield exactly one row and the number of provided
+ variables must be less than the total number of columns of the results.
+ This meta command does not end the current SQL command.
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+SELECT abalance \into abalance
+ FROM pgbench_accounts WHERE aid=5432;
+</programlisting>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 87fb006..667b63c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -302,11 +302,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char ***intos; /* per-compound command \into variables */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1148,6 +1151,107 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char ** intos[])
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ if (intos[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "cannot apply \\into to non SELECT statement\n",
+ st->id, st->state, compound);
+ st->ecnt++;
+ return false;
+ }
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (intos[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f = 0;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->state, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ while (intos[compound][f] != NULL && f < nfields)
+ {
+ /* store result as a string */
+ if (!putVariable(st, "into", intos[compound][f],
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->state, compound, intos[compound][f]);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ f++;
+ }
+
+ if (intos[compound][f] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: missing results"
+ " to fill into variable %s\n",
+ st->id, st->state, compound, intos[compound][f]);
+ st->ecnt++;
+ return false;
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr, "client %d aborted in state %d compound %d: %s",
+ st->id, st->state, compound, PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d state %d: no results\n", st->id, st->state);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1764,7 +1868,6 @@ chooseScript(TState *thread)
static bool
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command **commands;
bool trans_needs_throttle = false;
instr_time now;
@@ -1891,22 +1994,11 @@ top:
{
/*
* Read and discard the query result; note this is not included in
- * the statement latency numbers.
+ * the statement latency numbers (above), thus if reading the
+ * response fails the transaction is counted nevertheless.
*/
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- break; /* OK */
- default:
- fprintf(stderr, "client %d aborted in state %d: %s",
- st->id, st->state, PQerrorMessage(st->con));
- PQclear(res);
- return clientDone(st);
- }
- PQclear(res);
- discard_response(st);
+ if (!read_response(st, commands[st->state]->intos))
+ return clientDone(st);
}
if (commands[st->state + 1] == NULL)
@@ -2701,22 +2793,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char * p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2736,23 +2816,94 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->intos = pg_malloc0(sizeof(char**) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int space;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* add a separator if needed */
+ space = isspace(my_command->lines[len-1]) ? 0 : 1;
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + space + lmore + 1);
+ if (space > 0)
+ my_command->lines[len] = '\n';
+ memcpy(my_command->lines + len + space, more, lmore+1);
+
+ if (compounds > 0)
+ {
+ int nc = my_command->compound + compounds;
+ my_command->intos =
+ pg_realloc(my_command->intos, sizeof(char**) * (nc+1));
+ memset(my_command->intos + my_command->compound + 1, 0,
+ sizeof(char**) * compounds);
+ my_command->compound = nc;
+ }
+}
+
+static void
+postprocess_sql_command(Command * my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
+ my_command->line = pg_malloc(nlpos - p + 1 + 3);
memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->line[nlpos - p] = '.';
+ my_command->line[nlpos - p + 1] = '.';
+ my_command->line[nlpos - p + 2] = '.';
+ my_command->line[nlpos - p + 3] = '\0';
}
else
my_command->line = pg_strdup(p);
@@ -2760,19 +2911,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -2930,6 +3079,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "into") == 0)
+ {
+ /* at least one variable name must be provided */
+ if (my_command->argc < 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "missing variable names", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -2941,6 +3097,15 @@ process_backslash_command(PsqlScanState sstate, const char *source)
return my_command;
}
+static bool
+ends_with_semicolon(const char * buf)
+{
+ int i = strlen(buf)-1;
+ while (i > 0 && isspace(buf[i]))
+ i--;
+ return i > 0 && buf[i] == ';';
+}
+
/*
* Parse a script (either the contents of a file, or a built-in script)
* and add it to the list of scripts.
@@ -2953,6 +3118,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool sql_command_in_progress = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -2976,6 +3144,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -2985,31 +3154,27 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (sql_command_in_progress)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command was interrupted by an \into */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ sql_command_in_progress = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3024,6 +3189,75 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ /* Merge \into into preceeding SQL command ... */
+ if (strncmp(command->argv[0], "into", 4) == 0)
+ {
+ int cindex, i;
+ Command *into_cmd;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot start a script",
+ NULL, -1);
+
+ into_cmd = ps.commands[index-1];
+
+ if (into_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into must follow an SQL command",
+ into_cmd->line, -1);
+
+ /* this \into applies to this sub-command */
+ cindex = into_cmd->compound -
+ (ends_with_semicolon(line_buf.data) ? 1 : 0);
+
+ if (into_cmd->intos[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot follow an \\into",
+ NULL, -1);
+
+ /* check that all variable names are valid */
+ for (i = 1; i < command->argc; i++)
+ {
+ if (!isLegalVariableName(command->argv[i]))
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into invalid variable name",
+ command->argv[i], -1);
+ }
+
+ into_cmd->intos[cindex] =
+ pg_malloc0(sizeof(char *) * (command->argc+1));
+
+ memcpy(into_cmd->intos[cindex], command->argv + 1,
+ sizeof(char*) * command->argc);
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+
+ /* "SELECT ... \into ..." follow-up */
+ sql_command_in_progress = *line_buf.data != '\0';
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3031,6 +3265,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index ab0f822..4d28fdb 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 55067b4..43b5a4d 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,6 +680,9 @@ other .
"\\"[;:] {
/* Force a semicolon or colon into the query buffer */
+ if (yytext[1] == ';')
+ /* count compound commands */
+ cur_state->semicolons++;
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index a52929d..a184fc7 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
[application/x-sql] into.sql (295B, ../../alpine.DEB.2.20.1607151820160.7291@sto/3-into.sql)
download
[application/x-sql] into2.sql (733B, ../../alpine.DEB.2.20.1607151820160.7291@sto/4-into2.sql)
download
[application/x-sql] into-err-1.sql (31B, ../../alpine.DEB.2.20.1607151820160.7291@sto/5-into-err-1.sql)
download
[application/x-sql] into-err-2.sql (47B, ../../alpine.DEB.2.20.1607151820160.7291@sto/6-into-err-2.sql)
download
[application/x-sql] into-err-3.sql (76B, ../../alpine.DEB.2.20.1607151820160.7291@sto/7-into-err-3.sql)
download
[application/x-sql] into-err-4.sql (42B, ../../alpine.DEB.2.20.1607151820160.7291@sto/8-into-err-4.sql)
download
[application/x-sql] into-err-5.sql (75B, ../../alpine.DEB.2.20.1607151820160.7291@sto/9-into-err-5.sql)
download
[application/x-sql] into-err-6.sql (88B, ../../alpine.DEB.2.20.1607151820160.7291@sto/10-into-err-6.sql)
download
[application/x-sql] into-err-7.sql (46B, ../../alpine.DEB.2.20.1607151820160.7291@sto/11-into-err-7.sql)
download
[application/x-sql] into-err-8.sql (56B, ../../alpine.DEB.2.20.1607151820160.7291@sto/12-into-err-8.sql)
download
[application/x-sql] into-err-9.sql (37B, ../../alpine.DEB.2.20.1607151820160.7291@sto/13-into-err-9.sql)
download
[application/x-sql] into-err-10.sql (35B, ../../alpine.DEB.2.20.1607151820160.7291@sto/14-into-err-10.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-02 02:58 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-02 02:58 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hi Fabien,
On 2016/07/16 1:33, Fabien COELHO wrote:
> Here is a v2 with more or less this approach, although \into does not end
> the query, but applies to the current or last sql command. A query is
> still terminated with a ";".
This patch needs to be rebased because of commit 64710452 (on 2016-08-19).
Thanks,
Amit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-02 17:47 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-09-02 17:47 UTC (permalink / raw)
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Amit,
> This patch needs to be rebased because of commit 64710452 (on 2016-08-19).
Here it is!
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-3.patch (16.2K, ../../alpine.DEB.2.20.1609021943130.9507@lancre/2-pgbench-into-3.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 285608d..a879e59 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -809,6 +809,30 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-into'>
+ <term>
+ <literal>\into <replaceable>var1</> [<replaceable>var2</> ...]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the current or preceding
+ <command>SELECT</> command into these variables.
+ The queries must yield exactly one row and the number of provided
+ variables must be less than the total number of columns of the results.
+ This meta command does not end the current SQL command.
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+SELECT abalance \into abalance
+ FROM pgbench_accounts WHERE aid=5432;
+</programlisting>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 56c37d5..6433df4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -302,11 +302,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char ***intos; /* per-compound command \into variables */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1150,6 +1153,107 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char ** intos[])
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (intos[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "cannot apply \\into to non SELECT statement\n",
+ st->id, st->state, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (intos[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f = 0;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->state, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ while (intos[compound][f] != NULL && f < nfields)
+ {
+ /* store result as a string */
+ if (!putVariable(st, "into", intos[compound][f],
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->state, compound, intos[compound][f]);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ f++;
+ }
+
+ if (intos[compound][f] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: missing results"
+ " to fill into variable %s\n",
+ st->id, st->state, compound, intos[compound][f]);
+ st->ecnt++;
+ return false;
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr, "client %d aborted in state %d compound %d: %s",
+ st->id, st->state, compound, PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d state %d: no results\n", st->id, st->state);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1766,7 +1870,6 @@ chooseScript(TState *thread)
static bool
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command **commands;
bool trans_needs_throttle = false;
instr_time now;
@@ -1893,23 +1996,11 @@ top:
{
/*
* Read and discard the query result; note this is not included in
- * the statement latency numbers.
+ * the statement latency numbers (above), thus if reading the
+ * response fails the transaction is counted nevertheless.
*/
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- break; /* OK */
- default:
- fprintf(stderr, "client %d aborted in state %d: %s",
- st->id, st->state, PQerrorMessage(st->con));
- PQclear(res);
- return clientDone(st);
- }
- PQclear(res);
- discard_response(st);
+ if (!read_response(st, commands[st->state]->intos))
+ return clientDone(st);
}
if (commands[st->state + 1] == NULL)
@@ -2704,22 +2795,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char * p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2739,23 +2818,94 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->intos = pg_malloc0(sizeof(char**) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int space;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* add a separator if needed */
+ space = isspace(my_command->lines[len-1]) ? 0 : 1;
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + space + lmore + 1);
+ if (space > 0)
+ my_command->lines[len] = '\n';
+ memcpy(my_command->lines + len + space, more, lmore+1);
+
+ if (compounds > 0)
+ {
+ int nc = my_command->compound + compounds;
+ my_command->intos =
+ pg_realloc(my_command->intos, sizeof(char**) * (nc+1));
+ memset(my_command->intos + my_command->compound + 1, 0,
+ sizeof(char**) * compounds);
+ my_command->compound = nc;
+ }
+}
+
+static void
+postprocess_sql_command(Command * my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
+ my_command->line = pg_malloc(nlpos - p + 1 + 3);
memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->line[nlpos - p] = '.';
+ my_command->line[nlpos - p + 1] = '.';
+ my_command->line[nlpos - p + 2] = '.';
+ my_command->line[nlpos - p + 3] = '\0';
}
else
my_command->line = pg_strdup(p);
@@ -2763,19 +2913,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -2933,6 +3081,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "into") == 0)
+ {
+ /* at least one variable name must be provided */
+ if (my_command->argc < 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "missing variable names", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -2944,6 +3099,15 @@ process_backslash_command(PsqlScanState sstate, const char *source)
return my_command;
}
+static bool
+ends_with_semicolon(const char * buf)
+{
+ int i = strlen(buf)-1;
+ while (i > 0 && isspace(buf[i]))
+ i--;
+ return i > 0 && buf[i] == ';';
+}
+
/*
* Parse a script (either the contents of a file, or a built-in script)
* and add it to the list of scripts.
@@ -2956,6 +3120,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool sql_command_in_progress = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -2979,6 +3146,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -2988,31 +3156,27 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (sql_command_in_progress)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command was interrupted by an \into */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ sql_command_in_progress = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3027,6 +3191,75 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ /* Merge \into into preceeding SQL command ... */
+ if (strncmp(command->argv[0], "into", 4) == 0)
+ {
+ int cindex, i;
+ Command *into_cmd;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot start a script",
+ NULL, -1);
+
+ into_cmd = ps.commands[index-1];
+
+ if (into_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into must follow an SQL command",
+ into_cmd->line, -1);
+
+ /* this \into applies to this sub-command */
+ cindex = into_cmd->compound -
+ (ends_with_semicolon(line_buf.data) ? 1 : 0);
+
+ if (into_cmd->intos[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot follow an \\into",
+ NULL, -1);
+
+ /* check that all variable names are valid */
+ for (i = 1; i < command->argc; i++)
+ {
+ if (!isLegalVariableName(command->argv[i]))
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into invalid variable name",
+ command->argv[i], -1);
+ }
+
+ into_cmd->intos[cindex] =
+ pg_malloc0(sizeof(char *) * (command->argc+1));
+
+ memcpy(into_cmd->intos[cindex], command->argv + 1,
+ sizeof(char*) * command->argc);
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+
+ /* "SELECT ... \into ..." follow-up */
+ sql_command_in_progress = *line_buf.data != '\0';
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3034,6 +3267,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index ab0f822..4d28fdb 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 55067b4..43b5a4d 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,6 +680,9 @@ other .
"\\"[;:] {
/* Force a semicolon or colon into the query buffer */
+ if (yytext[1] == ';')
+ /* count compound commands */
+ cur_state->semicolons++;
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index a52929d..a184fc7 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-05 08:09 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-05 08:09 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hi Fabien,
On 2016/09/03 2:47, Fabien COELHO wrote:
>> This patch needs to be rebased because of commit 64710452 (on 2016-08-19).
>
> Here it is!
Thanks for sending the updated patch. Here are some (mostly cosmetic)
comments. Before the comments, let me confirm whether the following
result is odd (a bug) or whether I am just using it wrong:
Custom script looks like:
\;
select a \into a
from tab where a = 1;
\set i debug(:a)
I get the following error:
undefined variable "a"
client 0 aborted in state 1; execution of meta-command failed
Even the following script gives the same result:
\;
select a from a where a = 1;
\into a
\set t debug(:a)
However with the following there is no error and a gets set to 2:
select a from a where a = 1
\;
select a+1 from a where a = 1;
\into a
\set t debug(:a)
Comments on the patch follow:
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the current or
preceding
+ <command>SELECT</> command into these variables.
Any command returning rows ought to work, no? For example, the following
works:
update a set a = a+1 returning *;
\into a
\set t debug(:a)
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
When I looked at this and related hunks (and also the hunks related to
semicolons), it made me wonder whether this patch contains two logical
changes. Is this a just a refactoring for the \into implementation or
does this provide some new externally visible feature?
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char ***intos; /* per-compound command \into variables */
Need an extra space for intos to align with earlier fields. Also I wonder
if intonames or intoargs would be a slightly better name for the field.
+static bool
+read_response(CState *st, char ** intos[])
Usual style seems to be to use ***intos here.
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "cannot apply \\into to non SELECT statement\n",
+ st->id, st->state, compound);
How about make this error message say something like other messages
related to \into, perhaps something like: "\\into cannot follow non SELECT
statements\n"
/*
* Read and discard the query result; note this is not
included in
- * the statement latency numbers.
+ * the statement latency numbers (above), thus if reading the
+ * response fails the transaction is counted nevertheless.
*/
Does this comment need to mention that the result is not discarded when
\into is specified?
+ my_command->intos = pg_malloc0(sizeof(char**) * (compounds+1));
Need space: s/char**/char **/g
This comments applies also to a couple of nearby places.
- my_command->line = pg_malloc(nlpos - p + 1);
+ my_command->line = pg_malloc(nlpos - p + 1 + 3);
A comment mentioning what the above means would be helpful.
+ bool sql_command_in_progress = false;
This variable's name could be slightly confusing to readers. If I
understand its purpose correctly, perhaps it could be called
sql_command_continues.
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot start a script",
+ NULL, -1);
How about: "\\into cannot be at the beginning of a script" ?
Thanks,
Amit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-07 14:01 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-09-07 14:01 UTC (permalink / raw)
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Amit,
> Custom script looks like:
>
> \;
> select a \into a
> from tab where a = 1;
> \set i debug(:a)
>
> I get the following error:
>
> undefined variable "a"
> client 0 aborted in state 1; execution of meta-command failed
Good catch!
Indeed, there is a problem with empty commands which are simply ignored by
libpq/postgres if there are other commands around, so my synchronization
between results & commands was too naive.
In order to fix this, I made the scanner also count empty commands and
ignore comments. I guessed that proposing to change libpq/postgres
behavior was not an option.
> Comments on the patch follow:
>
> + <listitem>
> + <para>
> + Stores the first fields of the resulting row from the current or
> preceding
> + <command>SELECT</> command into these variables.
>
> Any command returning rows ought to work, no?
Yes. I put "SQL command" instead.
>
> - char *line; /* text of command line */
> + char *line; /* first line for short display */
> + char *lines; /* full multi-line text of command */
>
> When I looked at this and related hunks (and also the hunks related to
> semicolons), it made me wonder whether this patch contains two logical
> changes. Is this a just a refactoring for the \into implementation or
> does this provide some new externally visible feature?
There is essentially a refactoring that I did when updating how Command is
managed because it has to be done in several stages to fit "into" into it
and to take care of compounds.
However there was small "new externally visible feature": on -r, instead
of cutting abruptly a multiline command at the end of the first line it
appended "..." as an ellipsis because it looked nicer.
I've removed this small visible changed.
> char *argv[MAX_ARGS]; /* command word list */
> + int compound; /* last compound command (number of \;) */
> + char ***intos; /* per-compound command \into variables */
> Need an extra space for intos to align with earlier fields.
Ok.
> Also I wonder if intonames or intoargs would be a slightly better name
> for the field.
I put "intovars" as they are variable names.
> +static bool
> +read_response(CState *st, char ** intos[])
>
> Usual style seems to be to use ***intos here.
Ok.
> + fprintf(stderr,
> + "client %d state %d compound %d: "
> + "cannot apply \\into to non SELECT statement\n",
> + st->id, st->state, compound);
>
> How about make this error message say something like other messages
> related to \into, perhaps something like: "\\into cannot follow non SELECT
> statements\n"
As you pointed out above, there may be statements without "SELECT" which
which return a row. I wrote "\\into expects a row" instead.
> /*
> * Read and discard the query result; note this is not included in
> - * the statement latency numbers.
> + * the statement latency numbers (above), thus if reading the
> + * response fails the transaction is counted nevertheless.
> */
>
> Does this comment need to mention that the result is not discarded when
> \into is specified?
Hmmm. The result structure is discarded, but the results are copied into
variables before that, so the comments seems ok...
> + my_command->intos = pg_malloc0(sizeof(char**) * (compounds+1));
>
> Need space: s/char**/char **/g
Ok.
> This comments applies also to a couple of nearby places.
Indeed.
> - my_command->line = pg_malloc(nlpos - p + 1);
> + my_command->line = pg_malloc(nlpos - p + 1 + 3);
>
> A comment mentioning what the above means would be helpful.
Ok. I removed the "+ 3" anyway.
> + bool sql_command_in_progress = false;
>
> This variable's name could be slightly confusing to readers. If I
> understand its purpose correctly, perhaps it could be called
> sql_command_continues.
It is possible. I like 'in progress' though. Why is it confusing?
It means that the current command is not finished yet and more is
expected, that is the final ';' has not been encountered.
> + if (index == 0)
> + syntax_error(desc, lineno, NULL, NULL,
> + "\\into cannot start a script",
> + NULL, -1);
>
> How about: "\\into cannot be at the beginning of a script" ?
It is possible, but it's quite longer... I'm not a native speaker, so I'm
do not know whether it would be better.
The attached patch takes into all your comments but:
- comment about discarded results...
- the sql_command_in_progress variable name change
- the longer message on into at the start of a script
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-4.patch (18.3K, ../../alpine.DEB.2.20.1609062144220.28392@lancre/2-pgbench-into-4.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 285608d..0a474e1 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -809,6 +809,30 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-into'>
+ <term>
+ <literal>\into <replaceable>var1</> [<replaceable>var2</> ...]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the current or preceding
+ SQL command into these variables.
+ The queries must yield exactly one row and the number of provided
+ variables must be less than the total number of columns of the results.
+ This meta command does not end the current SQL command.
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+SELECT abalance \into abalance
+ FROM pgbench_accounts WHERE aid=5432;
+</programlisting>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 56c37d5..42a9d2c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -302,11 +302,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char ***intovars; /* per-compound command \into variables */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1150,6 +1153,107 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char ***intovars)
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (intovars[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "\\into expects a row\n",
+ st->id, st->state, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (intovars[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f = 0;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->state, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ while (intovars[compound][f] != NULL && f < nfields)
+ {
+ /* store result as a string */
+ if (!putVariable(st, "into", intovars[compound][f],
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->state, compound, intovars[compound][f]);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ f++;
+ }
+
+ if (intovars[compound][f] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: missing results"
+ " to fill into variable %s\n",
+ st->id, st->state, compound, intovars[compound][f]);
+ st->ecnt++;
+ return false;
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr, "client %d aborted in state %d compound %d: %s",
+ st->id, st->state, compound, PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d state %d: no results\n", st->id, st->state);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1766,7 +1870,6 @@ chooseScript(TState *thread)
static bool
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command **commands;
bool trans_needs_throttle = false;
instr_time now;
@@ -1893,23 +1996,11 @@ top:
{
/*
* Read and discard the query result; note this is not included in
- * the statement latency numbers.
+ * the statement latency numbers (above), thus if reading the
+ * response fails the transaction is counted nevertheless.
*/
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- break; /* OK */
- default:
- fprintf(stderr, "client %d aborted in state %d: %s",
- st->id, st->state, PQerrorMessage(st->con));
- PQclear(res);
- return clientDone(st);
- }
- PQclear(res);
- discard_response(st);
+ if (!read_response(st, commands[st->state]->intovars))
+ return clientDone(st);
}
if (commands[st->state + 1] == NULL)
@@ -2704,22 +2795,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2739,17 +2818,85 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->intovars = pg_malloc0(sizeof(char **) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int space;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* add a separator if needed */
+ space = isspace(my_command->lines[len-1]) ? 0 : 1;
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + space + lmore + 1);
+ if (space > 0)
+ my_command->lines[len] = '\n';
+ memcpy(my_command->lines + len + space, more, lmore+1);
+
+ if (compounds > 0)
+ {
+ int nc = my_command->compound + compounds;
+ my_command->intovars =
+ pg_realloc(my_command->intovars, sizeof(char **) * (nc+1));
+ memset(my_command->intovars + my_command->compound + 1, 0,
+ sizeof(char **) * compounds);
+ my_command->compound = nc;
+ }
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -2763,19 +2910,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -2933,6 +3078,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "into") == 0)
+ {
+ /* at least one variable name must be provided */
+ if (my_command->argc < 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "missing variable names", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -2944,6 +3096,15 @@ process_backslash_command(PsqlScanState sstate, const char *source)
return my_command;
}
+static bool
+ends_with_semicolon(const char *buf)
+{
+ int i = strlen(buf)-1;
+ while (i > 0 && isspace(buf[i]))
+ i--;
+ return i > 0 && buf[i] == ';';
+}
+
/*
* Parse a script (either the contents of a file, or a built-in script)
* and add it to the list of scripts.
@@ -2956,6 +3117,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool sql_command_in_progress = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -2968,6 +3132,7 @@ ParseScript(const char *script, const char *desc, int weight)
/* Prepare to parse script */
sstate = psql_scan_create(&pgbench_callbacks);
+ sstate->rm_c_comments = true;
/*
* Ideally, we'd scan scripts using the encoding and stdstrings settings
@@ -2979,6 +3144,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -2988,31 +3154,32 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
+ sstate->empty_cmds = 0;
+ sstate->last_cmd_start = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (sql_command_in_progress)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command was interrupted by an \into */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons - sstate->empty_cmds);
+ sql_command_in_progress = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command =
+ create_sql_command(&line_buf, desc,
+ sstate->semicolons - sstate->empty_cmds);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3027,6 +3194,75 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ /* Merge \into into preceeding SQL command ... */
+ if (pg_strcasecmp(command->argv[0], "into") == 0)
+ {
+ int cindex, i;
+ Command *into_cmd;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot start a script",
+ NULL, -1);
+
+ into_cmd = ps.commands[index-1];
+
+ if (into_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into must follow an SQL command",
+ into_cmd->line, -1);
+
+ /* this \into applies to this sub-command */
+ cindex = into_cmd->compound -
+ (ends_with_semicolon(line_buf.data) ? 1 : 0);
+
+ if (into_cmd->intovars[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot follow an \\into",
+ NULL, -1);
+
+ /* check that all variable names are valid */
+ for (i = 1; i < command->argc; i++)
+ {
+ if (!isLegalVariableName(command->argv[i]))
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into invalid variable name",
+ command->argv[i], -1);
+ }
+
+ into_cmd->intovars[cindex] =
+ pg_malloc0(sizeof(char *) * (command->argc+1));
+
+ memcpy(into_cmd->intovars[cindex], command->argv + 1,
+ sizeof(char*) * command->argc);
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+
+ /* "SELECT ... \into ..." follow-up */
+ sql_command_in_progress = *line_buf.data != '\0';
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3034,6 +3270,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index ab0f822..4d28fdb 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 55067b4..aa1ab6d 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -71,6 +71,8 @@ typedef int YYSTYPE;
extern int psql_yyget_column(yyscan_t yyscanner);
extern void psql_yyset_column(int column_no, yyscan_t yyscanner);
+static bool buffer_empty_since(PQExpBuffer buf, size_t len);
+
%}
%option reentrant
@@ -388,14 +390,16 @@ other .
BEGIN(xc);
/* Put back any characters past slash-star; see above */
yyless(2);
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcstart} {
cur_state->xcdepth++;
/* Put back any characters past slash-star; see above */
yyless(2);
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcstop} {
@@ -403,19 +407,23 @@ other .
BEGIN(INITIAL);
else
cur_state->xcdepth--;
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcinside} {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{op_chars} {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>\*+ {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
{xbstart} {
@@ -680,6 +688,16 @@ other .
"\\"[;:] {
/* Force a semicolon or colon into the query buffer */
+ if (yytext[1] == ';')
+ {
+ /* count compound commands */
+ cur_state->semicolons++;
+ /* detect empty commands */
+ if (buffer_empty_since(cur_state->output_buf, cur_state->last_cmd_start))
+ cur_state->empty_cmds++;
+ cur_state->last_cmd_start =
+ cur_state->output_buf->len + 2;
+ }
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1323,6 +1341,20 @@ psqlscan_prepare_buffer(PsqlScanState state, const char *txt, int len,
return yy_scan_buffer(newtxt, len + 2, state->scanner);
}
+/* tell whether there were only spaces inserted since len
+ */
+static bool
+buffer_empty_since(PQExpBuffer buf, size_t len)
+{
+ while (len < buf->len)
+ {
+ if (!isspace(buf->data[len]))
+ return false;
+ len++;
+ }
+ return true;
+}
+
/*
* psqlscan_emit() --- body for ECHO macro
*
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index a52929d..9ec5f51 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,10 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
+ int empty_cmds; /* number of empty commands (\;\;) */
+ size_t last_cmd_start; /* position in output buffer */
+ bool rm_c_comments; /* whether to remove C comments */
char *dolqstart; /* current $foo$ quote start string */
/*
[application/x-sql] into-8.sql (190B, ../../alpine.DEB.2.20.1609062144220.28392@lancre/3-into-8.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-12 07:33 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-12 07:33 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hi Fabien,
On 2016/09/07 23:01, Fabien COELHO wrote:
>> Custom script looks like:
>>
>> \;
>> select a \into a
>> from tab where a = 1;
>> \set i debug(:a)
>>
>> I get the following error:
>>
>> undefined variable "a"
>> client 0 aborted in state 1; execution of meta-command failed
>
> Good catch!
>
> Indeed, there is a problem with empty commands which are simply ignored by
> libpq/postgres if there are other commands around, so my synchronization
> between results & commands was too naive.
>
> In order to fix this, I made the scanner also count empty commands and
> ignore comments. I guessed that proposing to change libpq/postgres
> behavior was not an option.
Seems to be fixed.
>> Comments on the patch follow:
>>
>> + <listitem>
>> + <para>
>> + Stores the first fields of the resulting row from the current or
>> preceding
>> + <command>SELECT</> command into these variables.
>>
>> Any command returning rows ought to work, no?
>
> Yes. I put "SQL command" instead.
Check.
>> - char *line; /* text of command line */
>> + char *line; /* first line for short display */
>> + char *lines; /* full multi-line text of command */
>>
>> When I looked at this and related hunks (and also the hunks related to
>> semicolons), it made me wonder whether this patch contains two logical
>> changes. Is this a just a refactoring for the \into implementation or
>> does this provide some new externally visible feature?
>
> There is essentially a refactoring that I did when updating how Command is
> managed because it has to be done in several stages to fit "into" into it
> and to take care of compounds.
>
> However there was small "new externally visible feature": on -r, instead
> of cutting abruptly a multiline command at the end of the first line it
> appended "..." as an ellipsis because it looked nicer.
> I've removed this small visible changed.
There still seems to be a change in behavior of the -r option due to the
patch. Consider the following example:
# no \into used in the script
$ cat /tmp/into.sql
select a from a where a = 1 \;
select a+1 from a where a = 1;
\set a 1
\set b 2
\set i debug(:a)
\set i debug(:b)
$ pgbench -r -n -t 1 -f /tmp/into.sql postgres
<snip>
- statement latencies in milliseconds:
2.889 select a from a where a = 1 ;
0.012 \set a 1
0.009 \set b 2
0.031 \set i debug(:a)
0.014 \set i debug(:b)
# behavior wrt compound statement changes when \into is used
$ cat /tmp/into.sql
select a from a where a = 1 \; \into a
select a+1 from a where a = 1; \into b
\set i debug(:a)
\set i debug(:b)
$ pgbench -r -n -t 1 -f /tmp/into.sql postgres
<snip>
- statement latencies in milliseconds:
2.093 select a from a where a = 1 ; select a+1 from a where a = 1;
0.034 \set i debug(:a)
0.015 \set i debug(:b)
One more thing I observed which I am not sure if it's a fault of this
patch is illustrated below:
$ cat /tmp/into.sql
\;
select a from a where a = 1 \;
select a+1 from a where a = 1;
\set a 1
\set b 2
\set i debug(:a)
\set i debug(:b)
$ pgbench -r -n -t 1 -f /tmp/into.sql postgres
<snip>
- statement latencies in milliseconds:
2.349 ;
0.013 \set a 1
0.009 \set b 2
0.029 \set i debug(:a)
0.015 \set i debug(:b)
Note that the compound select statement is nowhere to be seen in the
latencies output. The output remains the same even if I use the \into's.
What seems to be going on is that the empty statement on the first line
(\;) is the only part kept of the compound statement spanning lines 1-3.
>> Also I wonder if intonames or intoargs would be a slightly better name
>> for the field.
>
> I put "intovars" as they are variable names.
Sounds fine.
>> + fprintf(stderr,
>> + "client %d state %d compound %d: "
>> + "cannot apply \\into to non SELECT statement\n",
>> + st->id, st->state, compound);
>>
>> How about make this error message say something like other messages
>> related to \into, perhaps something like: "\\into cannot follow non SELECT
>> statements\n"
>
> As you pointed out above, there may be statements without "SELECT" which
> which return a row. I wrote "\\into expects a row" instead.
Sounds fine.
>
>> /*
>> * Read and discard the query result; note this is not
>> included in
>> - * the statement latency numbers.
>> + * the statement latency numbers (above), thus if reading the
>> + * response fails the transaction is counted nevertheless.
>> */
>>
>> Does this comment need to mention that the result is not discarded when
>> \into is specified?
>
> Hmmm. The result structure is discarded, but the results are copied into
> variables before that, so the comments seems ok...
Hmm, OK.
>> + bool sql_command_in_progress = false;
>>
>> This variable's name could be slightly confusing to readers. If I
>> understand its purpose correctly, perhaps it could be called
>> sql_command_continues.
>
> It is possible. I like 'in progress' though. Why is it confusing?
> It means that the current command is not finished yet and more is
> expected, that is the final ';' has not been encountered.
I understood that it refers to what you explain here. But to me it
sounded like the name is referring to the progress of *execution* of a SQL
command whereas the code in question is simply expecting to finish
*parsing* the SQL command using the next lines. It may be fine though.
>> + if (index == 0)
>> + syntax_error(desc, lineno, NULL, NULL,
>> + "\\into cannot start a script",
>> + NULL, -1);
>>
>> How about: "\\into cannot be at the beginning of a script" ?
>
> It is possible, but it's quite longer... I'm not a native speaker, so I'm
> do not know whether it would be better.
Me neither, let's leave it for the committer to decide.
> The attached patch takes into all your comments but:
> - comment about discarded results...
> - the sql_command_in_progress variable name change
> - the longer message on into at the start of a script
The patch seems fine without these, although please consider the concern I
raised with regard to the -r option above.
Thanks,
Amit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-13 08:41 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-09-13 08:41 UTC (permalink / raw)
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Amit,
> [...]
> There still seems to be a change in behavior of the -r option due to the
> patch. Consider the following example:
>
> select a from a where a = 1 \;
> select a+1 from a where a = 1;
> ...
> - statement latencies in milliseconds:
> 2.889 select a from a where a = 1 ;
vs
> select a from a where a = 1 \; \into a
> select a+1 from a where a = 1; \into b
> ...
> 2.093 select a from a where a = 1 ; select a+1 from a where a = 1;
Yep.
Note that there is a small logical conumdrum in this argument: As the
script is not the same, especially as into was not possible before,
strictly speaking there is no behavior "change".
This said, what you suggest can be done.
After giving it some thought, I suggest that it is not needed nor
desirable. If you want to achieve the initial effect, you just have to put
the "into a" on the next line:
select a from a where a = 1 \;
\into a
select a+1 from a where a = 1; \into b
Then you would get the -r cut at the end of the compound command. Thus the
current version gives full control of what will appear in the summary. If
I change "\into xxx\n" to mean "also cut here", then there is less control
on when the cut occurs when into is used.
> One more thing I observed which I am not sure if it's a fault of this
> patch is illustrated below:
>
> $ cat /tmp/into.sql
> \;
> select a from a where a = 1 \;
> select a+1 from a where a = 1;
>
> $ pgbench -r -n -t 1 -f /tmp/into.sql postgres
> <snip>
> - statement latencies in milliseconds:
> 2.349 ;
>
> Note that the compound select statement is nowhere to be seen in the
> latencies output. The output remains the same even if I use the \into's.
> What seems to be going on is that the empty statement on the first line
> (\;) is the only part kept of the compound statement spanning lines 1-3.
Yes.
This is really the (debatable) current behavior, and is not affected by
the patch. The "-r" summary takes the first line of the command, whatever
it is. In your example the first line is "\;", so you get what you asked
for, even if it looks rather strange, obviously.
>>> + bool sql_command_in_progress = false;
> [...]
> I understood that it refers to what you explain here. But to me it
> sounded like the name is referring to the progress of *execution* of a SQL
> command whereas the code in question is simply expecting to finish
> *parsing* the SQL command using the next lines.
Ok. I changed it "sql_command_lexing_in_progress".
>> The attached patch takes into all your comments but:
>> - comment about discarded results...
>> - the sql_command_in_progress variable name change
>> - the longer message on into at the start of a script
>
> The patch seems fine without these, although please consider the concern I
> raised with regard to the -r option above.
I have considered it. As the legitimate behavior you suggested can be
achieved just by putting the into on the next line, ISTM that the current
proposition gives more control than doing a mandatory cut when into is
used.
Attached is a new version with the boolean renaming.
The other thing I have considered is whether to implemented a "\gset"
syntax, as suggested by Pavel and Tom. Bar the aesthetic, the main issue I
have with it is that it does not work with compound commands, and what I
want is to get the values out of compound commands... because of my focus
on latency... so basically "\gset" does not do the job I want... Now I
recognize that other people would like it, so probably I'll do it anyway
in another patch.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-5.patch (18.3K, ../../alpine.DEB.2.20.1609130730380.10870@lancre/2-pgbench-into-5.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 285608d..0a474e1 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -809,6 +809,30 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-into'>
+ <term>
+ <literal>\into <replaceable>var1</> [<replaceable>var2</> ...]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the current or preceding
+ SQL command into these variables.
+ The queries must yield exactly one row and the number of provided
+ variables must be less than the total number of columns of the results.
+ This meta command does not end the current SQL command.
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+SELECT abalance \into abalance
+ FROM pgbench_accounts WHERE aid=5432;
+</programlisting>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 56c37d5..8817ac5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -302,11 +302,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char ***intovars; /* per-compound command \into variables */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1150,6 +1153,107 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char ***intovars)
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (intovars[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "\\into expects a row\n",
+ st->id, st->state, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (intovars[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f = 0;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->state, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ while (intovars[compound][f] != NULL && f < nfields)
+ {
+ /* store result as a string */
+ if (!putVariable(st, "into", intovars[compound][f],
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d state %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->state, compound, intovars[compound][f]);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ f++;
+ }
+
+ if (intovars[compound][f] != NULL)
+ {
+ fprintf(stderr,
+ "client %d state %d compound %d: missing results"
+ " to fill into variable %s\n",
+ st->id, st->state, compound, intovars[compound][f]);
+ st->ecnt++;
+ return false;
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr, "client %d aborted in state %d compound %d: %s",
+ st->id, st->state, compound, PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d state %d: no results\n", st->id, st->state);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1766,7 +1870,6 @@ chooseScript(TState *thread)
static bool
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command **commands;
bool trans_needs_throttle = false;
instr_time now;
@@ -1893,23 +1996,11 @@ top:
{
/*
* Read and discard the query result; note this is not included in
- * the statement latency numbers.
+ * the statement latency numbers (above), thus if reading the
+ * response fails the transaction is counted nevertheless.
*/
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- break; /* OK */
- default:
- fprintf(stderr, "client %d aborted in state %d: %s",
- st->id, st->state, PQerrorMessage(st->con));
- PQclear(res);
- return clientDone(st);
- }
- PQclear(res);
- discard_response(st);
+ if (!read_response(st, commands[st->state]->intovars))
+ return clientDone(st);
}
if (commands[st->state + 1] == NULL)
@@ -2704,22 +2795,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2739,17 +2818,85 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->intovars = pg_malloc0(sizeof(char **) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int space;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* add a separator if needed */
+ space = isspace(my_command->lines[len-1]) ? 0 : 1;
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + space + lmore + 1);
+ if (space > 0)
+ my_command->lines[len] = '\n';
+ memcpy(my_command->lines + len + space, more, lmore+1);
+
+ if (compounds > 0)
+ {
+ int nc = my_command->compound + compounds;
+ my_command->intovars =
+ pg_realloc(my_command->intovars, sizeof(char **) * (nc+1));
+ memset(my_command->intovars + my_command->compound + 1, 0,
+ sizeof(char **) * compounds);
+ my_command->compound = nc;
+ }
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -2763,19 +2910,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -2933,6 +3078,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "into") == 0)
+ {
+ /* at least one variable name must be provided */
+ if (my_command->argc < 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "missing variable names", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -2944,6 +3096,15 @@ process_backslash_command(PsqlScanState sstate, const char *source)
return my_command;
}
+static bool
+ends_with_semicolon(const char *buf)
+{
+ int i = strlen(buf)-1;
+ while (i > 0 && isspace(buf[i]))
+ i--;
+ return i > 0 && buf[i] == ';';
+}
+
/*
* Parse a script (either the contents of a file, or a built-in script)
* and add it to the list of scripts.
@@ -2956,6 +3117,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool sql_command_lexing_in_progress = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -2968,6 +3132,7 @@ ParseScript(const char *script, const char *desc, int weight)
/* Prepare to parse script */
sstate = psql_scan_create(&pgbench_callbacks);
+ sstate->rm_c_comments = true;
/*
* Ideally, we'd scan scripts using the encoding and stdstrings settings
@@ -2979,6 +3144,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -2988,31 +3154,32 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
+ sstate->empty_cmds = 0;
+ sstate->last_cmd_start = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (sql_command_lexing_in_progress)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command was interrupted by an \into */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons - sstate->empty_cmds);
+ sql_command_lexing_in_progress = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command =
+ create_sql_command(&line_buf, desc,
+ sstate->semicolons - sstate->empty_cmds);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3027,6 +3194,75 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ /* Merge \into into preceeding SQL command ... */
+ if (pg_strcasecmp(command->argv[0], "into") == 0)
+ {
+ int cindex, i;
+ Command *into_cmd;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot start a script",
+ NULL, -1);
+
+ into_cmd = ps.commands[index-1];
+
+ if (into_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into must follow an SQL command",
+ into_cmd->line, -1);
+
+ /* this \into applies to this sub-command */
+ cindex = into_cmd->compound -
+ (ends_with_semicolon(line_buf.data) ? 1 : 0);
+
+ if (into_cmd->intovars[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot follow an \\into",
+ NULL, -1);
+
+ /* check that all variable names are valid */
+ for (i = 1; i < command->argc; i++)
+ {
+ if (!isLegalVariableName(command->argv[i]))
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into invalid variable name",
+ command->argv[i], -1);
+ }
+
+ into_cmd->intovars[cindex] =
+ pg_malloc0(sizeof(char *) * (command->argc+1));
+
+ memcpy(into_cmd->intovars[cindex], command->argv + 1,
+ sizeof(char*) * command->argc);
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+
+ /* "SELECT ... \into ..." follow-up */
+ sql_command_lexing_in_progress = *line_buf.data != '\0';
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3034,6 +3270,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index ab0f822..4d28fdb 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 55067b4..aa1ab6d 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -71,6 +71,8 @@ typedef int YYSTYPE;
extern int psql_yyget_column(yyscan_t yyscanner);
extern void psql_yyset_column(int column_no, yyscan_t yyscanner);
+static bool buffer_empty_since(PQExpBuffer buf, size_t len);
+
%}
%option reentrant
@@ -388,14 +390,16 @@ other .
BEGIN(xc);
/* Put back any characters past slash-star; see above */
yyless(2);
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcstart} {
cur_state->xcdepth++;
/* Put back any characters past slash-star; see above */
yyless(2);
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcstop} {
@@ -403,19 +407,23 @@ other .
BEGIN(INITIAL);
else
cur_state->xcdepth--;
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcinside} {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{op_chars} {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>\*+ {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
{xbstart} {
@@ -680,6 +688,16 @@ other .
"\\"[;:] {
/* Force a semicolon or colon into the query buffer */
+ if (yytext[1] == ';')
+ {
+ /* count compound commands */
+ cur_state->semicolons++;
+ /* detect empty commands */
+ if (buffer_empty_since(cur_state->output_buf, cur_state->last_cmd_start))
+ cur_state->empty_cmds++;
+ cur_state->last_cmd_start =
+ cur_state->output_buf->len + 2;
+ }
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1323,6 +1341,20 @@ psqlscan_prepare_buffer(PsqlScanState state, const char *txt, int len,
return yy_scan_buffer(newtxt, len + 2, state->scanner);
}
+/* tell whether there were only spaces inserted since len
+ */
+static bool
+buffer_empty_since(PQExpBuffer buf, size_t len)
+{
+ while (len < buf->len)
+ {
+ if (!isspace(buf->data[len]))
+ return false;
+ len++;
+ }
+ return true;
+}
+
/*
* psqlscan_emit() --- body for ECHO macro
*
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index a52929d..9ec5f51 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,10 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
+ int empty_cmds; /* number of empty commands (\;\;) */
+ size_t last_cmd_start; /* position in output buffer */
+ bool rm_c_comments; /* whether to remove C comments */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-14 10:39 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-14 10:39 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hi Fabien,
On 2016/09/13 17:41, Fabien COELHO wrote:
>
> Hello Amit,
>
>> [...]
>> There still seems to be a change in behavior of the -r option due to the
>> patch. Consider the following example:
>>
>> select a from a where a = 1 \;
>> select a+1 from a where a = 1;
>> ...
>> - statement latencies in milliseconds:
>> 2.889 select a from a where a = 1 ;
>
> vs
>
>> select a from a where a = 1 \; \into a
>> select a+1 from a where a = 1; \into b
>> ...
>> 2.093 select a from a where a = 1 ; select a+1 from a where a = 1;
>
> Yep.
>
> Note that there is a small logical conumdrum in this argument: As the
> script is not the same, especially as into was not possible before,
> strictly speaking there is no behavior "change".
Sure, scripts are not the same but it seemed like showing the whole
compound query whereas previously only part of it was shown may be an
implementation artifact of \into.
> This said, what you suggest can be done.
>
> After giving it some thought, I suggest that it is not needed nor
> desirable. If you want to achieve the initial effect, you just have to put
> the "into a" on the next line:
>
> select a from a where a = 1 \;
> \into a
> select a+1 from a where a = 1; \into b
>
> Then you would get the -r cut at the end of the compound command. Thus the
> current version gives full control of what will appear in the summary. If
> I change "\into xxx\n" to mean "also cut here", then there is less control
> on when the cut occurs when into is used.
So it means that position of \into affects where a compound command gets
cut for -r display. I was just wondering if that was unintentional.
>> One more thing I observed which I am not sure if it's a fault of this
>> patch is illustrated below:
>>
>> $ cat /tmp/into.sql
>> \;
>> select a from a where a = 1 \;
>> select a+1 from a where a = 1;
>>
>> $ pgbench -r -n -t 1 -f /tmp/into.sql postgres
>> <snip>
>> - statement latencies in milliseconds:
>> 2.349 ;
>>
>> Note that the compound select statement is nowhere to be seen in the
>> latencies output. The output remains the same even if I use the \into's.
>> What seems to be going on is that the empty statement on the first line
>> (\;) is the only part kept of the compound statement spanning lines 1-3.
>
> Yes.
>
> This is really the (debatable) current behavior, and is not affected by
> the patch. The "-r" summary takes the first line of the command, whatever
> it is. In your example the first line is "\;", so you get what you asked
> for, even if it looks rather strange, obviously.
Yep, perhaps it's strange to write a script like that anyway, :)
>>>> + bool sql_command_in_progress = false;
>> [...]
>> I understood that it refers to what you explain here. But to me it
>> sounded like the name is referring to the progress of *execution* of a SQL
>> command whereas the code in question is simply expecting to finish
>> *parsing* the SQL command using the next lines.
>
> Ok. I changed it "sql_command_lexing_in_progress".
>
>>> The attached patch takes into all your comments but:
>>> - comment about discarded results...
>>> - the sql_command_in_progress variable name change
>>> - the longer message on into at the start of a script
>>
>> The patch seems fine without these, although please consider the concern I
>> raised with regard to the -r option above.
>
> I have considered it. As the legitimate behavior you suggested can be
> achieved just by putting the into on the next line, ISTM that the current
> proposition gives more control than doing a mandatory cut when into is used.
>
> Attached is a new version with the boolean renaming.
Thanks.
> The other thing I have considered is whether to implemented a "\gset"
> syntax, as suggested by Pavel and Tom. Bar the aesthetic, the main issue I
> have with it is that it does not work with compound commands, and what I
> want is to get the values out of compound commands... because of my focus
> on latency... so basically "\gset" does not do the job I want... Now I
> recognize that other people would like it, so probably I'll do it anyway
> in another patch.
So, psql's \gset does not work as desired for compound commands (viz. it
is only able to save the result of the last sub-command). You need to use
\gset with each sub-command separately if no result should be discarded.
Because of undesirable latency characteristics of sending multiple
commands, you want to be able to modify compound command handling such
that every sub-command's result could be saved. In that regard, it's good
that pgbench does not use PQexec() which only returns the result of the
last sub-command if a compound command was issued through it. pgbench's
doCustom() currently discards all results by issuing discard_response().
You propose to change things such that results are intercepted in a new
function read_response(), values assigned to intovars corresponding to
each sub-command, and then discarded. The intovars arrays are allocated
within each sub-command's Command struct when parsing the compound command
based on where \into is located wrt to the sub-command.
So, most of the code in the patch is about handling compound statements to
be be able to save results of all sub-commands, not just the last. Do you
think it would be OK to suffer the bad latency of multiple round trips and
implement a version of \into (or \gset or \ginto) for pgbench scripts that
behaves exactly like psql's \gset as a first step? But you say you will
do it as another patch.
By the way, I tend to agree with your point about \gset syntax being
suitable (only) in a interactive context such as psql; it's not as
readable as \into x y ... when used in a script.
Thanks,
Amit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-18 11:13 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2016-09-18 11:13 UTC (permalink / raw)
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Amit,
>> [...]
>> Then you would get the -r cut at the end of the compound command. Thus the
>> current version gives full control of what will appear in the summary. If
>> I change "\into xxx\n" to mean "also cut here", then there is less control
>> on when the cut occurs when into is used.
>
> So it means that position of \into affects where a compound command gets
> cut for -r display. I was just wondering if that was unintentional.
Yes, but it happens to work reasonnably:-)
>> The other thing I have considered is whether to implemented a "\gset"
>> syntax, as suggested by Pavel and Tom. Bar the aesthetic, the main issue I
>> have with it is that it does not work with compound commands, and what I
>> want is to get the values out of compound commands... because of my focus
>> on latency... so basically "\gset" does not do the job I want... Now I
>> recognize that other people would like it, so probably I'll do it anyway
>> in another patch.
>
> So, psql's \gset does not work as desired for compound commands (viz. it
> is only able to save the result of the last sub-command).
Yes.
> You need to use \gset with each sub-command separately if no result
> should be discarded. Because of undesirable latency characteristics of
> sending multiple commands, you want to be able to modify compound
> command handling such that every sub-command's result could be saved.
Exactly.
> In that regard, it's good that pgbench does not use PQexec() which only
> returns the result of the last sub-command if a compound command was
> issued through it.
Indeed!
> pgbench's
> doCustom() currently discards all results by issuing discard_response().
> You propose to change things such that results are intercepted in a new
> function read_response(), values assigned to intovars corresponding to
> each sub-command, and then discarded. The intovars arrays are allocated
> within each sub-command's Command struct when parsing the compound command
> based on where \into is located wrt to the sub-command.
Yep.
> So, most of the code in the patch is about handling compound statements to
> be be able to save results of all sub-commands, not just the last.
Yep. Previously pgbench did not need to handle compounds commands which
where just seen as one large string.
Note that the added machinery is also a first step to allow the handling
of prepared statements on compounds command, which I think is a desirable
feature for benchmarks.
> Do you think it would be OK to suffer the bad latency of multiple round
> trips and implement a version of \into (or \gset or \ginto) for pgbench
> scripts that behaves exactly like psql's \gset as a first step?
I do not see gset as a reasonnable "first step" because: (0) "\into"
already works while "\gset" in pgbench will need some time that I do not
have at the moment (1) it is not what I want/need to do a clean bench (2)
the feature is not orthogonal to compounds statements -- which is what I
want -- (3) I do not like the "implicit" naming thing -- but this is
really just a matter of taste.
I simply recognize that Peter & Tom have a point: whatever I think of gset
it is there in "psql" so it makes some sense to have it as well in
"pgbench", so I agree to put that on my pgbench todo list.
> But you say you will do it as another patch.
Yep. I suggested another patch because it is a different feature and
previous submissions where I mixed features, even closely related ones,
all resulted in me having to separate the features in distinct patches.
> By the way, I tend to agree with your point about \gset syntax being
> suitable (only) in a interactive context such as psql; it's not as
> readable as \into x y ... when used in a script.
Yep, but as people already know it, it makes sense to provide it as well
at some point.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-26 07:12 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-26 07:12 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hi Fabien,
I am marking the pgbench-into-5.patch [1] as "Ready for Committer" as I
have no further comments at the moment.
Thanks,
Amit
[1]
https://www.postgresql.org/message-id/alpine.DEB.2.20.1609130730380.10870%40lancre
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-26 08:19 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-26 08:19 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On 2016/09/26 16:12, Amit Langote wrote:
> I am marking the pgbench-into-5.patch [1] as "Ready for Committer" as I
> have no further comments at the moment.
Wait... Heikki's latest commit now requires this patch to be rebased.
commit 12788ae49e1933f463bc59a6efe46c4a01701b76
Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date: Mon Sep 26 10:56:02 2016 +0300
Refactor script execution state machine in pgbench.
So, will change the status to "Waiting on Author".
Thanks,
Amit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-26 11:27 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 2 replies; 120+ messages in thread
From: Fabien COELHO @ 2016-09-26 11:27 UTC (permalink / raw)
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Amit,
>> I am marking the pgbench-into-5.patch [1] as "Ready for Committer" as I
>> have no further comments at the moment.
>
> Wait... Heikki's latest commit now requires this patch to be rebased.
Indeed. Here is the rebased version, which still get through my various
tests.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-6.patch (18.5K, ../../alpine.DEB.2.20.1609261324480.11795@lancre/2-pgbench-into-6.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 285608d..0a474e1 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -809,6 +809,30 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-into'>
+ <term>
+ <literal>\into <replaceable>var1</> [<replaceable>var2</> ...]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ Stores the first fields of the resulting row from the current or preceding
+ SQL command into these variables.
+ The queries must yield exactly one row and the number of provided
+ variables must be less than the total number of columns of the results.
+ This meta command does not end the current SQL command.
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+SELECT abalance \into abalance
+ FROM pgbench_accounts WHERE aid=5432;
+</programlisting>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 1fb4ae4..9c13a18 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -373,11 +373,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char ***intovars; /* per-compound command \into variables */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1221,6 +1224,110 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char ***intovars)
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (intovars[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\into expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (intovars[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f = 0;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ while (intovars[compound][f] != NULL && f < nfields)
+ {
+ /* store result as a string */
+ if (!putVariable(st, "into", intovars[compound][f],
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ intovars[compound][f]);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ f++;
+ }
+
+ if (intovars[compound][f] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: missing results"
+ " to fill into variable %s\n",
+ st->id, st->use_file, st->command, compound,
+ intovars[compound][f]);
+ st->ecnt++;
+ return false;
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr, "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1953,7 +2060,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2291,26 +2397,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->intovars))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2943,22 +3035,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2978,17 +3058,85 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->intovars = pg_malloc0(sizeof(char **) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int space;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* add a separator if needed */
+ space = isspace(my_command->lines[len-1]) ? 0 : 1;
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + space + lmore + 1);
+ if (space > 0)
+ my_command->lines[len] = '\n';
+ memcpy(my_command->lines + len + space, more, lmore+1);
+
+ if (compounds > 0)
+ {
+ int nc = my_command->compound + compounds;
+ my_command->intovars =
+ pg_realloc(my_command->intovars, sizeof(char **) * (nc+1));
+ memset(my_command->intovars + my_command->compound + 1, 0,
+ sizeof(char **) * compounds);
+ my_command->compound = nc;
+ }
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3002,19 +3150,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -3172,6 +3318,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "into") == 0)
+ {
+ /* at least one variable name must be provided */
+ if (my_command->argc < 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "missing variable names", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3183,6 +3336,15 @@ process_backslash_command(PsqlScanState sstate, const char *source)
return my_command;
}
+static bool
+ends_with_semicolon(const char *buf)
+{
+ int i = strlen(buf)-1;
+ while (i > 0 && isspace(buf[i]))
+ i--;
+ return i > 0 && buf[i] == ';';
+}
+
/*
* Parse a script (either the contents of a file, or a built-in script)
* and add it to the list of scripts.
@@ -3195,6 +3357,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool sql_command_lexing_in_progress = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3207,6 +3372,7 @@ ParseScript(const char *script, const char *desc, int weight)
/* Prepare to parse script */
sstate = psql_scan_create(&pgbench_callbacks);
+ sstate->rm_c_comments = true;
/*
* Ideally, we'd scan scripts using the encoding and stdstrings settings
@@ -3218,6 +3384,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3227,31 +3394,32 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
+ sstate->empty_cmds = 0;
+ sstate->last_cmd_start = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (sql_command_lexing_in_progress)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command was interrupted by an \into */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons - sstate->empty_cmds);
+ sql_command_lexing_in_progress = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command =
+ create_sql_command(&line_buf, desc,
+ sstate->semicolons - sstate->empty_cmds);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3266,6 +3434,75 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ /* Merge \into into preceeding SQL command ... */
+ if (pg_strcasecmp(command->argv[0], "into") == 0)
+ {
+ int cindex, i;
+ Command *into_cmd;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot start a script",
+ NULL, -1);
+
+ into_cmd = ps.commands[index-1];
+
+ if (into_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into must follow an SQL command",
+ into_cmd->line, -1);
+
+ /* this \into applies to this sub-command */
+ cindex = into_cmd->compound -
+ (ends_with_semicolon(line_buf.data) ? 1 : 0);
+
+ if (into_cmd->intovars[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into cannot follow an \\into",
+ NULL, -1);
+
+ /* check that all variable names are valid */
+ for (i = 1; i < command->argc; i++)
+ {
+ if (!isLegalVariableName(command->argv[i]))
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\into invalid variable name",
+ command->argv[i], -1);
+ }
+
+ into_cmd->intovars[cindex] =
+ pg_malloc0(sizeof(char *) * (command->argc+1));
+
+ memcpy(into_cmd->intovars[cindex], command->argv + 1,
+ sizeof(char*) * command->argc);
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+
+ /* "SELECT ... \into ..." follow-up */
+ sql_command_lexing_in_progress = *line_buf.data != '\0';
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3273,6 +3510,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index ab0f822..4d28fdb 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 55067b4..aa1ab6d 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -71,6 +71,8 @@ typedef int YYSTYPE;
extern int psql_yyget_column(yyscan_t yyscanner);
extern void psql_yyset_column(int column_no, yyscan_t yyscanner);
+static bool buffer_empty_since(PQExpBuffer buf, size_t len);
+
%}
%option reentrant
@@ -388,14 +390,16 @@ other .
BEGIN(xc);
/* Put back any characters past slash-star; see above */
yyless(2);
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcstart} {
cur_state->xcdepth++;
/* Put back any characters past slash-star; see above */
yyless(2);
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcstop} {
@@ -403,19 +407,23 @@ other .
BEGIN(INITIAL);
else
cur_state->xcdepth--;
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{xcinside} {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>{op_chars} {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
<xc>\*+ {
- ECHO;
+ if (!cur_state->rm_c_comments)
+ ECHO;
}
{xbstart} {
@@ -680,6 +688,16 @@ other .
"\\"[;:] {
/* Force a semicolon or colon into the query buffer */
+ if (yytext[1] == ';')
+ {
+ /* count compound commands */
+ cur_state->semicolons++;
+ /* detect empty commands */
+ if (buffer_empty_since(cur_state->output_buf, cur_state->last_cmd_start))
+ cur_state->empty_cmds++;
+ cur_state->last_cmd_start =
+ cur_state->output_buf->len + 2;
+ }
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1323,6 +1341,20 @@ psqlscan_prepare_buffer(PsqlScanState state, const char *txt, int len,
return yy_scan_buffer(newtxt, len + 2, state->scanner);
}
+/* tell whether there were only spaces inserted since len
+ */
+static bool
+buffer_empty_since(PQExpBuffer buf, size_t len)
+{
+ while (len < buf->len)
+ {
+ if (!isspace(buf->data[len]))
+ return false;
+ len++;
+ }
+ return true;
+}
+
/*
* psqlscan_emit() --- body for ECHO macro
*
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index a52929d..9ec5f51 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,10 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
+ int empty_cmds; /* number of empty commands (\;\;) */
+ size_t last_cmd_start; /* position in output buffer */
+ bool rm_c_comments; /* whether to remove C comments */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-09-27 01:41 Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 120+ messages in thread
From: Amit Langote @ 2016-09-27 01:41 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On 2016/09/26 20:27, Fabien COELHO wrote:
>
> Hello Amit,
>
>>> I am marking the pgbench-into-5.patch [1] as "Ready for Committer" as I
>>> have no further comments at the moment.
>>
>> Wait... Heikki's latest commit now requires this patch to be rebased.
>
> Indeed. Here is the rebased version, which still get through my various
> tests.
Thanks, Fabien. It seems to work here too.
Regards,
Amit
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-10-02 13:43 Michael Paquier <michael.paquier@gmail.com>
parent: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Michael Paquier @ 2016-10-02 13:43 UTC (permalink / raw)
To: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Tue, Sep 27, 2016 at 10:41 AM, Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> On 2016/09/26 20:27, Fabien COELHO wrote:
>>
>> Hello Amit,
>>
>>>> I am marking the pgbench-into-5.patch [1] as "Ready for Committer" as I
>>>> have no further comments at the moment.
>>>
>>> Wait... Heikki's latest commit now requires this patch to be rebased.
>>
>> Indeed. Here is the rebased version, which still get through my various
>> tests.
>
> Thanks, Fabien. It seems to work here too.
Moved to next CF with same status, ready for committer.
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2016-12-02 06:33 Haribabu Kommi <kommi.haribabu@gmail.com>
parent: Michael Paquier <michael.paquier@gmail.com>
0 siblings, 0 replies; 120+ messages in thread
From: Haribabu Kommi @ 2016-12-02 06:33 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Fabien COELHO <coelho@cri.ensmp.fr>; Tom Lane <tgl@sss.pgh.pa.us>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Mon, Oct 3, 2016 at 12:43 AM, Michael Paquier <michael.paquier@gmail.com>
wrote:
> On Tue, Sep 27, 2016 at 10:41 AM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
> > On 2016/09/26 20:27, Fabien COELHO wrote:
> >>
> >> Hello Amit,
> >>
> >>>> I am marking the pgbench-into-5.patch [1] as "Ready for Committer" as
> I
> >>>> have no further comments at the moment.
> >>>
> >>> Wait... Heikki's latest commit now requires this patch to be rebased.
> >>
> >> Indeed. Here is the rebased version, which still get through my various
> >> tests.
> >
> > Thanks, Fabien. It seems to work here too.
>
> Moved to next CF with same status, ready for committer.
>
Moved to next CF with same status (ready for committer).
Regards,
Hari Babu
Fujitsu Australia
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-01-04 21:15 Tom Lane <tgl@sss.pgh.pa.us>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 2 replies; 120+ messages in thread
From: Tom Lane @ 2017-01-04 21:15 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Fabien COELHO <coelho@cri.ensmp.fr> writes:
> Indeed. Here is the rebased version, which still get through my various
> tests.
I looked through this again, and I still think that the syntactic design
of the new command is seriously misguided, leading to an ugly and
unmaintainable implementation that may well block further innovation
in pgbench. I will not commit it in this form.
Possibly you can find some other committer whom you can convince this
is a good design --- but since the patch has gone untouched for two full
commitfest cycles, I rather imagine that whoever else has looked at it
has likewise decided they didn't want to be responsible for it.
Please look at changing \into to be a SQL-command-ending backslash
command as we previously discussed. I think you will find that the
implementation is a great deal simpler that way and doesn't require
weird hackery on the shared lexer.
(BTW, said hackery is not just weird but broken. You can't simply
remove comments. Consider something like "SELECT foo/*as*/bar".
This code reduces that to "SELECT foobar" which is wrong.)
If you won't do that, and you can't find another committer who will
accept responsibility for this patch before the end of the current
commitfest, I think we should mark it Rejected.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-01-05 09:47 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-01-05 09:47 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Tom,
> Please look at changing \into to be a SQL-command-ending backslash
> command as we previously discussed.
Hmmm.
I do want storing results & compound command ending to be orthogonal.
In order to keep this feature, I think that I can move the
"into/ginto/gset/..." at the end of the command. For the compound command
list to necessarily end, I can probably do some reassembly as a post phase
on Commands in pgbench so that the impact on the lexer is much reduced, in
particular without undue "hackery" as you put it.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-01-07 09:25 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 2 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-01-07 09:25 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Tom,
> Please look at changing \into to be a SQL-command-ending backslash
> command as we previously discussed.
Done.
There are two variants: \gset & \gcset for compound (end SQL query, set
variables, but do not end command, so that several settings are allowed in
a compound command, a key feature for performance testing).
Personnally, I find the end-of-query semicolon-replacing syntax ugly.
However I'm more interested in feature than in elegance on this one, so
I'll put up with it.
> I think you will find that the implementation is a great deal simpler
> that way and doesn't require weird hackery on the shared lexer.
I have removed the "hackery", only counting embedded semicolons remains to
keep track of compound queries.
Note that the (somehow buggy and indeed not too clean) hackery was not
related to the into syntax, but to detecting empty queries which are
silently skipped by the server.
> If you won't do that, [...]
I think that I have done what you required.
I have documented the fact that now the feature does not work if compound
commands contain empty queries, which is a very minor drawback for a
pgbench script anyway.
Attached are the patch, a test script for the feature, and various test
scripts to trigger error cases.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-7.patch (16.0K, ../../alpine.DEB.2.20.1701070938300.10378@lancre/2-pgbench-into-7.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 3fb29f8..9a0fb12 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -815,6 +815,51 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gcset [<replaceable>prefix</>]</literal> or
+ <literal>\gset [<replaceable>prefix</>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\gcset</> replaces an embedded semicolon (<literal>\;</>) within
+ a compound SQL command, and <literal>\gset</> replaces a final
+ (<literal>;</>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</>, and fills variables
+ <replaceable>one</>, <replaceable>two</> and <replaceable>p_three</> with
+ integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \gcset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gcset</> and <literal>\gset</> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index f6cb5d4..d57eb31 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1225,6 +1228,105 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1957,7 +2059,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2295,26 +2396,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2944,22 +3031,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2979,17 +3054,79 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3003,19 +3140,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -3173,6 +3308,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "gcset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most on argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3196,6 +3338,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3219,6 +3364,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3228,31 +3374,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \gcset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3267,6 +3410,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "gcset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[1] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3274,6 +3478,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 38b3af5..00bf432 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1b29341..d923f70 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -678,8 +678,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0fddc7a..c1899df 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
[application/x-sql] gset-1.sql (473B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/3-gset-1.sql)
download
[application/x-sql] gset-err-1.sql (36B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/4-gset-err-1.sql)
download
[application/x-sql] gset-err-2.sql (46B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/5-gset-err-2.sql)
download
[application/x-sql] gset-err-3.sql (35B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/6-gset-err-3.sql)
download
[application/x-sql] gset-err-4.sql (15B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/7-gset-err-4.sql)
download
[application/x-sql] gset-err-5.sql (31B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/8-gset-err-5.sql)
download
[application/x-sql] gset-err-6.sql (13B, ../../alpine.DEB.2.20.1701070938300.10378@lancre/9-gset-err-6.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-01-29 08:15 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-01-29 08:15 UTC (permalink / raw)
To: pgsql-hackers
<APOLOGY>
Please pardon the redondance: this is a slightly edited repost
from another thread where motivation for this patch was discussed, so
that it appear in the relevant thread.
</APOLOGY>
Tom> [...] there was immediately objection as to whether his idea of TPC-B
Tom> compliance was actually right.
From my point of view TPC-* are simply objective examples of typical
benchmark requirements to show which features are needed in a tool for
doing this activity. Once features are available, I think that pgbench
should also be a show-case for their usage. Currently a few functions (for
implementing the bench as specified) and actually extracting results into
variables (for suspicious auditors and bench relevance, see below) are
missing.
Tom> I remember complaining that he had a totally artificial idea of what
Tom> "fetching a data value" requires.
Yep.
I think that the key misunderstanding is that you are honest and assume
that other people are honest too. This is naïve: There is a long history
of vendors creatively "cheating" to get better than deserve benchmark
results. Benchmark specifications try to prevent such behaviors by laying
careful requirements and procedures.
In this instance, you "know" that when pg has returned the result of the
query the data is actually on the client side, so you considered it is
fetched. That is fine for you, but from a benchmarking perspective with
external auditors your belief/knowledge is not good enough.
For instance, the vendor could implement a new version of the protocol
where the data are only transfered on demand, and the result just tells
that the data is indeed somewhere on the server (eg on "SELECT abalance"
it could just check that the key exists, no need to actually fetch the
data from the table, so no need to read the table, the index is
enough...). That would be pretty stupid for real application performance,
but the benchmark would get better tps by doing so.
Without even intentionnaly cheating, this could be part of a useful
"streaming mode" protocol option which make sense for very large results
but would be activated for a small result.
Another point is that decoding the message may be a little expensive, so
that by not actually extracting the data into the client but just keeping
it in the connection/OS one gets better performance.
Thus, TPC-B 2.0.0 benchmark specification says:
"1.3.2 Each transaction shall return to the driver the Account_Balance
resulting from successful commit of the transaction.
Comment: It is the intent of this clause that the account balance in the
database be returned to the driver, i.e., that the application retrieve
the account balance."
For me the correct interpretation of "the APPLICATION retrieve the account
balance" is that the client application code, pgbench in this context, did
indeed get the value from the vendor code, here "libpq" which is handling
the connection.
Having the value discarded from libpq by calling PQclear instead of
PQntuples/PQgetvalue/... skips a key part of the client code that no real
application would skip. This looks strange and is not representative of
real client code: as a potential auditor, because of this performance
impact doubt and lack of relevance, I would not check the corresponding
item in the audit check list:
"11.3.1.2 Verify that transaction inputs and outputs satisfy Clause 1.3."
So the benchmark implementation would not be validated.
Another trivial reason to be able to actually retrieve data is that for
benchmarking purpose it is very easy to want to test a scenario where you
do different things based on data received, which imply that the data can
be manipulated somehow on the benchmarking client side, which is currently
not possible.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-01-31 06:13 Michael Paquier <michael.paquier@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 120+ messages in thread
From: Michael Paquier @ 2017-01-31 06:13 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Sat, Jan 7, 2017 at 6:25 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
> I think that I have done what you required.
>
> I have documented the fact that now the feature does not work if compound
> commands contain empty queries, which is a very minor drawback for a pgbench
> script anyway.
>
> Attached are the patch, a test script for the feature, and various test
> scripts to trigger error cases.
I have moved this patch to next CF as the last status is a new patch
set with no further reviews. I did not check if the comments have been
applied though, this is a bit too much for me now..
--
Michael
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-01-31 06:24 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Michael Paquier <michael.paquier@gmail.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-01-31 06:24 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Bonjour Michaël,
>> Attached are the patch, a test script for the feature, and various test
>> scripts to trigger error cases.
>
> I have moved this patch to next CF
Ok.
> as the last status is a new patch set with no further reviews.
Indeed.
> I did not check if the comments have been applied though, this is a bit
> too much for me now...
Sure.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-03-15 05:53 Rafia Sabih <rafia.sabih@enterprisedb.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Rafia Sabih @ 2017-03-15 05:53 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Tue, Jan 31, 2017 at 11:54 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> Bonjour Michaël,
>
> Attached are the patch, a test script for the feature, and various test
>>> scripts to trigger error cases.
>>>
>>
>> I have moved this patch to next CF
>>
>
> Ok.
>
> as the last status is a new patch set with no further reviews.
>>
>
> Indeed.
>
> I did not check if the comments have been applied though, this is a bit
>> too much for me now...
>>
>
> Sure.
I was reviewing v7 of this patch, to start with I found following white
space errors when applying with git apply,
/home/edb/Desktop/patches/others/pgbench-into-7.patch:66: trailing
whitespace.
char *line; /* first line for short display */
/home/edb/Desktop/patches/others/pgbench-into-7.patch:67: trailing
whitespace.
char *lines; /* full multi-line text of command */
/home/edb/Desktop/patches/others/pgbench-into-7.patch:72: trailing
whitespace.
int compound; /* last compound command (number of \;) */
/home/edb/Desktop/patches/others/pgbench-into-7.patch:73: trailing
whitespace.
char **gset; /* per-compound command prefix */
/home/edb/Desktop/patches/others/pgbench-into-7.patch:81: trailing
whitespace.
/* read all responses from backend */
error: patch failed: doc/src/sgml/ref/pgbench.sgml:815
error: doc/src/sgml/ref/pgbench.sgml: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.c:375
error: src/bin/pgbench/pgbench.c: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.h:11
error: src/bin/pgbench/pgbench.h: patch does not apply
error: patch failed: src/fe_utils/psqlscan.l:678
error: src/fe_utils/psqlscan.l: patch does not apply
error: patch failed: src/include/fe_utils/psqlscan_int.h:112
error: src/include/fe_utils/psqlscan_int.h: patch does not apply
Apart from that, on executing SELECT 1 AS a \gset \set i debug(:a) SELECT 2
AS a \gcset SELECT 3; given in your provided script gset-1.sql. it is
giving error Invalid command \gcset. Not sure what is the intention of this
script anyway? Also, instead of so many different files for error why don't
you combine it into one.
--
Regards,
Rafia Sabih
EnterpriseDB: http://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-03-15 19:15 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Rafia Sabih <rafia.sabih@enterprisedb.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-03-15 19:15 UTC (permalink / raw)
To: Rafia Sabih <rafia.sabih@enterprisedb.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Rafia,
> I was reviewing v7 of this patch, to start with I found following white
> space errors when applying with git apply,
> /home/edb/Desktop/patches/others/pgbench-into-7.patch:66: trailing
> whitespace.
Yep.
I do not know why "git apply" sometimes complains. All is fine for me both
with "git apply" and "patch".
Last time it was because my mailer uses text/x-diff for the mime type, as
define by the system in "/etc/mime.types", which some mailer then
interpret as a license to change eol-style when saving, resulting in this
kind of behavior. Could you tell your mailer just to save the file as is?
> Apart from that, on executing SELECT 1 AS a \gset \set i debug(:a) SELECT 2
> AS a \gcset SELECT 3; given in your provided script gset-1.sql. it is
> giving error Invalid command \gcset.
Are you sure that you are using the compiled pgbench, not a previously
installed one?
bin/pgbench> pgbench -t 1 -f SQL/gset-1.sql
SQL/gset-1.sql:1: invalid command in command "gset"
\gset
bin/pgbench> ./pgbench -t 1 -f SQL/gset-1.sql
starting vacuum...end.
debug(script=0,command=2): int 1
debug(script=0,command=4): int 2
...
> Not sure what is the intention of this script anyway?
The intention is to test that gset & gcset work as expected in various
settings, especially with combined queries (\;) the right result must be
extracted in the sequence.
> Also, instead of so many different files for error why don't you combine
> it into one.
Because a pgbench scripts stops on the first error, and I wanted to test
what happens with several kind of errors.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-03-23 10:59 Rafia Sabih <rafia.sabih@enterprisedb.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Rafia Sabih @ 2017-03-23 10:59 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Thu, Mar 16, 2017 at 12:45 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> Hello Rafia,
>
>> I was reviewing v7 of this patch, to start with I found following white
>> space errors when applying with git apply,
>> /home/edb/Desktop/patches/others/pgbench-into-7.patch:66: trailing
>> whitespace.
>
>
> Yep.
>
> I do not know why "git apply" sometimes complains. All is fine for me both
> with "git apply" and "patch".
>
> Last time it was because my mailer uses text/x-diff for the mime type, as
> define by the system in "/etc/mime.types", which some mailer then interpret
> as a license to change eol-style when saving, resulting in this kind of
> behavior. Could you tell your mailer just to save the file as is?
>
>> Apart from that, on executing SELECT 1 AS a \gset \set i debug(:a) SELECT
>> 2
>> AS a \gcset SELECT 3; given in your provided script gset-1.sql. it is
>> giving error Invalid command \gcset.
>
>
> Are you sure that you are using the compiled pgbench, not a previously
> installed one?
>
> bin/pgbench> pgbench -t 1 -f SQL/gset-1.sql
> SQL/gset-1.sql:1: invalid command in command "gset"
> \gset
>
> bin/pgbench> ./pgbench -t 1 -f SQL/gset-1.sql
> starting vacuum...end.
> debug(script=0,command=2): int 1
> debug(script=0,command=4): int 2
> ...
>
>> Not sure what is the intention of this script anyway?
>
>
> The intention is to test that gset & gcset work as expected in various
> settings, especially with combined queries (\;) the right result must be
> extracted in the sequence.
>
>> Also, instead of so many different files for error why don't you combine
>> it into one.
>
>
> Because a pgbench scripts stops on the first error, and I wanted to test
> what happens with several kind of errors.
>
if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most on argument expected", NULL, -1);
I suppose you mean 'one' argument here.
Apart from that indentation is not correct as per pgindent, please check.
--
Regards,
Rafia Sabih
EnterpriseDB: http://www.enterprisedb.com/
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-03-24 15:29 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Rafia Sabih <rafia.sabih@enterprisedb.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-03-24 15:29 UTC (permalink / raw)
To: Rafia Sabih <rafia.sabih@enterprisedb.com>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
Hello Rafia,
> if (my_command->argc > 2)
> + syntax_error(source, lineno, my_command->line, my_command->argv[0],
> + "at most on argument expected", NULL, -1);
>
> I suppose you mean 'one' argument here.
Indeed.
> Apart from that indentation is not correct as per pgindent, please check.
I guess that you are refering to switch/case indentation which my emacs
does not do as expected.
Please find attached a v8 which hopefully fixes these two issues.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-8.patch (16.0K, ../../alpine.DEB.2.20.1703241616580.28545@lancre/2-pgbench-into-8.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..1108fcb 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -816,6 +816,51 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gcset [<replaceable>prefix</>]</literal> or
+ <literal>\gset [<replaceable>prefix</>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\gcset</> replaces an embedded semicolon (<literal>\;</>) within
+ a compound SQL command, and <literal>\gset</> replaces a final
+ (<literal>;</>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</>, and fills variables
+ <replaceable>one</>, <replaceable>two</> and <replaceable>p_three</> with
+ integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \gcset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gcset</> and <literal>\gset</> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 78f1e6b..ac9289b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1225,6 +1228,105 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1957,7 +2059,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2295,26 +2396,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2944,22 +3031,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2979,17 +3054,79 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3003,19 +3140,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -3173,6 +3308,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "gcset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3196,6 +3338,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3219,6 +3364,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3228,31 +3374,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \gcset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3267,6 +3410,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "gcset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[1] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3274,6 +3478,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 38b3af5..00bf432 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 19b3e57..d5e7062 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -678,8 +678,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index b4044e8..85f57c5 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-03-27 10:49 Rafia Sabih <rafia.sabih@enterprisedb.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Rafia Sabih @ 2017-03-27 10:49 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; Pavel Stehule <pavel.stehule@gmail.com>; pgsql-hackers
On Fri, Mar 24, 2017 at 8:59 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> Hello Rafia,
>
>> if (my_command->argc > 2)
>> + syntax_error(source, lineno, my_command->line, my_command->argv[0],
>> + "at most on argument expected", NULL, -1);
>>
>> I suppose you mean 'one' argument here.
>
>
> Indeed.
>
>> Apart from that indentation is not correct as per pgindent, please check.
>
>
> I guess that you are refering to switch/case indentation which my emacs does
> not do as expected.
>
> Please find attached a v8 which hopefully fixes these two issues.
Looks good to me, marking as ready for committer.
--
Regards,
Rafia Sabih
EnterpriseDB: http://www.enterprisedb.com/
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-05 00:45 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Rafia Sabih <rafia.sabih@enterprisedb.com>
0 siblings, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-05 00:45 UTC (permalink / raw)
To: rafia.sabih@enterprisedb.com; +Cc: coelho@cri.ensmp.fr; michael.paquier@gmail.com; tgl@sss.pgh.pa.us; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
>> Please find attached a v8 which hopefully fixes these two issues.
> Looks good to me, marking as ready for committer.
I have looked into this a little bit.
It seems the new feature \gset doesn't work with tables having none
ascii column names:
$ src/bin/pgbench/pgbench -t 1 -f /tmp/f test
starting vacuum...end.
gset: invalid variable name: "$B?t;z(B"
client 0 file 0 command 0 compound 0: error storing into var $B?t;z(B
transaction type: /tmp/f
scaling factor: 1
query mode: simple
number of clients: 1
number of threads: 1
number of transactions per client: 1
number of transactions actually processed: 0/1
This is because pgbench variable names are limited to ascii
ranges. IMO the limitation is unnecessary and should be removed. (I
know that the limitation was brought in by someone long time ago and
the patch author is not responsible for that).
Anyway, now that the feature reveals the undocumented limitation, we
should document the limitation of \gset at least.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-05 18:28 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 2 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-04-05 18:28 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
Hello Tatsuo-san,
> It seems the new feature \gset doesn't work with tables having none
> ascii column names:
Indeed. The same error is triggered with the \set syntax, which does not
involve any query execution.
I have added a sentence mentionning the restriction when variables are
first discussed in the documentation, see attached patch.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-9.patch (16.6K, ../../alpine.DEB.2.20.1704052019100.1967@lancre/2-pgbench-into-9.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..c40f73e 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -773,6 +773,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
There is a simple variable-substitution facility for script files.
Variables can be set by the command-line <option>-D</> option,
explained above, or by the meta commands explained below.
+ Variable names are restricted to ASCII strings.
In addition to any variables preset by <option>-D</> command-line options,
there are a few variables that are preset automatically, listed in
<xref linkend="pgbench-automatic-variables">. A value specified for these
@@ -816,6 +817,51 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gcset [<replaceable>prefix</>]</literal> or
+ <literal>\gset [<replaceable>prefix</>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\gcset</> replaces an embedded semicolon (<literal>\;</>) within
+ a compound SQL command, and <literal>\gset</> replaces a final
+ (<literal>;</>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</>, and fills variables
+ <replaceable>one</>, <replaceable>two</> and <replaceable>p_three</> with
+ integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \gcset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gcset</> and <literal>\gset</> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 78f1e6b..ac9289b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1225,6 +1228,105 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = -1;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ compound += 1;
+
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ }
+
+ if (compound == -1)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1957,7 +2059,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2295,26 +2396,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2944,22 +3031,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2979,17 +3054,79 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3003,19 +3140,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -3173,6 +3308,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "gcset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3196,6 +3338,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3219,6 +3364,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3228,31 +3374,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \gcset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3267,6 +3410,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "gcset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[1] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3274,6 +3478,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 38b3af5..00bf432 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 27689d7..9974722 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -678,8 +678,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index af62f5e..3fe7789 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-06 00:07 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-06 00:07 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: ishii@sraoss.co.jp; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; tgl@sss.pgh.pa.us; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
Tom and others,
I still wonder whether I should commit this or not because this patch
does not allow none ascii column names. We know pgbench variable name
has been restricted since the functionality was born. When users
explicitly define a pgbench variable using \set, it is not a too
strong limitation, because it's in a "pgbench world" anyway and
nothing is related to PostgreSQL core specs. However, \gset is not
happy with perfectly valid column names in PostgreSQL core, which
looks too inconsistent and confusing for users.
So the choices are:
1) commit the patch now with documenting the limitation.
(the patch looks good to me except the issue above)
2) move it to next cf hoping that someone starts the implementation to
eliminate the limitation of none ascii variable names.
Comments?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
> Hello Tatsuo-san,
>
>> It seems the new feature \gset doesn't work with tables having none
>> ascii column names:
>
> Indeed. The same error is triggered with the \set syntax, which does
> not involve any query execution.
>
> I have added a sentence mentionning the restriction when variables are
> first discussed in the documentation, see attached patch.
>
> --
> Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-06 00:24 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 2 replies; 120+ messages in thread
From: Tom Lane @ 2017-04-06 00:24 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
> I still wonder whether I should commit this or not because this patch
> does not allow none ascii column names.
Well, personally, as an all-ASCII guy I'm not too fussed about that,
but I can see that other people might be annoyed.
The main problem in dealing with it seems to be whether you're willing
to support pgbench running in non-backend-safe encodings (eg SJIS).
If we rejected that case then it'd be a relatively simple change to allow
pgbench to treat any high-bit-set byte as a valid variable name character.
(I think anyway, haven't checked the code.)
Although ... actually, psql allows any high-bit-set byte in variable
names (cf valid_variable_name()) without concern about encoding.
That means it's formally incorrect in SJIS, but it's been like that
for an awful lot of years and nobody's complained. Maybe it'd be fine
for pgbench to act the same.
Having said all that, I think we're at the point in the commitfest
where if there's any design question at all about a patch, it should
get booted to the next cycle. We are going to have more than enough
to do to stabilize what's already committed, we don't need to be
adding more uncertainty.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-06 00:28 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 120+ messages in thread
From: Andres Freund @ 2017-04-06 00:28 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; coelho@cri.ensmp.fr; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
On 2017-04-05 20:24:19 -0400, Tom Lane wrote:
> Having said all that, I think we're at the point in the commitfest
> where if there's any design question at all about a patch, it should
> get booted to the next cycle. We are going to have more than enough
> to do to stabilize what's already committed, we don't need to be
> adding more uncertainty.
+1
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-06 00:36 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-06 00:36 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: ishii@sraoss.co.jp; coelho@cri.ensmp.fr; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
> Well, personally, as an all-ASCII guy I'm not too fussed about that,
> but I can see that other people might be annoyed.
>
> The main problem in dealing with it seems to be whether you're willing
> to support pgbench running in non-backend-safe encodings (eg SJIS).
> If we rejected that case then it'd be a relatively simple change to allow
> pgbench to treat any high-bit-set byte as a valid variable name character.
> (I think anyway, haven't checked the code.)
>
> Although ... actually, psql allows any high-bit-set byte in variable
> names (cf valid_variable_name()) without concern about encoding.
> That means it's formally incorrect in SJIS, but it's been like that
> for an awful lot of years and nobody's complained. Maybe it'd be fine
> for pgbench to act the same.
That's my thought too.
> Having said all that, I think we're at the point in the commitfest
> where if there's any design question at all about a patch, it should
> get booted to the next cycle. We are going to have more than enough
> to do to stabilize what's already committed, we don't need to be
> adding more uncertainty.
Ok, I will move the patch to the next cf.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-07 00:17 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-07 00:17 UTC (permalink / raw)
To: ishii@sraoss.co.jp; +Cc: tgl@sss.pgh.pa.us; coelho@cri.ensmp.fr; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
>> Having said all that, I think we're at the point in the commitfest
>> where if there's any design question at all about a patch, it should
>> get booted to the next cycle. We are going to have more than enough
>> to do to stabilize what's already committed, we don't need to be
>> adding more uncertainty.
>
> Ok, I will move the patch to the next cf.
Done.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-07 07:55 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-04-07 07:55 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
Hello Tatsuo,
>> Ok, I will move the patch to the next cf.
>
> Done.
If I understand correctly, the patch is moved because of the unrelated
issue that variables cannot be utf8 in pgbench, and it is a condition to
consider this patch that existing pgbench variables (set with \set) can be
utf8?
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-07 08:06 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-07 08:06 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: tgl@sss.pgh.pa.us; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
> If I understand correctly, the patch is moved because of the unrelated
> issue that variables cannot be utf8 in pgbench, and it is a condition
> to consider this patch that existing pgbench variables (set with \set)
> can be utf8?
I'm not sure if it is "unrelated" because the new feature relies on
existing pgbench variable infrastructure.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-07 09:43 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-04-07 09:43 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
>> If I understand correctly, the patch is moved because of the unrelated
>> issue that variables cannot be utf8 in pgbench, and it is a condition
>> to consider this patch that existing pgbench variables (set with \set)
>> can be utf8?
>
> I'm not sure if it is "unrelated" because the new feature relies on
> existing pgbench variable infrastructure.
Sure. I meant that the constraint on variable names exists before the
patch and the patch is not related to variable names, but the patch is
about variables, obviously.
As "psql" variables can be utf8 and that the same scanner is used, but the
variables values are not stritcly the same (they are typed in pgbench),
I'm wondering whether the effort should be do share more code/abstraction
between psql & pgbench or just adjust/replicate the needed small
functions/code excerpts.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-08 20:24 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-04-08 20:24 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
Hello Tatsuo-san,
>>> If I understand correctly, the patch is moved because of the unrelated
>>> issue that variables cannot be utf8 in pgbench, and it is a condition
>>> to consider this patch that existing pgbench variables (set with \set)
>>> can be utf8?
>>
>> I'm not sure if it is "unrelated" because the new feature relies on
>> existing pgbench variable infrastructure.
>
> Sure. I meant that the constraint on variable names exists before the patch
> and the patch is not related to variable names, but the patch is about
> variables, obviously.
>
> As "psql" variables can be utf8 and that the same scanner is used, but the
> variables values are not stritcly the same (they are typed in pgbench), I'm
> wondering whether the effort should be do share more code/abstraction between
> psql & pgbench or just adjust/replicate the needed small functions/code
> excerpts.
As the variable infrastructures are pretty different between psql &
pgbench (typed vs untyped values, sorted array vs linked list data
structure, no hook vs 2 hooks, name spaces vs no such thing...), I have
chosen the simplest option of just copying the name checking function and
extending the lexer to authorize non-ascii letters, so that psql/pgbench
would accept the same variable names with the same constraint about
encodings.
See patch attached & test script.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-sql] pgbench-utf8-vars-1.sql (2.8K, ../../alpine.DEB.2.20.1704090514480.26549@lancre/2-pgbench-utf8-vars-1.sql)
download
[application/x-sql] pgbench-utf8-vars.sql (101B, ../../alpine.DEB.2.20.1704090514480.26549@lancre/3-pgbench-utf8-vars.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-08 20:32 Fabien COELHO <fabien.coelho@mines-paristech.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-04-08 20:32 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
> As the variable infrastructures are pretty different between psql & pgbench
> (typed vs untyped values, sorted array vs linked list data structure, no hook
> vs 2 hooks, name spaces vs no such thing...), I have chosen the simplest
> option of just copying the name checking function and extending the lexer to
> authorize non-ascii letters, so that psql/pgbench would accept the same
> variable names with the same constraint about encodings.
>
> See patch attached & test script.
Argh, I'm jet-lagged, wrong patch suffix... Here it is with the right
suffix.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-utf8-vars-1.patch (2.8K, ../../alpine.DEB.2.20.1704090529570.26549@lancre/2-pgbench-utf8-vars-1.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..502d31b 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -825,6 +825,8 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<para>
Sets variable <replaceable>varname</> to a value calculated
from <replaceable>expression</>.
+ The variable name must consist of letters (including non-Latin letters),
+ digits, and underscores.
The expression may contain integer constants such as <literal>5432</>,
double constants such as <literal>3.14159</>,
references to variables <literal>:</><replaceable>variablename</>,
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index dc1367b..1862fe4 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -58,9 +58,9 @@ extern void expr_yyset_column(int column_no, yyscan_t yyscanner);
%option prefix="expr_yy"
/* Character classes */
-alpha [a-zA-Z_]
+alpha [a-zA-Z\200-\377_]
digit [0-9]
-alnum [a-zA-Z0-9_]
+alnum [A-Za-z\200-\377_0-9]
/* {space} + {nonspace} + {newline} should cover all characters */
space [ \t\r\f\v]
nonspace [^ \t\r\f\v\n]
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 78f1e6b..85f2edb 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1020,19 +1020,35 @@ makeVariableNumeric(Variable *var)
return true;
}
-/* check whether the name consists of alphabets, numerals and underscores. */
+/*
+ * Check whether a variable's name is allowed.
+ *
+ * We allow any non-ASCII character, as well as ASCII letters, digits, and
+ * underscore. Keep this in sync with the definition of variable_char in
+ * psqlscan.l and psqlscanslash.l.
+ *
+ * This static function is copied from "src/bin/psql/variables.c"
+ */
static bool
-isLegalVariableName(const char *name)
+valid_variable_name(const char *name)
{
- int i;
+ const unsigned char *ptr = (const unsigned char *) name;
- for (i = 0; name[i] != '\0'; i++)
+ /* Mustn't be zero-length */
+ if (*ptr == '\0')
+ return false;
+
+ while (*ptr)
{
- if (!isalnum((unsigned char) name[i]) && name[i] != '_')
+ if (IS_HIGHBIT_SET(*ptr) ||
+ strchr("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz"
+ "_0123456789", *ptr) != NULL)
+ ptr++;
+ else
return false;
}
- return (i > 0); /* must be non-empty */
+ return true;
}
/*
@@ -1054,7 +1070,7 @@ lookupCreateVariable(CState *st, const char *context, char *name)
* Check for the name only when declaring a new variable to avoid
* overhead.
*/
- if (!isLegalVariableName(name))
+ if (!valid_variable_name(name))
{
fprintf(stderr, "%s: invalid variable name: \"%s\"\n",
context, name);
[application/x-sql] pgbench-utf8-vars.sql (101B, ../../alpine.DEB.2.20.1704090529570.26549@lancre/3-pgbench-utf8-vars.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-08 20:37 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-04-08 20:37 UTC (permalink / raw)
To: pgsql-hackers
> As the variable infrastructures are pretty different between psql & pgbench
> (typed vs untyped values, sorted array vs linked list data structure, no hook
> vs 2 hooks, name spaces vs no such thing...), I have chosen the simplest
> option of just copying the name checking function and extending the lexer to
> authorize non-ascii letters, so that psql/pgbench would accept the same
> variable names with the same constraint about encodings.
>
> See patch attached & test script.
Argh, I should be asleep:-(
Wrong patch suffix, wrong from, multiple apology:-(
Here it is again.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-utf8-vars-1.patch (2.8K, ../../alpine.DEB.2.20.1704090534460.26549@lancre/2-pgbench-utf8-vars-1.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..502d31b 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -825,6 +825,8 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<para>
Sets variable <replaceable>varname</> to a value calculated
from <replaceable>expression</>.
+ The variable name must consist of letters (including non-Latin letters),
+ digits, and underscores.
The expression may contain integer constants such as <literal>5432</>,
double constants such as <literal>3.14159</>,
references to variables <literal>:</><replaceable>variablename</>,
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index dc1367b..1862fe4 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -58,9 +58,9 @@ extern void expr_yyset_column(int column_no, yyscan_t yyscanner);
%option prefix="expr_yy"
/* Character classes */
-alpha [a-zA-Z_]
+alpha [a-zA-Z\200-\377_]
digit [0-9]
-alnum [a-zA-Z0-9_]
+alnum [A-Za-z\200-\377_0-9]
/* {space} + {nonspace} + {newline} should cover all characters */
space [ \t\r\f\v]
nonspace [^ \t\r\f\v\n]
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 78f1e6b..85f2edb 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1020,19 +1020,35 @@ makeVariableNumeric(Variable *var)
return true;
}
-/* check whether the name consists of alphabets, numerals and underscores. */
+/*
+ * Check whether a variable's name is allowed.
+ *
+ * We allow any non-ASCII character, as well as ASCII letters, digits, and
+ * underscore. Keep this in sync with the definition of variable_char in
+ * psqlscan.l and psqlscanslash.l.
+ *
+ * This static function is copied from "src/bin/psql/variables.c"
+ */
static bool
-isLegalVariableName(const char *name)
+valid_variable_name(const char *name)
{
- int i;
+ const unsigned char *ptr = (const unsigned char *) name;
- for (i = 0; name[i] != '\0'; i++)
+ /* Mustn't be zero-length */
+ if (*ptr == '\0')
+ return false;
+
+ while (*ptr)
{
- if (!isalnum((unsigned char) name[i]) && name[i] != '_')
+ if (IS_HIGHBIT_SET(*ptr) ||
+ strchr("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz"
+ "_0123456789", *ptr) != NULL)
+ ptr++;
+ else
return false;
}
- return (i > 0); /* must be non-empty */
+ return true;
}
/*
@@ -1054,7 +1070,7 @@ lookupCreateVariable(CState *st, const char *context, char *name)
* Check for the name only when declaring a new variable to avoid
* overhead.
*/
- if (!isLegalVariableName(name))
+ if (!valid_variable_name(name))
{
fprintf(stderr, "%s: invalid variable name: \"%s\"\n",
context, name);
[application/x-sql] pgbench-utf8-vars.sql (101B, ../../alpine.DEB.2.20.1704090534460.26549@lancre/3-pgbench-utf8-vars.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-09 23:32 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Fabien COELHO <fabien.coelho@mines-paristech.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-09 23:32 UTC (permalink / raw)
To: fabien.coelho@mines-paristech.fr; +Cc: tgl@sss.pgh.pa.us; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
Fabien,
>> As the variable infrastructures are pretty different between psql &
>> pgbench (typed vs untyped values, sorted array vs linked list data
>> structure, no hook vs 2 hooks, name spaces vs no such thing...), I
>> have chosen the simplest option of just copying the name checking
>> function and extending the lexer to authorize non-ascii letters, so
>> that psql/pgbench would accept the same variable names with the same
>> constraint about encodings.
>>
>> See patch attached & test script.
>
> Argh, I'm jet-lagged, wrong patch suffix... Here it is with the right
> suffix.
Thank you for the patch. I tested a little bit and found that it does
not allow value replacement against non ascii variables in given SQL
statements . Is it intentional? If not, I think you need to fix
parseVariable() as well.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-17 06:34 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-04-17 06:34 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
Hello Tatsuo-san,
> Thank you for the patch. I tested a little bit and found that it does
> not allow value replacement against non ascii variables in given SQL
> statements . Is it intentional?
No, this is a bug.
> If not, I think you need to fix parseVariable() as well.
Indeed. Here is v2.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-utf8-vars-2.patch (3.1K, ../../alpine.DEB.2.20.1704171524050.4025@lancre/2-pgbench-utf8-vars-2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..502d31b 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -825,6 +825,8 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<para>
Sets variable <replaceable>varname</> to a value calculated
from <replaceable>expression</>.
+ The variable name must consist of letters (including non-Latin letters),
+ digits, and underscores.
The expression may contain integer constants such as <literal>5432</>,
double constants such as <literal>3.14159</>,
references to variables <literal>:</><replaceable>variablename</>,
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index dc1367b..1862fe4 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -58,9 +58,9 @@ extern void expr_yyset_column(int column_no, yyscan_t yyscanner);
%option prefix="expr_yy"
/* Character classes */
-alpha [a-zA-Z_]
+alpha [a-zA-Z\200-\377_]
digit [0-9]
-alnum [a-zA-Z0-9_]
+alnum [A-Za-z\200-\377_0-9]
/* {space} + {nonspace} + {newline} should cover all characters */
space [ \t\r\f\v]
nonspace [^ \t\r\f\v\n]
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae36247..f49e8c6 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1020,19 +1020,35 @@ makeVariableNumeric(Variable *var)
return true;
}
-/* check whether the name consists of alphabets, numerals and underscores. */
+/*
+ * Check whether a variable's name is allowed.
+ *
+ * We allow any non-ASCII character, as well as ASCII letters, digits, and
+ * underscore. Keep this in sync with the definition of variable_char in
+ * psqlscan.l and psqlscanslash.l.
+ *
+ * This static function is copied from "src/bin/psql/variables.c"
+ */
static bool
-isLegalVariableName(const char *name)
+valid_variable_name(const char *name)
{
- int i;
+ const unsigned char *ptr = (const unsigned char *) name;
- for (i = 0; name[i] != '\0'; i++)
+ /* Mustn't be zero-length */
+ if (*ptr == '\0')
+ return false;
+
+ while (*ptr)
{
- if (!isalnum((unsigned char) name[i]) && name[i] != '_')
+ if (IS_HIGHBIT_SET(*ptr) ||
+ strchr("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz"
+ "_0123456789", *ptr) != NULL)
+ ptr++;
+ else
return false;
}
- return (i > 0); /* must be non-empty */
+ return true;
}
/*
@@ -1054,7 +1070,7 @@ lookupCreateVariable(CState *st, const char *context, char *name)
* Check for the name only when declaring a new variable to avoid
* overhead.
*/
- if (!isLegalVariableName(name))
+ if (!valid_variable_name(name))
{
fprintf(stderr, "%s: invalid variable name: \"%s\"\n",
context, name);
@@ -1148,7 +1164,7 @@ parseVariable(const char *sql, int *eaten)
do
{
i++;
- } while (isalnum((unsigned char) sql[i]) || sql[i] == '_');
+ } while (IS_HIGHBIT_SET(sql[i]) || isalnum((unsigned char) sql[i]) || sql[i] == '_');
if (i == 1)
return NULL;
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-17 06:40 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
1 sibling, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-04-17 06:40 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
>> It seems the new feature \gset doesn't work with tables having none
>> ascii column names:
>
> Indeed. The same error is triggered with the \set syntax, which does not
> involve any query execution.
>
> I have added a sentence mentionning the restriction when variables are first
> discussed in the documentation, see attached patch.
Here is a v10:
- does not talk about ASCII variable name constraint, as a patch has been
submitted independently to lift this constraint.
- rename gcset to cset (compound set, \; + \set), where gset is ; + \set,
because "\gcset" looked really strange.
- simplify the code a little bit.
Also attached is an updated test script.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-10.patch (16.0K, ../../alpine.DEB.2.20.1704171535210.4025@lancre/2-pgbench-into-10.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..862e1d4 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -816,6 +816,51 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</>]</literal> or
+ <literal>\gset [<replaceable>prefix</>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</> replaces an embedded semicolon (<literal>\;</>) within
+ a compound SQL command, and <literal>\gset</> replaces a final
+ (<literal>;</>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</>, and fills variables
+ <replaceable>one</>, <replaceable>two</> and <replaceable>p_three</> with
+ integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</> and <literal>\gset</> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae36247..1e2402a 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1225,6 +1228,104 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1957,7 +2058,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2295,26 +2395,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2944,22 +3030,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2979,17 +3053,79 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
+
/*
* If SQL command is multi-line, we only want to save the first line as
* the "line" label.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3003,19 +3139,17 @@ process_sql_command(PQExpBuffer buf, const char *source)
switch (querymode)
{
case QUERY_SIMPLE:
- my_command->argv[0] = pg_strdup(p);
+ my_command->argv[0] = my_command->lines;
my_command->argc++;
break;
case QUERY_EXTENDED:
case QUERY_PREPARED:
- if (!parseQuery(my_command, p))
+ if (!parseQuery(my_command, my_command->lines))
exit(1);
break;
default:
exit(1);
}
-
- return my_command;
}
/*
@@ -3173,6 +3307,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3196,6 +3337,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3219,6 +3363,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3228,31 +3373,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3267,6 +3409,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3274,6 +3477,11 @@ ParseScript(const char *script, const char *desc, int weight)
ps.commands[index] = NULL;
+ /* complete SQL command initializations */
+ while (--index >= 0)
+ if (ps.commands[index]->type == SQL_COMMAND)
+ postprocess_sql_command(ps.commands[index]);
+
addScript(ps);
termPQExpBuffer(&line_buf);
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 38b3af5..00bf432 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 27689d7..9974722 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -678,8 +678,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index af62f5e..3fe7789 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
[application/x-sql] gset-1.sql (468B, ../../alpine.DEB.2.20.1704171535210.4025@lancre/3-gset-1.sql)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-17 09:11 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-17 09:11 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: tgl@sss.pgh.pa.us; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
Fabien,
> Hello Tatsuo-san,
>
>> Thank you for the patch. I tested a little bit and found that it does
>> not allow value replacement against non ascii variables in given SQL
>> statements . Is it intentional?
>
> No, this is a bug.
>
>> If not, I think you need to fix parseVariable() as well.
>
> Indeed. Here is v2.
I think you'd better to change the following comments because there's
no psqlscan.l or psqlscanslash.l in pgbench source tree.
+ * underscore. Keep this in sync with the definition of variable_char in
+ * psqlscan.l and psqlscanslash.l.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-17 11:00 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-04-17 11:00 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
Hello,
> I think you'd better to change the following comments because there's
> no psqlscan.l or psqlscanslash.l in pgbench source tree.
>
> + * underscore. Keep this in sync with the definition of variable_char in
> + * psqlscan.l and psqlscanslash.l.
Here is a v3 with a more precise comment.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-utf8-vars-3.patch (3.2K, ../../alpine.DEB.2.20.1704171959080.1741@lancre/2-pgbench-utf8-vars-3.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 1eee8dc..502d31b 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -825,6 +825,8 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
<para>
Sets variable <replaceable>varname</> to a value calculated
from <replaceable>expression</>.
+ The variable name must consist of letters (including non-Latin letters),
+ digits, and underscores.
The expression may contain integer constants such as <literal>5432</>,
double constants such as <literal>3.14159</>,
references to variables <literal>:</><replaceable>variablename</>,
diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index dc1367b..1862fe4 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -58,9 +58,9 @@ extern void expr_yyset_column(int column_no, yyscan_t yyscanner);
%option prefix="expr_yy"
/* Character classes */
-alpha [a-zA-Z_]
+alpha [a-zA-Z\200-\377_]
digit [0-9]
-alnum [a-zA-Z0-9_]
+alnum [A-Za-z\200-\377_0-9]
/* {space} + {nonspace} + {newline} should cover all characters */
space [ \t\r\f\v]
nonspace [^ \t\r\f\v\n]
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae36247..44ae29e 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1020,19 +1020,38 @@ makeVariableNumeric(Variable *var)
return true;
}
-/* check whether the name consists of alphabets, numerals and underscores. */
+/*
+ * Check whether a variable's name is allowed.
+ *
+ * We allow any non-ASCII character, as well as ASCII letters, digits, and
+ * underscore.
+ *
+ * Keep this in sync with the definitions of variable name characters in
+ * "src/fe_utils/psqlscan.l", "src/bin/psql/psqlscanslash.l" and
+ * "src/bin/pgbench/exprscan.l".
+ *
+ * Note: this static function is copied from "src/bin/psql/variables.c"
+ */
static bool
-isLegalVariableName(const char *name)
+valid_variable_name(const char *name)
{
- int i;
+ const unsigned char *ptr = (const unsigned char *) name;
- for (i = 0; name[i] != '\0'; i++)
+ /* Mustn't be zero-length */
+ if (*ptr == '\0')
+ return false;
+
+ while (*ptr)
{
- if (!isalnum((unsigned char) name[i]) && name[i] != '_')
+ if (IS_HIGHBIT_SET(*ptr) ||
+ strchr("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz"
+ "_0123456789", *ptr) != NULL)
+ ptr++;
+ else
return false;
}
- return (i > 0); /* must be non-empty */
+ return true;
}
/*
@@ -1054,7 +1073,7 @@ lookupCreateVariable(CState *st, const char *context, char *name)
* Check for the name only when declaring a new variable to avoid
* overhead.
*/
- if (!isLegalVariableName(name))
+ if (!valid_variable_name(name))
{
fprintf(stderr, "%s: invalid variable name: \"%s\"\n",
context, name);
@@ -1148,7 +1167,7 @@ parseVariable(const char *sql, int *eaten)
do
{
i++;
- } while (isalnum((unsigned char) sql[i]) || sql[i] == '_');
+ } while (IS_HIGHBIT_SET(sql[i]) || isalnum((unsigned char) sql[i]) || sql[i] == '_');
if (i == 1)
return NULL;
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-20 08:06 Tatsuo Ishii <ishii@sraoss.co.jp>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 120+ messages in thread
From: Tatsuo Ishii @ 2017-04-20 08:06 UTC (permalink / raw)
To: coelho@cri.ensmp.fr; +Cc: tgl@sss.pgh.pa.us; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
> Hello,
>
>> I think you'd better to change the following comments because there's
>> no psqlscan.l or psqlscanslash.l in pgbench source tree.
>>
>> + * underscore. Keep this in sync with the definition of
>> variable_char in
>> + * psqlscan.l and psqlscanslash.l.
>
> Here is a v3 with a more precise comment.
Looks good to me. I have marked the patch status as "Ready for
committer".
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-04-22 04:58 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-04-22 04:58 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
>> Here is a v3 with a more precise comment.
>
> Looks good to me. I have marked the patch status as "Ready for
> committer".
Ok. Thanks. When/if committed, it might trigger a few rebase of other
pending patches. I'll see about that then.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-08-13 18:33 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-08-13 18:33 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: pgsql-hackers
Here is a v11.
It is basically a simple rebase after Tom committed the "pgbench -M order"
patch. It interfered because the compound command management also needs
to delay part of the SQL command initialization. Some patch are luckier
than others:-)
> Here is a v10:
>
> - does not talk about ASCII variable name constraint, as a patch has been
> submitted independently to lift this constraint.
>
> - rename gcset to cset (compound set, \; + \set), where gset is ; + \set,
> because "\gcset" looked really strange.
>
> - simplify the code a little bit.
>
> Also attached is an updated test script.
>
>
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-11.patch (17.7K, ../../alpine.DEB.2.20.1708132029250.15606@lancre/2-pgbench-into-11.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 03e1212..ea154bc 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -816,6 +816,51 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</>]</literal> or
+ <literal>\gset [<replaceable>prefix</>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</> replaces an embedded semicolon (<literal>\;</>) within
+ a compound SQL command, and <literal>\gset</> replaces a final
+ (<literal>;</>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</>, and fills variables
+ <replaceable>one</>, <replaceable>two</> and <replaceable>p_three</> with
+ integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</> and <literal>\gset</> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index ae78c7b..b1488c0 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1225,6 +1228,104 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1957,7 +2058,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2295,26 +2395,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2853,8 +2939,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -2878,7 +2963,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -2890,7 +2975,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -2949,22 +3034,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -2984,23 +3057,83 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append more text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3011,7 +3144,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -3169,6 +3316,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3192,6 +3346,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3215,6 +3372,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3224,31 +3382,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3263,6 +3418,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -3997,28 +4213,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index abc13e9..5e2e3fd 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 27689d7..9974722 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -678,8 +678,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index c70ff29..e26eac8 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-08-13 19:45 Pavel Stehule <pavel.stehule@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Pavel Stehule @ 2017-08-13 19:45 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hi
2017-08-13 20:33 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:
>
> Here is a v11.
>
> It is basically a simple rebase after Tom committed the "pgbench -M order"
> patch. It interfered because the compound command management also needs
> to delay part of the SQL command initialization. Some patch are luckier
> than others:-)
>
> Here is a v10:
>>
>> - does not talk about ASCII variable name constraint, as a patch has been
>> submitted independently to lift this constraint.
>>
>> - rename gcset to cset (compound set, \; + \set), where gset is ; + \set,
>> because "\gcset" looked really strange.
>>
>> - simplify the code a little bit.
>>
>> Also attached is an updated test script.
>>
>>
looks little bit strange, but it has sense
+1
Pavel
>>
> --
> Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-09-04 17:46 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tatsuo Ishii <ishii@sraoss.co.jp>
1 sibling, 1 reply; 120+ messages in thread
From: Tom Lane @ 2017-09-04 17:46 UTC (permalink / raw)
To: Tatsuo Ishii <ishii@sraoss.co.jp>; +Cc: coelho@cri.ensmp.fr; rafia.sabih@enterprisedb.com; michael.paquier@gmail.com; Langote_Amit_f8@lab.ntt.co.jp; robertmhaas@gmail.com; pavel.stehule@gmail.com; pgsql-hackers
Tatsuo Ishii <ishii@sraoss.co.jp> writes:
>> Here is a v3 with a more precise comment.
> Looks good to me. I have marked the patch status as "Ready for
> committer".
LGTM too. Pushed with a minor adjustment to make parseVariable()
have exactly the same test as valid_variable_name().
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-09-04 19:12 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-09-04 19:12 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; pgsql-hackers
>> Looks good to me. I have marked the patch status as "Ready for
>> committer".
>
> LGTM too. Pushed with a minor adjustment to make parseVariable()
> have exactly the same test as valid_variable_name().
\set ありがとうございました 1
\set 谢谢 2
\set dankeschön 3
:-)
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-09-08 17:17 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Pavel Stehule <pavel.stehule@gmail.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-09-08 17:17 UTC (permalink / raw)
To: Pavel Stehule <pavel.stehule@gmail.com>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Here is a v12.
There is no changes in the code or documentation, only TAP tests are
added.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-12.patch (20.8K, ../../alpine.DEB.2.20.1709081916410.3568@lancre/2-pgbench-into-12.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index f5db8d1..9ad82d4 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -818,6 +818,51 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</>
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</>]</literal> or
+ <literal>\gset [<replaceable>prefix</>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</> replaces an embedded semicolon (<literal>\;</>) within
+ a compound SQL command, and <literal>\gset</> replaces a final
+ (<literal>;</>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</>, and fills variables
+ <replaceable>one</>, <replaceable>two</> and <replaceable>p_three</> with
+ integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</> and <literal>\gset</> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</> <replaceable>expression</></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index e37496c..454127c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1254,6 +1257,104 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1986,7 +2087,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2332,26 +2432,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2892,8 +2978,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -2917,7 +3002,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -2929,7 +3014,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -2988,22 +3073,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3023,23 +3096,83 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append more text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3050,7 +3183,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -3203,6 +3350,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3226,6 +3380,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3249,6 +3406,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3258,31 +3416,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3297,6 +3452,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4030,28 +4246,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index fd428af..656f214 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 66df4bc..26514d1 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -280,6 +280,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -388,15 +430,43 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
# MISC
[ 'misc invalid backslash command', 1,
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
- [ 'misc empty script', 1, [qr{empty command list for script}], q{} ],);
+ [ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
+
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],);
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -M prepared',
+ '-n -t 1 -Dfoo=bla ' . ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 27689d7..9974722 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -678,8 +678,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index c70ff29..e26eac8 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-10-20 16:37 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2017-10-20 16:37 UTC (permalink / raw)
To: Pavel Stehule <pavel.stehule@gmail.com>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
> Here is a v12.
Here is a v13, which is just a rebase after the documentation xml-ization.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-13.patch (21.0K, ../../alpine.DEB.2.20.1710201836530.15170@lancre/2-pgbench-into-13.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index e509e6c..44e8896 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -818,6 +818,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</replaceable> <replaceable>expression</replaceable></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 5d8a01c..37ed07b 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -375,11 +375,14 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1254,6 +1257,104 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -1986,7 +2087,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2328,26 +2428,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2888,8 +2974,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -2913,7 +2998,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -2925,7 +3010,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -2984,22 +3069,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3019,23 +3092,83 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append more text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3046,7 +3179,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -3199,6 +3346,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
syntax_error(source, lineno, my_command->line, my_command->argv[0],
@@ -3222,6 +3376,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3245,6 +3402,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3254,31 +3412,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3293,6 +3448,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4026,28 +4242,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index fd428af..656f214 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 11bc0fe..8cb7971 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -275,6 +275,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -383,15 +425,43 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
# MISC
[ 'misc invalid backslash command', 1,
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
- [ 'misc empty script', 1, [qr{empty command list for script}], q{} ],);
+ [ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
+
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],);
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -M prepared',
+ '-n -t 1 -Dfoo=bla ' . ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 44fcf7e..9374d32 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index e9b3517..435b094 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-10-25 09:55 Pavel Stehule <pavel.stehule@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Pavel Stehule @ 2017-10-25 09:55 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hi
2017-10-20 18:37 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:
>
> Here is a v12.
>>
>
> Here is a v13, which is just a rebase after the documentation xml-ization.
>
I am looking to this patch.
Not sure if "cset" is best name - maybe "eset" .. like embeded set?
The code of append_sql_command is not too readable and is not enough
commented.
I don't understand why you pass a param compounds to append_sql_command,
when this value is stored in my_command->compound from create_sql_command?
Or maybe some unhappy field or variable names was chosen.
Regards
Pavel
>
> --
> Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: pgbench - allow to store select results into variables
@ 2017-11-04 11:05 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Pavel Stehule <pavel.stehule@gmail.com>
0 siblings, 3 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-11-04 11:05 UTC (permalink / raw)
To: Pavel Stehule <pavel.stehule@gmail.com>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hello Pavel,
>> Here is a v13, which is just a rebase after the documentation xml-ization.
Here is a v14, after yet another rebase, and some comments added to answer
your new comments.
> I am looking to this patch.
>
> Not sure if "cset" is best name - maybe "eset" .. like embeded set?
I used c for "compound", because they compound SQL commands.
Now I do not have a very strong opinion, only that it should be some
letter which some logic followed by "set".
The variables and fields in the source currently use "compound"
everywhere, if this is changed they should be updated accordingly.
ISTM that the ";" is embedded, but the commands are compound, so
"compound" seems better word to me. However it is debatable.
If there some standard naming for the concept, it should be used.
> The code of append_sql_command is not too readable and is not enough
> commented.
Ok. I have added comments in the code.
> I don't understand why you pass a param compounds to append_sql_command,
> when this value is stored in my_command->compound from create_sql_command?
This is the number of compound commands in the "more" string. It must be
updated as well as the command text, so that the my_command text and
number of compounds is consistant.
Think of one initialization followed by two appends:
SELECT 1 AS x \cset
SELECT 2 \; SELECT 3 AS y \cset
SELECT 4 \; SELECT 5 \; SELECT 6 AS z \gset
In the end, we must have the full 6 queries
"SELECT 1 AS x \; SELECT 2 \; SELECT 3 AS y \; SELECT 4 \; SELECT 5 \; SELECT 6 AS z"
and know that we want to set variables from queries 1, 3 and 6 and ignore
the 3 others.
> Or maybe some unhappy field or variable names was chosen.
It seems ok to me. What would your suggest?
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] pgbench-into-14.patch (21.2K, ../../alpine.DEB.2.20.1710251209330.24229@lancre/2-pgbench-into-14.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index e509e6c..44e8896 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -818,6 +818,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</replaceable> <replaceable>expression</replaceable></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index d4a6035..32262df 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -384,12 +384,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1264,6 +1267,104 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2019,7 +2120,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2361,26 +2461,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -2921,8 +3007,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -2946,7 +3031,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -2958,7 +3043,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3017,22 +3102,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3052,24 +3125,86 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, const char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3080,7 +3215,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -3236,6 +3385,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
@@ -3260,6 +3416,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3283,6 +3442,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3292,31 +3452,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -3331,6 +3488,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4064,28 +4282,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index fd428af..656f214 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 11bc0fe..8cb7971 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -275,6 +275,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -383,15 +425,43 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
# MISC
[ 'misc invalid backslash command', 1,
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
- [ 'misc empty script', 1, [qr{empty command list for script}], q{} ],);
+ [ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
+
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],);
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -M prepared',
+ '-n -t 1 -Dfoo=bla ' . ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 44fcf7e..9374d32 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index e9b3517..435b094 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2017-12-01 02:57 Michael Paquier <michael.paquier@gmail.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
2 siblings, 1 reply; 120+ messages in thread
From: Michael Paquier @ 2017-12-01 02:57 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On Sat, Nov 4, 2017 at 8:05 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>>> Here is a v13, which is just a rebase after the documentation
>>> xml-ization.
>
> Here is a v14, after yet another rebase, and some comments added to answer
> your new comments.
The last patch sent still applies, but its status of "ready for
committer" does not look adapted as this version got no reviews. So I
am switching back the patch as "needs review". Feel free to change if
that's not adapted, The patch is moved as well to next CF.
--
Michael
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2017-12-01 05:48 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Michael Paquier <michael.paquier@gmail.com>
0 siblings, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2017-12-01 05:48 UTC (permalink / raw)
To: Michael Paquier <michael.paquier@gmail.com>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hello Michaël,
>> Here is a v14, after yet another rebase, and some comments added to answer
>> your new comments.
>
> The last patch sent still applies, but its status of "ready for
> committer" does not look adapted as this version got no reviews. So I
> am switching back the patch as "needs review". Feel free to change if
> that's not adapted,
Indeed, I think that is not needed, as the edit was beyond mundane
(comments lightly edited, one const added after a wind of const got
committed, patch context diff changes probably after a pgindent run that
triggered the need of a rebase), which is why I did not updated the status
for this version.
So I put it back to "ready".
> The patch is moved as well to next CF.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-01-10 08:54 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
2 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-01-10 08:54 UTC (permalink / raw)
To: Pavel Stehule <pavel.stehule@gmail.com>; +Cc: Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
> Here is a v14, after yet another rebase, and some comments added to answer
> your new comments.
Attached v15 is a simple rebase after Teodor push of new functions &
operators in pgbench.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-15.patch (21.2K, ../../alpine.DEB.2.20.1801100952040.2283@lancre/2-pgbench-into-15.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 3dd492c..b51399e 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -895,6 +895,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry id='pgbench-metacommand-set'>
<term>
<literal>\set <replaceable>varname</replaceable> <replaceable>expression</replaceable></literal>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 31ea6ca..e6daa9f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -416,12 +416,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1523,6 +1526,104 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2565,7 +2666,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -2907,26 +3007,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3592,8 +3678,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3617,7 +3702,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3629,7 +3714,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3688,22 +3773,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3723,24 +3796,86 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3751,7 +3886,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -3907,6 +4056,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"missing command", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
@@ -3931,6 +4087,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -3954,6 +4113,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -3963,31 +4123,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4002,6 +4159,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4759,28 +4977,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 0705ccd..6dcd2f4 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index e579334..4c518bf 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -371,6 +371,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -523,16 +565,44 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0, [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
- );
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-03-24 09:25 Fabien COELHO <fabien.coelho@mines-paristech.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 120+ messages in thread
From: Fabien COELHO @ 2018-03-24 09:25 UTC (permalink / raw)
To: pgsql-hackers
>> Here is a v14, after yet another rebase, and some comments added to answer
>> your new comments.
>
> Attached v15 is a simple rebase after Teodor push of new functions &
> operators in pgbench.
Patch v16 is a rebase.
--
Fabien.
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index d52d324..203b6bc 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -900,6 +900,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 894571e..4a8595f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -434,12 +434,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1591,6 +1594,104 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2666,7 +2767,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3142,26 +3242,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3824,8 +3910,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3849,7 +3934,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3861,7 +3946,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3920,22 +4005,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3955,24 +4028,86 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3983,7 +4118,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4151,6 +4300,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
@@ -4231,6 +4387,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4254,6 +4413,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4263,31 +4423,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4302,6 +4459,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -5061,28 +5279,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865..c349477 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 7448a96..0b39dee 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -417,6 +417,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -569,16 +611,44 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0, [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
- );
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
Attachments:
[text/plain] pgbench-into-16.patch (21.2K, ../../alpine.DEB.2.20.1803241024500.13537@lancre/2-pgbench-into-16.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index d52d324..203b6bc 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -900,6 +900,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 894571e..4a8595f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -434,12 +434,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1591,6 +1594,104 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2666,7 +2767,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3142,26 +3242,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read and discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3824,8 +3910,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3849,7 +3934,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3861,7 +3946,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3920,22 +4005,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3955,24 +4028,86 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
@@ -3983,7 +4118,21 @@ process_sql_command(PQExpBuffer buf, const char *source)
else
my_command->line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4151,6 +4300,13 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
@@ -4231,6 +4387,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4254,6 +4413,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4263,31 +4423,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4302,6 +4459,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset must follow a SQL command",
+ sql_cmd->line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -5061,28 +5279,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865..c349477 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 7448a96..0b39dee 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -417,6 +417,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -569,16 +611,44 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0, [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
- );
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-04-06 20:55 Stephen Frost <sfrost@snowman.net>
parent: Fabien COELHO <fabien.coelho@mines-paristech.fr>
1 sibling, 0 replies; 120+ messages in thread
From: Stephen Frost @ 2018-04-06 20:55 UTC (permalink / raw)
To: Fabien COELHO <fabien.coelho@mines-paristech.fr>; +Cc: pgsql-hackers
Greetings Fabien,
* Fabien COELHO (fabien.coelho@mines-paristech.fr) wrote:
> >>Here is a v14, after yet another rebase, and some comments added to
> >>answer your new comments.
> >
> >Attached v15 is a simple rebase after Teodor push of new functions &
> >operators in pgbench.
>
> Patch v16 is a rebase.
Thank you for the diligent efforts to keep this patch moving forward, I
know it's been a long time coming but it's also been through a number of
reviews and improvements.
With the UTF bits addressed previously, the rest of this patch doesn't
strike me as overly controversial. I'm working through reviewing it and
barring any big issues I'll see if I can get it committed before feature
freeze, yet again, ends up swallowing it.
I expect to get the review done tonight, at which point I'll either
announce plans to commit it sometime tomorrow, or not, if there's issues
still to be resovled. :) As such, taking it in the CF app as committer.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (818B, ../../20180406205526.GT27724@tamriel.snowman.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-04-07 03:17 Stephen Frost <sfrost@snowman.net>
parent: Fabien COELHO <fabien.coelho@mines-paristech.fr>
1 sibling, 1 reply; 120+ messages in thread
From: Stephen Frost @ 2018-04-07 03:17 UTC (permalink / raw)
To: Fabien COELHO <fabien.coelho@mines-paristech.fr>; +Cc: pgsql-hackers
Fabien,
* Fabien COELHO (fabien.coelho@mines-paristech.fr) wrote:
> Patch v16 is a rebase.
Here's that review.
> diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
> index d52d324..203b6bc 100644
> --- a/doc/src/sgml/ref/pgbench.sgml
> +++ b/doc/src/sgml/ref/pgbench.sgml
> @@ -900,6 +900,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
> </para>
>
> <variablelist>
> + <varlistentry id='pgbench-metacommand-gset'>
> + <term>
> + <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
> + <literal>\gset [<replaceable>prefix</replaceable>]</literal>
> + </term>
Seems like this should really be moved down below the section for
'\set'.
> diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
> index 894571e..4a8595f 100644
> --- a/src/bin/pgbench/pgbench.c
> +++ b/src/bin/pgbench/pgbench.c
> @@ -434,12 +434,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
>
> typedef struct
> {
> - char *line; /* text of command line */
> + char *line; /* first line for short display */
> + char *lines; /* full multi-line text of command */
Not really a fan of such closely-named variables... Maybe first_line
instead?
> +/* read all responses from backend */
> +static bool
> +read_response(CState *st, char **gset)
> +{
> + PGresult *res;
> + int compound = 0;
> +
> + while ((res = PQgetResult(st->con)) != NULL)
> + {
> + switch (PQresultStatus(res))
> + {
> + case PGRES_COMMAND_OK: /* non-SELECT commands */
> + case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
> + if (gset[compound] != NULL)
> + {
> + fprintf(stderr,
> + "client %d file %d command %d compound %d: "
> + "\\gset expects a row\n",
> + st->id, st->use_file, st->command, compound);
> + st->ecnt++;
> + return false;
> + }
> + break; /* OK */
> +
> + case PGRES_TUPLES_OK:
> + if (gset[compound] != NULL)
Probably be good to add some comments here, eh:
/*
* The results are intentionally thrown away if we aren't under a gset
* call.
*/
> + {
> + /* store result into variables */
> + int ntuples = PQntuples(res),
> + nfields = PQnfields(res),
> + f;
> +
> + if (ntuples != 1)
> + {
> + fprintf(stderr,
> + "client %d file %d command %d compound %d: "
> + "expecting one row, got %d\n",
> + st->id, st->use_file, st->command, compound, ntuples);
> + st->ecnt++;
> + PQclear(res);
> + discard_response(st);
> + return false;
> + }
Might be interesting to support mutli-row (or no row?) returns in the
future, but not something this patch needs to do, so this looks fine to
me.
> + for (f = 0; f < nfields ; f++)
> + {
> + char *varname = PQfname(res, f);
> + if (*gset[compound] != '\0')
> + varname = psprintf("%s%s", gset[compound], varname);
> +
> + /* store result as a string */
> + if (!putVariable(st, "gset", varname,
> + PQgetvalue(res, 0, f)))
> + {
> + /* internal error, should it rather abort? */
> + fprintf(stderr,
> + "client %d file %d command %d compound %d: "
> + "error storing into var %s\n",
> + st->id, st->use_file, st->command, compound,
> + varname);
> + st->ecnt++;
> + PQclear(res);
> + discard_response(st);
> + return false;
> + }
I'm a bit on the fence about if we should abort in this case or not. A
failure here seems likely to happen consistently (whereas the ntuples
case might be a fluke of some kind), which tends to make me think we
should abort, but still thinking about it.
> + if (*gset[compound] != '\0')
> + free(varname);
A comment here, and above where we're assigning the result of the
psprintf(), to varname probably wouldn't hurt, explaining that the
variable is sometimes pointing into the query result structure and
sometimes not...
Thinking about it a bit more, wouldn't it be cleaner to just always use
psprintf()? eg:
char *varname;
varname = psprintf("%s%s", gset[compound] != '\0' ? gset[compound] : "", varname);
...
free(varname);
> + /* read and discard the query results */
That comment doesn't feel quite right now. ;)
> @@ -3824,8 +3910,7 @@ parseQuery(Command *cmd)
> char *sql,
> *p;
>
> - /* We don't want to scribble on cmd->argv[0] until done */
> - sql = pg_strdup(cmd->argv[0]);
> + sql = pg_strdup(cmd->lines);
The function-header comment for parseQuery() could really stand to be
improved.
> + /* merge gset variants into preceeding SQL command */
> + if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
> + pg_strcasecmp(bs_cmd, "cset") == 0)
> + {
> + int cindex;
> + Command *sql_cmd;
> +
> + is_compound = bs_cmd[0] == 'c';
> +
> + if (index == 0)
> + syntax_error(desc, lineno, NULL, NULL,
> + "\\gset cannot start a script",
> + NULL, -1);
> +
> + sql_cmd = ps.commands[index-1];
> +
> + if (sql_cmd->type != SQL_COMMAND)
> + syntax_error(desc, lineno, NULL, NULL,
> + "\\gset must follow a SQL command",
> + sql_cmd->line, -1);
> +
> + /* this \gset applies to the last sub-command */
> + cindex = sql_cmd->compound;
> +
> + if (sql_cmd->gset[cindex] != NULL)
> + syntax_error(desc, lineno, NULL, NULL,
> + "\\gset cannot follow one another",
> + NULL, -1);
> +
> + /* get variable prefix */
> + if (command->argc <= 1 || command->argv[1][0] == '\0')
> + sql_cmd->gset[cindex] = "";
> + else
> + sql_cmd->gset[cindex] = command->argv[1];
> +
> + /* cleanup unused backslash command */
> + pg_free(command);
These errors should probably be '\\gset and \\cset' or similar, no?
Since we fall into this for both.. Probably not a big deal to someone
using pgbench, but still.
So, overall, looks pretty good to me. There's definitely some cleanup
work to be done with variable names and comments and such, but nothing
too terrible and I should have time to go through those changes and then
go back over the patch again tomorrow with an eye towards committing it
tomorrow afternoon, barring objections, etc.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (818B, ../../20180407031725.GX27724@tamriel.snowman.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-04-07 16:06 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-04-07 16:06 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-hackers
Hello Stephen,
> Here's that review.
Thanks for the review.
>> <variablelist>
>> + <varlistentry id='pgbench-metacommand-gset'>
>> + <term>
>> + <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
>> + <literal>\gset [<replaceable>prefix</replaceable>]</literal>
>> + </term>
>
> Seems like this should really be moved down below the section for
> '\set'.
Dunno.
I put them there because it is in alphabetical order (for cset at least)
and because "set" documentation is heavy about expressions (operators,
functions, constants, ...) which have nothing to do with cset/gset, so I
felt that having them clearly separated and in abc order was best.
>> - char *line; /* text of command line */
>> + char *line; /* first line for short display */
>> + char *lines; /* full multi-line text of command */
>
> Not really a fan of such closely-named variables... Maybe first_line
> instead?
Indeed, looks better.
>> + case PGRES_TUPLES_OK:
>> + if (gset[compound] != NULL)
>
> Probably be good to add some comments here, eh:
> /*
> * The results are intentionally thrown away if we aren't under a gset
> * call.
> */
I added a (shorter) comment.
>> + if (ntuples != 1)
>> + {
>> + fprintf(stderr,
>> + "client %d file %d command %d compound %d: "
>> + "expecting one row, got %d\n",
>> + st->id, st->use_file, st->command, compound, ntuples);
>> + st->ecnt++;
>> + PQclear(res);
>> + discard_response(st);
>> + return false;
>> + }
>
> Might be interesting to support mutli-row (or no row?) returns in the
> future, but not something this patch needs to do, so this looks fine to
> me.
It could match PL/pgSQL's INTO, that is first row or NULL if none.
>> +
>> + /* store result as a string */
>> + if (!putVariable(st, "gset", varname,
>> + PQgetvalue(res, 0, f)))
>> + {
>> + /* internal error, should it rather abort? */
>
> I'm a bit on the fence about if we should abort in this case or not. A
> failure here seems likely to happen consistently (whereas the ntuples
> case might be a fluke of some kind), which tends to make me think we
> should abort, but still thinking about it.
I'm also still thinking about it:-) For me it is an abort, but there is
this whole "return false" internal error management in pgbench the purpose
of which fails me a little bit, so I stick to that anyway.
>> + if (*gset[compound] != '\0')
>> + free(varname);
>
> A comment here, and above where we're assigning the result of the
> psprintf(), to varname probably wouldn't hurt, explaining that the
> variable is sometimes pointing into the query result structure and
> sometimes not...
I added two comments to avoid a malloc/free when there are no prefixes.
ISTM that although it might be a border-line over-optimization case, it is
a short one, the free is a few lines away, so it might be ok.
>> + /* read and discard the query results */
>
> That comment doesn't feel quite right now. ;)
Indeed. Changed with "store or discard".
>>
>> - /* We don't want to scribble on cmd->argv[0] until done */
>> - sql = pg_strdup(cmd->argv[0]);
>> + sql = pg_strdup(cmd->lines);
>
> The function-header comment for parseQuery() could really stand to be
> improved.
Indeed.
>> + /* merge gset variants into preceeding SQL command */
>> + if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
>> + pg_strcasecmp(bs_cmd, "cset") == 0)
>> + {
>> + "\\gset cannot start a script",
>> + "\\gset must follow a SQL command",
>> + "\\gset cannot follow one another",
>
> These errors should probably be '\\gset and \\cset' or similar, no?
> Since we fall into this for both..
Indeed.
Attached an improved version, mostly comments and error message fixes.
I have not changed the 1 row constraint for now. Could be an later
extension.
If this patch get through, might be handy for simplifying tests in
current pgbench submissions, especially the "error handling" one.
--
Fabien.
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 41d9030..dffbae2 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index fd18568..47bc40c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -437,12 +437,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1595,6 +1598,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2670,7 +2774,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3146,26 +3249,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3820,7 +3909,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3828,8 +3917,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3853,7 +3941,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3865,7 +3953,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3924,22 +4012,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3959,35 +4035,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4045,7 +4197,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4066,10 +4218,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4082,7 +4235,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4091,19 +4244,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4132,7 +4286,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4140,25 +4294,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4235,6 +4396,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4258,6 +4422,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4267,31 +4432,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4306,6 +4468,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4661,7 +4884,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5129,28 +5352,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865..c349477 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index be08b20..7f86df5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -474,6 +474,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -626,16 +668,44 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0, [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
- );
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
Attachments:
[text/plain] pgbench-into-17.patch (26.1K, ../../alpine.DEB.2.20.1804071734480.4721@lancre/2-pgbench-into-17.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 41d9030..dffbae2 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index fd18568..47bc40c 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -437,12 +437,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1595,6 +1598,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2670,7 +2774,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3146,26 +3249,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3820,7 +3909,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3828,8 +3917,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3853,7 +3941,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3865,7 +3953,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3924,22 +4012,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3959,35 +4035,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4045,7 +4197,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4066,10 +4218,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4082,7 +4235,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4091,19 +4244,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4132,7 +4286,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4140,25 +4294,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4235,6 +4396,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4258,6 +4422,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4267,31 +4432,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4306,6 +4468,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4661,7 +4884,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5129,28 +5352,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865..c349477 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index be08b20..7f86df5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -474,6 +474,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -626,16 +668,44 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0, [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
- );
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-04-08 00:00 Stephen Frost <sfrost@snowman.net>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Stephen Frost @ 2018-04-08 00:00 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: pgsql-hackers
Fabien,
* Fabien COELHO (coelho@cri.ensmp.fr) wrote:
> >> <variablelist>
> >>+ <varlistentry id='pgbench-metacommand-gset'>
> >>+ <term>
> >>+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
> >>+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
> >>+ </term>
> >
> >Seems like this should really be moved down below the section for
> >'\set'.
>
> Dunno.
>
> I put them there because it is in alphabetical order (for cset at least) and
> because "set" documentation is heavy about expressions (operators,
> functions, constants, ...) which have nothing to do with cset/gset, so I
> felt that having them clearly separated and in abc order was best.
Ah, hmmm, yes, alphabetical order is sensible, certainly.
> >>- char *line; /* text of command line */
> >>+ char *line; /* first line for short display */
> >>+ char *lines; /* full multi-line text of command */
> >
> >Not really a fan of such closely-named variables... Maybe first_line
> >instead?
>
> Indeed, looks better.
Great, thanks.
> >>+ case PGRES_TUPLES_OK:
> >>+ if (gset[compound] != NULL)
> >
> >Probably be good to add some comments here, eh:
>
> >/*
> >* The results are intentionally thrown away if we aren't under a gset
> >* call.
> >*/
>
> I added a (shorter) comment.
Ok.
> >>+ if (ntuples != 1)
> >>+ {
> >>+ fprintf(stderr,
> >>+ "client %d file %d command %d compound %d: "
> >>+ "expecting one row, got %d\n",
> >>+ st->id, st->use_file, st->command, compound, ntuples);
> >>+ st->ecnt++;
> >>+ PQclear(res);
> >>+ discard_response(st);
> >>+ return false;
> >>+ }
> >
> >Might be interesting to support mutli-row (or no row?) returns in the
> >future, but not something this patch needs to do, so this looks fine to
> >me.
>
> It could match PL/pgSQL's INTO, that is first row or NULL if none.
Yeah, but that's not really something that needs to go into this patch.
> >>+
> >>+ /* store result as a string */
> >>+ if (!putVariable(st, "gset", varname,
> >>+ PQgetvalue(res, 0, f)))
> >>+ {
> >>+ /* internal error, should it rather abort? */
> >
> >I'm a bit on the fence about if we should abort in this case or not. A
> >failure here seems likely to happen consistently (whereas the ntuples
> >case might be a fluke of some kind), which tends to make me think we
> >should abort, but still thinking about it.
>
> I'm also still thinking about it:-) For me it is an abort, but there is this
> whole "return false" internal error management in pgbench the purpose of
> which fails me a little bit, so I stick to that anyway.
Yeah.
> >>+ if (*gset[compound] != '\0')
> >>+ free(varname);
> >
> >A comment here, and above where we're assigning the result of the
> >psprintf(), to varname probably wouldn't hurt, explaining that the
> >variable is sometimes pointing into the query result structure and
> >sometimes not...
>
> I added two comments to avoid a malloc/free when there are no prefixes. ISTM
> that although it might be a border-line over-optimization case, it is a
> short one, the free is a few lines away, so it might be ok.
Ok, having the comments is definitely good as it was a bit confusing as
to what was going on. :)
> >>+ /* read and discard the query results */
> >
> >That comment doesn't feel quite right now. ;)
>
> Indeed. Changed with "store or discard".
Ok.
> >>
> >>- /* We don't want to scribble on cmd->argv[0] until done */
> >>- sql = pg_strdup(cmd->argv[0]);
> >>+ sql = pg_strdup(cmd->lines);
> >
> >The function-header comment for parseQuery() could really stand to be
> >improved.
>
> Indeed.
>
> >>+ /* merge gset variants into preceeding SQL command */
> >>+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
> >>+ pg_strcasecmp(bs_cmd, "cset") == 0)
> >>+ {
> >>+ "\\gset cannot start a script",
> >>+ "\\gset must follow a SQL command",
> >>+ "\\gset cannot follow one another",
> >
> >These errors should probably be '\\gset and \\cset' or similar, no?
> >Since we fall into this for both..
>
> Indeed.
>
> Attached an improved version, mostly comments and error message fixes.
>
> I have not changed the 1 row constraint for now. Could be an later
> extension.
>
> If this patch get through, might be handy for simplifying tests in
> current pgbench submissions, especially the "error handling" one.
Glad to hear that. Unfortunately, I didn't end up having time to wrap
this up to a satisfactory level for myself to get it into PG11. I know
it's been a long time coming, and thank you for continuing to push on
it; I'll try to make sure that I take some time in the first CF for PG12
to wrap this up and get it in. I'm all for these improvements in
pgbench in general, it's a really useful tool and it's great to see work
going into it.
Thanks again!
Stephen
Attachments:
[application/pgp-signature] signature.asc (818B, ../../20180408000001.GD27724@tamriel.snowman.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-04-08 05:58 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-04-08 05:58 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-hackers
Hello Stephen,
>>> Might be interesting to support mutli-row (or no row?) returns in the
>>> future, but not something this patch needs to do, so this looks fine to
>>> me.
>>
>> It could match PL/pgSQL's INTO, that is first row or NULL if none.
>
> Yeah, but that's not really something that needs to go into this patch.
Sure. I did not. I checked psql \gset behavior:
psql> SELECT 1 AS stuff WHERE false \gset
no rows returned for \gset
psql> \echo :stuff
:stuff -- "stuff" var was not set
psql> SELECT i AS stuff FROM generate_series(1,5) AS i \gset
more than one row returned for \gset
psql> \echo :stuff
:stuff -- "stuff" var was not set either
If the semantics is changed in anyway, ISTM that psql & pgbench should be
kept consistent.
>> If this patch get through, might be handy for simplifying tests in
>> current pgbench submissions, especially the "error handling" one.
>
> Glad to hear that. Unfortunately, I didn't end up having time to wrap
> this up to a satisfactory level for myself to get it into PG11.
No problem with waiting for PG<N+1>. Whatever N:-)
> I know it's been a long time coming, and thank you for continuing to
> push on it;
Yeah, I'm kind of stubborn. Sometimes a quality, often a flaw.
> I'll try to make sure that I take some time in the first CF
> for PG12 to wrap this up and get it in. I'm all for these improvements
> in pgbench in general, it's a really useful tool and it's great to see
> work going into it.
Thanks for scheduling a try! :-)
When it gets in, I'll submit, eventually, a "tpcb-strict" builtin
benchmarking script for illustration, which would implement the bench
requirement that clients more often query in their own branch. This would
take advantage of recently (PG11) added \if and logical expressions (for
correlating clients to their branch) and gset (the benchmark states that
the client must retrieve the value, whereas it is currently discarded).
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-04-27 16:28 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-04-27 16:28 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-hackers
Hello Stephen,
Attached is v18, another basic rebase after some perl automatic
reindentation.
--
Fabien.
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index e4b37dd..28a1387 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 78b8f17..7bf4331 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -437,12 +437,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1595,6 +1598,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2679,7 +2783,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3172,26 +3275,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3846,7 +3935,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3854,8 +3943,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3879,7 +3967,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3891,7 +3979,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3950,22 +4038,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3985,35 +4061,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4071,7 +4223,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4092,10 +4244,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4108,7 +4261,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4117,19 +4270,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4158,7 +4312,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4166,25 +4320,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4263,6 +4424,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4286,6 +4450,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4295,31 +4460,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4334,6 +4496,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4689,7 +4912,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5159,28 +5382,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865..c349477 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 947f13d..c1e86da 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -484,6 +484,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -618,16 +660,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],);
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
Attachments:
[text/plain] pgbench-into-18.patch (26.2K, ../../alpine.DEB.2.20.1804271819510.8704@lancre/2-pgbench-into-18.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index e4b37dd..28a1387 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 78b8f17..7bf4331 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -437,12 +437,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1595,6 +1598,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2679,7 +2783,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3172,26 +3275,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3846,7 +3935,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3854,8 +3943,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3879,7 +3967,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3891,7 +3979,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3950,22 +4038,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3985,35 +4061,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4071,7 +4223,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4092,10 +4244,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4108,7 +4261,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4117,19 +4270,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4158,7 +4312,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4166,25 +4320,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4263,6 +4424,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4286,6 +4450,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4295,31 +4460,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4334,6 +4496,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4689,7 +4912,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5159,28 +5382,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865..c349477 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 947f13d..c1e86da 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -484,6 +484,48 @@ pgbench(
\shell echo shell-echo-output
} });
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -618,16 +660,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand} ],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
[ 'bad boolean', 0,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],);
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587b..efca525 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db6..8ef4abd 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-13 22:04 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Andrew Dunstan @ 2018-08-13 22:04 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-hackers
On 04/27/2018 12:28 PM, Fabien COELHO wrote:
>
> Hello Stephen,
>
> Attached is v18, another basic rebase after some perl automatic
> reindentation.
>
This patch contains CRLF line endings - and in any case it doesn't apply
any more. Please fix those things.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-13 22:30 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-08-13 22:30 UTC (permalink / raw)
To: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-hackers
Hello Andrew,
>> Attached is v18, another basic rebase after some perl automatic
>> reindentation.
>
> This patch contains CRLF line endings
Alas, not according to "file" nor "hexdump" (only 0A, no 0D) on my local
version, AFAICS.
What happens on the path and what is done by mail clients depending on the
mime type is another question (eg text/x-diff or text/plain).
> - and in any case it doesn't apply any more. Please fix those things.
Here is the new generated version, v19, that I just tested on my linux
ubuntu bionic laptop:
sh> git checkout -b test master
Switched to a new branch 'test'
sh> cksum ~/pgbench-into-19.patch
3375461661 26024 ~/pgbench-into-19.patch
sh> hexdump ~/pgbench-into-19.patch
0000000 6964 6666 2d20 672d 7469 6120 642f 636f
0000010 732f 6372 732f 6d67 2f6c 6572 2f66 6770
0000020 6562 636e 2e68 6773 6c6d 6220 642f 636f
0000030 732f 6372 732f 6d67 2f6c 6572 2f66 6770
0000040 6562 636e 2e68 6773 6c6d 690a 646e ....
# no 0d in front of 0a ^^
sh> git apply ~/pgbench-into-19.patch
sh> git status
On branch test ...
modified: doc/src/sgml/ref/pgbench.sgml
modified: src/bin/pgbench/pgbench.c
modified: src/bin/pgbench/pgbench.h
modified: src/bin/pgbench/t/001_pgbench_with_server.pl
modified: src/fe_utils/psqlscan.l
modified: src/include/fe_utils/psqlscan_int.h
--
Fabien.
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 88cf8b3933..946f08005d 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 41b756c089..4c2c263db4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -438,12 +438,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1596,6 +1599,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2683,7 +2787,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3176,26 +3279,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3850,7 +3939,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3858,8 +3947,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3883,7 +3971,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3895,7 +3983,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3954,22 +4042,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3989,35 +4065,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4075,7 +4227,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4096,10 +4248,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4112,7 +4265,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4121,19 +4274,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4162,7 +4316,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4170,25 +4324,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4267,6 +4428,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4290,6 +4454,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4299,31 +4464,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4338,6 +4500,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4693,7 +4916,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5163,28 +5386,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865b92..c349477ebe 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 2fc021dde7..39c7bbfdc5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -527,6 +527,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -728,19 +770,46 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 0,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 0,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587be34..efca525acc 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
Attachments:
[text/plain] pgbench-into-19.patch (26.2K, ../../alpine.DEB.2.21.1808140012360.2727@lancre/2-pgbench-into-19.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 88cf8b3933..946f08005d 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 41b756c089..4c2c263db4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -438,12 +438,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1596,6 +1599,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2683,7 +2787,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3176,26 +3279,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3850,7 +3939,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3858,8 +3947,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3883,7 +3971,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3895,7 +3983,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3954,22 +4042,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3989,35 +4065,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4075,7 +4227,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4096,10 +4248,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4112,7 +4265,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4121,19 +4274,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4162,7 +4316,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4170,25 +4324,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4267,6 +4428,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4290,6 +4454,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4299,31 +4464,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4338,6 +4500,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4693,7 +4916,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5163,28 +5386,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865b92..c349477ebe 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 2fc021dde7..39c7bbfdc5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -527,6 +527,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -728,19 +770,46 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 0,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 0,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587be34..efca525acc 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
[application/gzip] pgbench-into-19.patch.gz (7.6K, ../../alpine.DEB.2.21.1808140012360.2727@lancre/3-pgbench-into-19.patch.gz)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 10:47 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 2 replies; 120+ messages in thread
From: Andrew Dunstan @ 2018-08-14 10:47 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-hackers
On 08/13/2018 06:30 PM, Fabien COELHO wrote:
>
> Hello Andrew,
>
>>> Attached is v18, another basic rebase after some perl automatic
>>> reindentation.
>>
>> This patch contains CRLF line endings
>
> Alas, not according to "file" nor "hexdump" (only 0A, no 0D) on my
> local version, AFAICS.
>
> What happens on the path and what is done by mail clients depending on
> the mime type is another question (eg text/x-diff or text/plain).
It's not done by my MUA, and it's present in your latest posted patch.
If anything I'd suspect your MUA:
andrew@emma*$ curl -s
https://www.postgresql.org/message-id/attachment/64237/pgbench-into-19.patch
| head -n 3 | od -c
0000000 d i f f - - g i t a / d o c
0000020 / s r c / s g m l / r e f / p g
0000040 b e n c h . s g m l b / d o c
0000060 / s r c / s g m l / r e f / p g
0000100 b e n c h . s g m l \r \n i n d e
0000120 x 8 8 c f 8 b 3 9 3 3 . . 9 4
0000140 6 f 0 8 0 0 5 d 1 0 0 6 4 4 \r
0000160 \n - - - a / d o c / s r c / s
0000200 g m l / r e f / p g b e n c h .
0000220 s g m l \r \n
The gzipped version you also attached did not have the CRs.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 11:37 Andrew Gierth <andrew@tao11.riddles.org.uk>
parent: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
1 sibling, 2 replies; 120+ messages in thread
From: Andrew Gierth @ 2018-08-14 11:37 UTC (permalink / raw)
To: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
>>>>> "Andrew" == Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>>> This patch contains CRLF line endings
>>
>> Alas, not according to "file" nor "hexdump" (only 0A, no 0D) on my
>> local version, AFAICS.
>>
>> What happens on the path and what is done by mail clients depending
>> on the mime type is another question (eg text/x-diff or text/plain).
Andrew> It's not done by my MUA, and it's present in your latest posted
Andrew> patch. If anything I'd suspect your MUA:
The patch in the original email is in text/plain with base64 transfer
encoding, which means that CRLF line endings are mandatory. It's
actually up to the receiving MUA (or the archives webserver) to undo
that.
If the archives webserver isn't handling that then it's a bug there.
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 11:38 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
1 sibling, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-08-14 11:38 UTC (permalink / raw)
To: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-hackers
Hello Andrew,
> It's not done by my MUA, and it's present in your latest posted patch. If
> anything I'd suspect your MUA:
>
> andrew@emma*$ curl -s
> https://www.postgresql.org/message-id/attachment/64237/pgbench-into-19.patch
Argh. Indeed, this downloaded version has CRLF. Now when I save the
attachment in my MUA, I only have LF... Let us look at the raw format:
Content-Type: text/plain; name=pgbench-into-19.patch
Content-Transfer-Encoding: BASE64
...
ZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9yZWYvcGdiZW5jaC5zZ21sIGIv
ZG9jL3NyYy9zZ21sL3JlZi9wZ2JlbmNoLnNnbWwNCmluZGV4IDg4Y2Y4YjM5
...
Where you immediatly see that it has indeed CRLF at the end of the second
line:-).
So you are right, and my trusted mailer is encoding *AND* decoding
silently.
Why would it do that? After some googling, this is because RFC 2046 (MIME)
says you "MUST":
https://tools.ietf.org/html/rfc2046#section-4.1.1
So I'm right in the end, and the whole world is wrong, which is a
relief:-)
As I cannot except everybody to have a RFC 2046 compliant MUA, and after
some meddling in "/etc/mime.types", I now have:
Content-Type: application/octet-stream; name=pgbench-into-19.patch
Content-Transfer-Encoding: BASE64
...
ZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9yZWYvcGdiZW5jaC5zZ21sIGIv
ZG9jL3NyYy9zZ21sL3JlZi9wZ2JlbmNoLnNnbWwKaW5kZXggODhjZjhiMzkz
Which is much better:-)
I re-attached the v19 for a check on the list.
--
Fabien.
Attachments:
[application/octet-stream] pgbench-into-19.patch (25.4K, ../../alpine.DEB.2.21.1808141307140.27691@lancre/2-pgbench-into-19.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 88cf8b3933..946f08005d 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 41b756c089..4c2c263db4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -438,12 +438,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1596,6 +1599,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2683,7 +2787,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3176,26 +3279,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3850,7 +3939,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3858,8 +3947,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3883,7 +3971,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3895,7 +3983,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -3954,22 +4042,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -3989,35 +4065,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4075,7 +4227,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4096,10 +4248,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4112,7 +4265,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4121,19 +4274,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4162,7 +4316,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4170,25 +4324,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4267,6 +4428,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4290,6 +4454,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4299,31 +4464,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4338,6 +4500,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4693,7 +4916,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5163,28 +5386,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index 6983865b92..c349477ebe 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 2fc021dde7..39c7bbfdc5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -527,6 +527,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -728,19 +770,46 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 0,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 0,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX -M prepared',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 1cc587be34..efca525acc 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -680,8 +680,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 11:56 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-08-14 11:56 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
> Andrew> It's not done by my MUA, and it's present in your latest posted
> Andrew> patch. If anything I'd suspect your MUA:
>
> The patch in the original email is in text/plain with base64 transfer
> encoding, which means that CRLF line endings are mandatory. It's
> actually up to the receiving MUA (or the archives webserver) to undo
> that.
I came to the same conclusion. This is hidden because most people post
patches as "application/octet-stream", where no meddling is allowed. I'll
try to do that in the future.
> If the archives webserver isn't handling that then it's a bug there.
I'm not sure that the webserver is at fault either: it sends "CRLF" on
"text/plain", which seems okay, even required, by MIME. Maybe the web
user agent should do the translating if appropriate to the receiving
system... Obviously "curl" does not do it, nor "firefox" on "save as".
I have no doubt that some MUA around would forget to do the conversion.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 12:35 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
parent: Andrew Gierth <andrew@tao11.riddles.org.uk>
1 sibling, 1 reply; 120+ messages in thread
From: Andrew Dunstan @ 2018-08-14 12:35 UTC (permalink / raw)
To: Andrew Gierth <andrew@tao11.riddles.org.uk>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
On 08/14/2018 07:37 AM, Andrew Gierth wrote:
>>>>>> "Andrew" == Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> >>> This patch contains CRLF line endings
> >>
> >> Alas, not according to "file" nor "hexdump" (only 0A, no 0D) on my
> >> local version, AFAICS.
> >>
> >> What happens on the path and what is done by mail clients depending
> >> on the mime type is another question (eg text/x-diff or text/plain).
>
> Andrew> It's not done by my MUA, and it's present in your latest posted
> Andrew> patch. If anything I'd suspect your MUA:
>
> The patch in the original email is in text/plain with base64 transfer
> encoding, which means that CRLF line endings are mandatory. It's
> actually up to the receiving MUA (or the archives webserver) to undo
> that.
>
> If the archives webserver isn't handling that then it's a bug there.
>
Probably a good reason not to use text/plain for patches, ISTM. I do
note that my MUA (Thunderbird) uses text/x-patch and probably violates
RFC2046 4.1.1
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 12:43 Andrew Gierth <andrew@tao11.riddles.org.uk>
parent: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
0 siblings, 0 replies; 120+ messages in thread
From: Andrew Gierth @ 2018-08-14 12:43 UTC (permalink / raw)
To: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
>>>>> "Andrew" == Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> The patch in the original email is in text/plain with base64 transfer
>> encoding, which means that CRLF line endings are mandatory. It's
>> actually up to the receiving MUA (or the archives webserver) to undo
>> that.
>>
>> If the archives webserver isn't handling that then it's a bug there.
Andrew> Probably a good reason not to use text/plain for patches, ISTM.
Andrew> I do note that my MUA (Thunderbird) uses text/x-patch and
Andrew> probably violates RFC2046 4.1.1
The first patch of yours I found was in text/x-patch with 7bit transfer
encoding, so the line endings are actually those of the mail message
itself (i.e. CRLF on the wire).
--
Andrew (irc:RhodiumToad)
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 17:44 Tom Lane <tgl@sss.pgh.pa.us>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 3 replies; 120+ messages in thread
From: Tom Lane @ 2018-08-14 17:44 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
Fabien COELHO <coelho@cri.ensmp.fr> writes:
> I have no doubt that some MUA around would forget to do the conversion.
FWIW, one reason that I invariably use patch(1) to apply submitted patches
is that it will take care of stripping any CRs that may have snuck in.
So I'm not particularly fussed about the problem.
I'm not excited about encouraging people to use application/octet-stream
rather than text/something for submitted patches. If you use text then
the patch can easily be examined in the web archives; with
application/octet-stream the patch has to be downloaded, adding a lot of
manual overhead. (At least, that's true with my preferred web browser,
maybe it's different for other people?)
regards, tom lane
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 18:23 Robert Haas <robertmhaas@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 0 replies; 120+ messages in thread
From: Robert Haas @ 2018-08-14 18:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
On Tue, Aug 14, 2018 at 1:44 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Fabien COELHO <coelho@cri.ensmp.fr> writes:
>> I have no doubt that some MUA around would forget to do the conversion.
>
> FWIW, one reason that I invariably use patch(1) to apply submitted patches
> is that it will take care of stripping any CRs that may have snuck in.
> So I'm not particularly fussed about the problem.
Yeah. I think that we shouldn't care about this, or about
context/unified diffs, or really anything other than that patch can
apply it. Once you apply it, you can issue the correct incantation to
see it in whatever format you prefer. If it's a whole patch stack, it
makes sense to use 'git format-patch' to generate the patches, because
then it's a lot easier to apply the whole stack, but for a single
patch it really doesn't matter.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 19:10 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 0 replies; 120+ messages in thread
From: Andrew Dunstan @ 2018-08-14 19:10 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
On 08/14/2018 01:44 PM, Tom Lane wrote:
> Fabien COELHO <coelho@cri.ensmp.fr> writes:
>> I have no doubt that some MUA around would forget to do the conversion.
> FWIW, one reason that I invariably use patch(1) to apply submitted patches
> is that it will take care of stripping any CRs that may have snuck in.
> So I'm not particularly fussed about the problem.
I also use patch(1), although probably mainly out of laziness in not
changing habits going back to the CVS days ;-)
You obviously commit far more patches that I do, but I don't normally
see patch complaining about CRs, which is why I raised the issue.
>
> I'm not excited about encouraging people to use application/octet-stream
> rather than text/something for submitted patches. If you use text then
> the patch can easily be examined in the web archives; with
> application/octet-stream the patch has to be downloaded, adding a lot of
> manual overhead. (At least, that's true with my preferred web browser,
> maybe it's different for other people?)
>
>
You might have a point. Compressed patched can be particularly annoying.
Most other things my browser will download and pop unto geany for me.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 20:44 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-08-14 20:44 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Andrew Gierth <andrew@tao11.riddles.org.uk>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
>> I have no doubt that some MUA around would forget to do the conversion.
>
> FWIW, one reason that I invariably use patch(1) to apply submitted patches
> is that it will take care of stripping any CRs that may have snuck in.
> So I'm not particularly fussed about the problem.
Good to know.
> I'm not excited about encouraging people to use application/octet-stream
> rather than text/something for submitted patches.
I'm not happy with that either, it is just to avoid complaints.
> If you use text then the patch can easily be examined in the web
> archives; with application/octet-stream the patch has to be downloaded,
> adding a lot of manual overhead.
Indeed.
> (At least, that's true with my preferred web browser, maybe it's
> different for other people?)
So if I send with text/x-diff or text/plain I've got complaints, if I send
with application/octet-stream, it is not right either:-) Everybody being
somehow right.
Sigh.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-08-14 20:53 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 0 replies; 120+ messages in thread
From: Alvaro Herrera @ 2018-08-14 20:53 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Andrew Gierth <andrew@tao11.riddles.org.uk>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; pgsql-hackers
On 2018-Aug-14, Fabien COELHO wrote:
> > (At least, that's true with my preferred web browser, maybe it's
> > different for other people?)
>
> So if I send with text/x-diff or text/plain I've got complaints, if I send
> with application/octet-stream, it is not right either:-) Everybody being
> somehow right.
I like that I can look at the text/* ones directly in the browser
instead of having to download, but I can handle whatever (and I expect
the same for most people, except maybe those who work directly on
Windows). I just wish people would not send tarballs, which aren't as
comfy to page through with "zcat | cdiff" ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-09-12 06:35 Michael Paquier <michael@paquier.xyz>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Michael Paquier @ 2018-09-12 06:35 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers; Fabien COELHO <coelho@cri.ensmp.fr>
Hi Stephen,
On Tue, Aug 14, 2018 at 01:38:21PM +0200, Fabien COELHO wrote:
> I re-attached the v19 for a check on the list.
You are marked as the committer of this patch in the CF app since last
April and this patch is marked as ready for committer. Are you planning
to look at it soon?
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../20180912063542.GH25160@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-10-02 06:43 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-10-02 06:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Bonjour Michaël,
> On Tue, Aug 14, 2018 at 01:38:21PM +0200, Fabien COELHO wrote:
>> I re-attached the v19 for a check on the list.
>
> You are marked as the committer of this patch in the CF app since last
> April and this patch is marked as ready for committer. Are you planning
> to look at it soon?
Here is yet another rebase.
Whether Stephen will have time to commit this patch is unclear. I'd
suggest that I remove his name from the committer column so that another
committer may consider it.
What do you think?
For me this patch is fundamental because it allows a test script to
interact both ways with the database, and to act on database data (in
particular thanks to \if and expressions already added), and also actually
retrieving results is a key benchmark compliance constraint that pgbench
does not meet.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-20.patch (26.1K, ../../alpine.DEB.2.21.1810020832440.27693@lancre/2-pgbench-into-20.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 8c464c9d42..d9cdec15da 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -942,6 +942,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 436764b9c9..eb9478b3d5 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -467,12 +467,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1681,6 +1684,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2784,7 +2888,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3277,26 +3380,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3951,7 +4040,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3959,8 +4048,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3984,7 +4072,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3996,7 +4084,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4055,22 +4143,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4090,35 +4166,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4176,7 +4328,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4197,10 +4349,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4213,7 +4366,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4222,19 +4375,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4263,7 +4417,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4271,25 +4425,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4368,6 +4529,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4391,6 +4555,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4400,31 +4565,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4439,6 +4601,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4794,7 +5017,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5264,28 +5487,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index de50340434..f4a7327dc7 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index d972903f2a..66dfca6325 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -530,6 +530,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -740,17 +782,46 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[
'bad boolean', 0,
[qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ ],
+ # GSET & CSET
+ [ 'gset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 0,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT',
+ 0,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name',
+ 0,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name',
+ 0,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index fdf49875a7..6b3e34a1fb 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -689,8 +689,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-10-15 13:09 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-10-15 13:09 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
> Here is yet another rebase.
Here is yet another rebase after Peter's exit status changes on pgbench.
> Whether Stephen will have time to commit this patch is unclear. I'd suggest
> that I remove his name from the committer column so that another committer
> may consider it.
>
> What do you think?
?
> For me this patch is fundamental because it allows a test script to interact
> both ways with the database, and to act on database data (in particular
> thanks to \if and expressions already added), and also actually retrieving
> results is a key benchmark compliance constraint that pgbench does not meet.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-21.patch (26.4K, ../../alpine.DEB.2.21.1810151506270.13477@lancre/2-pgbench-into-21.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..049fcaf839 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,51 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal> or
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ These commands may be used to end SQL queries, replacing a semicolon.
+ <literal>\cset</literal> replaces an embedded semicolon (<literal>\;</literal>) within
+ a compound SQL command, and <literal>\gset</literal> replaces a final
+ (<literal>;</literal>) semicolon which ends the SQL command.
+ </para>
+
+ <para>
+ When these commands are used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>one</replaceable>, <replaceable>two</replaceable> and
+ <replaceable>p_three</replaceable> with integers from a compound query.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 AS one, 2 AS two \cset
+SELECT 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> and <literal>\gset</literal> commands do not work when
+ empty SQL queries appear within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 81bc6d8a6e..112716ca1f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -467,12 +467,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1681,6 +1684,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2784,7 +2888,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3277,26 +3380,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3951,7 +4040,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3959,8 +4048,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -3984,7 +4072,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -3996,7 +4084,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4055,22 +4143,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4090,35 +4166,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4176,7 +4328,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4197,10 +4349,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4213,7 +4366,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4222,19 +4375,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4263,7 +4417,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4271,25 +4425,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4368,6 +4529,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4391,6 +4555,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4400,31 +4565,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4439,6 +4601,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4794,7 +5017,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5266,28 +5489,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index de50340434..f4a7327dc7 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 0081989026..79ec6fb6f5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -530,6 +530,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -737,21 +779,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index fdf49875a7..6b3e34a1fb 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -689,8 +689,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-16 21:43 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
2 siblings, 2 replies; 120+ messages in thread
From: Alvaro Herrera @ 2018-11-16 21:43 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 2017-Nov-04, Fabien COELHO wrote:
> Think of one initialization followed by two appends:
>
> SELECT 1 AS x \cset
> SELECT 2 \; SELECT 3 AS y \cset
> SELECT 4 \; SELECT 5 \; SELECT 6 AS z \gset
>
> In the end, we must have the full 6 queries
>
> "SELECT 1 AS x \; SELECT 2 \; SELECT 3 AS y \; SELECT 4 \; SELECT 5 \; SELECT 6 AS z"
>
> and know that we want to set variables from queries 1, 3 and 6 and ignore
> the 3 others.
I'm not sure I understand this. Why is the "SELECT 2" ignored? (I can
see why the 4 and 5 are ignored: they are not processed by gset).
What exactly does \cset do? I thought "SELECT 2 \; SELECT 3 AS y \cset"
would search for the \; and process *both* queries.
I think the doc addition should be split.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-16 22:13 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 120+ messages in thread
From: Alvaro Herrera @ 2018-11-16 22:13 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
On 2018-Nov-16, Alvaro Herrera wrote:
> On 2017-Nov-04, Fabien COELHO wrote:
>
> > Think of one initialization followed by two appends:
> >
> > SELECT 1 AS x \cset
> > SELECT 2 \; SELECT 3 AS y \cset
> > SELECT 4 \; SELECT 5 \; SELECT 6 AS z \gset
> >
> > In the end, we must have the full 6 queries
> >
> > "SELECT 1 AS x \; SELECT 2 \; SELECT 3 AS y \; SELECT 4 \; SELECT 5 \; SELECT 6 AS z"
> >
> > and know that we want to set variables from queries 1, 3 and 6 and ignore
> > the 3 others.
>
> I'm not sure I understand this. Why is the "SELECT 2" ignored? (I can
> see why the 4 and 5 are ignored: they are not processed by gset).
>
> What exactly does \cset do?
Oh! I understand it now. You say "replace a semicolon" to mean "works
as if it were a semicolon, and also captures the result". So \cset
means "works as if it were an escaped semicolon". It all suddenly makes
sense now! I think I'll propose some rewording of that explanation, as
it was very confusing to me.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-16 22:36 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2018-11-16 22:36 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
I think this patch's Command->lines would benefit from using PQExpBuffer
(or maybe StringInfo?) for the command string instead of open-coding
string manipulation and allocation.
I'm not sure that Command->first_line is really all that useful. It
seems we go to a lot of trouble to keep it up to date. Isn't it easier
to chop Command->lines at the first newline when it is needed?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-17 10:25 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2018-11-17 10:25 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Pavel Stehule <pavel.stehule@gmail.com>; Tatsuo Ishii <ishii@sraoss.co.jp>; rafia.sabih@enterprisedb.com, Michael Paquier <michael.paquier@gmail.com>; Tom Lane <tgl@sss.pgh.pa.us>; Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>; Robert Haas <robertmhaas@gmail.com>; pgsql-hackers
Hello Alvaro,
Thanks for having a look at this patch.
>> Think of one initialization followed by two appends:
>>
>> SELECT 1 AS x \cset
>> SELECT 2 \; SELECT 3 AS y \cset
>> SELECT 4 \; SELECT 5 \; SELECT 6 AS z \gset
>>
>> In the end, we must have the full 6 queries
>>
>> "SELECT 1 AS x \; SELECT 2 \; SELECT 3 AS y \; SELECT 4 \; SELECT 5 \; SELECT 6 AS z"
>>
>> and know that we want to set variables from queries 1, 3 and 6 and ignore
>> the 3 others.
>
> I'm not sure I understand this. Why is the "SELECT 2" ignored?
Because there is no \cset nor \gset attached to it, so the command does
not say to put the result into variables.
> (I can see why the 4 and 5 are ignored: they are not processed by gset).
Same thing with SELECT 2, which is followed by "\;", meaning execute and
that's all.
> What exactly does \cset do? I thought "SELECT 2 \; SELECT 3 AS y \cset"
> would search for the \; and process *both* queries.
No, "\cset" does not end the compound query, only ";" or "\gset" do that.
\cset separates queries (like \;) and adds the fact that the just
preceding query result is to be put into variables when received.
\cset = \; + store result into variables
\gset = ; + store result into variables
From a performance point of view, the point is to be able to use compound
queries which reduce the number of round trips so should impact latency.
> I think the doc addition should be split.
Indeed, as the current version is confusing.
Attached an attempt at clarifying the documentation on this point. The doc
is split as suggested, descriptions and examples are specific to each
presented case.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-22.patch (27.4K, ../../alpine.DEB.2.21.1811171039400.16105@lancre/2-pgbench-into-22.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..246944ea79 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,87 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends three queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the second query is stored into variable <replaceable>two</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of 3 queries
+SELECT 1 AS one \; SELECT 2 AS two \cset
+SELECT 2;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 73d3de0677..f1e84860b4 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -490,12 +490,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ char *lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1718,6 +1721,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2825,7 +2929,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3319,26 +3422,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3993,7 +4082,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -4001,8 +4090,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines);
cmd->argc = 1;
@@ -4026,7 +4114,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines);
pg_free(name);
return false;
}
@@ -4038,7 +4126,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4097,22 +4185,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4132,35 +4208,111 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->lines = pg_strdup(p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ size_t lmore;
+ size_t len = strlen(my_command->lines);
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ lmore = strlen(more);
+ my_command->lines = pg_realloc(my_command->lines, len + lmore + 2);
+ my_command->lines[len] = ';';
+ memcpy(my_command->lines + len + 1, more, lmore + 1);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char *nlpos;
+ char *p;
+
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
+ p = my_command->lines;
nlpos = strchr(p, '\n');
if (nlpos)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ my_command->first_line = pg_malloc(nlpos - p + 1);
+ memcpy(my_command->first_line, p, nlpos - p);
+ my_command->first_line[nlpos - p] = '\0';
}
else
- my_command->line = pg_strdup(p);
+ my_command->first_line = pg_strdup(p);
- return my_command;
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
}
/*
@@ -4218,7 +4370,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4239,10 +4391,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4255,7 +4408,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4264,19 +4417,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4305,7 +4459,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4313,25 +4467,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4410,6 +4571,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4433,6 +4597,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4442,31 +4607,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4481,6 +4643,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4836,7 +5059,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5307,28 +5530,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index de50340434..f4a7327dc7 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 0081989026..79ec6fb6f5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -530,6 +530,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -737,21 +779,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 02d95498e4..d9ac285d1e 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-17 10:43 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-11-17 10:43 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
> I think this patch's Command->lines would benefit from using PQExpBuffer
> (or maybe StringInfo?) for the command string instead of open-coding
> string manipulation and allocation.
Indeed it could be used, but it is not used anywhere in "pgbench": not for
lines, not for variable subtitutions, not for the PREPARE stuff... So I
did not think that it was time to start, I just kept the current style.
Probably it could be a refactoring patch to replace all basic string stuff
with PQExpBuffer infrastructure within pgbench.
> I'm not sure that Command->first_line is really all that useful. It
> seems we go to a lot of trouble to keep it up to date. Isn't it easier
> to chop Command->lines at the first newline when it is needed?
Hmmm, it is needed quite often (about 12 times) to report errors, that
would mean having to handle the truncation in many places, so I felt it
was worth the trouble.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-17 11:06 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2018-11-17 11:06 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
On 2018-Nov-17, Fabien COELHO wrote:
>
> > I think this patch's Command->lines would benefit from using PQExpBuffer
> > (or maybe StringInfo?) for the command string instead of open-coding
> > string manipulation and allocation.
>
> Indeed it could be used, but it is not used anywhere in "pgbench": not for
> lines, not for variable subtitutions, not for the PREPARE stuff... So I did
> not think that it was time to start, I just kept the current style.
>
> Probably it could be a refactoring patch to replace all basic string stuff
> with PQExpBuffer infrastructure within pgbench.
Well, I think command handling was mostly straightforward before, but
it's not as straightforward after your patch. Also, PQExpBuffer is
already in use in pgbench when interacting with the lexer, so yeah I
think we should fix that. I don't think we should replace *ALL* string
handling in pgbench with PQExpBuffer, just the command stuff.
> > I'm not sure that Command->first_line is really all that useful. It
> > seems we go to a lot of trouble to keep it up to date. Isn't it easier
> > to chop Command->lines at the first newline when it is needed?
>
> Hmmm, it is needed quite often (about 12 times) to report errors, that would
> mean having to handle the truncation in many places, so I felt it was worth
> the trouble.
Ok, as long as we don't repeat the work during script execution.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-18 10:23 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-11-18 10:23 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Hello Alvaro,
>>> I think this patch's Command->lines would benefit from using PQExpBuffer
>>> (or maybe StringInfo?) for the command string instead of open-coding
>>> string manipulation and allocation.
>
> [...]
Ok.
>>> I'm not sure that Command->first_line is really all that useful. It
>>> seems we go to a lot of trouble to keep it up to date. Isn't it easier
>>> to chop Command->lines at the first newline when it is needed?
>>
>> Hmmm, it is needed quite often (about 12 times) to report errors, that would
>> mean having to handle the truncation in many places, so I felt it was worth
>> the trouble.
>
> Ok, as long as we don't repeat the work during script execution.
Sure, the point of first_line is that it is computed once at parse time.
Attached a v23 with PQExpBuffer for managing lines.
I've also added a function to compute the summary first line, which
handles carriage-return.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-23.patch (27.6K, ../../alpine.DEB.2.21.1811180851210.19159@lancre/2-pgbench-into-23.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..246944ea79 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,87 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends three queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the second query is stored into variable <replaceable>two</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of 3 queries
+SELECT 1 AS one \; SELECT 2 AS two \cset
+SELECT 2;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 73d3de0677..60d4c95be7 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -490,12 +490,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ PQExpBufferData lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1718,6 +1721,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2825,7 +2929,6 @@ evaluateSleep(CState *st, int argc, char **argv, int *usecs)
static void
doCustom(TState *thread, CState *st, StatsData *agg)
{
- PGresult *res;
Command *command;
instr_time now;
bool end_tx_processed = false;
@@ -3319,26 +3422,12 @@ doCustom(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /*
- * Read and discard the query result;
- */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, command->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
+
break;
/*
@@ -3993,7 +4082,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -4001,8 +4090,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines.data);
cmd->argc = 1;
@@ -4026,7 +4114,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines.data);
pg_free(name);
return false;
}
@@ -4038,7 +4126,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4097,22 +4185,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4132,35 +4208,121 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->first_line = NULL;
+
+ initPQExpBuffer(&my_command->lines);
+ appendPQExpBufferStr(&my_command->lines, p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ appendPQExpBufferChar(&my_command->lines, ';');
+ appendPQExpBufferStr(&my_command->lines, more);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+/*
+ * compute a shorten summary line for display
+ */
+static char *
+summary_line(const char *lines)
+{
+#define BUFLEN 60
+ char buf[BUFLEN];
+ char *pos;
+
+ strncpy(buf, lines, BUFLEN-1);
+ buf[BUFLEN-1] = '\0';
+
+ pos = strchr(buf, '\n');
+ if (pos != NULL)
+ buf[pos - buf] = '\0';
+ pos = strchr(buf, '\r');
+ if (pos != NULL)
+ buf[pos - buf] = '\0';
+
+ return pg_strdup(buf);
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
- nlpos = strchr(p, '\n');
- if (nlpos)
+ my_command->first_line = summary_line(my_command->lines.data);
+
+ /* parse query if necessary */
+ switch (querymode)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines.data;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
}
- else
- my_command->line = pg_strdup(p);
-
- return my_command;
}
/*
@@ -4218,7 +4380,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4239,10 +4401,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4255,7 +4418,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4264,19 +4427,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4305,7 +4469,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4313,25 +4477,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4410,6 +4581,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4433,6 +4607,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4442,31 +4617,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4481,6 +4653,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4836,7 +5069,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5307,28 +5540,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index de50340434..f4a7327dc7 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 0081989026..79ec6fb6f5 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -530,6 +530,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -737,21 +779,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 02d95498e4..d9ac285d1e 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-21 19:42 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2018-11-21 19:42 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
On 2018-Nov-18, Fabien COELHO wrote:
> Attached a v23 with PQExpBuffer for managing lines.
>
> I've also added a function to compute the summary first line, which handles
> carriage-return.
Thanks.
Please when you rebase, consider these (minor) changes.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-diff] pgbench-into-23-delta.patch (4.5K, ../../20181121194252.g4ugrnjesxfmewmz@alvherre.pgsql/2-pgbench-into-23-delta.patch)
download | inline diff:
commit 32076235c0d7fc59ad4b1b1ff876c74c40b34efb[m
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
AuthorDate: Mon Nov 19 14:29:14 2018 -0300
CommitDate: Mon Nov 19 14:29:17 2018 -0300
tweaks
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 246944ea79..3d9dd512bb 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -961,25 +961,33 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
<listitem>
<para>
- This command may be used to end SQL queries, replacing an embedded
- semicolon (<literal>\;</literal>) within a compound SQL command.
+ This command may be used to end individual SQL queries within a
+ compound SQL command, acting as a command separator
+ like an escaped semicolon (<literal>\;</literal>).
+ A query terminated with <literal>\cset</literal> will
+ have its results stored into variables instead of the
+ usual behavior of discarding the result.
</para>
<para>
- When this command is used, the preceding SQL query is expected to
- return one row, the columns of which are stored into variables named after
- column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ Queries terminated by <literal>\cset</literal> are expected
+ to return exactly one row. The values in the row are saved
+ into <application>pgbench</application> variables, the name
+ of each variable being determined by prefixing each column name
+ with <replaceable>prefix</replaceable> if one is provided.
</para>
<para>
- The following example sends three queries as one compound SQL command,
- inducing one message sent at the protocol level.
- The result of the second query is stored into variable <replaceable>two</replaceable>,
- whereas the results of the other queries are discarded.
+ The following example sends one message to the server, containing three
+ SQL queries. The result of the second query is processed and stored into
+ variables <replaceable>v_two</replaceable> and
+ <replaceable>v_three</replaceable>.
+ The results of the other queries are discarded in the usual way.
<programlisting>
-- compound of 3 queries
-SELECT 1 AS one \; SELECT 2 AS two \cset
-SELECT 2;
+SELECT * AS one FROM generate_series(1,100) \;
+select count(*) as accounts from pgbench_accounts \cset num_
+TABLE pgbench_accounts LIMIT 1;
</programlisting>
</para>
@@ -999,14 +1007,12 @@ SELECT 2;
<listitem>
<para>
- This commands may be used to end SQL queries, replacing a final semicolon
+ This command may be used to end an SQL query, like a semicolon
(<literal>;</literal>).
- </para>
-
- <para>
When this command is used, the preceding SQL query is expected to
- return one row, the columns of which are stored into variables named after
- column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ return one row, the columns of which are stored into variables named
+ after each column name prefixed with <replaceable>prefix</replaceable>
+ if provided.
</para>
<para>
@@ -1019,10 +1025,7 @@ SELECT 2;
UPDATE pgbench_accounts
SET abalance = abalance + :delta
WHERE aid = :aid
- RETURNING abalance \gset
--- compound of two queries
-SELECT 1 \;
-SELECT 2 AS two, 3 AS three \gset p_
+ RETURNING abalance \gset p_
</programlisting>
</para>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 3467598aa9..60e5fe972a 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -34,6 +34,7 @@
#include "postgres_fe.h"
#include "common/int.h"
#include "fe_utils/conditional.h"
+#include "fe_utils/psqlscan_int.h"
#include "getopt_long.h"
#include "libpq-fe.h"
#include "portability/instr_time.h"
@@ -1630,14 +1631,12 @@ assignVariables(CState *st, char *sql)
if (name == NULL)
{
while (*p == ':')
- {
p++;
- }
continue;
}
val = getVariable(st, name);
- free(name);
+ pg_free(name);
if (val == NULL)
{
p++;
@@ -4106,9 +4105,7 @@ parseQuery(Command *cmd)
if (name == NULL)
{
while (*p == ':')
- {
p++;
- }
continue;
}
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index f4a7327dc7..de50340434 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,7 +11,6 @@
#ifndef PGBENCH_H
#define PGBENCH_H
-#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-24 07:45 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-11-24 07:45 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Hello Alvaro,
> Please when you rebase, consider these (minor) changes.
Here it is.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-24.patch (27.6K, ../../alpine.DEB.2.21.1811240844130.12627@lancre/2-pgbench-into-24.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..246944ea79 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,87 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends three queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the second query is stored into variable <replaceable>two</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of 3 queries
+SELECT 1 AS one \; SELECT 2 AS two \cset
+SELECT 2;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index c64e16187a..fd0de34d49 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -501,12 +501,15 @@ static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
typedef struct
{
- char *line; /* text of command line */
+ char *first_line; /* first line for short display */
+ PQExpBufferData lines; /* full multi-line text of command */
int command_num; /* unique index of this Command struct */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int compound; /* last compound command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -1732,6 +1735,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields ; f++)
+ {
+ char *varname = PQfname(res, f);
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2862,8 +2966,6 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
for (;;)
{
- PGresult *res;
-
switch (st->state)
{
/* Select transaction (script) to run. */
@@ -3141,24 +3243,11 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /* Read and discard the query result */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, sql_script[st->use_file].commands[st->command]->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
break;
/*
@@ -3976,7 +4065,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3984,8 +4073,7 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
+ sql = pg_strdup(cmd->lines.data);
cmd->argc = 1;
@@ -4009,7 +4097,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines.data);
pg_free(name);
return false;
}
@@ -4021,7 +4109,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4080,22 +4168,10 @@ syntax_error(const char *source, int lineno,
exit(1);
}
-/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
- */
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *p)
{
- Command *my_command;
- char *p;
- char *nlpos;
-
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4115,35 +4191,121 @@ process_sql_command(PQExpBuffer buf, const char *source)
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int compounds)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
my_command->command_num = num_commands++;
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ my_command->compound = compounds;
+ my_command->gset = pg_malloc0(sizeof(char *) * (compounds+1));
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
+ my_command->first_line = NULL;
+
+ initPQExpBuffer(&my_command->lines);
+ appendPQExpBufferStr(&my_command->lines, p);
+
+ return my_command;
+}
+
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int compounds)
+{
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
+
+ more = skip_sql_comments(more);
+
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ appendPQExpBufferChar(&my_command->lines, ';');
+ appendPQExpBufferStr(&my_command->lines, more);
+
+ /* update number of compounds and extend array of prefixes */
+ nc = my_command->compound + 1 + compounds;
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc+1));
+ memset(my_command->gset + my_command->compound + 1, 0,
+ sizeof(char *) * (compounds + 1));
+ my_command->compound = nc;
+}
+
+/*
+ * compute a shorten summary line for display
+ */
+static char *
+summary_line(const char *lines)
+{
+#define BUFLEN 60
+ char buf[BUFLEN];
+ char *pos;
+
+ strncpy(buf, lines, BUFLEN-1);
+ buf[BUFLEN-1] = '\0';
+
+ pos = strchr(buf, '\n');
+ if (pos != NULL)
+ buf[pos - buf] = '\0';
+ pos = strchr(buf, '\r');
+ if (pos != NULL)
+ buf[pos - buf] = '\0';
+
+ return pg_strdup(buf);
+}
+
+static void
+postprocess_sql_command(Command *my_command)
+{
+ Assert(my_command->type == SQL_COMMAND);
/*
* If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
+ * the "line" label for display.
*/
- nlpos = strchr(p, '\n');
- if (nlpos)
+ my_command->first_line = summary_line(my_command->lines.data);
+
+ /* parse query if necessary */
+ switch (querymode)
{
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines.data;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
}
- else
- my_command->line = pg_strdup(p);
-
- return my_command;
}
/*
@@ -4201,7 +4363,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4222,10 +4384,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4238,7 +4401,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4247,19 +4410,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4288,7 +4452,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4296,25 +4460,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4393,6 +4564,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4416,6 +4590,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4425,31 +4600,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
+
+ sstate->semicolons = 0;
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* a multi-line command ended with \cset */
+ append_sql_command(ps.commands[index-1], line_buf.data,
+ sstate->semicolons);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, sstate->semicolons);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4464,6 +4636,67 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char * bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = bs_cmd[0] == 'c';
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index-1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->compound;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4819,7 +5052,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5290,28 +5523,19 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initializations and collect total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+ int j;
+
+ for (j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/pgbench.h b/src/bin/pgbench/pgbench.h
index de50340434..f4a7327dc7 100644
--- a/src/bin/pgbench/pgbench.h
+++ b/src/bin/pgbench/pgbench.h
@@ -11,6 +11,7 @@
#ifndef PGBENCH_H
#define PGBENCH_H
+#include "fe_utils/psqlscan_int.h"
#include "fe_utils/psqlscan.h"
/*
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 89678e7f3f..0eaf18c2f8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -528,6 +528,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -735,21 +777,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 02d95498e4..d9ac285d1e 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count compound commands */
+ cur_state->semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 0be0db69ab..8ef4abd92d 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int semicolons; /* number of embedded (\;) semi-colons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-24 15:36 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2018-11-24 15:36 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
FWIW I think the terminology in this patch is wrong. You use the term
"compound" to mean "one query within a string containing multiple
queries", but that's not what compound means. Compound is the whole
thing, comprised of the multiple queries. Maybe "query" is the right
word to use there, not sure.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2018-11-24 16:57 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2018-11-24 16:57 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
> FWIW I think the terminology in this patch is wrong. You use the term
> "compound" to mean "one query within a string containing multiple
> queries", but that's not what compound means. Compound is the whole
> thing, comprised of the multiple queries.
Indeed. Compounded query?
> Maybe "query" is the right word to use there, not sure.
I do not understand, "query queries"?
I think it should avoid using sql-related words, such as "group",
"aggregate", "merge", "join"...
I thought of "combined", meaning the queries are combined together in a
single message at the protocol level.
Basically I'm ok with any better idea.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-03 17:51 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2019-01-03 17:51 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
I revised this patch a bit. Here's v25, where some finishing touches
are needed -- see below. I think with these changes the patch would
become committable, at least for me.
I didn't like that you were adding an #include of psqlscan_int.h into
pgbench.c, when there's a specific comment in the header saying not to
do that, so I opted for adding a new accessor function on psqlscan.h.
I renamed some of your parameter additions. I think the new names are
clearer, but they meant the +1's in your code are now in illogical
places. (I moved some; probably not enough). Please review/fix that.
I think "gset" is not a great name for the new struct member; please
find a better name. I suggest "targetvar" but feel free to choose a
name that suits your taste.
There are two XXX comments. One is about a function needing a comment
atop it. The other is about realloc behavior. To fix this one I would
add a new struct member indicating the allocated size of the array, then
growing exponentially instead of one at a time. For most cases you can
probably get away with never reallocating beyond an initial allocation
of, say, 8 members.
In the docs, the [prefix] part needs to be explained in the \cset entry;
right now it's in \gset, which comes afterwards. Let's move the
explanation up, and then in \gset say "prefix behaves as in \cset".
Thanks
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-diff] pgbench-into-25.patch (29.8K, ../../201901031751.h7siijzvd3ve@alvherre.pgsql/2-pgbench-into-25.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..246944ea79 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,87 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends three queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the second query is stored into variable <replaceable>two</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of 3 queries
+SELECT 1 AS one \; SELECT 2 AS two \cset
+SELECT 2;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 649297ae4f..7901fcffca 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -499,14 +499,16 @@ typedef enum QueryMode
static QueryMode querymode = QUERY_SIMPLE;
static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
-typedef struct
+typedef struct Command
{
- char *line; /* text of command line */
- int command_num; /* unique index of this Command struct */
+ PQExpBufferData lines; /* raw command text (possibly multi-line) */
+ char *first_line; /* first line for short display */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int last_cmd; /* last command (number of \;) */
+ char **gset; /* per-compound command prefix */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -521,7 +523,6 @@ typedef struct ParsedScript
static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
-static int num_commands = 0; /* total number of Command structs */
static int64 total_weight = 0;
static int debug = 0; /* debug flag */
@@ -1732,6 +1733,108 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **gset)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (gset[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (gset[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res),
+ f;
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (f = 0; f < nfields; f++)
+ {
+ char *varname = PQfname(res, f);
+
+ /* prefix varname if required, will be freed below */
+ if (*gset[compound] != '\0')
+ varname = psprintf("%s%s", gset[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*gset[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2862,8 +2965,6 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
for (;;)
{
- PGresult *res;
-
switch (st->state)
{
/* Select transaction (script) to run. */
@@ -3141,24 +3242,11 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /* Read and discard the query result */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, sql_script[st->use_file].commands[st->command]->gset))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
break;
/*
@@ -3191,7 +3279,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
INSTR_TIME_SET_CURRENT_LAZY(now);
- command = sql_script[st->use_file].commands[st->command];
+ command = sql_script[st->use_file].commands[st->command];
/* XXX could use a mutex here, but we choose not to */
addToSimpleStats(&command->stats,
INSTR_TIME_GET_DOUBLE(now) -
@@ -3235,9 +3323,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->state = CSTATE_CHOOSE_SCRIPT;
/*
- * Ensure that we always return on this point, so as to
- * avoid an infinite loop if the script only contains meta
- * commands.
+ * Ensure that we always return on this point, so as to avoid
+ * an infinite loop if the script only contains meta commands.
*/
return;
@@ -3976,7 +4063,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3984,12 +4071,9 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
-
cmd->argc = 1;
- p = sql;
+ p = sql = pg_strdup(cmd->lines.data);
while ((p = strchr(p, ':')) != NULL)
{
char var[13];
@@ -4009,7 +4093,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines.data);
pg_free(name);
return false;
}
@@ -4021,7 +4105,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4081,21 +4165,16 @@ syntax_error(const char *source, int lineno,
}
/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
+ * Return a pointer to the start of the SQL command, after skipping over
+ * whitespace and "--" comments.
+ * If the end of the string is reached, return NULL.
*/
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *sql_command)
{
- Command *my_command;
- char *p;
- char *nlpos;
+ char *p = sql_command;
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4111,42 +4190,105 @@ process_sql_command(PQExpBuffer buf, const char *source)
break;
}
- /* If there's nothing but whitespace and comments, we're done */
+ /* NULL if there's nothing but whitespace and comments */
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int numqueries)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
- my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
+ my_command = (Command *) pg_malloc(sizeof(Command));
+ initPQExpBuffer(&my_command->lines);
+ appendPQExpBufferStr(&my_command->lines, p);
+ my_command->first_line = NULL; /* this is set later */
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ memset(my_command->argv, 0, sizeof(my_command->argv));
+ my_command->last_cmd = numqueries;
+ my_command->gset = pg_malloc0(sizeof(char *) * numqueries);
+ my_command->expr = NULL;
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
-
- /*
- * If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
- */
- nlpos = strchr(p, '\n');
- if (nlpos)
- {
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
- }
- else
- my_command->line = pg_strdup(p);
-
return my_command;
}
/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int numqueries)
+{
+ int nc;
+
+ Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
+
+ more = skip_sql_comments(more);
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ appendPQExpBuffer(&my_command->lines, ";\n%s", more);
+
+ /* update number of commands and extend array of prefixes */
+ nc = my_command->last_cmd + 1 + numqueries;
+ /* XXX don't reallocate one by one; use exponential grow as everyhere else */
+ my_command->gset =
+ pg_realloc(my_command->gset, sizeof(char *) * (nc + 1));
+ memset(my_command->gset + my_command->last_cmd + 1, 0,
+ sizeof(char *) * (numqueries + 1));
+ my_command->last_cmd = nc;
+}
+
+/* XXX explain what we do here */
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char buffer[128];
+
+ Assert(my_command->type == SQL_COMMAND);
+
+ /* Save the first line for error display. */
+ strlcpy(buffer, my_command->lines.data, sizeof(buffer));
+ buffer[strcspn(buffer, "\n\r")] = '\0';
+ my_command->first_line = pg_strdup(buffer);
+
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines.data;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
+}
+
+/*
* Parse a backslash command; return a Command struct, or NULL if comment
*
* At call, we have scanned only the initial backslash.
@@ -4177,7 +4319,6 @@ process_backslash_command(PsqlScanState sstate, const char *source)
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
my_command->type = META_COMMAND;
my_command->argc = 0;
initSimpleStats(&my_command->stats);
@@ -4201,7 +4342,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4222,10 +4363,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4238,7 +4380,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4247,19 +4389,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4288,7 +4431,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4296,25 +4439,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4393,6 +4543,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4416,6 +4569,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4425,31 +4579,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
+ int semicolons;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
sr = psql_scan(sstate, &line_buf, &prompt);
+ semicolons = psql_scan_get_escaped_semicolons(sstate);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* the previous multi-line command ended with \cset */
+ append_sql_command(ps.commands[index - 1], line_buf.data,
+ semicolons + 1);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, semicolons + 1);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4464,6 +4615,68 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char *bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ bool is_gset = bs_cmd[0] == 'g';
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = !is_gset;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index - 1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->last_cmd;
+
+ if (sql_cmd->gset[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->gset[cindex] = "";
+ else
+ sql_cmd->gset[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4819,7 +5032,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5286,28 +5499,18 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initialization and compute total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+
+ for (int j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 89678e7f3f..0eaf18c2f8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -528,6 +528,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -735,21 +777,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 9ea32c319f..321744cddb 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count semicolons in compound commands */
+ cur_state->escaped_semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1065,6 +1072,9 @@ psql_scan(PsqlScanState state,
/* Set current output target */
state->output_buf = query_buf;
+ /* Reset number of escaped semicolons seen */
+ state->escaped_semicolons = 0;
+
/* Set input source */
if (state->buffer_stack != NULL)
yy_switch_to_buffer(state->buffer_stack->buf, state->scanner);
@@ -1209,6 +1219,16 @@ psql_scan_reset(PsqlScanState state)
}
/*
+ * Return the number of escaped semicolons in the lexed string seen by the
+ * previous psql_scan call.
+ */
+int
+psql_scan_get_escaped_semicolons(PsqlScanState state)
+{
+ return state->escaped_semicolons;
+}
+
+/*
* Reselect this lexer (psqlscan.l) after using another one.
*
* Currently and for foreseeable uses, it's sufficient to reset to INITIAL
diff --git a/src/include/fe_utils/psqlscan.h b/src/include/fe_utils/psqlscan.h
index 1cf5b2e7fa..d6fef9ff77 100644
--- a/src/include/fe_utils/psqlscan.h
+++ b/src/include/fe_utils/psqlscan.h
@@ -90,6 +90,8 @@ extern PsqlScanResult psql_scan(PsqlScanState state,
extern void psql_scan_reset(PsqlScanState state);
+extern int psql_scan_get_escaped_semicolons(PsqlScanState state);
+
extern void psql_scan_reselect_sql_lexer(PsqlScanState state);
extern bool psql_scan_in_quote(PsqlScanState state);
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 42a738f422..752cc9406a 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int escaped_semicolons; /* number of embedded (\;) semicolons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-06 18:29 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2019-01-06 18:29 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Hello Alvaro,
> I revised this patch a bit. Here's v25, where some finishing touches
> are needed -- see below. I think with these changes the patch would
> become committable, at least for me.
Thanks a lot for having a look a this patch, and improving it.
The updated version did not work, though, but the fix was easy.
I do not understand why you have removed the num_commands stuff, which
indeed is not very useful but could be for debugging. No big deal.
> I didn't like that you were adding an #include of psqlscan_int.h into
> pgbench.c, when there's a specific comment in the header saying not to
> do that,
Oops, I did not notice the comment.
> so I opted for adding a new accessor function on psqlscan.h.
Ok.
> I renamed some of your parameter additions. I think the new names are
> clearer, but they meant the +1's in your code are now in illogical
> places. (I moved some; probably not enough). Please review/fix that.
It needed some fixing. I understood that you suggest to avoid keeping the
last index and prefer the number of elements instead, so I applied it to
the Command struct as well.
> I think "gset" is not a great name for the new struct member;
Indeed.
> please find a better name. I suggest "targetvar" but feel free to
> choose a name that suits your taste.
Ok. Note that it is not a variable name but a prefix, so I named it
"prefix".
> There are two XXX comments. One is about a function needing a comment
> atop it.
Ok.
> The other is about realloc behavior. To fix this one I would add a new
> struct member indicating the allocated size of the array, then growing
> exponentially instead of one at a time. For most cases you can probably
> get away with never reallocating beyond an initial allocation of, say, 8
> members.
Yep. I guess I did it 1 by 1 because it should be a rare case and it was
saving a counter. I've done some exponential thing instead.
> In the docs, the [prefix] part needs to be explained in the \cset entry;
> right now it's in \gset, which comes afterwards. Let's move the
> explanation up, and then in \gset say "prefix behaves as in \cset".
I do not understand, the very same explanation text about prefix appears
in both entry? The examples are different, is that what you mean? They are
about different backslash commands, so they have an interest of their own.
Attached a v26 with I hope everything ok, but for the documentation that
I'm unsure how to improve.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-26.patch (31.0K, ../../alpine.DEB.2.21.1901061737560.30093@lancre/2-pgbench-into-26.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..246944ea79 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,87 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends three queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the second query is stored into variable <replaceable>two</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of 3 queries
+SELECT 1 AS one \; SELECT 2 AS two \cset
+SELECT 2;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 649297ae4f..4b8b6ba10f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -499,14 +499,17 @@ typedef enum QueryMode
static QueryMode querymode = QUERY_SIMPLE;
static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
-typedef struct
+typedef struct Command
{
- char *line; /* text of command line */
- int command_num; /* unique index of this Command struct */
+ PQExpBufferData lines; /* raw command text (possibly multi-line) */
+ char *first_line; /* first line for short display */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int nqueries; /* number of compounds within an SQL command */
+ int prefix_size; /* allocated size of prefix, >= nqueries */
+ char **prefix; /* per-compound command prefix for [cg]set */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -521,7 +524,6 @@ typedef struct ParsedScript
static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
-static int num_commands = 0; /* total number of Command structs */
static int64 total_weight = 0;
static int debug = 0; /* debug flag */
@@ -1732,6 +1734,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **prefix)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (prefix[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (prefix[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res);
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (int f = 0; f < nfields; f++)
+ {
+ char *varname = PQfname(res, f);
+
+ /* prefix varname if required, will be freed below */
+ if (*prefix[compound] != '\0')
+ varname = psprintf("%s%s", prefix[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*prefix[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2862,8 +2965,6 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
for (;;)
{
- PGresult *res;
-
switch (st->state)
{
/* Select transaction (script) to run. */
@@ -3141,24 +3242,11 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /* Read and discard the query result */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, sql_script[st->use_file].commands[st->command]->prefix))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
break;
/*
@@ -3191,7 +3279,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
INSTR_TIME_SET_CURRENT_LAZY(now);
- command = sql_script[st->use_file].commands[st->command];
+ command = sql_script[st->use_file].commands[st->command];
/* XXX could use a mutex here, but we choose not to */
addToSimpleStats(&command->stats,
INSTR_TIME_GET_DOUBLE(now) -
@@ -3235,9 +3323,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->state = CSTATE_CHOOSE_SCRIPT;
/*
- * Ensure that we always return on this point, so as to
- * avoid an infinite loop if the script only contains meta
- * commands.
+ * Ensure that we always return on this point, so as to avoid
+ * an infinite loop if the script only contains meta commands.
*/
return;
@@ -3976,7 +4063,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3984,12 +4071,9 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
-
cmd->argc = 1;
- p = sql;
+ p = sql = pg_strdup(cmd->lines.data);
while ((p = strchr(p, ':')) != NULL)
{
char var[13];
@@ -4009,7 +4093,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines.data);
pg_free(name);
return false;
}
@@ -4021,7 +4105,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4081,21 +4165,16 @@ syntax_error(const char *source, int lineno,
}
/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
+ * Return a pointer to the start of the SQL command, after skipping over
+ * whitespace and "--" comments.
+ * If the end of the string is reached, return NULL.
*/
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *sql_command)
{
- Command *my_command;
- char *p;
- char *nlpos;
+ char *p = sql_command;
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4111,41 +4190,114 @@ process_sql_command(PQExpBuffer buf, const char *source)
break;
}
- /* If there's nothing but whitespace and comments, we're done */
+ /* NULL if there's nothing but whitespace and comments */
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int numqueries)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
- my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
+ my_command = (Command *) pg_malloc(sizeof(Command));
+ initPQExpBuffer(&my_command->lines);
+ appendPQExpBufferStr(&my_command->lines, p);
+ my_command->first_line = NULL; /* this is set later */
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ memset(my_command->argv, 0, sizeof(my_command->argv));
+ my_command->nqueries = numqueries;
+ my_command->prefix_size = numqueries + 8;
+ my_command->prefix = pg_malloc0(sizeof(char *) * my_command->prefix_size);
+ my_command->expr = NULL;
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
-
- /*
- * If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
- */
- nlpos = strchr(p, '\n');
- if (nlpos)
- {
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
- }
- else
- my_command->line = pg_strdup(p);
-
return my_command;
}
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int numqueries)
+{
+ int nq;
+
+ Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
+
+ more = skip_sql_comments(more);
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ appendPQExpBuffer(&my_command->lines, ";\n%s", more);
+
+ /* update number of commands and extend array of prefixes */
+ nq = my_command->nqueries + numqueries;
+ if (nq > my_command->prefix_size)
+ {
+ my_command->prefix_size = 2 * nq;
+ my_command->prefix =
+ pg_realloc(my_command->prefix,
+ sizeof(char *) * my_command->prefix_size);
+ }
+
+ /* fill with NULL */
+ memset(my_command->prefix + my_command->nqueries, 0,
+ sizeof(char *) * numqueries);
+ my_command->nqueries = nq;
+}
+
+/*
+ * Once an SQL command is fully parsed, possibly by accumulating several
+ * parts, complete other fields of the Command structure.
+ */
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char buffer[128];
+
+ Assert(my_command->type == SQL_COMMAND);
+
+ /* Save the first line for error display. */
+ strlcpy(buffer, my_command->lines.data, sizeof(buffer));
+ buffer[strcspn(buffer, "\n\r")] = '\0';
+ my_command->first_line = pg_strdup(buffer);
+
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines.data;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
+}
+
/*
* Parse a backslash command; return a Command struct, or NULL if comment
*
@@ -4177,7 +4329,6 @@ process_backslash_command(PsqlScanState sstate, const char *source)
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
my_command->type = META_COMMAND;
my_command->argc = 0;
initSimpleStats(&my_command->stats);
@@ -4201,7 +4352,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4222,10 +4373,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4238,7 +4390,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4247,19 +4399,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4288,7 +4441,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4296,25 +4449,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4393,6 +4553,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4416,6 +4579,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4425,31 +4589,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
+ int semicolons;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
sr = psql_scan(sstate, &line_buf, &prompt);
+ semicolons = psql_scan_get_escaped_semicolons(sstate);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* the previous multi-line command ended with \cset */
+ append_sql_command(ps.commands[index - 1], line_buf.data,
+ semicolons + 1);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, semicolons + 1);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4464,6 +4625,68 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char *bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ bool is_gset = bs_cmd[0] == 'g';
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = !is_gset;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index - 1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->nqueries - 1;
+
+ if (sql_cmd->prefix[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->prefix[cindex] = "";
+ else
+ sql_cmd->prefix[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4819,7 +5042,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5286,28 +5509,18 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initialization and compute total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+
+ for (int j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 89678e7f3f..0eaf18c2f8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -528,6 +528,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -735,21 +777,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 9ea32c319f..321744cddb 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count semicolons in compound commands */
+ cur_state->escaped_semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1065,6 +1072,9 @@ psql_scan(PsqlScanState state,
/* Set current output target */
state->output_buf = query_buf;
+ /* Reset number of escaped semicolons seen */
+ state->escaped_semicolons = 0;
+
/* Set input source */
if (state->buffer_stack != NULL)
yy_switch_to_buffer(state->buffer_stack->buf, state->scanner);
@@ -1208,6 +1218,16 @@ psql_scan_reset(PsqlScanState state)
state->dolqstart = NULL;
}
+/*
+ * Return the number of escaped semicolons in the lexed string seen by the
+ * previous psql_scan call.
+ */
+int
+psql_scan_get_escaped_semicolons(PsqlScanState state)
+{
+ return state->escaped_semicolons;
+}
+
/*
* Reselect this lexer (psqlscan.l) after using another one.
*
diff --git a/src/include/fe_utils/psqlscan.h b/src/include/fe_utils/psqlscan.h
index 1cf5b2e7fa..d6fef9ff77 100644
--- a/src/include/fe_utils/psqlscan.h
+++ b/src/include/fe_utils/psqlscan.h
@@ -90,6 +90,8 @@ extern PsqlScanResult psql_scan(PsqlScanState state,
extern void psql_scan_reset(PsqlScanState state);
+extern int psql_scan_get_escaped_semicolons(PsqlScanState state);
+
extern void psql_scan_reselect_sql_lexer(PsqlScanState state);
extern bool psql_scan_in_quote(PsqlScanState state);
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 42a738f422..752cc9406a 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int escaped_semicolons; /* number of embedded (\;) semicolons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-09 07:16 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2019-01-09 07:16 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
> Attached a v26 with I hope everything ok, but for the documentation that I'm
> unsure how to improve.
Attached v27 is the same but with an improved documentation where full
examples, with and without prefix, are provided for both cset & gset.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-27.patch (31.2K, ../../alpine.DEB.2.21.1901090814400.6213@lancre/2-pgbench-into-27.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..cc369c423e 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,91 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends four queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the first query is stored into variable <replaceable>one</replaceable>,
+ the results of the third query are stored into variables <replaceable>z_three</replaceable>
+ and <replaceable>z_four</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of four queries
+SELECT 1 AS one \cset
+SELECT 2 AS two \;
+SELECT 3 AS three, 4 AS four \cset z_
+SELECT 5;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 649297ae4f..4b8b6ba10f 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -499,14 +499,17 @@ typedef enum QueryMode
static QueryMode querymode = QUERY_SIMPLE;
static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
-typedef struct
+typedef struct Command
{
- char *line; /* text of command line */
- int command_num; /* unique index of this Command struct */
+ PQExpBufferData lines; /* raw command text (possibly multi-line) */
+ char *first_line; /* first line for short display */
int type; /* command type (SQL_COMMAND or META_COMMAND) */
MetaCommand meta; /* meta command identifier, or META_NONE */
int argc; /* number of command words */
char *argv[MAX_ARGS]; /* command word list */
+ int nqueries; /* number of compounds within an SQL command */
+ int prefix_size; /* allocated size of prefix, >= nqueries */
+ char **prefix; /* per-compound command prefix for [cg]set */
PgBenchExpr *expr; /* parsed expression, if needed */
SimpleStats stats; /* time spent in this command */
} Command;
@@ -521,7 +524,6 @@ typedef struct ParsedScript
static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
-static int num_commands = 0; /* total number of Command structs */
static int64 total_weight = 0;
static int debug = 0; /* debug flag */
@@ -1732,6 +1734,107 @@ valueTruth(PgBenchValue *pval)
}
}
+/* read all responses from backend, storing into variable or discarding */
+static bool
+read_response(CState *st, char **prefix)
+{
+ PGresult *res;
+ int compound = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (prefix[compound] != NULL)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "\\gset/cset expects a row\n",
+ st->id, st->use_file, st->command, compound);
+ st->ecnt++;
+ return false;
+ }
+ break; /* OK */
+
+ case PGRES_TUPLES_OK:
+ if (prefix[compound] != NULL)
+ {
+ /* store result into variables if required */
+ int ntuples = PQntuples(res),
+ nfields = PQnfields(res);
+
+ if (ntuples != 1)
+ {
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "expecting one row, got %d\n",
+ st->id, st->use_file, st->command, compound, ntuples);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ for (int f = 0; f < nfields; f++)
+ {
+ char *varname = PQfname(res, f);
+
+ /* prefix varname if required, will be freed below */
+ if (*prefix[compound] != '\0')
+ varname = psprintf("%s%s", prefix[compound], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, f)))
+ {
+ /* internal error, should it rather abort? */
+ fprintf(stderr,
+ "client %d file %d command %d compound %d: "
+ "error storing into var %s\n",
+ st->id, st->use_file, st->command, compound,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* free varname only if allocated because of prefix */
+ if (*prefix[compound] != '\0')
+ free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break; /* OK */
+
+ default:
+ /* everything else is unexpected, so probably an error */
+ fprintf(stderr,
+ "client %d file %d aborted in command %d compound %d: %s",
+ st->id, st->use_file, st->command, compound,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ compound += 1;
+ }
+
+ if (compound == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2862,8 +2965,6 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
for (;;)
{
- PGresult *res;
-
switch (st->state)
{
/* Select transaction (script) to run. */
@@ -3141,24 +3242,11 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /* Read and discard the query result */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* read, store or discard the query results */
+ if (read_response(st, sql_script[st->use_file].commands[st->command]->prefix))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
break;
/*
@@ -3191,7 +3279,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
INSTR_TIME_SET_CURRENT_LAZY(now);
- command = sql_script[st->use_file].commands[st->command];
+ command = sql_script[st->use_file].commands[st->command];
/* XXX could use a mutex here, but we choose not to */
addToSimpleStats(&command->stats,
INSTR_TIME_GET_DOUBLE(now) -
@@ -3235,9 +3323,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->state = CSTATE_CHOOSE_SCRIPT;
/*
- * Ensure that we always return on this point, so as to
- * avoid an infinite loop if the script only contains meta
- * commands.
+ * Ensure that we always return on this point, so as to avoid
+ * an infinite loop if the script only contains meta commands.
*/
return;
@@ -3976,7 +4063,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3984,12 +4071,9 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
-
cmd->argc = 1;
- p = sql;
+ p = sql = pg_strdup(cmd->lines.data);
while ((p = strchr(p, ':')) != NULL)
{
char var[13];
@@ -4009,7 +4093,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines.data);
pg_free(name);
return false;
}
@@ -4021,7 +4105,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4081,21 +4165,16 @@ syntax_error(const char *source, int lineno,
}
/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
+ * Return a pointer to the start of the SQL command, after skipping over
+ * whitespace and "--" comments.
+ * If the end of the string is reached, return NULL.
*/
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *sql_command)
{
- Command *my_command;
- char *p;
- char *nlpos;
+ char *p = sql_command;
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4111,41 +4190,114 @@ process_sql_command(PQExpBuffer buf, const char *source)
break;
}
- /* If there's nothing but whitespace and comments, we're done */
+ /* NULL if there's nothing but whitespace and comments */
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comment and set up a
+ * Command struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int numqueries)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
- my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
+ my_command = (Command *) pg_malloc(sizeof(Command));
+ initPQExpBuffer(&my_command->lines);
+ appendPQExpBufferStr(&my_command->lines, p);
+ my_command->first_line = NULL; /* this is set later */
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ memset(my_command->argv, 0, sizeof(my_command->argv));
+ my_command->nqueries = numqueries;
+ my_command->prefix_size = numqueries + 8;
+ my_command->prefix = pg_malloc0(sizeof(char *) * my_command->prefix_size);
+ my_command->expr = NULL;
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
-
- /*
- * If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
- */
- nlpos = strchr(p, '\n');
- if (nlpos)
- {
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
- }
- else
- my_command->line = pg_strdup(p);
-
return my_command;
}
+/*
+ * append "more" text to current compound command which may have been
+ * interrupted by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int numqueries)
+{
+ int nq;
+
+ Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
+
+ more = skip_sql_comments(more);
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ appendPQExpBuffer(&my_command->lines, ";\n%s", more);
+
+ /* update number of commands and extend array of prefixes */
+ nq = my_command->nqueries + numqueries;
+ if (nq > my_command->prefix_size)
+ {
+ my_command->prefix_size = 2 * nq;
+ my_command->prefix =
+ pg_realloc(my_command->prefix,
+ sizeof(char *) * my_command->prefix_size);
+ }
+
+ /* fill with NULL */
+ memset(my_command->prefix + my_command->nqueries, 0,
+ sizeof(char *) * numqueries);
+ my_command->nqueries = nq;
+}
+
+/*
+ * Once an SQL command is fully parsed, possibly by accumulating several
+ * parts, complete other fields of the Command structure.
+ */
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char buffer[128];
+
+ Assert(my_command->type == SQL_COMMAND);
+
+ /* Save the first line for error display. */
+ strlcpy(buffer, my_command->lines.data, sizeof(buffer));
+ buffer[strcspn(buffer, "\n\r")] = '\0';
+ my_command->first_line = pg_strdup(buffer);
+
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines.data;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
+}
+
/*
* Parse a backslash command; return a Command struct, or NULL if comment
*
@@ -4177,7 +4329,6 @@ process_backslash_command(PsqlScanState sstate, const char *source)
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
my_command->type = META_COMMAND;
my_command->argc = 0;
initSimpleStats(&my_command->stats);
@@ -4201,7 +4352,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4222,10 +4373,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4238,7 +4390,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4247,19 +4399,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4288,7 +4441,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4296,25 +4449,32 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
+ pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "at most one argument expected", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4393,6 +4553,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool is_compound = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4416,6 +4579,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4425,31 +4589,28 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
+ int semicolons;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
sr = psql_scan(sstate, &line_buf, &prompt);
+ semicolons = psql_scan_get_escaped_semicolons(sstate);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ if (is_compound)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* the previous multi-line command ended with \cset */
+ append_sql_command(ps.commands[index - 1], line_buf.data,
+ semicolons + 1);
+ is_compound = false;
}
-
- /* If we reached a backslash, process that */
- if (sr == PSCAN_BACKSLASH)
+ else
{
- command = process_backslash_command(sstate, desc);
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, semicolons + 1);
+
+ /* store new command */
if (command)
{
ps.commands[index] = command;
@@ -4464,6 +4625,68 @@ ParseScript(const char *script, const char *desc, int weight)
}
}
+ if (sr == PSCAN_BACKSLASH)
+ {
+ command = process_backslash_command(sstate, desc);
+
+ if (command)
+ {
+ char *bs_cmd = command->argv[0];
+
+ /* merge gset variants into preceeding SQL command */
+ if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
+ pg_strcasecmp(bs_cmd, "cset") == 0)
+ {
+ bool is_gset = bs_cmd[0] == 'g';
+ int cindex;
+ Command *sql_cmd;
+
+ is_compound = !is_gset;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ sql_cmd = ps.commands[index - 1];
+
+ if (sql_cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ sql_cmd->first_line, -1);
+
+ /* this \gset applies to the last sub-command */
+ cindex = sql_cmd->nqueries - 1;
+
+ if (sql_cmd->prefix[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ sql_cmd->prefix[cindex] = "";
+ else
+ sql_cmd->prefix[cindex] = command->argv[1];
+
+ /* cleanup unused backslash command */
+ pg_free(command);
+ }
+ else /* any other backslash command is a Command */
+ {
+ ps.commands[index] = command;
+ index++;
+
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+ }
+ }
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4819,7 +5042,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5286,28 +5509,18 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initialization and compute total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+
+ for (int j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 89678e7f3f..0eaf18c2f8 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -528,6 +528,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -735,21 +777,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many args', 1,
+ [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{gset/cset expects a row}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into var \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into var bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 9ea32c319f..321744cddb 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count semicolons in compound commands */
+ cur_state->escaped_semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1065,6 +1072,9 @@ psql_scan(PsqlScanState state,
/* Set current output target */
state->output_buf = query_buf;
+ /* Reset number of escaped semicolons seen */
+ state->escaped_semicolons = 0;
+
/* Set input source */
if (state->buffer_stack != NULL)
yy_switch_to_buffer(state->buffer_stack->buf, state->scanner);
@@ -1208,6 +1218,16 @@ psql_scan_reset(PsqlScanState state)
state->dolqstart = NULL;
}
+/*
+ * Return the number of escaped semicolons in the lexed string seen by the
+ * previous psql_scan call.
+ */
+int
+psql_scan_get_escaped_semicolons(PsqlScanState state)
+{
+ return state->escaped_semicolons;
+}
+
/*
* Reselect this lexer (psqlscan.l) after using another one.
*
diff --git a/src/include/fe_utils/psqlscan.h b/src/include/fe_utils/psqlscan.h
index 1cf5b2e7fa..d6fef9ff77 100644
--- a/src/include/fe_utils/psqlscan.h
+++ b/src/include/fe_utils/psqlscan.h
@@ -90,6 +90,8 @@ extern PsqlScanResult psql_scan(PsqlScanState state,
extern void psql_scan_reset(PsqlScanState state);
+extern int psql_scan_get_escaped_semicolons(PsqlScanState state);
+
extern void psql_scan_reselect_sql_lexer(PsqlScanState state);
extern bool psql_scan_in_quote(PsqlScanState state);
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 42a738f422..752cc9406a 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int escaped_semicolons; /* number of embedded (\;) semicolons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-09 13:41 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2019-01-09 13:41 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
On 2019-Jan-09, Fabien COELHO wrote:
> > Attached a v26 with I hope everything ok, but for the documentation that
> > I'm unsure how to improve.
>
> Attached v27 is the same but with an improved documentation where full
> examples, with and without prefix, are provided for both cset & gset.
I have already made changes on top of v26. Can you please submit this
as a delta patch on top of v26?
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-09 15:31 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2019-01-09 15:31 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
>> Attached v27 is the same but with an improved documentation where full
>> examples, with and without prefix, are provided for both cset & gset.
>
> I have already made changes on top of v26. Can you please submit this
> as a delta patch on top of v26?
Attached.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-26-to-27.patch (1.2K, ../../alpine.DEB.2.21.1901091625120.6213@lancre/2-pgbench-into-26-to-27.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index 246944ea79..cc369c423e 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -972,14 +972,18 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<para>
- The following example sends three queries as one compound SQL command,
+ The following example sends four queries as one compound SQL command,
inducing one message sent at the protocol level.
- The result of the second query is stored into variable <replaceable>two</replaceable>,
+ The result of the first query is stored into variable <replaceable>one</replaceable>,
+ the results of the third query are stored into variables <replaceable>z_three</replaceable>
+ and <replaceable>z_four</replaceable>,
whereas the results of the other queries are discarded.
<programlisting>
--- compound of 3 queries
-SELECT 1 AS one \; SELECT 2 AS two \cset
-SELECT 2;
+-- compound of four queries
+SELECT 1 AS one \cset
+SELECT 2 AS two \;
+SELECT 3 AS three, 4 AS four \cset z_
+SELECT 5;
</programlisting>
</para>
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-09 23:00 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2019-01-09 23:00 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Here are my proposed final changes. I noticed that you were allocating
the prefixes for all cases even when there were no cset/gset in the
command; I changed it to delay the allocation until needed. I also
noticed you were skipping the Meta enum dance for no good reason; added
that makes conditionals simpler. The read_response routine seemed
misplaced and I gave it a name in a style closer to the rest of the
pgbench code. Also, you were missing to free the ->lines pqexpbuffer
when the command is discarded. I grant that the free_command() stuff
might be bogus since it's only tested with a command that's barely
initialized, but it seems better to make it bogus in this way (fixable
if we ever extend its use) than to forever leak memory silently.
I didn't test this beyond running "make check".
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
[text/x-diff] v27-delta.patch (20.1K, ../../201901092300.lmmnu2abxmuq@alvherre.pgsql/2-v27-delta.patch)
download | inline diff:
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 4b8b6ba10f..3e111876fb 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -482,6 +482,8 @@ typedef enum MetaCommand
META_SETSHELL, /* \setshell */
META_SHELL, /* \shell */
META_SLEEP, /* \sleep */
+ META_CSET, /* \cset */
+ META_GSET, /* \gset */
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
@@ -499,19 +501,39 @@ typedef enum QueryMode
static QueryMode querymode = QUERY_SIMPLE;
static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
+/*
+ * struct Command represents one command in a script.
+ *
+ * lines The raw, possibly multi-line command text. Variable substitution
+ * not applied.
+ * first_line A short, single-line extract of 'lines', for error reporting.
+ * type SQL_COMMAND or META_COMMAND
+ * meta The type of meta-command, or META_NONE if command is SQL
+ * argc Number of arguments of the command, 0 if not yet processed.
+ * argv Command arguments, the first of which is the command or SQL
+ * string itself. For SQL commands, after post-processing
+ * argv[0] is the same as 'lines' with variables substituted.
+ * nqueries In a multi-command SQL line, the number of queries.
+ * varprefix SQL commands terminated with \gset or \cset have this set
+ * to a non NULL value. If nonempty, it's used to prefix the
+ * variable name that receives the value.
+ * varprefix_max Allocated size of the varprefix array.
+ * expr Parsed expression, if needed.
+ * stats Time spent in this command.
+ */
typedef struct Command
{
- PQExpBufferData lines; /* raw command text (possibly multi-line) */
- char *first_line; /* first line for short display */
- int type; /* command type (SQL_COMMAND or META_COMMAND) */
- MetaCommand meta; /* meta command identifier, or META_NONE */
- int argc; /* number of command words */
- char *argv[MAX_ARGS]; /* command word list */
- int nqueries; /* number of compounds within an SQL command */
- int prefix_size; /* allocated size of prefix, >= nqueries */
- char **prefix; /* per-compound command prefix for [cg]set */
- PgBenchExpr *expr; /* parsed expression, if needed */
- SimpleStats stats; /* time spent in this command */
+ PQExpBufferData lines;
+ char *first_line;
+ int type;
+ MetaCommand meta;
+ int argc;
+ char *argv[MAX_ARGS];
+ int nqueries;
+ char **varprefix;
+ int varprefix_max;
+ PgBenchExpr *expr;
+ SimpleStats stats;
} Command;
typedef struct ParsedScript
@@ -589,6 +611,7 @@ static void doLog(TState *thread, CState *st,
static void processXactStats(TState *thread, CState *st, instr_time *now,
bool skipped, StatsData *agg);
static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
+static void allocate_command_varprefix(Command *cmd, int totalqueries);
static void addScript(ParsedScript script);
static void *threadRun(void *arg);
static void finishCon(CState *st);
@@ -1734,107 +1757,6 @@ valueTruth(PgBenchValue *pval)
}
}
-/* read all responses from backend, storing into variable or discarding */
-static bool
-read_response(CState *st, char **prefix)
-{
- PGresult *res;
- int compound = 0;
-
- while ((res = PQgetResult(st->con)) != NULL)
- {
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK: /* non-SELECT commands */
- case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
- if (prefix[compound] != NULL)
- {
- fprintf(stderr,
- "client %d file %d command %d compound %d: "
- "\\gset/cset expects a row\n",
- st->id, st->use_file, st->command, compound);
- st->ecnt++;
- return false;
- }
- break; /* OK */
-
- case PGRES_TUPLES_OK:
- if (prefix[compound] != NULL)
- {
- /* store result into variables if required */
- int ntuples = PQntuples(res),
- nfields = PQnfields(res);
-
- if (ntuples != 1)
- {
- fprintf(stderr,
- "client %d file %d command %d compound %d: "
- "expecting one row, got %d\n",
- st->id, st->use_file, st->command, compound, ntuples);
- st->ecnt++;
- PQclear(res);
- discard_response(st);
- return false;
- }
-
- for (int f = 0; f < nfields; f++)
- {
- char *varname = PQfname(res, f);
-
- /* prefix varname if required, will be freed below */
- if (*prefix[compound] != '\0')
- varname = psprintf("%s%s", prefix[compound], varname);
-
- /* store result as a string */
- if (!putVariable(st, "gset", varname,
- PQgetvalue(res, 0, f)))
- {
- /* internal error, should it rather abort? */
- fprintf(stderr,
- "client %d file %d command %d compound %d: "
- "error storing into var %s\n",
- st->id, st->use_file, st->command, compound,
- varname);
- st->ecnt++;
- PQclear(res);
- discard_response(st);
- return false;
- }
-
- /* free varname only if allocated because of prefix */
- if (*prefix[compound] != '\0')
- free(varname);
- }
- }
- /* otherwise the result is simply thrown away by PQclear below */
- break; /* OK */
-
- default:
- /* everything else is unexpected, so probably an error */
- fprintf(stderr,
- "client %d file %d aborted in command %d compound %d: %s",
- st->id, st->use_file, st->command, compound,
- PQerrorMessage(st->con));
- st->ecnt++;
- PQclear(res);
- discard_response(st);
- return false;
- }
-
- PQclear(res);
- compound += 1;
- }
-
- if (compound == 0)
- {
- fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
- st->ecnt++;
- return false;
- }
-
- return true;
-}
-
/* get a value as an int, tell if there is a problem */
static bool
coerceToInt(PgBenchValue *pval, int64 *ival)
@@ -2672,6 +2594,10 @@ getMetaCommand(const char *cmd)
mc = META_ELSE;
else if (pg_strcasecmp(cmd, "endif") == 0)
mc = META_ENDIF;
+ else if (pg_strcasecmp(cmd, "cset") == 0)
+ mc = META_CSET;
+ else if (pg_strcasecmp(cmd, "gset") == 0)
+ mc = META_GSET;
else
mc = META_NONE;
return mc;
@@ -2900,6 +2826,109 @@ sendCommand(CState *st, Command *command)
}
/*
+ * Process query response from the backend.
+ *
+ * If varprefix is not NULL, it's the array of variable prefix names where to
+ * store the results.
+ *
+ * Returns true if everything is A-OK, false if any error occurs.
+ */
+static bool
+readCommandResponse(CState *st, char **varprefix)
+{
+ PGresult *res;
+ int qrynum = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (varprefix && varprefix[qrynum] != NULL)
+ {
+ fprintf(stderr,
+ "client %d script %d command %d query %d: expected one row, got %d\n",
+ st->id, st->use_file, st->command, qrynum, 0);
+ st->ecnt++;
+ return false;
+ }
+ break;
+
+ case PGRES_TUPLES_OK:
+ if (varprefix && varprefix[qrynum] != NULL)
+ {
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr,
+ "client %d script %d command %d query %d: expected one row, got %d\n",
+ st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* store results into variables */
+ for (int fld = 0; fld < PQnfields(res); fld++)
+ {
+ char *varname = PQfname(res, fld);
+
+ /* allocate varname only if necessary, freed below */
+ if (*varprefix[qrynum] != '\0')
+ varname =
+ psprintf("%s%s", varprefix[qrynum], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, fld)))
+ {
+ /* internal error */
+ fprintf(stderr,
+ "client %d script %d command %d query %d: error storing into variable %s\n",
+ st->id, st->use_file, st->command, qrynum,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*varprefix[qrynum] != '\0')
+ pg_free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break;
+
+ default:
+ /* anything else is unexpected */
+ fprintf(stderr,
+ "client %d script %d aborted in command %d query %d: %s",
+ st->id, st->use_file, st->command, qrynum,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ qrynum++;
+ }
+
+ if (qrynum == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
+
+/*
* Parse the argument to a \sleep command, and return the requested amount
* of delay, in microseconds. Returns true on success, false on error.
*/
@@ -3242,8 +3271,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /* read, store or discard the query results */
- if (read_response(st, sql_script[st->use_file].commands[st->command]->prefix))
+ /* store or discard the query results */
+ if (readCommandResponse(st, sql_script[st->use_file].commands[st->command]->varprefix))
st->state = CSTATE_END_COMMAND;
else
st->state = CSTATE_ABORTED;
@@ -4201,8 +4230,8 @@ skip_sql_comments(char *sql_command)
* Parse a SQL command; return a Command struct, or NULL if it's a comment
*
* On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
+ * really need to do much here except check for comments and set up a Command
+ * struct.
*/
static Command *
create_sql_command(PQExpBuffer buf, const char *source, int numqueries)
@@ -4217,29 +4246,44 @@ create_sql_command(PQExpBuffer buf, const char *source, int numqueries)
my_command = (Command *) pg_malloc(sizeof(Command));
initPQExpBuffer(&my_command->lines);
appendPQExpBufferStr(&my_command->lines, p);
- my_command->first_line = NULL; /* this is set later */
+ my_command->first_line = NULL; /* this is set later */
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
my_command->argc = 0;
memset(my_command->argv, 0, sizeof(my_command->argv));
my_command->nqueries = numqueries;
- my_command->prefix_size = numqueries + 8;
- my_command->prefix = pg_malloc0(sizeof(char *) * my_command->prefix_size);
+ my_command->varprefix = NULL; /* allocated later, if needed */
+ my_command->varprefix_max = 0;
my_command->expr = NULL;
initSimpleStats(&my_command->stats);
return my_command;
}
+/* Free a Command structure and associated data */
+static void
+free_command(Command *command)
+{
+ pg_free(command->lines.data);
+ if (command->first_line)
+ pg_free(command->first_line);
+ if (command->argv)
+ for (int i = 0; i < command->argc; i++)
+ pg_free(command->argv[i]);
+ if (command->varprefix)
+ pg_free(command->varprefix);
+ if (command->expr)
+ pg_free(command->expr);
+ pg_free(command);
+}
+
/*
- * append "more" text to current compound command which may have been
- * interrupted by \cset.
+ * append "more" text to current compound command which had been interrupted
+ * by \cset.
*/
static void
append_sql_command(Command *my_command, char *more, int numqueries)
{
- int nq;
-
Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
more = skip_sql_comments(more);
@@ -4249,20 +4293,7 @@ append_sql_command(Command *my_command, char *more, int numqueries)
/* append command text, embedding a ';' in place of the \cset */
appendPQExpBuffer(&my_command->lines, ";\n%s", more);
- /* update number of commands and extend array of prefixes */
- nq = my_command->nqueries + numqueries;
- if (nq > my_command->prefix_size)
- {
- my_command->prefix_size = 2 * nq;
- my_command->prefix =
- pg_realloc(my_command->prefix,
- sizeof(char *) * my_command->prefix_size);
- }
-
- /* fill with NULL */
- memset(my_command->prefix + my_command->nqueries, 0,
- sizeof(char *) * numqueries);
- my_command->nqueries = nq;
+ my_command->nqueries += numqueries;
}
/*
@@ -4272,7 +4303,7 @@ append_sql_command(Command *my_command, char *more, int numqueries)
static void
postprocess_sql_command(Command *my_command)
{
- char buffer[128];
+ char buffer[128];
Assert(my_command->type == SQL_COMMAND);
@@ -4299,6 +4330,35 @@ postprocess_sql_command(Command *my_command)
}
/*
+ * Determine the command's varprefix size needed and allocate memory for it
+ */
+static void
+allocate_command_varprefix(Command *cmd, int totalqueries)
+{
+ int new_max;
+
+ /* sufficient space already allocated? */
+ if (totalqueries <= cmd->varprefix_max)
+ return;
+
+ /* Determine the new array size. */
+ new_max = Max(cmd->varprefix_max, 2);
+ while (new_max < totalqueries)
+ new_max *= 2;
+
+ /* and enlarge the array, zero-initializing the allocated space */
+ if (cmd->varprefix == NULL)
+ cmd->varprefix = pg_malloc0(sizeof(char *) * new_max);
+ else
+ {
+ cmd->varprefix = pg_realloc(cmd->varprefix, sizeof(char *) * new_max);
+ memset(cmd->varprefix + cmd->varprefix_max, 0,
+ sizeof(char *) * (new_max - cmd->varprefix_max));
+ }
+ cmd->varprefix_max = new_max;
+}
+
+/*
* Parse a backslash command; return a Command struct, or NULL if comment
*
* At call, we have scanned only the initial backslash.
@@ -4464,12 +4524,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
- else if (pg_strcasecmp(my_command->argv[0], "gset") == 0 ||
- pg_strcasecmp(my_command->argv[0], "cset") == 0)
+ else if (my_command->meta == META_CSET || my_command->meta == META_GSET)
{
if (my_command->argc > 2)
syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
- "at most one argument expected", NULL, -1);
+ "too many arguments", NULL, -1);
}
else
{
@@ -4553,7 +4612,7 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
- bool is_compound = false;
+ bool saw_cset = false;
int lineno;
int start_offset;
@@ -4596,14 +4655,15 @@ ParseScript(const char *script, const char *desc, int weight)
lineno = expr_scanner_get_lineno(sstate, start_offset);
sr = psql_scan(sstate, &line_buf, &prompt);
+
semicolons = psql_scan_get_escaped_semicolons(sstate);
- if (is_compound)
+ if (saw_cset)
{
/* the previous multi-line command ended with \cset */
append_sql_command(ps.commands[index - 1], line_buf.data,
semicolons + 1);
- is_compound = false;
+ saw_cset = false;
}
else
{
@@ -4612,81 +4672,91 @@ ParseScript(const char *script, const char *desc, int weight)
/* store new command */
if (command)
- {
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
- }
+ ps.commands[index++] = command;
}
+ /* If we reached a backslash, process that */
if (sr == PSCAN_BACKSLASH)
{
command = process_backslash_command(sstate, desc);
if (command)
{
- char *bs_cmd = command->argv[0];
-
- /* merge gset variants into preceeding SQL command */
- if (pg_strcasecmp(bs_cmd, "gset") == 0 ||
- pg_strcasecmp(bs_cmd, "cset") == 0)
+ /*
+ * If this is gset/cset, merge into the preceding command. (We
+ * don't use a command slot in this case).
+ */
+ if (command->meta == META_CSET ||
+ command->meta == META_GSET)
{
- bool is_gset = bs_cmd[0] == 'g';
int cindex;
- Command *sql_cmd;
+ Command *cmd;
- is_compound = !is_gset;
+ /*
+ * If \cset is seen, set flag to leave the command pending
+ * for the next iteration to process.
+ */
+ saw_cset = command->meta == META_CSET;
if (index == 0)
syntax_error(desc, lineno, NULL, NULL,
"\\gset/cset cannot start a script",
NULL, -1);
- sql_cmd = ps.commands[index - 1];
+ cmd = ps.commands[index - 1];
- if (sql_cmd->type != SQL_COMMAND)
+ if (cmd->type != SQL_COMMAND)
syntax_error(desc, lineno, NULL, NULL,
"\\gset/cset must follow a SQL command",
- sql_cmd->first_line, -1);
+ cmd->first_line, -1);
- /* this \gset applies to the last sub-command */
- cindex = sql_cmd->nqueries - 1;
+ /* this {g,c}set applies to the previous query */
+ cindex = cmd->nqueries - 1;
- if (sql_cmd->prefix[cindex] != NULL)
+ /*
+ * now that we know there's a {g,c}set in this command,
+ * allocate space for the variable name prefix array.
+ */
+ allocate_command_varprefix(cmd, cmd->nqueries);
+
+ /*
+ * Don't allow the previous command be a gset/cset; that
+ * would make no sense.
+ */
+ if (cmd->varprefix[cindex] != NULL)
syntax_error(desc, lineno, NULL, NULL,
"\\gset/cset cannot follow one another",
NULL, -1);
/* get variable prefix */
if (command->argc <= 1 || command->argv[1][0] == '\0')
- sql_cmd->prefix[cindex] = "";
+ cmd->varprefix[cindex] = "";
else
- sql_cmd->prefix[cindex] = command->argv[1];
+ cmd->varprefix[cindex] = pg_strdup(command->argv[1]);
- /* cleanup unused backslash command */
- pg_free(command);
- }
- else /* any other backslash command is a Command */
- {
- ps.commands[index] = command;
- index++;
+ /* cleanup unused command */
+ free_command(command);
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ continue;
}
+
+ /* Attach any other backslash command as a new command */
+ ps.commands[index++] = command;
}
}
+ /*
+ * Since we used a command slot, allocate more if needed. Note we
+ * always allocate one more in order to accomodate the NULL terminator
+ * below.
+ */
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 0eaf18c2f8..c87748086a 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -782,27 +782,27 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
# GSET & CSET
[ 'gset no row', 2,
- [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [qr{expected one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
[ 'cset no row', 2,
- [qr{expecting one row, got 0\b}], q{SELECT WHERE FALSE \cset
+ [qr{expected one row, got 0\b}], q{SELECT WHERE FALSE \cset
SELECT 1 AS i\gset}, 1 ],
[ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
[ 'gset no SQL', 1,
[qr{gset/cset must follow a SQL command}], q{\set i +1
\gset} ],
- [ 'gset too many args', 1,
- [qr{at most one argument expected}], q{SELECT 1 \gset a b} ],
+ [ 'gset too many arguments', 1,
+ [qr{too many arguments}], q{SELECT 1 \gset a b} ],
[ 'gset after gset', 1,
[qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
\gset} ],
[ 'gset non SELECT', 2,
- [qr{gset/cset expects a row}],
+ [qr{expected one row, got 0}],
q{DROP TABLE IF EXISTS no_such_table \gset} ],
[ 'gset bad default name', 2,
- [qr{error storing into var \?column\?}],
+ [qr{error storing into variable \?column\?}],
q{SELECT 1 \gset} ],
[ 'gset bad name', 2,
- [qr{error storing into var bad name!}],
+ [qr{error storing into variable bad name!}],
q{SELECT 1 AS "bad name!" \gset} ],
);
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 09:12 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Fabien COELHO @ 2019-01-10 09:12 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Hello Alvaro,
> Here are my proposed final changes.
Thanks again for improving the patch!
> I noticed that you were allocating the prefixes for all cases even when
> there were no cset/gset in the command; I changed it to delay the
> allocation until needed.
Ok, why not.
> I also noticed you were skipping the Meta enum dance for no good reason;
Indeed. I think that the initial version of the patch was before the
"dance" was added, and it did not keep up with it.
> added that makes conditionals simpler. The read_response routine seemed
> misplaced and I gave it a name in a style closer to the rest of the
> pgbench code.
Fine.
> Also, you were missing to free the ->lines pqexpbuffer when the command
> is discarded. I grant that the free_command() stuff might be bogus
> since it's only tested with a command that's barely initialized, but it
> seems better to make it bogus in this way (fixable if we ever extend its
> use) than to forever leak memory silently.
Ok.
However, I switched "pg_free" to "termPQExpBuffer", which seems more
appropriate, even if it just does the same thing. I also ensured that
prefixes are allocated & freed. I've commented about expr which is not
freed.
> I didn't test this beyond running "make check".
That's a good start.
I'm not keen on having the command array size checked and updated *after*
the command is appended, even if the initial allocation ensures that there
is no overflow, but I let it as is.
Further tests did not yield any new issue.
Attached a v29 with the above minor changes wrt your version.
--
Fabien.
Attachments:
[text/x-diff] pgbench-into-29.patch (34.8K, ../../alpine.DEB.2.21.1901100923110.13058@lancre/2-pgbench-into-29.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml
index b5e3a62a33..cc369c423e 100644
--- a/doc/src/sgml/ref/pgbench.sgml
+++ b/doc/src/sgml/ref/pgbench.sgml
@@ -954,6 +954,91 @@ pgbench <optional> <replaceable>options</replaceable> </optional> <replaceable>d
</para>
<variablelist>
+ <varlistentry id='pgbench-metacommand-cset'>
+ <term>
+ <literal>\cset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This command may be used to end SQL queries, replacing an embedded
+ semicolon (<literal>\;</literal>) within a compound SQL command.
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example sends four queries as one compound SQL command,
+ inducing one message sent at the protocol level.
+ The result of the first query is stored into variable <replaceable>one</replaceable>,
+ the results of the third query are stored into variables <replaceable>z_three</replaceable>
+ and <replaceable>z_four</replaceable>,
+ whereas the results of the other queries are discarded.
+<programlisting>
+-- compound of four queries
+SELECT 1 AS one \cset
+SELECT 2 AS two \;
+SELECT 3 AS three, 4 AS four \cset z_
+SELECT 5;
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\cset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id='pgbench-metacommand-gset'>
+ <term>
+ <literal>\gset [<replaceable>prefix</replaceable>]</literal>
+ </term>
+
+ <listitem>
+ <para>
+ This commands may be used to end SQL queries, replacing a final semicolon
+ (<literal>;</literal>).
+ </para>
+
+ <para>
+ When this command is used, the preceding SQL query is expected to
+ return one row, the columns of which are stored into variables named after
+ column names, and prefixed with <replaceable>prefix</replaceable> if provided.
+ </para>
+
+ <para>
+ The following example puts the final account balance from the first query
+ into variable <replaceable>abalance</replaceable>, and fills variables
+ <replaceable>p_two</replaceable> and <replaceable>p_three</replaceable>
+ with integers from the last query.
+ The result of the second query is discarded.
+<programlisting>
+UPDATE pgbench_accounts
+ SET abalance = abalance + :delta
+ WHERE aid = :aid
+ RETURNING abalance \gset
+-- compound of two queries
+SELECT 1 \;
+SELECT 2 AS two, 3 AS three \gset p_
+</programlisting>
+ </para>
+
+ <note>
+ <para>
+ <literal>\gset</literal> does not work when empty SQL queries appear
+ within a compound SQL command.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>\if</literal> <replaceable class="parameter">expression</replaceable></term>
<term><literal>\elif</literal> <replaceable class="parameter">expression</replaceable></term>
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 649297ae4f..8bac939ff8 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -482,6 +482,8 @@ typedef enum MetaCommand
META_SETSHELL, /* \setshell */
META_SHELL, /* \shell */
META_SLEEP, /* \sleep */
+ META_CSET, /* \cset */
+ META_GSET, /* \gset */
META_IF, /* \if */
META_ELIF, /* \elif */
META_ELSE, /* \else */
@@ -499,16 +501,39 @@ typedef enum QueryMode
static QueryMode querymode = QUERY_SIMPLE;
static const char *QUERYMODE[] = {"simple", "extended", "prepared"};
-typedef struct
+/*
+ * struct Command represents one command in a script.
+ *
+ * lines The raw, possibly multi-line command text. Variable substitution
+ * not applied.
+ * first_line A short, single-line extract of 'lines', for error reporting.
+ * type SQL_COMMAND or META_COMMAND
+ * meta The type of meta-command, or META_NONE if command is SQL
+ * argc Number of arguments of the command, 0 if not yet processed.
+ * argv Command arguments, the first of which is the command or SQL
+ * string itself. For SQL commands, after post-processing
+ * argv[0] is the same as 'lines' with variables substituted.
+ * nqueries In a multi-command SQL line, the number of queries.
+ * varprefix SQL commands terminated with \gset or \cset have this set
+ * to a non NULL value. If nonempty, it's used to prefix the
+ * variable name that receives the value.
+ * varprefix_max Allocated size of the varprefix array.
+ * expr Parsed expression, if needed.
+ * stats Time spent in this command.
+ */
+typedef struct Command
{
- char *line; /* text of command line */
- int command_num; /* unique index of this Command struct */
- int type; /* command type (SQL_COMMAND or META_COMMAND) */
- MetaCommand meta; /* meta command identifier, or META_NONE */
- int argc; /* number of command words */
- char *argv[MAX_ARGS]; /* command word list */
- PgBenchExpr *expr; /* parsed expression, if needed */
- SimpleStats stats; /* time spent in this command */
+ PQExpBufferData lines;
+ char *first_line;
+ int type;
+ MetaCommand meta;
+ int argc;
+ char *argv[MAX_ARGS];
+ int nqueries;
+ char **varprefix;
+ int varprefix_max;
+ PgBenchExpr *expr;
+ SimpleStats stats;
} Command;
typedef struct ParsedScript
@@ -521,7 +546,6 @@ typedef struct ParsedScript
static ParsedScript sql_script[MAX_SCRIPTS]; /* SQL script files */
static int num_scripts; /* number of scripts in sql_script[] */
-static int num_commands = 0; /* total number of Command structs */
static int64 total_weight = 0;
static int debug = 0; /* debug flag */
@@ -587,6 +611,7 @@ static void doLog(TState *thread, CState *st,
static void processXactStats(TState *thread, CState *st, instr_time *now,
bool skipped, StatsData *agg);
static void pgbench_error(const char *fmt,...) pg_attribute_printf(1, 2);
+static void allocate_command_varprefix(Command *cmd, int totalqueries);
static void addScript(ParsedScript script);
static void *threadRun(void *arg);
static void finishCon(CState *st);
@@ -2569,6 +2594,10 @@ getMetaCommand(const char *cmd)
mc = META_ELSE;
else if (pg_strcasecmp(cmd, "endif") == 0)
mc = META_ENDIF;
+ else if (pg_strcasecmp(cmd, "cset") == 0)
+ mc = META_CSET;
+ else if (pg_strcasecmp(cmd, "gset") == 0)
+ mc = META_GSET;
else
mc = META_NONE;
return mc;
@@ -2796,6 +2825,109 @@ sendCommand(CState *st, Command *command)
return true;
}
+/*
+ * Process query response from the backend.
+ *
+ * If varprefix is not NULL, it's the array of variable prefix names where to
+ * store the results.
+ *
+ * Returns true if everything is A-OK, false if any error occurs.
+ */
+static bool
+readCommandResponse(CState *st, char **varprefix)
+{
+ PGresult *res;
+ int qrynum = 0;
+
+ while ((res = PQgetResult(st->con)) != NULL)
+ {
+ switch (PQresultStatus(res))
+ {
+ case PGRES_COMMAND_OK: /* non-SELECT commands */
+ case PGRES_EMPTY_QUERY: /* may be used for testing no-op overhead */
+ if (varprefix && varprefix[qrynum] != NULL)
+ {
+ fprintf(stderr,
+ "client %d script %d command %d query %d: expected one row, got %d\n",
+ st->id, st->use_file, st->command, qrynum, 0);
+ st->ecnt++;
+ return false;
+ }
+ break;
+
+ case PGRES_TUPLES_OK:
+ if (varprefix && varprefix[qrynum] != NULL)
+ {
+ if (PQntuples(res) != 1)
+ {
+ fprintf(stderr,
+ "client %d script %d command %d query %d: expected one row, got %d\n",
+ st->id, st->use_file, st->command, qrynum, PQntuples(res));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ /* store results into variables */
+ for (int fld = 0; fld < PQnfields(res); fld++)
+ {
+ char *varname = PQfname(res, fld);
+
+ /* allocate varname only if necessary, freed below */
+ if (*varprefix[qrynum] != '\0')
+ varname =
+ psprintf("%s%s", varprefix[qrynum], varname);
+
+ /* store result as a string */
+ if (!putVariable(st, "gset", varname,
+ PQgetvalue(res, 0, fld)))
+ {
+ /* internal error */
+ fprintf(stderr,
+ "client %d script %d command %d query %d: error storing into variable %s\n",
+ st->id, st->use_file, st->command, qrynum,
+ varname);
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ if (*varprefix[qrynum] != '\0')
+ pg_free(varname);
+ }
+ }
+ /* otherwise the result is simply thrown away by PQclear below */
+ break;
+
+ default:
+ /* anything else is unexpected */
+ fprintf(stderr,
+ "client %d script %d aborted in command %d query %d: %s",
+ st->id, st->use_file, st->command, qrynum,
+ PQerrorMessage(st->con));
+ st->ecnt++;
+ PQclear(res);
+ discard_response(st);
+ return false;
+ }
+
+ PQclear(res);
+ qrynum++;
+ }
+
+ if (qrynum == 0)
+ {
+ fprintf(stderr, "client %d command %d: no results\n", st->id, st->command);
+ st->ecnt++;
+ return false;
+ }
+
+ return true;
+}
+
+
/*
* Parse the argument to a \sleep command, and return the requested amount
* of delay, in microseconds. Returns true on success, false on error.
@@ -2862,8 +2994,6 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
*/
for (;;)
{
- PGresult *res;
-
switch (st->state)
{
/* Select transaction (script) to run. */
@@ -3141,24 +3271,11 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
if (PQisBusy(st->con))
return; /* don't have the whole result yet */
- /* Read and discard the query result */
- res = PQgetResult(st->con);
- switch (PQresultStatus(res))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- /* OK */
- PQclear(res);
- discard_response(st);
- st->state = CSTATE_END_COMMAND;
- break;
- default:
- commandFailed(st, "SQL", PQerrorMessage(st->con));
- PQclear(res);
- st->state = CSTATE_ABORTED;
- break;
- }
+ /* store or discard the query results */
+ if (readCommandResponse(st, sql_script[st->use_file].commands[st->command]->varprefix))
+ st->state = CSTATE_END_COMMAND;
+ else
+ st->state = CSTATE_ABORTED;
break;
/*
@@ -3191,7 +3308,7 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
INSTR_TIME_SET_CURRENT_LAZY(now);
- command = sql_script[st->use_file].commands[st->command];
+ command = sql_script[st->use_file].commands[st->command];
/* XXX could use a mutex here, but we choose not to */
addToSimpleStats(&command->stats,
INSTR_TIME_GET_DOUBLE(now) -
@@ -3235,9 +3352,8 @@ advanceConnectionState(TState *thread, CState *st, StatsData *agg)
st->state = CSTATE_CHOOSE_SCRIPT;
/*
- * Ensure that we always return on this point, so as to
- * avoid an infinite loop if the script only contains meta
- * commands.
+ * Ensure that we always return on this point, so as to avoid
+ * an infinite loop if the script only contains meta commands.
*/
return;
@@ -3976,7 +4092,7 @@ runInitSteps(const char *initialize_steps)
/*
* Replace :param with $n throughout the command's SQL text, which
- * is a modifiable string in cmd->argv[0].
+ * is a modifiable string in cmd->lines.
*/
static bool
parseQuery(Command *cmd)
@@ -3984,12 +4100,9 @@ parseQuery(Command *cmd)
char *sql,
*p;
- /* We don't want to scribble on cmd->argv[0] until done */
- sql = pg_strdup(cmd->argv[0]);
-
cmd->argc = 1;
- p = sql;
+ p = sql = pg_strdup(cmd->lines.data);
while ((p = strchr(p, ':')) != NULL)
{
char var[13];
@@ -4009,7 +4122,7 @@ parseQuery(Command *cmd)
if (cmd->argc >= MAX_ARGS)
{
fprintf(stderr, "statement has too many arguments (maximum is %d): %s\n",
- MAX_ARGS - 1, cmd->argv[0]);
+ MAX_ARGS - 1, cmd->lines.data);
pg_free(name);
return false;
}
@@ -4021,7 +4134,7 @@ parseQuery(Command *cmd)
cmd->argc++;
}
- pg_free(cmd->argv[0]);
+ Assert(cmd->argv[0] == NULL);
cmd->argv[0] = sql;
return true;
}
@@ -4081,21 +4194,16 @@ syntax_error(const char *source, int lineno,
}
/*
- * Parse a SQL command; return a Command struct, or NULL if it's a comment
- *
- * On entry, psqlscan.l has collected the command into "buf", so we don't
- * really need to do much here except check for comment and set up a
- * Command struct.
+ * Return a pointer to the start of the SQL command, after skipping over
+ * whitespace and "--" comments.
+ * If the end of the string is reached, return NULL.
*/
-static Command *
-process_sql_command(PQExpBuffer buf, const char *source)
+static char *
+skip_sql_comments(char *sql_command)
{
- Command *my_command;
- char *p;
- char *nlpos;
+ char *p = sql_command;
/* Skip any leading whitespace, as well as "--" style comments */
- p = buf->data;
for (;;)
{
if (isspace((unsigned char) *p))
@@ -4111,41 +4219,152 @@ process_sql_command(PQExpBuffer buf, const char *source)
break;
}
- /* If there's nothing but whitespace and comments, we're done */
+ /* NULL if there's nothing but whitespace and comments */
if (*p == '\0')
return NULL;
+ return p;
+}
+
+/*
+ * Parse a SQL command; return a Command struct, or NULL if it's a comment
+ *
+ * On entry, psqlscan.l has collected the command into "buf", so we don't
+ * really need to do much here except check for comments and set up a Command
+ * struct.
+ */
+static Command *
+create_sql_command(PQExpBuffer buf, const char *source, int numqueries)
+{
+ Command *my_command;
+ char *p = skip_sql_comments(buf->data);
+
+ if (p == NULL)
+ return NULL;
+
/* Allocate and initialize Command structure */
- my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
+ my_command = (Command *) pg_malloc(sizeof(Command));
+ initPQExpBuffer(&my_command->lines);
+ appendPQExpBufferStr(&my_command->lines, p);
+ my_command->first_line = NULL; /* this is set later */
my_command->type = SQL_COMMAND;
my_command->meta = META_NONE;
+ my_command->argc = 0;
+ memset(my_command->argv, 0, sizeof(my_command->argv));
+ my_command->nqueries = numqueries;
+ my_command->varprefix = NULL; /* allocated later, if needed */
+ my_command->varprefix_max = 0;
+ my_command->expr = NULL;
initSimpleStats(&my_command->stats);
- /*
- * Install query text as the sole argv string. If we are using a
- * non-simple query mode, we'll extract parameters from it later.
- */
- my_command->argv[0] = pg_strdup(p);
- my_command->argc = 1;
-
- /*
- * If SQL command is multi-line, we only want to save the first line as
- * the "line" label.
- */
- nlpos = strchr(p, '\n');
- if (nlpos)
- {
- my_command->line = pg_malloc(nlpos - p + 1);
- memcpy(my_command->line, p, nlpos - p);
- my_command->line[nlpos - p] = '\0';
- }
- else
- my_command->line = pg_strdup(p);
-
return my_command;
}
+/* Free a Command structure and associated data */
+static void
+free_command(Command *command)
+{
+ termPQExpBuffer(&command->lines);
+ if (command->first_line)
+ pg_free(command->first_line);
+ if (command->argv)
+ for (int i = 0; i < command->argc; i++)
+ pg_free(command->argv[i]);
+ if (command->varprefix)
+ {
+ for (int i = 0; i < command->varprefix_max; i++)
+ if (command->varprefix[i] != NULL)
+ pg_free(command->varprefix[i]);
+ pg_free(command->varprefix);
+ }
+ /*
+ * It should also free expr recursively, but this is currently not needed
+ * as only "\.set" commands which do not have an expression are freed.
+ */
+ pg_free(command);
+}
+
+/*
+ * append "more" text to current compound command which had been interrupted
+ * by \cset.
+ */
+static void
+append_sql_command(Command *my_command, char *more, int numqueries)
+{
+ Assert(my_command->type == SQL_COMMAND && my_command->lines.len > 0);
+
+ more = skip_sql_comments(more);
+ if (more == NULL)
+ return;
+
+ /* append command text, embedding a ';' in place of the \cset */
+ appendPQExpBuffer(&my_command->lines, ";\n%s", more);
+
+ my_command->nqueries += numqueries;
+}
+
+/*
+ * Once an SQL command is fully parsed, possibly by accumulating several
+ * parts, complete other fields of the Command structure.
+ */
+static void
+postprocess_sql_command(Command *my_command)
+{
+ char buffer[128];
+
+ Assert(my_command->type == SQL_COMMAND);
+
+ /* Save the first line for error display. */
+ strlcpy(buffer, my_command->lines.data, sizeof(buffer));
+ buffer[strcspn(buffer, "\n\r")] = '\0';
+ my_command->first_line = pg_strdup(buffer);
+
+ /* parse query if necessary */
+ switch (querymode)
+ {
+ case QUERY_SIMPLE:
+ my_command->argv[0] = my_command->lines.data;
+ my_command->argc++;
+ break;
+ case QUERY_EXTENDED:
+ case QUERY_PREPARED:
+ if (!parseQuery(my_command))
+ exit(1);
+ break;
+ default:
+ exit(1);
+ }
+}
+
+/*
+ * Determine the command's varprefix size needed and allocate memory for it
+ */
+static void
+allocate_command_varprefix(Command *cmd, int totalqueries)
+{
+ int new_max;
+
+ /* sufficient space already allocated? */
+ if (totalqueries <= cmd->varprefix_max)
+ return;
+
+ /* Determine the new array size. */
+ new_max = Max(cmd->varprefix_max, 2);
+ while (new_max < totalqueries)
+ new_max *= 2;
+
+ /* and enlarge the array, zero-initializing the allocated space */
+ if (cmd->varprefix == NULL)
+ cmd->varprefix = pg_malloc0(sizeof(char *) * new_max);
+ else
+ {
+ cmd->varprefix = pg_realloc(cmd->varprefix, sizeof(char *) * new_max);
+ memset(cmd->varprefix + cmd->varprefix_max, 0,
+ sizeof(char *) * (new_max - cmd->varprefix_max));
+ }
+ cmd->varprefix_max = new_max;
+}
+
/*
* Parse a backslash command; return a Command struct, or NULL if comment
*
@@ -4177,7 +4396,6 @@ process_backslash_command(PsqlScanState sstate, const char *source)
/* Allocate and initialize Command structure */
my_command = (Command *) pg_malloc0(sizeof(Command));
- my_command->command_num = num_commands++;
my_command->type = META_COMMAND;
my_command->argc = 0;
initSimpleStats(&my_command->stats);
@@ -4201,7 +4419,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (my_command->meta == META_SET)
{
if (!expr_lex_one_word(sstate, &word_buf, &word_offset))
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
offsets[j] = word_offset;
@@ -4222,10 +4440,11 @@ process_backslash_command(PsqlScanState sstate, const char *source)
my_command->expr = expr_parse_result;
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
expr_scanner_finish(yyscanner);
@@ -4238,7 +4457,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
while (expr_lex_one_word(sstate, &word_buf, &word_offset))
{
if (j >= MAX_ARGS)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL, -1);
offsets[j] = word_offset;
@@ -4247,19 +4466,20 @@ process_backslash_command(PsqlScanState sstate, const char *source)
}
/* Save line, trimming any trailing newline */
- my_command->line = expr_scanner_get_substring(sstate,
- start_offset,
- expr_scanner_offset(sstate),
- true);
+ my_command->first_line =
+ expr_scanner_get_substring(sstate,
+ start_offset,
+ expr_scanner_offset(sstate),
+ true);
if (my_command->meta == META_SLEEP)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
if (my_command->argc > 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"too many arguments", NULL,
offsets[3] - start_offset);
@@ -4288,7 +4508,7 @@ process_backslash_command(PsqlScanState sstate, const char *source)
if (pg_strcasecmp(my_command->argv[2], "us") != 0 &&
pg_strcasecmp(my_command->argv[2], "ms") != 0 &&
pg_strcasecmp(my_command->argv[2], "s") != 0)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unrecognized time unit, must be us, ms or s",
my_command->argv[2], offsets[2] - start_offset);
}
@@ -4296,25 +4516,31 @@ process_backslash_command(PsqlScanState sstate, const char *source)
else if (my_command->meta == META_SETSHELL)
{
if (my_command->argc < 3)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing argument", NULL, -1);
}
else if (my_command->meta == META_SHELL)
{
if (my_command->argc < 2)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"missing command", NULL, -1);
}
else if (my_command->meta == META_ELSE || my_command->meta == META_ENDIF)
{
if (my_command->argc != 1)
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"unexpected argument", NULL, -1);
}
+ else if (my_command->meta == META_CSET || my_command->meta == META_GSET)
+ {
+ if (my_command->argc > 2)
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
+ "too many arguments", NULL, -1);
+ }
else
{
/* my_command->meta == META_NONE */
- syntax_error(source, lineno, my_command->line, my_command->argv[0],
+ syntax_error(source, lineno, my_command->first_line, my_command->argv[0],
"invalid command", NULL, -1);
}
@@ -4393,6 +4619,9 @@ ParseScript(const char *script, const char *desc, int weight)
PQExpBufferData line_buf;
int alloc_num;
int index;
+ bool saw_cset = false;
+ int lineno;
+ int start_offset;
#define COMMANDS_ALLOC_NUM 128
alloc_num = COMMANDS_ALLOC_NUM;
@@ -4416,6 +4645,7 @@ ParseScript(const char *script, const char *desc, int weight)
* stdstrings should be true, which is a bit riskier.
*/
psql_scan_setup(sstate, script, strlen(script), 0, true);
+ start_offset = expr_scanner_offset(sstate) - 1;
initPQExpBuffer(&line_buf);
@@ -4425,45 +4655,115 @@ ParseScript(const char *script, const char *desc, int weight)
{
PsqlScanResult sr;
promptStatus_t prompt;
- Command *command;
+ Command *command = NULL;
+ int semicolons;
resetPQExpBuffer(&line_buf);
+ lineno = expr_scanner_get_lineno(sstate, start_offset);
sr = psql_scan(sstate, &line_buf, &prompt);
- /* If we collected a SQL command, process that */
- command = process_sql_command(&line_buf, desc);
- if (command)
+ semicolons = psql_scan_get_escaped_semicolons(sstate);
+
+ if (saw_cset)
+ {
+ /* the previous multi-line command ended with \cset */
+ append_sql_command(ps.commands[index - 1], line_buf.data,
+ semicolons + 1);
+ saw_cset = false;
+ }
+ else
{
- ps.commands[index] = command;
- index++;
+ /* If we collected a new SQL command, process that */
+ command = create_sql_command(&line_buf, desc, semicolons + 1);
- if (index >= alloc_num)
- {
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
- }
+ /* store new command */
+ if (command)
+ ps.commands[index++] = command;
}
/* If we reached a backslash, process that */
if (sr == PSCAN_BACKSLASH)
{
command = process_backslash_command(sstate, desc);
+
if (command)
{
- ps.commands[index] = command;
- index++;
-
- if (index >= alloc_num)
+ /*
+ * If this is gset/cset, merge into the preceding command. (We
+ * don't use a command slot in this case).
+ */
+ if (command->meta == META_CSET ||
+ command->meta == META_GSET)
{
- alloc_num += COMMANDS_ALLOC_NUM;
- ps.commands = (Command **)
- pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ int cindex;
+ Command *cmd;
+
+ /*
+ * If \cset is seen, set flag to leave the command pending
+ * for the next iteration to process.
+ */
+ saw_cset = command->meta == META_CSET;
+
+ if (index == 0)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot start a script",
+ NULL, -1);
+
+ cmd = ps.commands[index - 1];
+
+ if (cmd->type != SQL_COMMAND)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset must follow a SQL command",
+ cmd->first_line, -1);
+
+ /* this {g,c}set applies to the previous query */
+ cindex = cmd->nqueries - 1;
+
+ /*
+ * now that we know there's a {g,c}set in this command,
+ * allocate space for the variable name prefix array.
+ */
+ allocate_command_varprefix(cmd, cmd->nqueries);
+
+ /*
+ * Don't allow the previous command be a gset/cset; that
+ * would make no sense.
+ */
+ if (cmd->varprefix[cindex] != NULL)
+ syntax_error(desc, lineno, NULL, NULL,
+ "\\gset/cset cannot follow one another",
+ NULL, -1);
+
+ /* get variable prefix */
+ if (command->argc <= 1 || command->argv[1][0] == '\0')
+ cmd->varprefix[cindex] = pg_strdup("");
+ else
+ cmd->varprefix[cindex] = pg_strdup(command->argv[1]);
+
+ /* cleanup unused command */
+ free_command(command);
+
+ continue;
}
+
+ /* Attach any other backslash command as a new command */
+ ps.commands[index++] = command;
}
}
+ /*
+ * Since we used a command slot, allocate more if needed. Note we
+ * always allocate one more in order to accomodate the NULL terminator
+ * below.
+ */
+ if (index >= alloc_num)
+ {
+ alloc_num += COMMANDS_ALLOC_NUM;
+ ps.commands = (Command **)
+ pg_realloc(ps.commands, sizeof(Command *) * alloc_num);
+ }
+
/* Done if we reached EOF */
if (sr == PSCAN_INCOMPLETE || sr == PSCAN_EOL)
break;
@@ -4819,7 +5119,7 @@ printResults(TState *threads, StatsData *total, instr_time total_time,
printf(" %11.3f %s\n",
(cstats->count > 0) ?
1000.0 * cstats->sum / cstats->count : 0.0,
- (*commands)->line);
+ (*commands)->first_line);
}
}
}
@@ -5286,28 +5586,18 @@ main(int argc, char **argv)
internal_script_used = true;
}
- /* if not simple query mode, parse the script(s) to find parameters */
- if (querymode != QUERY_SIMPLE)
- {
- for (i = 0; i < num_scripts; i++)
- {
- Command **commands = sql_script[i].commands;
- int j;
-
- for (j = 0; commands[j] != NULL; j++)
- {
- if (commands[j]->type != SQL_COMMAND)
- continue;
- if (!parseQuery(commands[j]))
- exit(1);
- }
- }
- }
-
- /* compute total_weight */
+ /* complete SQL command initialization and compute total weight */
for (i = 0; i < num_scripts; i++)
+ {
+ Command **commands = sql_script[i].commands;
+
+ for (int j = 0; commands[j] != NULL; j++)
+ if (commands[j]->type == SQL_COMMAND)
+ postprocess_sql_command(commands[j]);
+
/* cannot overflow: weight is 32b, total_weight 64b */
total_weight += sql_script[i].weight;
+ }
if (total_weight == 0 && !is_init_mode)
{
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 89678e7f3f..c87748086a 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -528,6 +528,48 @@ pgbench(
}
});
+# working \gset and \cset
+pgbench(
+ '-t 1', 0,
+ [ qr{type: .*/001_pgbench_gset_and_cset}, qr{processed: 1/1} ],
+ [ qr{command=3.: int 0\b},
+ qr{command=5.: int 1\b},
+ qr{command=6.: int 2\b},
+ qr{command=8.: int 3\b},
+ qr{command=9.: int 4\b},
+ qr{command=10.: int 5\b},
+ qr{command=12.: int 6\b},
+ qr{command=13.: int 7\b},
+ qr{command=14.: int 8\b},
+ qr{command=16.: int 9\b} ],
+ 'pgbench gset and cset commands',
+ { '001_pgbench_gset_and_cset' => q{-- test gset and cset
+-- no columns
+SELECT \gset
+-- one value
+SELECT 0 AS i0 \gset
+\set i debug(:i0)
+-- two values
+SELECT 1 AS i1, 2 AS i2 \gset
+\set i debug(:i1)
+\set i debug(:i2)
+-- cset & gset to follow
+SELECT :i2 + 1 AS i3, :i2 * :i2 AS i4 \cset
+ SELECT 5 AS i5 \gset
+\set i debug(:i3)
+\set i debug(:i4)
+\set i debug(:i5)
+-- with prefix
+SELECT 6 AS i6, 7 AS i7 \cset x_
+ SELECT 8 AS i8 \gset y_
+\set i debug(:x_i6)
+\set i debug(:x_i7)
+\set i debug(:y_i8)
+-- overwrite existing variable
+SELECT 0 AS i9, 9 AS i9 \gset
+\set i debug(:i9)
+} });
+
# trigger many expression errors
my @errors = (
@@ -735,21 +777,45 @@ SELECT LEAST(:i, :i, :i, :i, :i, :i, :i, :i, :i, :i, :i);
[qr{invalid command .* "nosuchcommand"}], q{\nosuchcommand}
],
[ 'misc empty script', 1, [qr{empty command list for script}], q{} ],
- [
- 'bad boolean', 2,
- [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true}
- ],);
+ [ 'bad boolean', 2,
+ [qr{malformed variable.*trueXXX}], q{\set b :badtrue or true} ],
+ # GSET & CSET
+ [ 'gset no row', 2,
+ [qr{expected one row, got 0\b}], q{SELECT WHERE FALSE \gset} ],
+ [ 'cset no row', 2,
+ [qr{expected one row, got 0\b}], q{SELECT WHERE FALSE \cset
+SELECT 1 AS i\gset}, 1 ],
+ [ 'gset alone', 1, [qr{gset/cset cannot start a script}], q{\gset} ],
+ [ 'gset no SQL', 1,
+ [qr{gset/cset must follow a SQL command}], q{\set i +1
+\gset} ],
+ [ 'gset too many arguments', 1,
+ [qr{too many arguments}], q{SELECT 1 \gset a b} ],
+ [ 'gset after gset', 1,
+ [qr{gset/cset cannot follow one another}], q{SELECT 1 AS i \gset
+\gset} ],
+ [ 'gset non SELECT', 2,
+ [qr{expected one row, got 0}],
+ q{DROP TABLE IF EXISTS no_such_table \gset} ],
+ [ 'gset bad default name', 2,
+ [qr{error storing into variable \?column\?}],
+ q{SELECT 1 \gset} ],
+ [ 'gset bad name', 2,
+ [qr{error storing into variable bad name!}],
+ q{SELECT 1 AS "bad name!" \gset} ],
+ );
for my $e (@errors)
{
- my ($name, $status, $re, $script) = @$e;
+ my ($name, $status, $re, $script, $no_prepare) = @$e;
$status != 0 or die "invalid expected status for test \"$name\"";
my $n = '001_pgbench_error_' . $name;
$n =~ s/ /_/g;
pgbench(
- '-n -t 1 -M prepared -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 ' .
- '-Dbadtrue=trueXXX -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808',
+ '-n -t 1 -Dfoo=bla -Dnull=null -Dtrue=true -Done=1 -Dzero=0.0 -Dbadtrue=trueXXX' .
+ ' -Dmaxint=9223372036854775807 -Dminint=-9223372036854775808' .
+ ($no_prepare ? '' : ' -M prepared'),
$status,
[ $status == 1 ? qr{^$} : qr{processed: 0/1} ],
$re,
diff --git a/src/fe_utils/psqlscan.l b/src/fe_utils/psqlscan.l
index 9ea32c319f..321744cddb 100644
--- a/src/fe_utils/psqlscan.l
+++ b/src/fe_utils/psqlscan.l
@@ -693,8 +693,15 @@ other .
* substitution. We want these before {self}, also.
*/
-"\\"[;:] {
- /* Force a semicolon or colon into the query buffer */
+"\\"; {
+ /* Count semicolons in compound commands */
+ cur_state->escaped_semicolons++;
+ /* Force a semicolon into the query buffer */
+ psqlscan_emit(cur_state, yytext + 1, 1);
+ }
+
+"\\": {
+ /* Force a colon into the query buffer */
psqlscan_emit(cur_state, yytext + 1, 1);
}
@@ -1065,6 +1072,9 @@ psql_scan(PsqlScanState state,
/* Set current output target */
state->output_buf = query_buf;
+ /* Reset number of escaped semicolons seen */
+ state->escaped_semicolons = 0;
+
/* Set input source */
if (state->buffer_stack != NULL)
yy_switch_to_buffer(state->buffer_stack->buf, state->scanner);
@@ -1208,6 +1218,16 @@ psql_scan_reset(PsqlScanState state)
state->dolqstart = NULL;
}
+/*
+ * Return the number of escaped semicolons in the lexed string seen by the
+ * previous psql_scan call.
+ */
+int
+psql_scan_get_escaped_semicolons(PsqlScanState state)
+{
+ return state->escaped_semicolons;
+}
+
/*
* Reselect this lexer (psqlscan.l) after using another one.
*
diff --git a/src/include/fe_utils/psqlscan.h b/src/include/fe_utils/psqlscan.h
index 1cf5b2e7fa..d6fef9ff77 100644
--- a/src/include/fe_utils/psqlscan.h
+++ b/src/include/fe_utils/psqlscan.h
@@ -90,6 +90,8 @@ extern PsqlScanResult psql_scan(PsqlScanState state,
extern void psql_scan_reset(PsqlScanState state);
+extern int psql_scan_get_escaped_semicolons(PsqlScanState state);
+
extern void psql_scan_reselect_sql_lexer(PsqlScanState state);
extern bool psql_scan_in_quote(PsqlScanState state);
diff --git a/src/include/fe_utils/psqlscan_int.h b/src/include/fe_utils/psqlscan_int.h
index 42a738f422..752cc9406a 100644
--- a/src/include/fe_utils/psqlscan_int.h
+++ b/src/include/fe_utils/psqlscan_int.h
@@ -112,6 +112,7 @@ typedef struct PsqlScanStateData
int start_state; /* yylex's starting/finishing state */
int paren_depth; /* depth of nesting in parentheses */
int xcdepth; /* depth of nesting in slash-star comments */
+ int escaped_semicolons; /* number of embedded (\;) semicolons */
char *dolqstart; /* current $foo$ quote start string */
/*
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 16:43 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Fabien COELHO <coelho@cri.ensmp.fr>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2019-01-10 16:43 UTC (permalink / raw)
To: Fabien COELHO <coelho@cri.ensmp.fr>; +Cc: Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
On 2019-Jan-10, Fabien COELHO wrote:
> However, I switched "pg_free" to "termPQExpBuffer", which seems more
> appropriate, even if it just does the same thing. I also ensured that
> prefixes are allocated & freed. I've commented about expr which is not
> freed.
Oops, of course, thanks.
> I'm not keen on having the command array size checked and updated *after*
> the command is appended, even if the initial allocation ensures that there
> is no overflow, but I let it as is.
It was already done that way, only it was done in two places rather than
one. I just refactored it. (In fairness, I think the assignment of the
new command to the array could also be done in one place instead of two,
but it seems slightly clearer like this.)
> Attached a v29 with the above minor changes wrt your version.
Thanks, pushed. I fixed a couple of very minor issues in the docs.
Now let's see how the buildfarm likes this ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 18:20 Tom Lane <tgl@sss.pgh.pa.us>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 1 reply; 120+ messages in thread
From: Tom Lane @ 2019-01-10 18:20 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Now let's see how the buildfarm likes this ...
This \cset thing seem like an incredibly badly thought out kluge.
What is its excuse to live?
regards, tom lane
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 19:00 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2019-01-10 19:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
On 2019-Jan-10, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Now let's see how the buildfarm likes this ...
>
> This \cset thing seem like an incredibly badly thought out kluge.
> What is its excuse to live?
The reason is that you can set variables from several queries in one
network trip.
We can take it out I guess, but my impression was that we already pretty
much had a consensus that it was wanted.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 19:14 Tom Lane <tgl@sss.pgh.pa.us>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 2 replies; 120+ messages in thread
From: Tom Lane @ 2019-01-10 19:14 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> On 2019-Jan-10, Tom Lane wrote:
>> This \cset thing seem like an incredibly badly thought out kluge.
>> What is its excuse to live?
> The reason is that you can set variables from several queries in one
> network trip.
So who needs that? Just merge the queries, if it's so important that
you avoid multiple round trips.
> We can take it out I guess, but my impression was that we already pretty
> much had a consensus that it was wanted.
Maybe if the implementation weren't a pile of junk it'd be all right,
but as-is this is a mess. The dependency on counting \; in particular
is setting me off, because that has little if anything to do with the
number of query results to be expected. I imagine the argument will
be that nobody would write the sort of queries that break that assumption
in a pgbench script; but I don't find that kind of design to be up
to project standards, especially not when the argument for the feature
is tissue-thin in the first place.
regards, tom lane
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 19:49 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 120+ messages in thread
From: Alvaro Herrera @ 2019-01-10 19:49 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Fabien COELHO <coelho@cri.ensmp.fr>; Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
On 2019-Jan-10, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > On 2019-Jan-10, Tom Lane wrote:
> >> This \cset thing seem like an incredibly badly thought out kluge.
> >> What is its excuse to live?
>
> > The reason is that you can set variables from several queries in one
> > network trip.
>
> So who needs that? Just merge the queries, if it's so important that
> you avoid multiple round trips.
Hmm, I suppose that's true.
> > We can take it out I guess, but my impression was that we already pretty
> > much had a consensus that it was wanted.
>
> Maybe if the implementation weren't a pile of junk it'd be all right,
> but as-is this is a mess. The dependency on counting \; in particular
> is setting me off, because that has little if anything to do with the
> number of query results to be expected. I imagine the argument will
> be that nobody would write the sort of queries that break that assumption
> in a pgbench script; but I don't find that kind of design to be up
> to project standards, especially not when the argument for the feature
> is tissue-thin in the first place.
There's a lot of the new code in pgbench that can be simplified if we
remove \cset.
I'll leave time for others to argue for or against cset, and then act
accordingly.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 21:01 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2019-01-10 21:01 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Hello Tom,
> So who needs that? Just merge the queries, if it's so important that
> you avoid multiple round trips.
Pgbench is about testing (measuring) performance in various settings and
realistic scenarii: queries prepared or not, possible combined, and so on.
As postgres allows to send several queries into one message, I think it
interesting to be able to test the performance impact of doing that (the
answer is very significant, esp wrt lantency), and it should not preclude
to get the results out as a client application could do.
Queries can be "merged", but ISTM syntax is not especially friendly
(UNION, SELECT of SELECT, CROSS JOIN not sure which one you had in
mind...) nor reprensentative of what a client application would really do,
and it would mess with readability, message size, planning and so. Also,
compound queries need to return all one row, but this constraint is
neeeded for variable.
>> We can take it out I guess, but my impression was that we already pretty
>> much had a consensus that it was wanted.
>
> Maybe if the implementation weren't a pile of junk it'd be all right,
> but as-is this is a mess.
Thanks:-)
> The dependency on counting \; in particular is setting me off, because
> that has little if anything to do with the number of query results to be
> expected.
The kludge is because there is a kludge (aka optimizations:-) server side
to silently ignore empty queries. On "SELECT 1 \; /**/ \; SELECT 2 ;" the
server sends two results back instead of 3, whereas it should logically
return an empty result on the empty query.
Now pgbench could detect that there is an empty query (possibly skipping
comments and so), an early version of the patch did that AFAICR, but the
code did not seem worth it, it seemed cleaner to just document the
restriction, so it was removed.
> I imagine the argument will be that nobody would write the sort of
> queries that break that assumption in a pgbench script;
Detecting empty queries is possible, although the code for doing that is
kind of ugly and would look bad in the lexer, on which it seemed desirable
to have minimum changes.
> but I don't find that kind of design to be up to project standards,
> especially not when the argument for the feature is tissue-thin in the
> first place.
The "first place" is to be able to implement more realistic scenarii, and
to have getting results into variables orthogonal to combined queries.
Although I'm not specially thrilled by the resulting syntax, the point is
to provide a feature pertinent to performance testing, not to have a
incredibly well designed syntax. It just goes on with the existing
backslash approach used by psql & pgbench, which has the significant
advantage that it is mostly SQL with a few things around.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
* Re: [HACKERS] pgbench - allow to store select results into variables
@ 2019-01-10 21:23 Fabien COELHO <coelho@cri.ensmp.fr>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 0 replies; 120+ messages in thread
From: Fabien COELHO @ 2019-01-10 21:23 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Michael Paquier <michael@paquier.xyz>; Stephen Frost <sfrost@snowman.net>; Andrew Dunstan <andrew.dunstan@2ndquadrant.com>; pgsql-hackers
Hello Alvaro,
> There's a lot of the new code in pgbench that can be simplified if we
> remove \cset.
I'm not very happy with the resulting syntax, but IMO the feature is
useful. My initial design was to copy PL/pgSQL "into" with some "\into"
orthogonal to \; and ;, but the implementation was not especially nice and
I was told to use psql's \gset approach, which I did.
If we do not provide \cset, then combined queries and getting results are
not orthogonal, although from a performance testing point of view an
application could do both, and the point is to allow pgbench to test the
performance impact of doing that.
There are other existing restrictions which are a arbitrary, eg you cannot
use prepared with combined. I wish not to add more of this kind of
restrictions, which are not "up to project standard" in my opinion. I may
try to remove this particular restriction in the future.
Not many people know that queries can be combined, but if you are
interested in latency that is really an interesting option, and being able
to check how much can be gained from doing that is a point of a tool like
pgbench.
--
Fabien.
^ permalink raw reply [nested|flat] 120+ messages in thread
end of thread, other threads:[~2019-01-10 21:23 UTC | newest]
Thread overview: 120+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-07-09 08:20 pgbench - allow to store select results into variables Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-09 08:44 ` Pavel Stehule <pavel.stehule@gmail.com>
2016-07-09 09:19 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-09 12:19 ` Pavel Stehule <pavel.stehule@gmail.com>
2016-07-09 12:52 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-13 19:34 ` Robert Haas <robertmhaas@gmail.com>
2016-07-13 20:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2016-07-13 20:07 ` Robert Haas <robertmhaas@gmail.com>
2016-07-13 20:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2016-07-13 21:31 ` David G. Johnston <david.g.johnston@gmail.com>
2016-07-14 08:02 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-15 05:13 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-15 16:33 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-09-02 02:58 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-09-02 17:47 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-09-05 08:09 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-09-07 14:01 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-09-12 07:33 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-09-13 08:41 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-09-14 10:39 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-09-18 11:13 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-09-26 07:12 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-09-26 08:19 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-09-26 11:27 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-09-27 01:41 ` Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
2016-10-02 13:43 ` Michael Paquier <michael.paquier@gmail.com>
2016-12-02 06:33 ` Haribabu Kommi <kommi.haribabu@gmail.com>
2017-01-04 21:15 ` Tom Lane <tgl@sss.pgh.pa.us>
2017-01-05 09:47 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-01-07 09:25 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-01-29 08:15 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-01-31 06:13 ` Michael Paquier <michael.paquier@gmail.com>
2017-01-31 06:24 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-03-15 05:53 ` Rafia Sabih <rafia.sabih@enterprisedb.com>
2017-03-15 19:15 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-03-23 10:59 ` Rafia Sabih <rafia.sabih@enterprisedb.com>
2017-03-24 15:29 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-03-27 10:49 ` Rafia Sabih <rafia.sabih@enterprisedb.com>
2017-04-05 00:45 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-05 18:28 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-06 00:07 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-06 00:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2017-04-06 00:28 ` Andres Freund <andres@anarazel.de>
2017-04-06 00:36 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-07 00:17 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-07 07:55 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-07 08:06 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-07 09:43 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-08 20:24 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-08 20:32 ` Fabien COELHO <fabien.coelho@mines-paristech.fr>
2017-04-09 23:32 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-17 06:34 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-17 09:11 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-17 11:00 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-20 08:06 ` Tatsuo Ishii <ishii@sraoss.co.jp>
2017-04-22 04:58 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-09-04 17:46 ` Tom Lane <tgl@sss.pgh.pa.us>
2017-09-04 19:12 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-08 20:37 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-04-17 06:40 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-08-13 18:33 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-08-13 19:45 ` Pavel Stehule <pavel.stehule@gmail.com>
2017-09-08 17:17 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-10-20 16:37 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-10-25 09:55 ` Pavel Stehule <pavel.stehule@gmail.com>
2017-11-04 11:05 ` Fabien COELHO <coelho@cri.ensmp.fr>
2017-12-01 02:57 ` Michael Paquier <michael.paquier@gmail.com>
2017-12-01 05:48 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-01-10 08:54 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-03-24 09:25 ` Fabien COELHO <fabien.coelho@mines-paristech.fr>
2018-04-06 20:55 ` Stephen Frost <sfrost@snowman.net>
2018-04-07 03:17 ` Stephen Frost <sfrost@snowman.net>
2018-04-07 16:06 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-04-08 00:00 ` Stephen Frost <sfrost@snowman.net>
2018-04-08 05:58 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-04-27 16:28 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-08-13 22:04 ` Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
2018-08-13 22:30 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-08-14 10:47 ` Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
2018-08-14 11:37 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-08-14 11:56 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-08-14 17:44 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-08-14 18:23 ` Robert Haas <robertmhaas@gmail.com>
2018-08-14 19:10 ` Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
2018-08-14 20:44 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-08-14 20:53 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-08-14 12:35 ` Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
2018-08-14 12:43 ` Andrew Gierth <andrew@tao11.riddles.org.uk>
2018-08-14 11:38 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-09-12 06:35 ` Michael Paquier <michael@paquier.xyz>
2018-10-02 06:43 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-10-15 13:09 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-11-16 22:36 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-11-17 10:43 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-11-17 11:06 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-11-18 10:23 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-11-21 19:42 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-11-24 07:45 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-11-24 15:36 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-11-24 16:57 ` Fabien COELHO <coelho@cri.ensmp.fr>
2019-01-03 17:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-01-06 18:29 ` Fabien COELHO <coelho@cri.ensmp.fr>
2019-01-09 07:16 ` Fabien COELHO <coelho@cri.ensmp.fr>
2019-01-09 13:41 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-01-09 15:31 ` Fabien COELHO <coelho@cri.ensmp.fr>
2019-01-09 23:00 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-01-10 09:12 ` Fabien COELHO <coelho@cri.ensmp.fr>
2019-01-10 16:43 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-01-10 18:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-10 19:00 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-01-10 19:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2019-01-10 19:49 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-01-10 21:23 ` Fabien COELHO <coelho@cri.ensmp.fr>
2019-01-10 21:01 ` Fabien COELHO <coelho@cri.ensmp.fr>
2018-11-16 21:43 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-11-16 22:13 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2018-11-17 10:25 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-13 20:16 ` Fabien COELHO <coelho@cri.ensmp.fr>
2016-07-13 20:26 ` Tom Lane <tgl@sss.pgh.pa.us>
2016-07-13 21:19 ` Fabien COELHO <coelho@cri.ensmp.fr>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox