public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 06/17] Allow user to use password instead of encryption key.
27+ messages / 6 participants
[nested] [flat]
* [PATCH 06/17] Allow user to use password instead of encryption key.
@ 2019-07-05 14:24 Antonin Houska <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Antonin Houska @ 2019-07-05 14:24 UTC (permalink / raw)
pg_keytool reads the password from stdin, uses "key derivation function (KDF)
parameters" stored in the data directory to derive the key and writes the key
to standard output.
The cluster can be created this way
initdb -D data -K "echo securepwd | pg_keytool -D data -w"
and started either this way
pg_ctl -D data -K "echo securepwd | pg_keytool -D data -w" start
or this way
postgres -D data
and (in another session)
echo securepwd | pg_keytool -D data -ws
Both initdb and pg_ctl can substitute the data directory for %D pattern in the
key encryption command. Thus user uses pg_keytool to derive the password, he
can in fact say
initdb -D data -K "echo securepwd | pg_keytool -D %D -w"
and
pg_ctl -D data -K "echo securepwd | pg_keytool -D %D -w" start
respectively.
---
src/bin/initdb/initdb.c | 26 +++-
src/bin/pg_ctl/pg_ctl.c | 2 +-
src/bin/pg_keytool/pg_keytool.c | 87 +++++++++---
src/fe_utils/encryption.c | 273 +++++++++++++++++++++++++++++++++++++-
src/include/fe_utils/encryption.h | 8 +-
src/include/storage/encryption.h | 8 ++
6 files changed, 379 insertions(+), 25 deletions(-)
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index f2a582e975..bfa8f5fad4 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -3004,11 +3004,31 @@ initialize_data_directory(void)
write_version_file(NULL);
/*
- * If the cluster will be encrypted, run the command to generate the
- * encryption key.
+ * If the cluster will be encrypted, write the KDF file so that encryption
+ * key can be derived from password.
*/
if (encryption_key_command)
- run_encryption_key_command(encryption_key);
+ {
+ /*
+ * XXX Since execution of encryption_key_command produce the key (as
+ * opposed to password), we don't know if the command received the key
+ * itself or a password. If DBA provided initdb with a key, he will
+ * never use password in the future (there was no KDF so far so the
+ * key could not be derived from password, and the password can hardly
+ * be derived from the key), so the KDF file may be useless. We don't
+ * have enough information to recognize this special case, so just
+ * initialize and write the KDF unconditionally.
+ */
+ init_kdf();
+ write_kdf_file(pg_data);
+
+ /*
+ * The key command is allowed to use pg_keytool, which in turn needs
+ * the KDF parameters. The KDF parameters are now available so we can
+ * run the command.
+ */
+ run_encryption_key_command(encryption_key, pg_data);
+ }
/* Select suitable configuration settings */
set_null_conf();
diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c
index 7e414ac048..e8e4eee162 100644
--- a/src/bin/pg_ctl/pg_ctl.c
+++ b/src/bin/pg_ctl/pg_ctl.c
@@ -881,7 +881,7 @@ do_start(void)
* If encryption key is needed, retrieve it before trying to start
* postmaster.
*/
- run_encryption_key_command(encryption_key);
+ run_encryption_key_command(encryption_key, pg_data);
/*
* Where should the key be sent?
diff --git a/src/bin/pg_keytool/pg_keytool.c b/src/bin/pg_keytool/pg_keytool.c
index 322625da41..c5151ad57c 100644
--- a/src/bin/pg_keytool/pg_keytool.c
+++ b/src/bin/pg_keytool/pg_keytool.c
@@ -43,6 +43,7 @@ usage(const char *progname)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -D, --pgdata=DATADIR data directory\n"));
/* Display default host */
env = getenv("PGHOST");
printf(_(" -h, --host=HOSTNAME database server host or socket directory (default: \"%s\")\n"),
@@ -52,8 +53,9 @@ usage(const char *progname)
printf(_(" -p, --port=PORT database server port (default: \"%s\")\n"),
env ? env : DEF_PGPORT_STR);
printf(_(" -s, send output to database server\n"));
+ printf(_(" -w expect password on input, not a key\n"));
printf(_(" -?, --help show this help, then exit\n\n"));
- printf(_("Key is read from stdin and sent either to stdout or to PostgreSQL server being started\n"));
+ printf(_("Password or key is read from stdin. Key is sent to PostgreSQL server being started\n"));
}
#endif /* USE_ENCRYPTION */
@@ -69,9 +71,12 @@ main(int argc, char **argv)
int c;
char *host = NULL;
char *port_str = NULL;
+ char *DataDir = NULL;
bool to_server = false;
+ bool expect_password = false;
int i, n;
int optindex;
+ char password[ENCRYPTION_PWD_MAX_LENGTH];
char key_chars[ENCRYPTION_KEY_CHARS];
static struct option long_options[] =
@@ -99,11 +104,15 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "h:p:s",
+ while ((c = getopt_long(argc, argv, "h:D:p:sw",
long_options, &optindex)) != -1)
{
switch (c)
{
+ case 'D':
+ DataDir = optarg;
+ break;
+
case 'h':
host = pg_strdup(optarg);
break;
@@ -116,6 +125,10 @@ main(int argc, char **argv)
to_server = true;
break;
+ case 'w':
+ expect_password = true;
+ break;
+
case '?':
/* Actual help option given */
if (strcmp(argv[optind - 1], "-?") == 0)
@@ -146,34 +159,78 @@ main(int argc, char **argv)
exit(1);
}
- /* Read the key. */
+ /*
+ * The KDF file is needed to derive the key from password, and this file
+ * is located in the data directory.
+ */
+ if (expect_password && DataDir == NULL)
+ {
+ pg_log_error("%s: no data directory specified", progname);
+ pg_log_error("Try \"%s --help\" for more information.", progname);
+ exit(EXIT_FAILURE);
+ }
+
+ /*
+ * Read the credentials (key or password).
+ */
n = 0;
/* Key length in characters (two characters per hexadecimal digit) */
while ((c = getchar()) != EOF && c != '\n')
{
- if (n >= ENCRYPTION_KEY_CHARS)
+ if (!expect_password)
{
- pg_log_error("The key is too long");
- exit(EXIT_FAILURE);
+ if (n >= ENCRYPTION_KEY_CHARS)
+ {
+ pg_log_error("The key is too long");
+ exit(EXIT_FAILURE);
+ }
+
+ key_chars[n++] = c;
}
+ else
+ {
+ if (n >= ENCRYPTION_PWD_MAX_LENGTH)
+ {
+ pg_log_error("The password is too long");
+ exit(EXIT_FAILURE);
+ }
- key_chars[n++] = c;
+ password[n++] = c;
+ }
}
- if (n < ENCRYPTION_KEY_CHARS)
+ /* If password was received, turn it into encryption key. */
+ if (!expect_password)
{
- pg_log_error("The key is too short");
- exit(EXIT_FAILURE);
- }
+ if (n < ENCRYPTION_KEY_CHARS)
+ {
+ pg_log_error("The key is too short");
+ exit(EXIT_FAILURE);
+ }
- for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++)
+ for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++)
+ {
+ if (sscanf(key_chars + 2 * i, "%2hhx", encryption_key + i) == 0)
+ {
+ pg_log_error("Invalid character in encryption key at position %d",
+ 2 * i);
+ exit(EXIT_FAILURE);
+ }
+ }
+ }
+ else
{
- if (sscanf(key_chars + 2 * i, "%2hhx", encryption_key + i) == 0)
+ if (n < ENCRYPTION_PWD_MIN_LENGTH)
{
- pg_log_error("Invalid character in encryption key at position %d",
- 2 * i);
+ pg_log_error("The password is too short");
exit(EXIT_FAILURE);
}
+
+ /* Read the KDF parameters. */
+ read_kdf_file(DataDir);
+
+ /* Run the KDF. */
+ derive_key_from_password(encryption_key, password, n);
}
/*
diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c
index ca37c9f373..134b3bde9b 100644
--- a/src/fe_utils/encryption.c
+++ b/src/fe_utils/encryption.c
@@ -21,31 +21,294 @@
#include "common/file_perm.h"
#include "common/logging.h"
#include "fe_utils/encryption.h"
-#include "storage/encryption.h"
#include "libpq-fe.h"
#include "libpq-int.h"
#include "libpq/pqcomm.h"
char *encryption_key_command = NULL;
+#define KDF_PARAMS_FILE "global/kdf_params"
+#define KDF_PARAMS_FILE_SIZE 512
+
+/*
+ * Key derivation function.
+ */
+typedef enum KDFKind
+{
+ KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA = 0
+} KFDKind;
+
+typedef struct KDFParamsPBKDF2
+{
+ unsigned long int niter;
+ unsigned char salt[ENCRYPTION_KDF_SALT_LEN];
+} KDFParamsPBKDF2;
+
+/*
+ * Parameters of the key derivation function.
+ *
+ * The parameters are generated by initdb and stored into a file, which is
+ * then read during PG startup. This is similar to storing various settings in
+ * pg_control. However an existing KDF file is read only, so it does not have
+ * to be stored in shared memory.
+ */
+typedef struct KDFParamsData
+{
+ KFDKind function;
+
+ /*
+ * Function-specific parameters.
+ */
+ union
+ {
+ KDFParamsPBKDF2 pbkdf2;
+ } data;
+
+ /* CRC of all above ... MUST BE LAST! */
+ pg_crc32c crc;
+} KDFParamsData;
+
+extern KDFParamsData *KDFParams;
+
+/*
+ * Pointer to the KDF parameters.
+ */
+KDFParamsData *KDFParams = NULL;
+
+/* Initialize KDF file. */
+void
+init_kdf(void)
+{
+ KDFParamsPBKDF2 *params;
+ struct timeval tv;
+ uint64 salt;
+
+ /*
+ * The initialization should not be repeated.
+ */
+ Assert(KDFParams == NULL);
+
+ KDFParams = palloc0(KDF_PARAMS_FILE_SIZE);
+ KDFParams->function = KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA;
+ params = &KDFParams->data.pbkdf2;
+
+ /*
+ * Currently we derive the salt in the same way as system identifier,
+ * however these two values are not supposed to match. XXX Is it worth the
+ * effort if initdb derives the system identifier, passes it to this
+ * function and also sends it to the bootstrap process? Not sure.
+ */
+ gettimeofday(&tv, NULL);
+ salt = ((uint64) tv.tv_sec) << 32;
+ salt |= ((uint64) tv.tv_usec) << 12;
+ salt |= getpid() & 0xFFF;
+
+ memcpy(params->salt, &salt, sizeof(uint64));
+ params->niter = ENCRYPTION_KDF_NITER;
+}
+
+/*
+ * Write KDFParamsData to file.
+ */
+void
+write_kdf_file(char *dir)
+{
+ char path[MAXPGPATH];
+ int fd;
+
+ Assert(KDFParams != NULL);
+
+ /* Account for both file separator and terminating NULL character. */
+ if ((strlen(dir) + 1 + strlen(KDF_PARAMS_FILE) + 1) > MAXPGPATH)
+ {
+ pg_log_fatal("KDF directory is too long");
+ exit(EXIT_FAILURE);
+ }
+
+ snprintf(path, MAXPGPATH, "%s/%s", dir, KDF_PARAMS_FILE);
+
+ /* Contents are protected with a CRC */
+ INIT_CRC32C(KDFParams->crc);
+ COMP_CRC32C(KDFParams->crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(KDFParams->crc);
+
+ fd = open(path, O_WRONLY | O_CREAT | PG_BINARY,
+ pg_file_create_mode);
+ if (fd < 0)
+ {
+ pg_log_fatal("could not create key derivation file \"%s\": %m", path);
+ exit(EXIT_FAILURE);
+ }
+
+ if (write(fd, KDFParams, KDF_PARAMS_FILE_SIZE) != KDF_PARAMS_FILE_SIZE)
+ {
+ /* if write didn't set errno, assume problem is no disk space */
+ if (errno == 0)
+ errno = ENOSPC;
+ pg_log_fatal("could not write to key derivation file \"%s\": %m",
+ path);
+ exit(EXIT_FAILURE);
+ }
+
+ if (close(fd))
+ {
+ pg_log_fatal("could not close key setup file: %m");
+ exit(EXIT_FAILURE);
+ }
+}
+
+/*
+ * Read KDFParamsData from file and store it in local memory.
+ *
+ * If dir is NULL, assume we're in the data directory.
+ *
+ * postmaster should call the function early enough for any other process to
+ * inherit valid pointer to the data.
+ */
+void
+read_kdf_file(char *dir)
+{
+ pg_crc32c crc;
+ char path[MAXPGPATH];
+ int fd;
+
+ /* Account for both file separator and terminating NULL character. */
+ if ((strlen(dir) + 1 + strlen(KDF_PARAMS_FILE) + 1) > MAXPGPATH)
+ {
+ pg_log_fatal("KDF directory is too long");
+ exit(EXIT_FAILURE);
+ }
+
+ snprintf(path, MAXPGPATH, "%s/%s", dir, KDF_PARAMS_FILE);
+
+ KDFParams = palloc0(KDF_PARAMS_FILE_SIZE);
+ fd = open(path, O_RDONLY | PG_BINARY, S_IRUSR);
+
+ if (fd < 0)
+ {
+ pg_log_fatal("could not open key setup file \"%s\": %m", path);
+ exit(EXIT_FAILURE);
+ }
+
+ if (read(fd, KDFParams, sizeof(KDFParamsData)) != sizeof(KDFParamsData))
+ {
+ pg_log_fatal("could not read from key setup file \"%s\": %m", path);
+ exit(EXIT_FAILURE);
+ }
+
+ close(fd);
+
+ /* Now check the CRC. */
+ INIT_CRC32C(crc);
+ COMP_CRC32C(crc,
+ (char *) KDFParams,
+ offsetof(KDFParamsData, crc));
+ FIN_CRC32C(crc);
+
+ if (!EQ_CRC32C(crc, KDFParams->crc))
+ {
+ pg_log_fatal("incorrect checksum in key setup file \"%s\"", path);
+ exit(EXIT_FAILURE);
+ }
+
+
+ if (KDFParams->function != KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA)
+ {
+ pg_log_fatal("unsupported KDF function");
+ exit(EXIT_FAILURE);
+ }
+}
+
+/*
+ * Run the key derivation function and initialize encryption_key variable.
+ */
+void
+derive_key_from_password(unsigned char *encryption_key, const char *password,
+ int len)
+{
+ KDFParamsPBKDF2 *params;
+ int rc;
+
+ params = &KDFParams->data.pbkdf2;
+ rc = PKCS5_PBKDF2_HMAC(password,
+ len,
+ params->salt,
+ ENCRYPTION_KDF_SALT_LEN,
+ params->niter,
+ EVP_sha1(),
+ ENCRYPTION_KEY_LENGTH,
+ encryption_key);
+
+ if (rc != 1)
+ {
+ pg_log_fatal("failed to derive key from password");
+ exit(EXIT_FAILURE);
+ }
+}
+
/*
* Run the command that is supposed to generate encryption key and store it
- * where encryption_key points to.
+ * where encryption_key points to. If valid string is passed for data_dir,
+ * it's used to replace '%D' pattern in the command.
*/
void
-run_encryption_key_command(unsigned char *encryption_key)
+run_encryption_key_command(unsigned char *encryption_key, char *data_dir)
{
FILE *fp;
+ char cmd[MAXPGPATH];
+ char *sp, *dp, *endp;
char *buf;
int read_len, i, c;
Assert(encryption_key_command != NULL &&
strlen(encryption_key_command) > 0);
- fp = popen(encryption_key_command, "r");
+ /*
+ * Replace %D pattern in the command with the actual data directory path.
+ */
+ dp = cmd;
+ endp = cmd + MAXPGPATH - 1;
+ *endp = '\0';
+ for (sp = encryption_key_command; *sp; sp++)
+ {
+ if (*sp == '%')
+ {
+ if (sp[1] == 'D')
+ {
+ if (data_dir == NULL)
+ {
+ pg_log_fatal("data directory is not known, %%D pattern cannot be replaced");
+ exit(EXIT_FAILURE);
+ }
+
+ sp++;
+ strlcpy(dp, data_dir, endp - dp);
+ make_native_path(dp);
+ dp += strlen(dp);
+ }
+ else if (dp < endp)
+ *dp++ = *sp;
+ else
+ break;
+ }
+ else
+ {
+ if (dp < endp)
+ *dp++ = *sp;
+ else
+ break;
+ }
+ }
+ *dp = '\0';
+
+ pg_log_debug("executing encryption key command \"%s\"", cmd);
+
+ fp = popen(cmd, "r");
if (fp == NULL)
{
- pg_log_fatal("Failed to execute \"%s\"", encryption_key_command);
+ pg_log_fatal("Failed to execute \"%s\"", cmd);
exit(EXIT_FAILURE);
}
diff --git a/src/include/fe_utils/encryption.h b/src/include/fe_utils/encryption.h
index 7307557ed6..da302fe494 100644
--- a/src/include/fe_utils/encryption.h
+++ b/src/include/fe_utils/encryption.h
@@ -15,6 +15,12 @@
/* Executable to retrieve the encryption key. */
extern char *encryption_key_command;
-extern void run_encryption_key_command(unsigned char *encryption_key);
+extern void init_kdf(void);
+extern void write_kdf_file(char *dir);
+extern void read_kdf_file(char *dir);
+extern void derive_key_from_password(unsigned char *encryption_key,
+ const char *password, int len);
+extern void run_encryption_key_command(unsigned char *encryption_key,
+ char *data_dir);
extern bool send_key_to_postmaster(const char *host, const char *port,
const unsigned char *encryption_Key);
diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h
index 4f7b96d4f3..72bcae0972 100644
--- a/src/include/storage/encryption.h
+++ b/src/include/storage/encryption.h
@@ -57,6 +57,14 @@ typedef enum CipherKind
PG_CIPHER_AES_BLOCK_CBC_256_STREAM_CTR_256
} CipherKind;
+/*
+ * TODO Tune these values.
+ */
+#define ENCRYPTION_PWD_MIN_LENGTH 8
+#define ENCRYPTION_PWD_MAX_LENGTH 16
+#define ENCRYPTION_KDF_NITER 1048576
+#define ENCRYPTION_KDF_SALT_LEN sizeof(uint64)
+
/* Key to encrypt / decrypt data. */
extern unsigned char encryption_key[];
--
2.13.7
--=-=-=
Content-Type: text/x-diff
Content-Disposition: attachment;
filename=v04-0007-Refactoring-patch.-This-only-moves-some-code-to-XLog.patch
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH 4/4] Support clauses of the form Var op Var
@ 2020-03-08 22:27 Tomas Vondra <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Tomas Vondra @ 2020-03-08 22:27 UTC (permalink / raw)
---
src/backend/statistics/extended_stats.c | 63 ++++++++++++----
src/backend/statistics/mcv.c | 75 ++++++++++++++++++-
.../statistics/extended_stats_internal.h | 2 +-
src/test/regress/expected/stats_ext.out | 72 ++++++++++++++++++
src/test/regress/sql/stats_ext.sql | 22 ++++++
5 files changed, 217 insertions(+), 17 deletions(-)
diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c
index daf95ff437..ddcb5d2955 100644
--- a/src/backend/statistics/extended_stats.c
+++ b/src/backend/statistics/extended_stats.c
@@ -986,14 +986,18 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause,
{
RangeTblEntry *rte = root->simple_rte_array[relid];
OpExpr *expr = (OpExpr *) clause;
- Var *var;
+ Var *var,
+ *var2;
/* Only expressions with two arguments are considered compatible. */
if (list_length(expr->args) != 2)
return false;
- /* Check if the expression the right shape (one Var, one Const) */
- if (!examine_opclause_expression(expr, &var, NULL, NULL))
+ /*
+ * Check if the expression the right shape (one Var and one Const,
+ * or two Vars).
+ */
+ if (!examine_opclause_expression(expr, &var, &var2, NULL, NULL))
return false;
/*
@@ -1033,7 +1037,20 @@ statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause,
!get_func_leakproof(get_opcode(expr->opno)))
return false;
- return statext_is_compatible_clause_internal(root, (Node *) var,
+ /*
+ * Check compatibility of the first Var - we get this one for both
+ * types of supported expressions (Var op Const) and (Var op Var).
+ */
+ if (!statext_is_compatible_clause_internal(root, (Node *) var,
+ relid, attnums))
+ return false;
+
+ /* For (Var op Const) we don't get the second Var, and we're done. */
+ if (!var2)
+ return true;
+
+ /* For (Var op Var) check compatibility of the second Var. */
+ return statext_is_compatible_clause_internal(root, (Node *) var2,
relid, attnums);
}
@@ -1422,19 +1439,21 @@ statext_clauselist_selectivity(PlannerInfo *root, List *clauses, int varRelid,
* examine_opclause_expression
* Split expression into Var and Const parts.
*
- * Attempts to match the arguments to either (Var op Const) or (Const op Var),
- * possibly with a RelabelType on top. When the expression matches this form,
- * returns true, otherwise returns false.
+ * Attempts to match the arguments to either (Var op Const) or (Const op Var)
+ * or (Var op Var), possibly with a RelabelType on top. When the expression
+ * matches this form, returns true, otherwise returns false.
*
* Optionally returns pointers to the extracted Var/Const nodes, when passed
* non-null pointers (varp, cstp and varonleftp). The varonleftp flag specifies
* on which side of the operator we found the Var node.
*/
bool
-examine_opclause_expression(OpExpr *expr, Var **varp, Const **cstp, bool *varonleftp)
+examine_opclause_expression(OpExpr *expr, Var **var1p, Var **var2p,
+ Const **cstp, bool *varonleftp)
{
- Var *var;
- Const *cst;
+ Var *var1 = NULL;
+ Var *var2 = NULL;
+ Const *cst = NULL;
bool varonleft;
Node *leftop,
*rightop;
@@ -1454,22 +1473,38 @@ examine_opclause_expression(OpExpr *expr, Var **varp, Const **cstp, bool *varonl
if (IsA(leftop, Var) && IsA(rightop, Const))
{
- var = (Var *) leftop;
+ var1 = (Var *) leftop;
cst = (Const *) rightop;
varonleft = true;
}
else if (IsA(leftop, Const) && IsA(rightop, Var))
{
- var = (Var *) rightop;
+ var1 = (Var *) rightop;
cst = (Const *) leftop;
varonleft = false;
}
+ else if (IsA(leftop, Var) && IsA(rightop, Var))
+ {
+ var1 = (Var *) leftop;
+ var2 = (Var *) rightop;
+ varonleft = false;
+
+ /*
+ * Both variables have to be for the same relation (otherwise it's
+ * a join clause, and we don't deal with those yet.
+ */
+ if (var1->varno != var2->varno)
+ return false;
+ }
else
return false;
/* return pointers to the extracted parts if requested */
- if (varp)
- *varp = var;
+ if (var1p)
+ *var1p = var1;
+
+ if (var2p)
+ *var2p = var2;
if (cstp)
*cstp = cst;
diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c
index 3f42713aa2..97d3083451 100644
--- a/src/backend/statistics/mcv.c
+++ b/src/backend/statistics/mcv.c
@@ -1581,16 +1581,25 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses,
/* valid only after examine_opclause_expression returns true */
Var *var;
+ Var *var2;
Const *cst;
bool varonleft;
fmgr_info(get_opcode(expr->opno), &opproc);
- /* extract the var and const from the expression */
- if (examine_opclause_expression(expr, &var, &cst, &varonleft))
+ /* extract the vars and const from the expression */
+ if (!examine_opclause_expression(expr, &var, &var2, &cst, &varonleft))
+ continue; /* XXX Can this actually happen? */
+
+ /* We should always get at least one Var. */
+ Assert(var);
+
+ if (cst)
{
int idx;
+ Assert(!var2);
+
/* match the attribute to a dimension of the statistic */
idx = bms_member_index(keys, var->varattno);
@@ -1651,6 +1660,68 @@ mcv_get_match_bitmap(PlannerInfo *root, List *clauses,
matches[i] = RESULT_MERGE(matches[i], is_or, match);
}
}
+ else
+ {
+ int idx;
+ int idx2;
+
+ Assert(var2);
+
+ /* match the attribute to a dimension of the statistic */
+ idx = bms_member_index(keys, var->varattno);
+ idx2 = bms_member_index(keys, var2->varattno);
+
+ /*
+ * Walk through the MCV items and evaluate the current clause.
+ * We can skip items that were already ruled out, and
+ * terminate if there are no remaining MCV items that might
+ * possibly match.
+ */
+ for (i = 0; i < mcvlist->nitems; i++)
+ {
+ bool match = true;
+ MCVItem *item = &mcvlist->items[i];
+
+ /*
+ * When either of the MCV items is NULL we can treat this
+ * as a mismatch. We must not call the operator because
+ * of strictness.
+ */
+ if (item->isnull[idx] || item->isnull[idx2])
+ {
+ matches[i] = RESULT_MERGE(matches[i], is_or, false);
+ continue;
+ }
+
+ /*
+ * Skip MCV items that can't change result in the bitmap.
+ * Once the value gets false for AND-lists, or true for
+ * OR-lists, we don't need to look at more clauses.
+ */
+ if (RESULT_IS_FINAL(matches[i], is_or))
+ continue;
+
+ /*
+ * First check whether the constant is below the lower
+ * boundary (in that case we can skip the bucket, because
+ * there's no overlap).
+ *
+ * We don't store collations used to build the statistics,
+ * but we can use the collation for the attribute itself,
+ * as stored in varcollid. We do reset the statistics after
+ * a type change (including collation change), so this is
+ * OK. We may need to relax this after allowing extended
+ * statistics on expressions.
+ */
+ match = DatumGetBool(FunctionCall2Coll(&opproc,
+ var->varcollid,
+ item->values[idx],
+ item->values[idx2]));
+
+ /* update the match bitmap with the result */
+ matches[i] = RESULT_MERGE(matches[i], is_or, match);
+ }
+ }
}
else if (IsA(clause, NullTest))
{
diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h
index 5171895bba..804089bc57 100644
--- a/src/include/statistics/extended_stats_internal.h
+++ b/src/include/statistics/extended_stats_internal.h
@@ -96,7 +96,7 @@ extern SortItem *build_sorted_items(int numrows, int *nitems, HeapTuple *rows,
TupleDesc tdesc, MultiSortSupport mss,
int numattrs, AttrNumber *attnums);
-extern bool examine_opclause_expression(OpExpr *expr, Var **varp,
+extern bool examine_opclause_expression(OpExpr *expr, Var **var1p, Var **var2p,
Const **cstp, bool *varonleftp);
extern Selectivity mcv_clauselist_selectivity(PlannerInfo *root,
diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out
index 383465e8cb..3eb1804cf3 100644
--- a/src/test/regress/expected/stats_ext.out
+++ b/src/test/regress/expected/stats_ext.out
@@ -603,6 +603,18 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = '
343 | 200
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a > c');
+ estimated | actual
+-----------+--------
+ 1667 | 3750
+(1 row)
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < c');
+ estimated | actual
+-----------+--------
+ 1667 | 0
+(1 row)
+
-- create statistics
CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
ANALYZE mcv_lists;
@@ -654,6 +666,18 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = '
200 | 200
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a > c');
+ estimated | actual
+-----------+--------
+ 3750 | 3750
+(1 row)
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < c');
+ estimated | actual
+-----------+--------
+ 1 | 0
+(1 row)
+
-- check change of unrelated column type does not reset the MCV statistics
ALTER TABLE mcv_lists ALTER COLUMN d TYPE VARCHAR(64);
SELECT d.stxdmcv IS NOT NULL
@@ -749,6 +773,12 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = ''x'' OR d
3750 | 2500
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = d');
+ estimated | actual
+-----------+--------
+ 25 | 2500
+(1 row)
+
-- create statistics
CREATE STATISTICS mcv_lists_stats (mcv) ON b, d FROM mcv_lists;
ANALYZE mcv_lists;
@@ -758,6 +788,12 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = ''x'' OR d
2500 | 2500
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = d');
+ estimated | actual
+-----------+--------
+ 2500 | 2500
+(1 row)
+
-- mcv with arrays
CREATE TABLE mcv_lists_arrays (
a TEXT[],
@@ -808,6 +844,18 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a AND
1094 | 0
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b');
+ estimated | actual
+-----------+--------
+ 9950 | 2500
+(1 row)
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b AND b = c');
+ estimated | actual
+-----------+--------
+ 50 | 2500
+(1 row)
+
CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c
FROM mcv_lists_bool;
ANALYZE mcv_lists_bool;
@@ -835,6 +883,18 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a AND
1 | 0
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b');
+ estimated | actual
+-----------+--------
+ 2500 | 2500
+(1 row)
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b AND b = c');
+ estimated | actual
+-----------+--------
+ 2500 | 2500
+(1 row)
+
-- check the ability to use multiple MCV lists
CREATE TABLE mcv_lists_multi (
a INTEGER,
@@ -893,6 +953,12 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR
2649 | 1572
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = b AND c = d');
+ estimated | actual
+-----------+--------
+ 1 | 5000
+(1 row)
+
-- create separate MCV statistics
CREATE STATISTICS mcv_lists_multi_1 (mcv) ON a, b FROM mcv_lists_multi;
CREATE STATISTICS mcv_lists_multi_2 (mcv) ON c, d FROM mcv_lists_multi;
@@ -939,6 +1005,12 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR
1571 | 1572
(1 row)
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = b AND c = d');
+ estimated | actual
+-----------+--------
+ 5000 | 5000
+(1 row)
+
DROP TABLE mcv_lists_multi;
-- Permission tests. Users should not be able to see specific data values in
-- the extended statistics, if they lack permission to see those values in
diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql
index 6fd8b016bd..42b67f7cde 100644
--- a/src/test/regress/sql/stats_ext.sql
+++ b/src/test/regress/sql/stats_ext.sql
@@ -381,6 +381,10 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = '
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a > c');
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < c');
+
-- create statistics
CREATE STATISTICS mcv_lists_stats (mcv) ON a, b, c FROM mcv_lists;
@@ -402,6 +406,10 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = '
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a = 1 OR b = ''1'' OR c = 1 OR d IS NOT NULL');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a > c');
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE a < c');
+
-- check change of unrelated column type does not reset the MCV statistics
ALTER TABLE mcv_lists ALTER COLUMN d TYPE VARCHAR(64);
@@ -473,6 +481,8 @@ ANALYZE mcv_lists;
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = ''x'' OR d = ''x''');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = d');
+
-- create statistics
CREATE STATISTICS mcv_lists_stats (mcv) ON b, d FROM mcv_lists;
@@ -480,6 +490,8 @@ ANALYZE mcv_lists;
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = ''x'' OR d = ''x''');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists WHERE b = d');
+
-- mcv with arrays
CREATE TABLE mcv_lists_arrays (
a TEXT[],
@@ -521,6 +533,10 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a AND
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b');
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b AND b = c');
+
CREATE STATISTICS mcv_lists_bool_stats (mcv) ON a, b, c
FROM mcv_lists_bool;
@@ -534,6 +550,10 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a AND
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a AND b AND NOT c');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b');
+
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_bool WHERE NOT a = b AND b = c');
+
-- check the ability to use multiple MCV lists
CREATE TABLE mcv_lists_multi (
a INTEGER,
@@ -560,6 +580,7 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE b = 0 OR
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 AND b = 0 AND c = 0 AND d = 0');
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE (a = 0 AND b = 0) OR (c = 0 AND d = 0)');
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR b = 0 OR c = 0 OR d = 0');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = b AND c = d');
-- create separate MCV statistics
CREATE STATISTICS mcv_lists_multi_1 (mcv) ON a, b FROM mcv_lists_multi;
@@ -574,6 +595,7 @@ SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE b = 0 OR
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 AND b = 0 AND c = 0 AND d = 0');
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE (a = 0 AND b = 0) OR (c = 0 AND d = 0)');
SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = 0 OR b = 0 OR c = 0 OR d = 0');
+SELECT * FROM check_estimated_rows('SELECT * FROM mcv_lists_multi WHERE a = b AND c = d');
DROP TABLE mcv_lists_multi;
--
2.21.1
--vcvacfll7i2xqkl4--
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v9 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..ea17ca4559 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(&p1, &p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--CE+1k2dSO48ffgeK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..95b8f69ecf 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--gBBFr7Ir9EOA20Yy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..95b8f69ecf 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--gBBFr7Ir9EOA20Yy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v10 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 198 ++++++++-------------------
1 file changed, 58 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..b30b49702d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,25 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ /* return opposite of qsort comparator for max-heap */
+ return -TocEntrySizeCompareQsort(&p1, &p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4262,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4319,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4365,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4383,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4626,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4648,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--y0ulUmNC+osPPQO6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v10 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 198 ++++++++-------------------
1 file changed, 58 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..b30b49702d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,25 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ /* return opposite of qsort comparator for max-heap */
+ return -TocEntrySizeCompareQsort(&p1, &p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4262,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4319,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4365,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4383,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4626,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4648,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--y0ulUmNC+osPPQO6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v6 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 39ebcfec32..4c86f7b448 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -110,16 +93,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -134,7 +113,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2380,7 +2359,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3980,7 +3959,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4023,24 +4002,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4054,7 +4035,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4064,7 +4045,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4075,7 +4056,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4089,7 +4070,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4118,10 +4099,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4221,80 +4202,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4314,17 +4224,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4340,38 +4257,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will usually be able to pick one of the first few items, which will
+ * generally be relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4397,7 +4314,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4443,7 +4360,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4461,7 +4378,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4704,11 +4621,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4726,18 +4643,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--YZ5djTAD1cGYuMQK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v9 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..ea17ca4559 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(&p1, &p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--CE+1k2dSO48ffgeK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v9 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..ea17ca4559 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(&p1, &p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--CE+1k2dSO48ffgeK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..95b8f69ecf 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--gBBFr7Ir9EOA20Yy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v10 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 198 ++++++++-------------------
1 file changed, 58 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..b30b49702d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,25 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ /* return opposite of qsort comparator for max-heap */
+ return -TocEntrySizeCompareQsort(&p1, &p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4262,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4319,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4365,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4383,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4626,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4648,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--y0ulUmNC+osPPQO6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue.
@ 2023-07-20 17:19 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-07-20 17:19 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us
---
src/bin/pg_dump/pg_backup_archiver.c | 197 ++++++++-------------------
1 file changed, 57 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 4d83381d84..95b8f69ecf 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -111,16 +94,12 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -135,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2384,7 +2363,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3984,7 +3963,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4027,24 +4006,26 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4058,7 +4039,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4068,7 +4049,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4079,7 +4060,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4093,7 +4074,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4122,10 +4103,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4225,80 +4206,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4318,17 +4228,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(void *p1, void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4344,38 +4261,38 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te = (TocEntry *) binaryheap_get_node(ready_heap, i);
bool conflicts = false;
/*
@@ -4401,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_node(ready_heap, i);
return te;
}
@@ -4447,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4465,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4708,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4730,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, otherte);
}
}
}
--
2.25.1
--gBBFr7Ir9EOA20Yy
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0004-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v7 4/5] Convert pg_restore's ready_list to a priority queue.
@ 2023-09-04 22:35 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-09-04 22:35 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
---
src/bin/pg_dump/pg_backup_archiver.c | 201 ++++++++-------------------
1 file changed, 61 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 39ebcfec32..ff7349537e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -110,16 +93,13 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(const void *p1, const void *p2,
+ void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -134,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2380,7 +2360,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3980,7 +3960,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4023,24 +4003,27 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ sizeof(TocEntry *),
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4054,7 +4037,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4064,7 +4047,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4075,7 +4058,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4089,7 +4072,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4118,10 +4101,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4221,80 +4204,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4314,17 +4226,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(const void *p1, const void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4340,40 +4259,42 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, &te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te;
bool conflicts = false;
+ binaryheap_nth(ready_heap, i, &te);
+
/*
* Check to see if the item would need exclusive lock on something
* that a currently running item also needs lock on, or vice versa. If
@@ -4397,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_nth(ready_heap, i);
return te;
}
@@ -4443,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4461,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4704,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4726,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, &otherte);
}
}
}
--
2.25.1
--DocE+STaALJfprDB
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0005-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v7 4/5] Convert pg_restore's ready_list to a priority queue.
@ 2023-09-04 22:35 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-09-04 22:35 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
---
src/bin/pg_dump/pg_backup_archiver.c | 201 ++++++++-------------------
1 file changed, 61 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 39ebcfec32..ff7349537e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -110,16 +93,13 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(const void *p1, const void *p2,
+ void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -134,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2380,7 +2360,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3980,7 +3960,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4023,24 +4003,27 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ sizeof(TocEntry *),
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4054,7 +4037,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4064,7 +4047,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4075,7 +4058,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4089,7 +4072,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4118,10 +4101,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4221,80 +4204,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4314,17 +4226,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(const void *p1, const void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4340,40 +4259,42 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, &te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te;
bool conflicts = false;
+ binaryheap_nth(ready_heap, i, &te);
+
/*
* Check to see if the item would need exclusive lock on something
* that a currently running item also needs lock on, or vice versa. If
@@ -4397,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_nth(ready_heap, i);
return te;
}
@@ -4443,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4461,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4704,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4726,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, &otherte);
}
}
}
--
2.25.1
--DocE+STaALJfprDB
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0005-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* [PATCH v7 4/5] Convert pg_restore's ready_list to a priority queue.
@ 2023-09-04 22:35 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Nathan Bossart @ 2023-09-04 22:35 UTC (permalink / raw)
Presently, we spend a lot of time sorting this list so that we pick
the largest items first. With many tables, this sorting can become
a significant bottleneck. There are a couple of reports from the
field about this, and it is easily reproducible, so this is not a
hypothetical issue.
This commit improves the performance of pg_restore with many tables
by converting its ready_list to a priority queue, i.e., a binary
heap. We will first try to run the highest priority item, but if
it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable. This means that we might end up picking an item with
much lower priority, but since we expect that we'll typically be
able to choose one of the first few nodes, we should usually pick
an item with a relatively high priority.
On my machine, a basic test with 100,000 tables takes 11.5 minutes
without this patch and 1.5 minutes with it. Pierre Ducroquet
claims to see a speedup from 30 minutes to 23 minutes for a
real-world dump of over 50,000 tables.
---
src/bin/pg_dump/pg_backup_archiver.c | 201 ++++++++-------------------
1 file changed, 61 insertions(+), 140 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 39ebcfec32..ff7349537e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -34,6 +34,7 @@
#include "compress_io.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "lib/binaryheap.h"
#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -44,24 +45,6 @@
#define TEXT_DUMP_HEADER "--\n-- PostgreSQL database dump\n--\n\n"
#define TEXT_DUMPALL_HEADER "--\n-- PostgreSQL database cluster dump\n--\n\n"
-/*
- * State for tracking TocEntrys that are ready to process during a parallel
- * restore. (This used to be a list, and we still call it that, though now
- * it's really an array so that we can apply qsort to it.)
- *
- * tes[] is sized large enough that we can't overrun it.
- * The valid entries are indexed first_te .. last_te inclusive.
- * We periodically sort the array to bring larger-by-dataLength entries to
- * the front; "sorted" is true if the valid entries are known sorted.
- */
-typedef struct _parallelReadyList
-{
- TocEntry **tes; /* Ready-to-dump TocEntrys */
- int first_te; /* index of first valid entry in tes[] */
- int last_te; /* index of last valid entry in tes[] */
- bool sorted; /* are valid entries currently sorted? */
-} ParallelReadyList;
-
static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt,
const pg_compress_specification compression_spec,
@@ -110,16 +93,13 @@ static void restore_toc_entries_postfork(ArchiveHandle *AH,
static void pending_list_header_init(TocEntry *l);
static void pending_list_append(TocEntry *l, TocEntry *te);
static void pending_list_remove(TocEntry *te);
-static void ready_list_init(ParallelReadyList *ready_list, int tocCount);
-static void ready_list_free(ParallelReadyList *ready_list);
-static void ready_list_insert(ParallelReadyList *ready_list, TocEntry *te);
-static void ready_list_remove(ParallelReadyList *ready_list, int i);
-static void ready_list_sort(ParallelReadyList *ready_list);
-static int TocEntrySizeCompare(const void *p1, const void *p2);
-static void move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+static int TocEntrySizeCompareQsort(const void *p1, const void *p2);
+static int TocEntrySizeCompareBinaryheap(const void *p1, const void *p2,
+ void *arg);
+static void move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass);
-static TocEntry *pop_next_work_item(ParallelReadyList *ready_list,
+static TocEntry *pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate);
static void mark_dump_job_done(ArchiveHandle *AH,
TocEntry *te,
@@ -134,7 +114,7 @@ static bool has_lock_conflicts(TocEntry *te1, TocEntry *te2);
static void repoint_table_dependencies(ArchiveHandle *AH);
static void identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te);
static void reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list);
+ binaryheap *ready_heap);
static void mark_create_done(ArchiveHandle *AH, TocEntry *te);
static void inhibit_data_for_failed_table(ArchiveHandle *AH, TocEntry *te);
@@ -2380,7 +2360,7 @@ WriteDataChunks(ArchiveHandle *AH, ParallelState *pstate)
}
if (ntes > 1)
- qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompare);
+ qsort(tes, ntes, sizeof(TocEntry *), TocEntrySizeCompareQsort);
for (int i = 0; i < ntes; i++)
DispatchJobForTocEntry(AH, pstate, tes[i], ACT_DUMP,
@@ -3980,7 +3960,7 @@ restore_toc_entries_prefork(ArchiveHandle *AH, TocEntry *pending_list)
(void) restore_toc_entry(AH, next_work_item, false);
- /* Reduce dependencies, but don't move anything to ready_list */
+ /* Reduce dependencies, but don't move anything to ready_heap */
reduce_dependencies(AH, next_work_item, NULL);
}
else
@@ -4023,24 +4003,27 @@ static void
restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
TocEntry *pending_list)
{
- ParallelReadyList ready_list;
+ binaryheap *ready_heap;
TocEntry *next_work_item;
pg_log_debug("entering restore_toc_entries_parallel");
- /* Set up ready_list with enough room for all known TocEntrys */
- ready_list_init(&ready_list, AH->tocCount);
+ /* Set up ready_heap with enough room for all known TocEntrys */
+ ready_heap = binaryheap_allocate(AH->tocCount,
+ sizeof(TocEntry *),
+ TocEntrySizeCompareBinaryheap,
+ NULL);
/*
* The pending_list contains all items that we need to restore. Move all
- * items that are available to process immediately into the ready_list.
+ * items that are available to process immediately into the ready_heap.
* After this setup, the pending list is everything that needs to be done
- * but is blocked by one or more dependencies, while the ready list
+ * but is blocked by one or more dependencies, while the ready heap
* contains items that have no remaining dependencies and are OK to
* process in the current restore pass.
*/
AH->restorePass = RESTORE_PASS_MAIN;
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/*
* main parent loop
@@ -4054,7 +4037,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
for (;;)
{
/* Look for an item ready to be dispatched to a worker */
- next_work_item = pop_next_work_item(&ready_list, pstate);
+ next_work_item = pop_next_work_item(ready_heap, pstate);
if (next_work_item != NULL)
{
/* If not to be restored, don't waste time launching a worker */
@@ -4064,7 +4047,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item->dumpId,
next_work_item->desc, next_work_item->tag);
/* Update its dependencies as though we'd completed it */
- reduce_dependencies(AH, next_work_item, &ready_list);
+ reduce_dependencies(AH, next_work_item, ready_heap);
/* Loop around to see if anything else can be dispatched */
continue;
}
@@ -4075,7 +4058,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Dispatch to some worker */
DispatchJobForTocEntry(AH, pstate, next_work_item, ACT_RESTORE,
- mark_restore_job_done, &ready_list);
+ mark_restore_job_done, ready_heap);
}
else if (IsEveryWorkerIdle(pstate))
{
@@ -4089,7 +4072,7 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
/* Advance to next restore pass */
AH->restorePass++;
/* That probably allows some stuff to be made ready */
- move_to_ready_list(pending_list, &ready_list, AH->restorePass);
+ move_to_ready_heap(pending_list, ready_heap, AH->restorePass);
/* Loop around to see if anything's now ready */
continue;
}
@@ -4118,10 +4101,10 @@ restore_toc_entries_parallel(ArchiveHandle *AH, ParallelState *pstate,
next_work_item ? WFW_ONE_IDLE : WFW_GOT_STATUS);
}
- /* There should now be nothing in ready_list. */
- Assert(ready_list.first_te > ready_list.last_te);
+ /* There should now be nothing in ready_heap. */
+ Assert(binaryheap_empty(ready_heap));
- ready_list_free(&ready_list);
+ binaryheap_free(ready_heap);
pg_log_info("finished main parallel loop");
}
@@ -4221,80 +4204,9 @@ pending_list_remove(TocEntry *te)
}
-/*
- * Initialize the ready_list with enough room for up to tocCount entries.
- */
-static void
-ready_list_init(ParallelReadyList *ready_list, int tocCount)
-{
- ready_list->tes = (TocEntry **)
- pg_malloc(tocCount * sizeof(TocEntry *));
- ready_list->first_te = 0;
- ready_list->last_te = -1;
- ready_list->sorted = false;
-}
-
-/*
- * Free storage for a ready_list.
- */
-static void
-ready_list_free(ParallelReadyList *ready_list)
-{
- pg_free(ready_list->tes);
-}
-
-/* Add te to the ready_list */
-static void
-ready_list_insert(ParallelReadyList *ready_list, TocEntry *te)
-{
- ready_list->tes[++ready_list->last_te] = te;
- /* List is (probably) not sorted anymore. */
- ready_list->sorted = false;
-}
-
-/* Remove the i'th entry in the ready_list */
-static void
-ready_list_remove(ParallelReadyList *ready_list, int i)
-{
- int f = ready_list->first_te;
-
- Assert(i >= f && i <= ready_list->last_te);
-
- /*
- * In the typical case where the item to be removed is the first ready
- * entry, we need only increment first_te to remove it. Otherwise, move
- * the entries before it to compact the list. (This preserves sortedness,
- * if any.) We could alternatively move the entries after i, but there
- * are typically many more of those.
- */
- if (i > f)
- {
- TocEntry **first_te_ptr = &ready_list->tes[f];
-
- memmove(first_te_ptr + 1, first_te_ptr, (i - f) * sizeof(TocEntry *));
- }
- ready_list->first_te++;
-}
-
-/* Sort the ready_list into the desired order */
-static void
-ready_list_sort(ParallelReadyList *ready_list)
-{
- if (!ready_list->sorted)
- {
- int n = ready_list->last_te - ready_list->first_te + 1;
-
- if (n > 1)
- qsort(ready_list->tes + ready_list->first_te, n,
- sizeof(TocEntry *),
- TocEntrySizeCompare);
- ready_list->sorted = true;
- }
-}
-
/* qsort comparator for sorting TocEntries by dataLength */
static int
-TocEntrySizeCompare(const void *p1, const void *p2)
+TocEntrySizeCompareQsort(const void *p1, const void *p2)
{
const TocEntry *te1 = *(const TocEntry *const *) p1;
const TocEntry *te2 = *(const TocEntry *const *) p2;
@@ -4314,17 +4226,24 @@ TocEntrySizeCompare(const void *p1, const void *p2)
return 0;
}
+/* binaryheap comparator for sorting TocEntries by dataLength */
+static int
+TocEntrySizeCompareBinaryheap(const void *p1, const void *p2, void *arg)
+{
+ return TocEntrySizeCompareQsort(p1, p2);
+}
+
/*
- * Move all immediately-ready items from pending_list to ready_list.
+ * Move all immediately-ready items from pending_list to ready_heap.
*
* Items are considered ready if they have no remaining dependencies and
* they belong in the current restore pass. (See also reduce_dependencies,
* which applies the same logic one-at-a-time.)
*/
static void
-move_to_ready_list(TocEntry *pending_list,
- ParallelReadyList *ready_list,
+move_to_ready_heap(TocEntry *pending_list,
+ binaryheap *ready_heap,
RestorePass pass)
{
TocEntry *te;
@@ -4340,40 +4259,42 @@ move_to_ready_list(TocEntry *pending_list,
{
/* Remove it from pending_list ... */
pending_list_remove(te);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, te);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, &te);
}
}
}
/*
* Find the next work item (if any) that is capable of being run now,
- * and remove it from the ready_list.
+ * and remove it from the ready_heap.
*
* Returns the item, or NULL if nothing is runnable.
*
* To qualify, the item must have no remaining dependencies
* and no requirements for locks that are incompatible with
- * items currently running. Items in the ready_list are known to have
+ * items currently running. Items in the ready_heap are known to have
* no remaining dependencies, but we have to check for lock conflicts.
*/
static TocEntry *
-pop_next_work_item(ParallelReadyList *ready_list,
+pop_next_work_item(binaryheap *ready_heap,
ParallelState *pstate)
{
/*
- * Sort the ready_list so that we'll tackle larger jobs first.
- */
- ready_list_sort(ready_list);
-
- /*
- * Search the ready_list until we find a suitable item.
+ * Search the ready_heap until we find a suitable item. Note that we do a
+ * sequential scan through the heap nodes, so even though we will first
+ * try to choose the highest-priority item, we might end up picking
+ * something with a much lower priority. However, it is expected that we
+ * will typically be able to pick one of the first few items, which should
+ * usually have a relatively high priority.
*/
- for (int i = ready_list->first_te; i <= ready_list->last_te; i++)
+ for (int i = 0; i < binaryheap_size(ready_heap); i++)
{
- TocEntry *te = ready_list->tes[i];
+ TocEntry *te;
bool conflicts = false;
+ binaryheap_nth(ready_heap, i, &te);
+
/*
* Check to see if the item would need exclusive lock on something
* that a currently running item also needs lock on, or vice versa. If
@@ -4397,7 +4318,7 @@ pop_next_work_item(ParallelReadyList *ready_list,
continue;
/* passed all tests, so this item can run */
- ready_list_remove(ready_list, i);
+ binaryheap_remove_nth(ready_heap, i);
return te;
}
@@ -4443,7 +4364,7 @@ mark_restore_job_done(ArchiveHandle *AH,
int status,
void *callback_data)
{
- ParallelReadyList *ready_list = (ParallelReadyList *) callback_data;
+ binaryheap *ready_heap = (binaryheap *) callback_data;
pg_log_info("finished item %d %s %s",
te->dumpId, te->desc, te->tag);
@@ -4461,7 +4382,7 @@ mark_restore_job_done(ArchiveHandle *AH,
pg_fatal("worker process failed: exit code %d",
status);
- reduce_dependencies(AH, te, ready_list);
+ reduce_dependencies(AH, te, ready_heap);
}
@@ -4704,11 +4625,11 @@ identify_locking_dependencies(ArchiveHandle *AH, TocEntry *te)
/*
* Remove the specified TOC entry from the depCounts of items that depend on
* it, thereby possibly making them ready-to-run. Any pending item that
- * becomes ready should be moved to the ready_list, if that's provided.
+ * becomes ready should be moved to the ready_heap, if that's provided.
*/
static void
reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
- ParallelReadyList *ready_list)
+ binaryheap *ready_heap)
{
int i;
@@ -4726,18 +4647,18 @@ reduce_dependencies(ArchiveHandle *AH, TocEntry *te,
* the current restore pass, and it is currently a member of the
* pending list (that check is needed to prevent double restore in
* some cases where a list-file forces out-of-order restoring).
- * However, if ready_list == NULL then caller doesn't want any list
+ * However, if ready_heap == NULL then caller doesn't want any list
* memberships changed.
*/
if (otherte->depCount == 0 &&
_tocEntryRestorePass(otherte) == AH->restorePass &&
otherte->pending_prev != NULL &&
- ready_list != NULL)
+ ready_heap != NULL)
{
/* Remove it from pending list ... */
pending_list_remove(otherte);
- /* ... and add to ready_list */
- ready_list_insert(ready_list, otherte);
+ /* ... and add to ready_heap */
+ binaryheap_add(ready_heap, &otherte);
}
}
}
--
2.25.1
--DocE+STaALJfprDB
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0005-Remove-open-coded-binary-heap-in-pg_dump_sort.c.patch"
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
@ 2024-10-04 20:46 Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Tom Lane @ 2024-10-04 20:46 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: John Naylor <[email protected]>; [email protected]; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
I wrote:
> The cfbot points out that I should probably have marked progname
> as "static" in 0008. I'm not going to repost the patchset just for
> that, though.
Rebase needed after f22e84df1, so here's an update that rebases
up to HEAD and adds the missing "static". No other changes.
(Anybody want to review this? I'm getting tired of rebasing it,
and we're missing out on the clang build time savings.)
regards, tom lane
From b09c06c0c1b30f7bfe87d00439277c25774ffa6b Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:02:58 -0400
Subject: [PATCH v5 1/9] Clean up documentation of parse.pl, and add more input
checking.
README.parser is the user's manual, such as it is, for parse.pl.
It's rather poorly written if you ask me; so try to improve it.
(More could be written here, but this at least covers the same
info in a more organized fashion.)
Also, the single solitary line of usage info in parse.pl itself
was a lie. Replace.
Add some error checks that the ecpg.addons entries meet the syntax
rules set forth in README.parser. One of them didn't, but
accidentally worked anyway because the logic in include_addon is
such that 'block' is the default behavior.
Also add a cross-check that each ecpg.addons entry is matched exactly
once in the backend grammar. This exposed that there are two dead
entries there --- they are dead because the %replace_types table in
parse.pl causes their nonterminals to be ignored altogether.
Removing them doesn't change the generated preproc.y file.
(This implies that check_rules.pl is completely worthless and should
be nuked: it adds build cycles and maintenance effort while failing
to reliably accomplish its one job of detecting dead rules. I've
not done that here, though.)
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/README.parser | 119 ++++++++++++++--------
src/interfaces/ecpg/preproc/ecpg.addons | 11 +-
src/interfaces/ecpg/preproc/parse.pl | 58 ++++++++---
3 files changed, 123 insertions(+), 65 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/README.parser b/src/interfaces/ecpg/preproc/README.parser
index ddc3061d48..5698f5ab32 100644
--- a/src/interfaces/ecpg/preproc/README.parser
+++ b/src/interfaces/ecpg/preproc/README.parser
@@ -1,42 +1,77 @@
-ECPG modifies and extends the core grammar in a way that
-1) every token in ECPG is <str> type. New tokens are
- defined in ecpg.tokens, types are defined in ecpg.type
-2) most tokens from the core grammar are simply converted
- to literals concatenated together to form the SQL string
- passed to the server, this is done by parse.pl.
-3) some rules need side-effects, actions are either added
- or completely overridden (compared to the basic token
- concatenation) for them, these are defined in ecpg.addons,
- the rules for ecpg.addons are explained below.
-4) new grammar rules are needed for ECPG metacommands.
- These are in ecpg.trailer.
-5) ecpg.header contains common functions, etc. used by
- actions for grammar rules.
-
-In "ecpg.addons", every modified rule follows this pattern:
- ECPG: dumpedtokens postfix
-where "dumpedtokens" is simply tokens from core gram.y's
-rules concatenated together. e.g. if gram.y has this:
- ruleA: tokenA tokenB tokenC {...}
-then "dumpedtokens" is "ruleAtokenAtokenBtokenC".
-"postfix" above can be:
-a) "block" - the automatic rule created by parse.pl is completely
- overridden, the code block has to be written completely as
- it were in a plain bison grammar
-b) "rule" - the automatic rule is extended on, so new syntaxes
- are accepted for "ruleA". E.g.:
- ECPG: ruleAtokenAtokenBtokenC rule
- | tokenD tokenE { action_code; }
- ...
- It will be substituted with:
- ruleA: <original syntax forms and actions up to and including
- "tokenA tokenB tokenC">
- | tokenD tokenE { action_code; }
- ...
-c) "addon" - the automatic action for the rule (SQL syntax constructed
- from the tokens concatenated together) is prepended with a new
- action code part. This code part is written as is's already inside
- the { ... }
-
-Multiple "addon" or "block" lines may appear together with the
-new code block if the code block is common for those rules.
+ECPG's grammar (preproc.y) is built by parse.pl from the
+backend's grammar (gram.y) plus various add-on rules.
+Some notes:
+
+1) Most input matching core grammar productions is simply converted
+ to strings and concatenated together to form the SQL string
+ passed to the server. parse.pl can automatically build the
+ grammar actions needed to do this.
+2) Some grammar rules need special actions that are added to or
+ completely override the default token-concatenation behavior.
+ This is controlled by ecpg.addons as explained below.
+3) Additional grammar rules are needed for ECPG's own commands.
+ These are in ecpg.trailer, as is the "epilogue" part of preproc.y.
+4) ecpg.header contains the "prologue" part of preproc.y, including
+ support functions, Bison options, etc.
+5) Additional terminals added by ECPG must be defined in ecpg.tokens.
+ Additional nonterminals added by ECPG must be defined in ecpg.type.
+
+ecpg.header, ecpg.tokens, ecpg.type, and ecpg.trailer are just
+copied verbatim into preproc.y at appropriate points.
+
+ecpg.addons contains entries that begin with a line like
+ ECPG: concattokens ruletype
+and typically have one or more following lines that are the code
+for a grammar action. Any line not starting with "ECPG:" is taken
+to be part of the code block for the preceding "ECPG:" line.
+
+"concattokens" identifies which gram.y production this entry affects.
+It is simply the target nonterminal and the tokens from the gram.y rule
+concatenated together. For example, to modify the action for a gram.y
+rule like this:
+ target: tokenA tokenB tokenC {...}
+"concattokens" would be "targettokenAtokenBtokenC". If we want to
+modify a non-first alternative for a nonterminal, we still write the
+nonterminal. For example, "concattokens" should be "targettokenDtokenE"
+to affect the second alternative in:
+ target: tokenA tokenB tokenC {...}
+ | tokenD tokenE {...}
+
+"ruletype" is one of:
+
+a) "block" - the automatic action that parse.pl would create is
+ completely overridden. Instead the entry's code block is emitted.
+ The code block must include the braces ({}) needed for a Bison action.
+
+b) "addon" - the entry's code block is inserted into the generated
+ action, ahead of the automatic token-concatenation code.
+ In this case the code block need not contain braces, since
+ it will be inserted within braces.
+
+c) "rule" - the automatic action is emitted, but then the entry's
+ code block is added verbatim afterwards. This typically is
+ used to add new alternatives to a nonterminal of the core grammar.
+ For example, given the entry:
+ ECPG: targettokenAtokenBtokenC rule
+ | tokenD tokenE { custom_action; }
+ what will be emitted is
+ target: tokenA tokenB tokenC { automatic_action; }
+ | tokenD tokenE { custom_action; }
+
+Multiple "ECPG:" entries can share the same code block, if the
+same action is needed for all. When an "ECPG:" line is immediately
+followed by another one, it is not assigned an empty code block;
+rather the next nonempty code block is assumed to apply to all
+immediately preceding "ECPG:" entries.
+
+In addition to the modifications specified by ecpg.addons,
+parse.pl contains some tables that list backend grammar
+productions to be ignored or modified.
+
+Nonterminals that construct strings (as described above) should be
+given <str> type, which is parse.pl's default assumption for
+nonterminals found in gram.y. That can be overridden at need by
+making an entry in parse.pl's %replace_types table. %replace_types
+can also be used to suppress output of a nonterminal's rules
+altogether (in which case ecpg.trailer had better provide replacement
+rules, since the nonterminal will still be referred to elsewhere).
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index e7dce4e404..6a1893553b 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -497,7 +497,7 @@ ECPG: opt_array_boundsopt_array_bounds'['']' block
$$.index2 = mm_strdup($3);
$$.str = cat_str(4, $1.str, mm_strdup("["), $3, mm_strdup("]"));
}
-ECPG: opt_array_bounds
+ECPG: opt_array_bounds block
{
$$.index1 = mm_strdup("-1");
$$.index2 = mm_strdup("-1");
@@ -510,15 +510,6 @@ ECPG: IconstICONST block
ECPG: AexprConstNULL_P rule
| civar { $$ = $1; }
| civarind { $$ = $1; }
-ECPG: ColIdcol_name_keyword rule
- | ECPGKeywords { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
- | CHAR_P { $$ = mm_strdup("char"); }
- | VALUES { $$ = mm_strdup("values"); }
-ECPG: type_function_nametype_func_name_keyword rule
- | ECPGKeywords { $$ = $1; }
- | ECPGTypeName { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
ECPG: VariableShowStmtSHOWALL block
{
mmerror(PARSE_ERROR, ET_ERROR, "SHOW ALL is not implemented");
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
index fe8d3e5178..86d0782d45 100644
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -1,7 +1,13 @@
#!/usr/bin/perl
# src/interfaces/ecpg/preproc/parse.pl
-# parser generator for ecpg version 2
-# call with backend parser as stdin
+# parser generator for ecpg
+#
+# See README.parser for some explanation of what this does.
+#
+# Command-line options:
+# --srcdir: where to find ecpg-provided input files (default ".")
+# --parser: the backend gram.y file to read (required, no default)
+# --output: where to write preproc.y (required, no default)
#
# Copyright (c) 2007-2024, PostgreSQL Global Development Group
#
@@ -148,6 +154,14 @@ dump_buffer('trailer');
close($parserfh);
+# Cross-check that we don't have dead or ambiguous addon rules.
+foreach (keys %addons)
+{
+ die "addon rule $_ was never used\n" if $addons{$_}{used} == 0;
+ die "addon rule $_ was matched multiple times\n" if $addons{$_}{used} > 1;
+}
+
+
sub main
{
line: while (<$parserfh>)
@@ -487,7 +501,10 @@ sub include_addon
my $rec = $addons{$block};
return 0 unless $rec;
- my $rectype = (defined $rec->{type}) ? $rec->{type} : '';
+ # Track usage for later cross-check
+ $rec->{used}++;
+
+ my $rectype = $rec->{type};
if ($rectype eq 'rule')
{
dump_fields($stmt_mode, $fields, ' { ');
@@ -668,10 +685,10 @@ sub dump_line
}
=top
- load addons into cache
+ load ecpg.addons into %addons hash. The result is something like
%addons = {
- stmtClosePortalStmt => { 'type' => 'block', 'lines' => [ "{", "if (INFORMIX_MODE)" ..., "}" ] },
- stmtViewStmt => { 'type' => 'rule', 'lines' => [ "| ECPGAllocateDescr", ... ] }
+ stmtClosePortalStmt => { 'type' => 'block', 'lines' => [ "{", "if (INFORMIX_MODE)" ..., "}" ], 'used' => 0 },
+ stmtViewStmt => { 'type' => 'rule', 'lines' => [ "| ECPGAllocateDescr", ... ], 'used' => 0 }
}
=cut
@@ -681,17 +698,25 @@ sub preload_addons
my $filename = $srcdir . "/ecpg.addons";
open(my $fh, '<', $filename) or die;
- # there may be multiple lines starting ECPG: and then multiple lines of code.
- # the code need to be add to all prior ECPG records.
- my (@needsRules, @code, $record);
+ # There may be multiple "ECPG:" lines and then multiple lines of code.
+ # The block of code needs to be added to each of the consecutively-
+ # preceding "ECPG:" records.
+ my (@needsRules, @code);
- # there may be comments before the first ECPG line, skip them
+ # there may be comments before the first "ECPG:" line, skip them
my $skip = 1;
while (<$fh>)
{
- if (/^ECPG:\s(\S+)\s?(\w+)?/)
+ if (/^ECPG:\s+(\S+)\s+(\w+)\s*$/)
{
+ # Found an "ECPG:" line, so we're done skipping the header
$skip = 0;
+ # Validate record type and target
+ die "invalid record type $2 in addon rule for $1\n"
+ unless ($2 eq 'block' or $2 eq 'addon' or $2 eq 'rule');
+ die "duplicate addon rule for $1\n" if (exists $addons{$1});
+ # If we had some preceding code lines, attach them to all
+ # as-yet-unfinished records.
if (@code)
{
for my $x (@needsRules)
@@ -701,20 +726,27 @@ sub preload_addons
@code = ();
@needsRules = ();
}
- $record = {};
+ my $record = {};
$record->{type} = $2;
$record->{lines} = [];
- if (exists $addons{$1}) { die "Ga! there are dups!\n"; }
+ $record->{used} = 0;
$addons{$1} = $record;
push(@needsRules, $record);
}
+ elsif (/^ECPG:/)
+ {
+ # Complain if preceding regex failed to match
+ die "incorrect syntax in ECPG line: $_\n";
+ }
else
{
+ # Non-ECPG line: add to @code unless we're still skipping
next if $skip;
push(@code, $_);
}
}
close($fh);
+ # Deal with final code block
if (@code)
{
for my $x (@needsRules)
--
2.43.5
Attachments:
[text/plain] v5-0001-Clean-up-documentation-of-parse.pl-and-add-more-i.patch (12.4K, ../../[email protected]/2-v5-0001-Clean-up-documentation-of-parse.pl-and-add-more-i.patch)
download | inline diff:
From b09c06c0c1b30f7bfe87d00439277c25774ffa6b Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:02:58 -0400
Subject: [PATCH v5 1/9] Clean up documentation of parse.pl, and add more input
checking.
README.parser is the user's manual, such as it is, for parse.pl.
It's rather poorly written if you ask me; so try to improve it.
(More could be written here, but this at least covers the same
info in a more organized fashion.)
Also, the single solitary line of usage info in parse.pl itself
was a lie. Replace.
Add some error checks that the ecpg.addons entries meet the syntax
rules set forth in README.parser. One of them didn't, but
accidentally worked anyway because the logic in include_addon is
such that 'block' is the default behavior.
Also add a cross-check that each ecpg.addons entry is matched exactly
once in the backend grammar. This exposed that there are two dead
entries there --- they are dead because the %replace_types table in
parse.pl causes their nonterminals to be ignored altogether.
Removing them doesn't change the generated preproc.y file.
(This implies that check_rules.pl is completely worthless and should
be nuked: it adds build cycles and maintenance effort while failing
to reliably accomplish its one job of detecting dead rules. I've
not done that here, though.)
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/README.parser | 119 ++++++++++++++--------
src/interfaces/ecpg/preproc/ecpg.addons | 11 +-
src/interfaces/ecpg/preproc/parse.pl | 58 ++++++++---
3 files changed, 123 insertions(+), 65 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/README.parser b/src/interfaces/ecpg/preproc/README.parser
index ddc3061d48..5698f5ab32 100644
--- a/src/interfaces/ecpg/preproc/README.parser
+++ b/src/interfaces/ecpg/preproc/README.parser
@@ -1,42 +1,77 @@
-ECPG modifies and extends the core grammar in a way that
-1) every token in ECPG is <str> type. New tokens are
- defined in ecpg.tokens, types are defined in ecpg.type
-2) most tokens from the core grammar are simply converted
- to literals concatenated together to form the SQL string
- passed to the server, this is done by parse.pl.
-3) some rules need side-effects, actions are either added
- or completely overridden (compared to the basic token
- concatenation) for them, these are defined in ecpg.addons,
- the rules for ecpg.addons are explained below.
-4) new grammar rules are needed for ECPG metacommands.
- These are in ecpg.trailer.
-5) ecpg.header contains common functions, etc. used by
- actions for grammar rules.
-
-In "ecpg.addons", every modified rule follows this pattern:
- ECPG: dumpedtokens postfix
-where "dumpedtokens" is simply tokens from core gram.y's
-rules concatenated together. e.g. if gram.y has this:
- ruleA: tokenA tokenB tokenC {...}
-then "dumpedtokens" is "ruleAtokenAtokenBtokenC".
-"postfix" above can be:
-a) "block" - the automatic rule created by parse.pl is completely
- overridden, the code block has to be written completely as
- it were in a plain bison grammar
-b) "rule" - the automatic rule is extended on, so new syntaxes
- are accepted for "ruleA". E.g.:
- ECPG: ruleAtokenAtokenBtokenC rule
- | tokenD tokenE { action_code; }
- ...
- It will be substituted with:
- ruleA: <original syntax forms and actions up to and including
- "tokenA tokenB tokenC">
- | tokenD tokenE { action_code; }
- ...
-c) "addon" - the automatic action for the rule (SQL syntax constructed
- from the tokens concatenated together) is prepended with a new
- action code part. This code part is written as is's already inside
- the { ... }
-
-Multiple "addon" or "block" lines may appear together with the
-new code block if the code block is common for those rules.
+ECPG's grammar (preproc.y) is built by parse.pl from the
+backend's grammar (gram.y) plus various add-on rules.
+Some notes:
+
+1) Most input matching core grammar productions is simply converted
+ to strings and concatenated together to form the SQL string
+ passed to the server. parse.pl can automatically build the
+ grammar actions needed to do this.
+2) Some grammar rules need special actions that are added to or
+ completely override the default token-concatenation behavior.
+ This is controlled by ecpg.addons as explained below.
+3) Additional grammar rules are needed for ECPG's own commands.
+ These are in ecpg.trailer, as is the "epilogue" part of preproc.y.
+4) ecpg.header contains the "prologue" part of preproc.y, including
+ support functions, Bison options, etc.
+5) Additional terminals added by ECPG must be defined in ecpg.tokens.
+ Additional nonterminals added by ECPG must be defined in ecpg.type.
+
+ecpg.header, ecpg.tokens, ecpg.type, and ecpg.trailer are just
+copied verbatim into preproc.y at appropriate points.
+
+ecpg.addons contains entries that begin with a line like
+ ECPG: concattokens ruletype
+and typically have one or more following lines that are the code
+for a grammar action. Any line not starting with "ECPG:" is taken
+to be part of the code block for the preceding "ECPG:" line.
+
+"concattokens" identifies which gram.y production this entry affects.
+It is simply the target nonterminal and the tokens from the gram.y rule
+concatenated together. For example, to modify the action for a gram.y
+rule like this:
+ target: tokenA tokenB tokenC {...}
+"concattokens" would be "targettokenAtokenBtokenC". If we want to
+modify a non-first alternative for a nonterminal, we still write the
+nonterminal. For example, "concattokens" should be "targettokenDtokenE"
+to affect the second alternative in:
+ target: tokenA tokenB tokenC {...}
+ | tokenD tokenE {...}
+
+"ruletype" is one of:
+
+a) "block" - the automatic action that parse.pl would create is
+ completely overridden. Instead the entry's code block is emitted.
+ The code block must include the braces ({}) needed for a Bison action.
+
+b) "addon" - the entry's code block is inserted into the generated
+ action, ahead of the automatic token-concatenation code.
+ In this case the code block need not contain braces, since
+ it will be inserted within braces.
+
+c) "rule" - the automatic action is emitted, but then the entry's
+ code block is added verbatim afterwards. This typically is
+ used to add new alternatives to a nonterminal of the core grammar.
+ For example, given the entry:
+ ECPG: targettokenAtokenBtokenC rule
+ | tokenD tokenE { custom_action; }
+ what will be emitted is
+ target: tokenA tokenB tokenC { automatic_action; }
+ | tokenD tokenE { custom_action; }
+
+Multiple "ECPG:" entries can share the same code block, if the
+same action is needed for all. When an "ECPG:" line is immediately
+followed by another one, it is not assigned an empty code block;
+rather the next nonempty code block is assumed to apply to all
+immediately preceding "ECPG:" entries.
+
+In addition to the modifications specified by ecpg.addons,
+parse.pl contains some tables that list backend grammar
+productions to be ignored or modified.
+
+Nonterminals that construct strings (as described above) should be
+given <str> type, which is parse.pl's default assumption for
+nonterminals found in gram.y. That can be overridden at need by
+making an entry in parse.pl's %replace_types table. %replace_types
+can also be used to suppress output of a nonterminal's rules
+altogether (in which case ecpg.trailer had better provide replacement
+rules, since the nonterminal will still be referred to elsewhere).
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index e7dce4e404..6a1893553b 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -497,7 +497,7 @@ ECPG: opt_array_boundsopt_array_bounds'['']' block
$$.index2 = mm_strdup($3);
$$.str = cat_str(4, $1.str, mm_strdup("["), $3, mm_strdup("]"));
}
-ECPG: opt_array_bounds
+ECPG: opt_array_bounds block
{
$$.index1 = mm_strdup("-1");
$$.index2 = mm_strdup("-1");
@@ -510,15 +510,6 @@ ECPG: IconstICONST block
ECPG: AexprConstNULL_P rule
| civar { $$ = $1; }
| civarind { $$ = $1; }
-ECPG: ColIdcol_name_keyword rule
- | ECPGKeywords { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
- | CHAR_P { $$ = mm_strdup("char"); }
- | VALUES { $$ = mm_strdup("values"); }
-ECPG: type_function_nametype_func_name_keyword rule
- | ECPGKeywords { $$ = $1; }
- | ECPGTypeName { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
ECPG: VariableShowStmtSHOWALL block
{
mmerror(PARSE_ERROR, ET_ERROR, "SHOW ALL is not implemented");
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
index fe8d3e5178..86d0782d45 100644
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -1,7 +1,13 @@
#!/usr/bin/perl
# src/interfaces/ecpg/preproc/parse.pl
-# parser generator for ecpg version 2
-# call with backend parser as stdin
+# parser generator for ecpg
+#
+# See README.parser for some explanation of what this does.
+#
+# Command-line options:
+# --srcdir: where to find ecpg-provided input files (default ".")
+# --parser: the backend gram.y file to read (required, no default)
+# --output: where to write preproc.y (required, no default)
#
# Copyright (c) 2007-2024, PostgreSQL Global Development Group
#
@@ -148,6 +154,14 @@ dump_buffer('trailer');
close($parserfh);
+# Cross-check that we don't have dead or ambiguous addon rules.
+foreach (keys %addons)
+{
+ die "addon rule $_ was never used\n" if $addons{$_}{used} == 0;
+ die "addon rule $_ was matched multiple times\n" if $addons{$_}{used} > 1;
+}
+
+
sub main
{
line: while (<$parserfh>)
@@ -487,7 +501,10 @@ sub include_addon
my $rec = $addons{$block};
return 0 unless $rec;
- my $rectype = (defined $rec->{type}) ? $rec->{type} : '';
+ # Track usage for later cross-check
+ $rec->{used}++;
+
+ my $rectype = $rec->{type};
if ($rectype eq 'rule')
{
dump_fields($stmt_mode, $fields, ' { ');
@@ -668,10 +685,10 @@ sub dump_line
}
=top
- load addons into cache
+ load ecpg.addons into %addons hash. The result is something like
%addons = {
- stmtClosePortalStmt => { 'type' => 'block', 'lines' => [ "{", "if (INFORMIX_MODE)" ..., "}" ] },
- stmtViewStmt => { 'type' => 'rule', 'lines' => [ "| ECPGAllocateDescr", ... ] }
+ stmtClosePortalStmt => { 'type' => 'block', 'lines' => [ "{", "if (INFORMIX_MODE)" ..., "}" ], 'used' => 0 },
+ stmtViewStmt => { 'type' => 'rule', 'lines' => [ "| ECPGAllocateDescr", ... ], 'used' => 0 }
}
=cut
@@ -681,17 +698,25 @@ sub preload_addons
my $filename = $srcdir . "/ecpg.addons";
open(my $fh, '<', $filename) or die;
- # there may be multiple lines starting ECPG: and then multiple lines of code.
- # the code need to be add to all prior ECPG records.
- my (@needsRules, @code, $record);
+ # There may be multiple "ECPG:" lines and then multiple lines of code.
+ # The block of code needs to be added to each of the consecutively-
+ # preceding "ECPG:" records.
+ my (@needsRules, @code);
- # there may be comments before the first ECPG line, skip them
+ # there may be comments before the first "ECPG:" line, skip them
my $skip = 1;
while (<$fh>)
{
- if (/^ECPG:\s(\S+)\s?(\w+)?/)
+ if (/^ECPG:\s+(\S+)\s+(\w+)\s*$/)
{
+ # Found an "ECPG:" line, so we're done skipping the header
$skip = 0;
+ # Validate record type and target
+ die "invalid record type $2 in addon rule for $1\n"
+ unless ($2 eq 'block' or $2 eq 'addon' or $2 eq 'rule');
+ die "duplicate addon rule for $1\n" if (exists $addons{$1});
+ # If we had some preceding code lines, attach them to all
+ # as-yet-unfinished records.
if (@code)
{
for my $x (@needsRules)
@@ -701,20 +726,27 @@ sub preload_addons
@code = ();
@needsRules = ();
}
- $record = {};
+ my $record = {};
$record->{type} = $2;
$record->{lines} = [];
- if (exists $addons{$1}) { die "Ga! there are dups!\n"; }
+ $record->{used} = 0;
$addons{$1} = $record;
push(@needsRules, $record);
}
+ elsif (/^ECPG:/)
+ {
+ # Complain if preceding regex failed to match
+ die "incorrect syntax in ECPG line: $_\n";
+ }
else
{
+ # Non-ECPG line: add to @code unless we're still skipping
next if $skip;
push(@code, $_);
}
}
close($fh);
+ # Deal with final code block
if (@code)
{
for my $x (@needsRules)
--
2.43.5
[text/x-diff] v5-0002-Major-cleanup-simplification-and-documentation-of.patch (24.0K, ../../[email protected]/3-v5-0002-Major-cleanup-simplification-and-documentation-of.patch)
download | inline diff:
From 8a24e86e6f158b31407ec35b029e33b4bdbac6ee Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:05:26 -0400
Subject: [PATCH v5 2/9] Major cleanup, simplification, and documentation of
parse.pl.
Remove a lot of cruft, clean up and document what's left.
This produces the same preproc.y output as before, except for
fewer blank lines. (It's not like we're making any attempt to
match the layout of gram.y, so I removed the one bit of logic
that seemed to have that in mind.)
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/parse.pl | 486 ++++++++++++++++-----------
1 file changed, 292 insertions(+), 194 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
index 86d0782d45..5a00271468 100644
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -31,27 +31,11 @@ GetOptions(
'output=s' => \$outfile,
'parser=s' => \$parser,) or die "wrong arguments";
-# open parser / output file early, to raise errors early
-open(my $parserfh, '<', $parser) or die "could not open parser file $parser";
-open(my $outfh, '>', $outfile) or die "could not open output file $outfile";
-
-my $copymode = 0;
-my $brace_indent = 0;
-my $yaccmode = 0;
-my $in_rule = 0;
-my $header_included = 0;
-my $has_feature_not_supported = 0;
-my $has_if_command = 0;
-my $tokenmode = 0;
-
-my (%buff, $infield, $comment, %tokens, %addons);
-my ($stmt_mode, @fields);
-my $line = '';
-my $non_term_id;
+# These hash tables define additional transformations to apply to
+# grammar rules.
-# some token have to be replaced by other symbols
-# either in the rule
+# Substitutions to apply to tokens whenever they are seen in a rule.
my %replace_token = (
'BCONST' => 'ecpg_bconst',
'FCONST' => 'ecpg_fconst',
@@ -60,7 +44,9 @@ my %replace_token = (
'IDENT' => 'ecpg_ident',
'PARAM' => 'ecpg_param',);
-# or in the block
+# Substitutions to apply to terminal token names to reconstruct the
+# literal form of the token. (There is also a hard-wired substitution
+# rule that strips trailing '_P'.)
my %replace_string = (
'FORMAT_LA' => 'format',
'NOT_LA' => 'not',
@@ -75,14 +61,16 @@ my %replace_string = (
'GREATER_EQUALS' => '>=',
'NOT_EQUALS' => '<>',);
-# specific replace_types for specific non-terminals - never include the ':'
-# ECPG-only replace_types are defined in ecpg-replace_types
+# This hash can provide a result type to override '<str>' for nonterminals
+# that need that, or it can specify 'ignore' to cause us to skip the rule
+# for that nonterminal. (In that case, ecpg.trailer had better provide
+# a substitute rule.)
my %replace_types = (
'PrepareStmt' => '<prep>',
'ExecuteStmt' => '<exec>',
'opt_array_bounds' => '<index>',
- # "ignore" means: do not create type and rules for this non-term-id
+ # "ignore" means: do not create type and rules for this nonterminal
'parse_toplevel' => 'ignore',
'stmtmulti' => 'ignore',
'CreateAsStmt' => 'ignore',
@@ -97,9 +85,12 @@ my %replace_types = (
'plassign_target' => 'ignore',
'plassign_equals' => 'ignore',);
-# these replace_line commands excise certain keywords from the core keyword
-# lists. Be sure to account for these in ColLabel and related productions.
+# This hash provides an "ignore" option or substitute expansion for any
+# rule or rule alternative. The hash key is the same "concattokens" tag
+# used for lookup in ecpg.addons.
my %replace_line = (
+ # These entries excise certain keywords from the core keyword lists.
+ # Be sure to account for these in ColLabel and related productions.
'unreserved_keywordCONNECTION' => 'ignore',
'unreserved_keywordCURRENT_P' => 'ignore',
'unreserved_keywordDAY_P' => 'ignore',
@@ -137,10 +128,77 @@ my %replace_line = (
'PREPARE prepared_name prep_type_clause AS PreparableStmt',
'var_nameColId' => 'ECPGColId');
+
+# Declare assorted state variables.
+
+# yaccmode counts the '%%' separator lines we have seen, so that we can
+# distinguish prologue, rules, and epilogue sections of gram.y.
+my $yaccmode = 0;
+# in /* ... */ comment?
+my $comment = 0;
+# in { ... } braced text?
+my $brace_indent = 0;
+# within a rule (production)?
+my $in_rule = 0;
+# count of alternatives processed within the current rule.
+my $alt_count = 0;
+# copymode = 1 when we want to emit the current rule to preproc.y.
+# If it's 0, we have decided to ignore the current rule, and should
+# skip all output until we get to the ending semicolon.
+my $copymode = 0;
+# tokenmode = 1 indicates we are processing %token and following declarations.
+my $tokenmode = 0;
+# stmt_mode = 1 indicates that we are processing the 'stmt:' rule.
+my $stmt_mode = 0;
+# Hacky state for emitting feature-not-supported warnings.
+my $has_feature_not_supported = 0;
+my $has_if_command = 0;
+
+# %addons holds the rules loaded from ecpg.addons.
+my %addons;
+
+# %buff holds various named "buffers", which are just strings that accumulate
+# the output destined for different sections of the preproc.y file. This
+# allows us to process the input in one pass even though the resulting output
+# needs to appear in various places. See dump_buffer calls below for the
+# set of buffer names and the order in which they'll be dumped.
+my %buff;
+
+# %tokens contains an entry for every name we have discovered to be a token.
+my %tokens;
+
+# $non_term_id is the name of the nonterminal that is the target of the
+# current rule.
+my $non_term_id;
+
+# $line holds the reconstructed rule text (that is, RHS token list) that
+# we plan to emit for the current rule.
+my $line = '';
+
+# @fields holds the items to be emitted in the token-concatenation action
+# for the current rule (assuming we emit one). "$N" refers to the N'th
+# input token of the rule; anything else is a string to emit literally.
+# (We assume no such string can need to start with '$'.)
+my @fields;
+
+
+# Open parser / output file early, to raise errors early.
+open(my $parserfh, '<', $parser) or die "could not open parser file $parser";
+open(my $outfh, '>', $outfile) or die "could not open output file $outfile";
+
+# Read the various ecpg-supplied input files.
+# ecpg.addons is loaded into the %addons hash, while the other files
+# are just copied into buffers for verbatim output later.
preload_addons();
+include_file('header', 'ecpg.header');
+include_file('tokens', 'ecpg.tokens');
+include_file('ecpgtype', 'ecpg.type');
+include_file('trailer', 'ecpg.trailer');
+# Read gram.y, and do the bulk of the processing.
main();
+# Emit data from the various buffers we filled.
dump_buffer('header');
dump_buffer('tokens');
dump_buffer('types');
@@ -149,7 +207,6 @@ dump_buffer('orig_tokens');
print $outfh '%%', "\n";
print $outfh 'prog: statements;', "\n";
dump_buffer('rules');
-include_file('trailer', 'ecpg.trailer');
dump_buffer('trailer');
close($parserfh);
@@ -162,83 +219,67 @@ foreach (keys %addons)
}
+# Read the backend grammar.
sub main
{
line: while (<$parserfh>)
{
chomp;
- # comment out the line below to make the result file match (blank line wise)
- # the prior version.
- #next if ($_ eq '');
-
- # Dump the action for a rule -
- # stmt_mode indicates if we are processing the 'stmt:'
- # rule (mode==0 means normal, mode==1 means stmt:)
- # flds are the fields to use. These may start with a '$' - in
- # which case they are the result of a previous non-terminal
- #
- # if they don't start with a '$' then they are token name
- #
- # len is the number of fields in flds...
- # leadin is the padding to apply at the beginning (just use for formatting)
-
if (/^%%/)
{
- $tokenmode = 2;
- $copymode = 1;
+ # New file section, so advance yaccmode.
$yaccmode++;
- $infield = 0;
+ # We are no longer examining %token and related commands.
+ $tokenmode = 0;
+ # Shouldn't be anything else on the line.
+ next line;
}
+ # Hacky check for rules that throw FEATURE_NOT_SUPPORTED
+ # (do this before $_ has a chance to get clobbered)
if ($yaccmode == 1)
{
- # Check for rules that throw FEATURE_NOT_SUPPORTED
$has_feature_not_supported = 1 if /ERRCODE_FEATURE_NOT_SUPPORTED/;
$has_if_command = 1 if /^\s*if/;
}
+ # We track %prec per-line, not per-rule, which is not quite right
+ # but there are no counterexamples in gram.y at present.
my $prec = 0;
- # Make sure any braces are split
+ # Make sure any braces are split into separate fields
s/{/ { /g;
s/}/ } /g;
- # Any comments are split
+ # Likewise for comment start/end markers
s|\/\*| /* |g;
s|\*\/| */ |g;
# Now split the line into individual fields
my @arr = split(' ');
+ # Ignore empty lines
if (!@arr)
{
- # empty line: in tokenmode 1, emit an empty line, else ignore
- if ($tokenmode == 1)
- {
- add_to_buffer('orig_tokens', '');
- }
next line;
}
- if ($arr[0] eq '%token' && $tokenmode == 0)
+ # Once we have seen %token in the prologue, we assume all that follows
+ # up to the '%%' separator is %token and associativity declarations.
+ # Collect and process that as necessary.
+ if ($arr[0] eq '%token' && $yaccmode == 0)
{
$tokenmode = 1;
- include_file('tokens', 'ecpg.tokens');
- }
- elsif ($arr[0] eq '%type' && $header_included == 0)
- {
- include_file('header', 'ecpg.header');
- include_file('ecpgtype', 'ecpg.type');
- $header_included = 1;
}
if ($tokenmode == 1)
{
+ # Collect everything of interest on this line into $str.
my $str = '';
- my $prior = '';
for my $a (@arr)
{
+ # Skip comments.
if ($a eq '/*')
{
$comment++;
@@ -253,40 +294,50 @@ sub main
{
next;
}
+
+ # If it's "<something>", it's a type in a %token declaration,
+ # which we can just drop.
if (substr($a, 0, 1) eq '<')
{
next;
-
- # its a type
}
+
+ # Remember that this is a token. This will also make entries
+ # for "%token" and the associativity keywords such as "%left",
+ # which should be harmless so it's not worth the trouble to
+ # avoid it. If a token appears both in %token and in an
+ # associativity declaration, we'll redundantly re-set its
+ # entry, which is also OK.
$tokens{$a} = 1;
+ # Accumulate the line in $str.
$str = $str . ' ' . $a;
- if ($a eq 'IDENT' && $prior eq '%nonassoc')
- {
- # add more tokens to the list
+ # HACK: insert our own %nonassoc line after IDENT.
+ # XXX: this seems pretty wrong, IDENT is not last on its line!
+ if ($a eq 'IDENT' && $arr[0] eq '%nonassoc')
+ {
$str = $str . "\n%nonassoc CSTRING";
}
- $prior = $a;
}
+ # Save the lightly-processed line in orig_tokens.
add_to_buffer('orig_tokens', $str);
next line;
}
- # Don't worry about anything if we're not in the right section of gram.y
+ # The rest is only appropriate if we're in the rules section of gram.y
if ($yaccmode != 1)
{
next line;
}
-
- # Go through each field in turn
+ # Go through each word of the rule in turn
for (
my $fieldIndexer = 0;
$fieldIndexer < scalar(@arr);
$fieldIndexer++)
{
+ # Detect and ignore comments and braced action text
if ($arr[$fieldIndexer] eq '*/' && $comment)
{
$comment = 0;
@@ -298,15 +349,10 @@ sub main
}
elsif ($arr[$fieldIndexer] eq '/*')
{
-
- # start of a multiline comment
+ # start of a possibly-multiline comment
$comment = 1;
next;
}
- elsif ($arr[$fieldIndexer] eq '//')
- {
- next line;
- }
elsif ($arr[$fieldIndexer] eq '}')
{
$brace_indent--;
@@ -317,29 +363,35 @@ sub main
$brace_indent++;
next;
}
-
if ($brace_indent > 0)
{
next;
}
+
+ # OK, it's not a comment or part of an action.
+ # Check for ';' ending the current rule, or '|' ending the
+ # current alternative.
if ($arr[$fieldIndexer] eq ';')
{
if ($copymode)
{
- if ($infield)
- {
- dump_line($stmt_mode, \@fields);
- }
+ # Print the accumulated rule.
+ emit_rule(\@fields);
add_to_buffer('rules', ";\n\n");
}
else
{
+ # End of an ignored rule; revert to copymode = 1.
$copymode = 1;
}
+
+ # Reset for the next rule.
@fields = ();
- $infield = 0;
$line = '';
$in_rule = 0;
+ $alt_count = 0;
+ $has_feature_not_supported = 0;
+ $has_if_command = 0;
next;
}
@@ -347,56 +399,68 @@ sub main
{
if ($copymode)
{
- if ($infield)
- {
- $infield = $infield + dump_line($stmt_mode, \@fields);
- }
- if ($infield > 1)
- {
- $line = '| ';
- }
+ # Print the accumulated alternative.
+ # Increment $alt_count for each non-ignored alternative.
+ $alt_count += emit_rule(\@fields);
}
+
+ # Reset for the next alternative.
@fields = ();
+ # Start the next line with '|' if we've printed at least one
+ # alternative.
+ if ($alt_count > 1)
+ {
+ $line = '| ';
+ }
+ else
+ {
+ $line = '';
+ }
+ $has_feature_not_supported = 0;
+ $has_if_command = 0;
next;
}
+ # Apply replace_token substitution if we have one.
if (exists $replace_token{ $arr[$fieldIndexer] })
{
$arr[$fieldIndexer] = $replace_token{ $arr[$fieldIndexer] };
}
- # Are we looking at a declaration of a non-terminal ?
- if (($arr[$fieldIndexer] =~ /[A-Za-z0-9]+:/)
+ # Are we looking at a declaration of a non-terminal?
+ # We detect that by seeing ':' on the end of the token or
+ # as the next token.
+ if (($arr[$fieldIndexer] =~ /[A-Za-z0-9]+:$/)
|| ( $fieldIndexer + 1 < scalar(@arr)
&& $arr[ $fieldIndexer + 1 ] eq ':'))
{
+ # Extract the non-terminal, sans : if any
$non_term_id = $arr[$fieldIndexer];
$non_term_id =~ tr/://d;
+ # Consume the ':' if it's separate
+ if (!($arr[$fieldIndexer] =~ /[A-Za-z0-9]+:$/))
+ {
+ $fieldIndexer++;
+ }
+
+ # Check for %replace_types override of nonterminal's type
if (not defined $replace_types{$non_term_id})
{
+ # By default, the type is <str>
$replace_types{$non_term_id} = '<str>';
- $copymode = 1;
}
elsif ($replace_types{$non_term_id} eq 'ignore')
{
+ # We'll ignore this nonterminal and rule altogether.
$copymode = 0;
- $line = '';
next line;
}
- $line = $line . ' ' . $arr[$fieldIndexer];
- # Do we have the : attached already ?
- # If yes, we'll have already printed the ':'
- if (!($arr[$fieldIndexer] =~ '[A-Za-z0-9]+:'))
- {
+ # OK, we want this rule.
+ $copymode = 1;
- # Consume the ':' which is next...
- $line = $line . ':';
- $fieldIndexer++;
- }
-
- # Special mode?
+ # Set special mode for the "stmt:" rule.
if ($non_term_id eq 'stmt')
{
$stmt_mode = 1;
@@ -405,69 +469,73 @@ sub main
{
$stmt_mode = 0;
}
+
+ # Emit appropriate %type declaration for this nonterminal.
my $tstr =
'%type '
. $replace_types{$non_term_id} . ' '
. $non_term_id;
add_to_buffer('types', $tstr);
- if ($copymode)
- {
- add_to_buffer('rules', $line);
- }
+ # Emit the target part of the rule.
+ # Note: the leading space is just to match
+ # the old, rather weird output logic.
+ $tstr = ' ' . $non_term_id . ':';
+ add_to_buffer('rules', $tstr);
+
+ # Prepare for reading the fields (tokens) of the rule.
$line = '';
@fields = ();
- $infield = 1;
die "unterminated rule at grammar line $.\n"
if $in_rule;
$in_rule = 1;
+ $alt_count = 1;
next;
}
elsif ($copymode)
{
+ # Not a nonterminal declaration, so just add it to $line.
$line = $line . ' ' . $arr[$fieldIndexer];
}
+
+ # %prec and whatever follows it should get added to $line,
+ # but not to @fields.
if ($arr[$fieldIndexer] eq '%prec')
{
$prec = 1;
next;
}
+ # Emit transformed version of token to @fields if appropriate.
if ( $copymode
&& !$prec
&& !$comment
- && $fieldIndexer < scalar(@arr)
- && length($arr[$fieldIndexer])
- && $infield)
+ && $in_rule)
{
- if ($arr[$fieldIndexer] ne 'Op'
- && (( defined $tokens{ $arr[$fieldIndexer] }
- && $tokens{ $arr[$fieldIndexer] } > 0)
- || $arr[$fieldIndexer] =~ /'.+'/)
- || $stmt_mode == 1)
+ my $S = $arr[$fieldIndexer];
+
+ # If it's a known terminal token (other than Op) or a literal
+ # character, we need to emit the equivalent string, which'll
+ # later get wrapped into a C string literal, perhaps after
+ # merging with adjacent strings.
+ if ($S ne 'Op'
+ && (defined $tokens{$S}
+ || $S =~ /^'.+'$/))
{
- my $S;
- if (exists $replace_string{ $arr[$fieldIndexer] })
- {
- $S = $replace_string{ $arr[$fieldIndexer] };
- }
- else
- {
- $S = $arr[$fieldIndexer];
- }
- $S =~ s/_P//g;
+ # Apply replace_string substitution if any.
+ $S = $replace_string{$S} if (exists $replace_string{$S});
+ # Automatically strip _P if present.
+ $S =~ s/_P$//;
+ # And get rid of quotes if it's a literal character.
$S =~ tr/'//d;
- if ($stmt_mode == 1)
- {
- push(@fields, $S);
- }
- else
- {
- push(@fields, lc($S));
- }
+ # Finally, downcase and push into @fields.
+ push(@fields, lc($S));
}
else
{
+ # Otherwise, push a $N reference to this input token.
+ # (We assume this cannot be confused with anything the
+ # above code would produce.)
push(@fields, '$' . (scalar(@fields) + 1));
}
}
@@ -495,94 +563,108 @@ sub include_file
return;
}
-sub include_addon
+# Emit the semantic action for the current rule.
+# This function mainly accounts for any modifications specified
+# by an ecpg.addons entry.
+sub emit_rule_action
{
- my ($buffer, $block, $fields, $stmt_mode) = @_;
- my $rec = $addons{$block};
- return 0 unless $rec;
+ my ($tag, $fields) = @_;
- # Track usage for later cross-check
+ # See if we have an addons entry; if not, just emit default action
+ my $rec = $addons{$tag};
+ if (!$rec)
+ {
+ emit_default_action($fields, 0);
+ return;
+ }
+
+ # Track addons entry usage for later cross-check
$rec->{used}++;
my $rectype = $rec->{type};
if ($rectype eq 'rule')
{
- dump_fields($stmt_mode, $fields, ' { ');
+ # Emit default action and then the code block.
+ emit_default_action($fields, 0);
}
elsif ($rectype eq 'addon')
{
+ # Emit the code block wrapped in the same braces as the default action.
add_to_buffer('rules', ' { ');
}
- #add_to_buffer( $stream, $_ );
- #We have an array to add to the buffer, we'll add it ourself instead of
- #calling add_to_buffer, which does not know about arrays
-
- push(@{ $buff{$buffer} }, @{ $rec->{lines} });
+ # Emit the addons entry's code block.
+ # We have an array to add to the buffer, we'll add it directly instead of
+ # calling add_to_buffer, which does not know about arrays.
+ push(@{ $buff{'rules'} }, @{ $rec->{lines} });
if ($rectype eq 'addon')
{
- dump_fields($stmt_mode, $fields, '');
+ emit_default_action($fields, 1);
}
-
-
- # if we added something (ie there are lines in our array), return 1
- return 1 if (scalar(@{ $rec->{lines} }) > 0);
- return 0;
+ return;
}
-
-# include_addon does this same thing, but does not call this
-# sub... so if you change this, you need to fix include_addon too
+# Add the given line to the specified buffer.
# Pass: buffer_name, string_to_append
+# Note we add a newline automatically.
sub add_to_buffer
{
push(@{ $buff{ $_[0] } }, "$_[1]\n");
return;
}
+# Dump the specified buffer to the output file.
sub dump_buffer
{
my ($buffer) = @_;
+ # Label the output for debugging purposes.
print $outfh '/* ', $buffer, ' */', "\n";
my $ref = $buff{$buffer};
print $outfh @$ref;
return;
}
-sub dump_fields
+# Emit the default action (usually token concatenation) for the current rule.
+# Pass: fields array, brace_printed boolean
+# brace_printed should be true if caller already printed action's open brace.
+sub emit_default_action
{
- my ($mode, $flds, $ln) = @_;
+ my ($flds, $brace_printed) = @_;
my $len = scalar(@$flds);
- if ($mode == 0)
+ if ($stmt_mode == 0)
{
-
- #Normal
- add_to_buffer('rules', $ln);
+ # Normal rule
if ($has_feature_not_supported and not $has_if_command)
{
# The backend unconditionally reports
# FEATURE_NOT_SUPPORTED in this rule, so let's emit
# a warning on the ecpg side.
+ if (!$brace_printed)
+ {
+ add_to_buffer('rules', ' { ');
+ $brace_printed = 1;
+ }
add_to_buffer('rules',
'mmerror(PARSE_ERROR, ET_WARNING, "unsupported feature will be passed to server");'
);
}
- $has_feature_not_supported = 0;
- $has_if_command = 0;
if ($len == 0)
{
-
- # We have no fields ?
+ # Empty rule
+ if (!$brace_printed)
+ {
+ add_to_buffer('rules', ' { ');
+ $brace_printed = 1;
+ }
add_to_buffer('rules', ' $$=EMPTY; }');
}
else
{
-
- # Go through each field and try to 'aggregate' the tokens
- # into a single 'mm_strdup' where possible
+ # Go through each field and aggregate consecutive literal tokens
+ # into a single 'mm_strdup' call.
my @flds_new;
my $str;
for (my $z = 0; $z < $len; $z++)
@@ -600,8 +682,10 @@ sub dump_fields
if ($z >= $len - 1
|| substr($flds->[ $z + 1 ], 0, 1) eq '$')
{
-
- # We're at the end...
+ # Can't combine any more literals; push to @flds_new.
+ # This code would need work if any literals contain
+ # backslash or double quote, but right now that never
+ # happens.
push(@flds_new, "mm_strdup(\"$str\")");
last;
}
@@ -614,49 +698,62 @@ sub dump_fields
$len = scalar(@flds_new);
if ($len == 1)
{
-
- # Straight assignment
+ # Single field can be handled by straight assignment
+ if (!$brace_printed)
+ {
+ add_to_buffer('rules', ' { ');
+ $brace_printed = 1;
+ }
$str = ' $$ = ' . $flds_new[0] . ';';
add_to_buffer('rules', $str);
}
else
{
-
- # Need to concatenate the results to form
- # our final string
+ # Need to concatenate the results to form our final string
+ if (!$brace_printed)
+ {
+ add_to_buffer('rules', ' { ');
+ $brace_printed = 1;
+ }
$str =
' $$ = cat_str(' . $len . ',' . join(',', @flds_new) . ');';
add_to_buffer('rules', $str);
}
- add_to_buffer('rules', '}');
+ add_to_buffer('rules', '}') if ($brace_printed);
}
}
else
{
-
- # we're in the stmt: rule
+ # We're in the "stmt:" rule, where we need to output special actions.
+ # This code assumes that no ecpg.addons entry applies.
if ($len)
{
-
- # or just the statement ...
+ # Any regular kind of statement calls output_statement
add_to_buffer('rules',
' { output_statement($1, 0, ECPGst_normal); }');
}
else
{
+ # The empty production for stmt: do nothing
add_to_buffer('rules', ' { $$ = NULL; }');
}
}
return;
}
-
-sub dump_line
+# Print the accumulated rule text (in $line) and the appropriate action.
+# Ordinarily return 1. However, if the rule matches an "ignore"
+# entry in %replace_line, then do nothing and return 0.
+sub emit_rule
{
- my ($stmt_mode, $fields) = @_;
- my $block = $non_term_id . $line;
- $block =~ tr/ |//d;
- my $rep = $replace_line{$block};
+ my ($fields) = @_;
+
+ # compute tag to be used as lookup key in %replace_line and %addons
+ my $tag = $non_term_id . $line;
+ $tag =~ tr/ |//d;
+
+ # apply replace_line substitution if any
+ my $rep = $replace_line{$tag};
if ($rep)
{
if ($rep eq 'ignore')
@@ -664,6 +761,7 @@ sub dump_line
return 0;
}
+ # non-ignore entries replace the line, but we'd better keep any '|'
if (index($line, '|') != -1)
{
$line = '| ' . $rep;
@@ -672,15 +770,15 @@ sub dump_line
{
$line = $rep;
}
- $block = $non_term_id . $line;
- $block =~ tr/ |//d;
+
+ # recompute tag for use in emit_rule_action
+ $tag = $non_term_id . $line;
+ $tag =~ tr/ |//d;
}
+
+ # Emit $line, then print the appropriate action.
add_to_buffer('rules', $line);
- my $i = include_addon('rules', $block, $fields, $stmt_mode);
- if ($i == 0)
- {
- dump_fields($stmt_mode, $fields, ' { ');
- }
+ emit_rule_action($tag, $fields);
return 1;
}
--
2.43.5
[text/x-diff] v5-0003-Re-implement-ecpg-preprocessor-s-string-managemen.patch (109.5K, ../../[email protected]/4-v5-0003-Re-implement-ecpg-preprocessor-s-string-managemen.patch)
download | inline diff:
From 01337b5b1a126544a5d7842e5f8cb82181ed568f Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:09:51 -0400
Subject: [PATCH v5 3/9] Re-implement ecpg preprocessor's string management.
Most productions in the preprocessor grammar construct strings
representing SQL or C statements or fragments thereof. Instead
of returning these as <str> results of the productions, return
them as "location" values, taking advantage of Bison's flexibility
about what a location is. We aren't really giving up anything
thereby, since ecpg's error reports have always just given line
numbers, and that's tracked separately. The advantage of this
is that a single instance of the YYLLOC_DEFAULT macro can
perform all the work needed by the vast majority of productions,
including all the ones made automatically by parse.pl. This
avoids having large numbers of effectively-identical productions,
which tickles an optimization inefficiency in recent versions of
clang. (This patch reduces the compilation time for preproc.o
by more than 100-fold with clang 16.) The compiled parser is
noticeably smaller as well.
A disadvantage of this approach is that YYLLOC_DEFAULT is applied
before running the production's semantic action (if any). This
means it cannot use the method favored by cat_str() of free'ing
all the input strings; if the action needs to look at the input
strings, it'd be looking at dangling storage. As this stands,
therefore, it leaks memory like a sieve. This is already a big
patch though, and fixing the memory management seems like a
separable problem, so let's leave that for the next step.
(This does remove some free() calls that I'd have had to touch
anyway, in the expectation that the next step will manage
memory reclamation quite differently.)
Most of the changes here are mindless substitution of "@N" for
"$N" in grammar rules; see the changes to README.parser for
an explanation.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/README.parser | 32 +-
src/interfaces/ecpg/preproc/ecpg.addons | 294 ++---
src/interfaces/ecpg/preproc/ecpg.header | 63 +-
src/interfaces/ecpg/preproc/ecpg.trailer | 1148 +++++++-----------
src/interfaces/ecpg/preproc/ecpg.type | 127 --
src/interfaces/ecpg/preproc/output.c | 16 +-
src/interfaces/ecpg/preproc/parse.pl | 215 +---
src/interfaces/ecpg/preproc/parser.c | 58 +-
src/interfaces/ecpg/preproc/preproc_extern.h | 15 +-
9 files changed, 752 insertions(+), 1216 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/README.parser b/src/interfaces/ecpg/preproc/README.parser
index 5698f5ab32..d6bb872165 100644
--- a/src/interfaces/ecpg/preproc/README.parser
+++ b/src/interfaces/ecpg/preproc/README.parser
@@ -4,8 +4,8 @@ Some notes:
1) Most input matching core grammar productions is simply converted
to strings and concatenated together to form the SQL string
- passed to the server. parse.pl can automatically build the
- grammar actions needed to do this.
+ passed to the server. This is handled mostly automatically,
+ as described below.
2) Some grammar rules need special actions that are added to or
completely override the default token-concatenation behavior.
This is controlled by ecpg.addons as explained below.
@@ -14,11 +14,31 @@ Some notes:
4) ecpg.header contains the "prologue" part of preproc.y, including
support functions, Bison options, etc.
5) Additional terminals added by ECPG must be defined in ecpg.tokens.
- Additional nonterminals added by ECPG must be defined in ecpg.type.
+ Additional nonterminals added by ECPG must be defined in ecpg.type,
+ but only if they have non-void result type, which most don't.
ecpg.header, ecpg.tokens, ecpg.type, and ecpg.trailer are just
copied verbatim into preproc.y at appropriate points.
+
+In the original implementation of ecpg, the strings constructed
+by grammar rules were returned as the Bison result of each rule.
+This led to a large number of effectively-identical rule actions,
+which caused compilation-time problems with some versions of clang.
+Now, rules that need to return a string are declared as having
+void type (which in Bison means leaving out any %type declaration
+for them). Instead, we abuse Bison's "location tracking" mechanism
+to carry the string results, which allows a single YYLLOC_DEFAULT
+call to handle the standard token-concatenation behavior for the
+vast majority of the rules. Rules that don't need to do anything
+else can omit a semantic action altogether. Rules that need to
+construct an output string specially can do so, but they should
+assign it to "@$" rather than the usual "$$"; also, to reference
+the string value of the N'th input token, write "@N" not "$N".
+(But rules that return something other than a simple string
+continue to use the normal Bison notations.)
+
+
ecpg.addons contains entries that begin with a line like
ECPG: concattokens ruletype
and typically have one or more following lines that are the code
@@ -69,9 +89,9 @@ parse.pl contains some tables that list backend grammar
productions to be ignored or modified.
Nonterminals that construct strings (as described above) should be
-given <str> type, which is parse.pl's default assumption for
-nonterminals found in gram.y. That can be overridden at need by
-making an entry in parse.pl's %replace_types table. %replace_types
+given void type, which is parse.pl's default assumption for
+nonterminals found in gram.y. If the result should be of some other
+type, make an entry in parse.pl's %replace_types table. %replace_types
can also be used to suppress output of a nonterminal's rules
altogether (in which case ecpg.trailer had better provide replacement
rules, since the nonterminal will still be referred to elsewhere).
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index 6a1893553b..24ee54554e 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -3,36 +3,35 @@ ECPG: stmtClosePortalStmt block
{
if (INFORMIX_MODE)
{
- if (pg_strcasecmp($1 + strlen("close "), "database") == 0)
+ if (pg_strcasecmp(@1 + strlen("close "), "database") == 0)
{
if (connection)
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in CLOSE DATABASE statement");
fprintf(base_yyout, "{ ECPGdisconnect(__LINE__, \"CURRENT\");");
whenever_action(2);
- free($1);
break;
}
}
- output_statement($1, 0, ECPGst_normal);
+ output_statement(@1, 0, ECPGst_normal);
}
ECPG: stmtDeallocateStmt block
{
- output_deallocate_prepare_statement($1);
+ output_deallocate_prepare_statement(@1);
}
ECPG: stmtDeclareCursorStmt block
{
- output_simple_statement($1, (strncmp($1, "ECPGset_var", strlen("ECPGset_var")) == 0) ? 4 : 0);
+ output_simple_statement(@1, (strncmp(@1, "ECPGset_var", strlen("ECPGset_var")) == 0) ? 4 : 0);
}
ECPG: stmtDiscardStmt block
ECPG: stmtFetchStmt block
- { output_statement($1, 1, ECPGst_normal); }
+ { output_statement(@1, 1, ECPGst_normal); }
ECPG: stmtDeleteStmt block
ECPG: stmtInsertStmt block
ECPG: stmtSelectStmt block
ECPG: stmtUpdateStmt block
- { output_statement($1, 1, ECPGst_prepnormal); }
+ { output_statement(@1, 1, ECPGst_prepnormal); }
ECPG: stmtExecuteStmt block
{
check_declared_list($1.name);
@@ -94,50 +93,45 @@ ECPG: stmtPrepareStmt block
}
ECPG: stmtTransactionStmt block
{
- fprintf(base_yyout, "{ ECPGtrans(__LINE__, %s, \"%s\");", connection ? connection : "NULL", $1);
+ fprintf(base_yyout, "{ ECPGtrans(__LINE__, %s, \"%s\");", connection ? connection : "NULL", @1);
whenever_action(2);
- free($1);
}
ECPG: toplevel_stmtTransactionStmtLegacy block
{
- fprintf(base_yyout, "{ ECPGtrans(__LINE__, %s, \"%s\");", connection ? connection : "NULL", $1);
+ fprintf(base_yyout, "{ ECPGtrans(__LINE__, %s, \"%s\");", connection ? connection : "NULL", @1);
whenever_action(2);
- free($1);
}
ECPG: stmtViewStmt rule
| ECPGAllocateDescr
{
- fprintf(base_yyout, "ECPGallocate_desc(__LINE__, %s);", $1);
+ fprintf(base_yyout, "ECPGallocate_desc(__LINE__, %s);", @1);
whenever_action(0);
- free($1);
}
| ECPGConnect
{
if (connection)
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in CONNECT statement");
- fprintf(base_yyout, "{ ECPGconnect(__LINE__, %d, %s, %d); ", compat, $1, autocommit);
+ fprintf(base_yyout, "{ ECPGconnect(__LINE__, %d, %s, %d); ", compat, @1, autocommit);
reset_variables();
whenever_action(2);
- free($1);
}
| ECPGDeclareStmt
{
- output_simple_statement($1, 0);
+ output_simple_statement(@1, 0);
}
| ECPGCursorStmt
{
- output_simple_statement($1, (strncmp($1, "ECPGset_var", strlen("ECPGset_var")) == 0) ? 4 : 0);
+ output_simple_statement(@1, (strncmp(@1, "ECPGset_var", strlen("ECPGset_var")) == 0) ? 4 : 0);
}
| ECPGDeallocateDescr
{
- fprintf(base_yyout, "ECPGdeallocate_desc(__LINE__, %s);", $1);
+ fprintf(base_yyout, "ECPGdeallocate_desc(__LINE__, %s);", @1);
whenever_action(0);
- free($1);
}
| ECPGDeclare
{
- output_simple_statement($1, 0);
+ output_simple_statement(@1, 0);
}
| ECPGDescribe
{
@@ -157,27 +151,25 @@ ECPG: stmtViewStmt rule
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in DISCONNECT statement");
fprintf(base_yyout, "{ ECPGdisconnect(__LINE__, %s);",
- $1 ? $1 : "\"CURRENT\"");
+ @1 ? @1 : "\"CURRENT\"");
whenever_action(2);
- free($1);
}
| ECPGExecuteImmediateStmt
{
- output_statement($1, 0, ECPGst_exec_immediate);
+ output_statement(@1, 0, ECPGst_exec_immediate);
}
| ECPGFree
{
const char *con = connection ? connection : "NULL";
- if (strcmp($1, "all") == 0)
+ if (strcmp(@1, "all") == 0)
fprintf(base_yyout, "{ ECPGdeallocate_all(__LINE__, %d, %s);", compat, con);
- else if ($1[0] == ':')
- fprintf(base_yyout, "{ ECPGdeallocate(__LINE__, %d, %s, %s);", compat, con, $1 + 1);
+ else if (@1[0] == ':')
+ fprintf(base_yyout, "{ ECPGdeallocate(__LINE__, %d, %s, %s);", compat, con, @1 + 1);
else
- fprintf(base_yyout, "{ ECPGdeallocate(__LINE__, %d, %s, \"%s\");", compat, con, $1);
+ fprintf(base_yyout, "{ ECPGdeallocate(__LINE__, %d, %s, \"%s\");", compat, con, @1);
whenever_action(2);
- free($1);
}
| ECPGGetDescriptor
{
@@ -188,15 +180,14 @@ ECPG: stmtViewStmt rule
}
| ECPGGetDescriptorHeader
{
- lookup_descriptor($1, connection);
- output_get_descr_header($1);
- free($1);
+ lookup_descriptor(@1, connection);
+ output_get_descr_header(@1);
}
| ECPGOpen
{
struct cursor *ptr;
- if ((ptr = add_additional_variables($1, true)) != NULL)
+ if ((ptr = add_additional_variables(@1, true)) != NULL)
{
connection = ptr->connection ? mm_strdup(ptr->connection) : NULL;
output_statement(mm_strdup(ptr->command), 0, ECPGst_normal);
@@ -205,18 +196,16 @@ ECPG: stmtViewStmt rule
}
| ECPGSetAutocommit
{
- fprintf(base_yyout, "{ ECPGsetcommit(__LINE__, \"%s\", %s);", $1, connection ? connection : "NULL");
+ fprintf(base_yyout, "{ ECPGsetcommit(__LINE__, \"%s\", %s);", @1, connection ? connection : "NULL");
whenever_action(2);
- free($1);
}
| ECPGSetConnection
{
if (connection)
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in SET CONNECTION statement");
- fprintf(base_yyout, "{ ECPGsetconn(__LINE__, %s);", $1);
+ fprintf(base_yyout, "{ ECPGsetconn(__LINE__, %s);", @1);
whenever_action(2);
- free($1);
}
| ECPGSetDescriptor
{
@@ -227,17 +216,15 @@ ECPG: stmtViewStmt rule
}
| ECPGSetDescriptorHeader
{
- lookup_descriptor($1, connection);
- output_set_descr_header($1);
- free($1);
+ lookup_descriptor(@1, connection);
+ output_set_descr_header(@1);
}
| ECPGTypedef
{
if (connection)
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in TYPE statement");
- fprintf(base_yyout, "%s", $1);
- free($1);
+ fprintf(base_yyout, "%s", @1);
output_line_number();
}
| ECPGVar
@@ -245,180 +232,169 @@ ECPG: stmtViewStmt rule
if (connection)
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in VAR statement");
- output_simple_statement($1, 0);
+ output_simple_statement(@1, 0);
}
| ECPGWhenever
{
if (connection)
mmerror(PARSE_ERROR, ET_ERROR, "AT option not allowed in WHENEVER statement");
- output_simple_statement($1, 0);
+ output_simple_statement(@1, 0);
}
ECPG: where_or_current_clauseWHERECURRENT_POFcursor_name block
{
- char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
+ char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
- $$ = cat_str(2, mm_strdup("where current of"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("where current of"), cursor_marker);
}
ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listcopy_fromopt_programcopy_file_namecopy_delimiteropt_withcopy_optionswhere_clause addon
- if (strcmp($6, "from") == 0 &&
- (strcmp($7, "stdin") == 0 || strcmp($7, "stdout") == 0))
+ if (strcmp(@6, "from") == 0 &&
+ (strcmp(@7, "stdin") == 0 || strcmp(@7, "stdout") == 0))
mmerror(PARSE_ERROR, ET_WARNING, "COPY FROM STDIN is not implemented");
ECPG: var_valueNumericOnly addon
- if ($1[0] == '$')
- {
- free($1);
- $1 = mm_strdup("$0");
- }
+ if (@1[0] == '$')
+ @$ = mm_strdup("$0");
ECPG: fetch_argscursor_name addon
- struct cursor *ptr = add_additional_variables($1, false);
+ struct cursor *ptr = add_additional_variables(@1, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- if ($1[0] == ':')
- {
- free($1);
- $1 = mm_strdup("$0");
- }
+ if (@1[0] == ':')
+ @$ = mm_strdup("$0");
ECPG: fetch_argsfrom_incursor_name addon
- struct cursor *ptr = add_additional_variables($2, false);
+ struct cursor *ptr = add_additional_variables(@2, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- if ($2[0] == ':')
- {
- free($2);
- $2 = mm_strdup("$0");
- }
+ if (@2[0] == ':')
+ @$ = cat2_str(mm_strdup(@1), mm_strdup("$0"));
ECPG: fetch_argsNEXTopt_from_incursor_name addon
ECPG: fetch_argsPRIORopt_from_incursor_name addon
ECPG: fetch_argsFIRST_Popt_from_incursor_name addon
ECPG: fetch_argsLAST_Popt_from_incursor_name addon
ECPG: fetch_argsALLopt_from_incursor_name addon
- struct cursor *ptr = add_additional_variables($3, false);
+ struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- if ($3[0] == ':')
- {
- free($3);
- $3 = mm_strdup("$0");
- }
+ if (@3[0] == ':')
+ @$ = cat_str(3, mm_strdup(@1), mm_strdup(@2), mm_strdup("$0"));
ECPG: fetch_argsSignedIconstopt_from_incursor_name addon
- struct cursor *ptr = add_additional_variables($3, false);
+ struct cursor *ptr = add_additional_variables(@3, false);
+ bool replace = false;
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- if ($3[0] == ':')
+ if (@3[0] == ':')
{
- free($3);
- $3 = mm_strdup("$0");
+ @3 = mm_strdup("$0");
+ replace = true;
}
- if ($1[0] == '$')
+ if (@1[0] == '$')
{
- free($1);
- $1 = mm_strdup("$0");
+ @1 = mm_strdup("$0");
+ replace = true;
}
+ if (replace)
+ @$ = cat_str(3, mm_strdup(@1), mm_strdup(@2), mm_strdup(@3));
ECPG: fetch_argsFORWARDALLopt_from_incursor_name addon
ECPG: fetch_argsBACKWARDALLopt_from_incursor_name addon
- struct cursor *ptr = add_additional_variables($4, false);
+ struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- if ($4[0] == ':')
- {
- free($4);
- $4 = mm_strdup("$0");
- }
+ if (@4[0] == ':')
+ @$ = cat_str(4, mm_strdup(@1), mm_strdup(@2), mm_strdup(@3), mm_strdup("$0"));
ECPG: fetch_argsABSOLUTE_PSignedIconstopt_from_incursor_name addon
ECPG: fetch_argsRELATIVE_PSignedIconstopt_from_incursor_name addon
ECPG: fetch_argsFORWARDSignedIconstopt_from_incursor_name addon
ECPG: fetch_argsBACKWARDSignedIconstopt_from_incursor_name addon
- struct cursor *ptr = add_additional_variables($4, false);
+ struct cursor *ptr = add_additional_variables(@4, false);
+ bool replace = false;
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- if ($4[0] == ':')
+ if (@4[0] == ':')
{
- free($4);
- $4 = mm_strdup("$0");
+ @4 = mm_strdup("$0");
+ replace = true;
}
- if ($2[0] == '$')
+ if (@2[0] == '$')
{
- free($2);
- $2 = mm_strdup("$0");
+ @2 = mm_strdup("$0");
+ replace = true;
}
-ECPG: cursor_namename rule
+ if (replace)
+ @$ = cat_str(4, mm_strdup(@1), mm_strdup(@2), mm_strdup(@3), mm_strdup(@4));
+ECPG: cursor_namename block
| char_civar
{
- char *curname = mm_alloc(strlen($1) + 2);
+ char *curname = mm_alloc(strlen(@1) + 2);
- sprintf(curname, ":%s", $1);
- free($1);
- $1 = curname;
- $$ = $1;
+ sprintf(curname, ":%s", @1);
+ @$ = curname;
}
ECPG: ExplainableStmtExecuteStmt block
{
- $$ = $1.name;
+ @$ = $1.name;
}
ECPG: PrepareStmtPREPAREprepared_nameprep_type_clauseASPreparableStmt block
{
- $$.name = $2;
- $$.type = $3;
- $$.stmt = $5;
+ $$.name = @2;
+ $$.type = @3;
+ $$.stmt = @5;
}
| PREPARE prepared_name FROM execstring
{
- $$.name = $2;
+ $$.name = @2;
$$.type = NULL;
- $$.stmt = $4;
+ $$.stmt = @4;
}
ECPG: ExecuteStmtEXECUTEprepared_nameexecute_param_clauseexecute_rest block
{
- $$.name = $2;
- $$.type = $3;
+ $$.name = @2;
+ $$.type = @3;
}
ECPG: ExecuteStmtCREATEOptTempTABLEcreate_as_targetASEXECUTEprepared_nameexecute_param_clauseopt_with_dataexecute_rest block
{
- $$.name = cat_str(8, mm_strdup("create"), $2, mm_strdup("table"), $4, mm_strdup("as execute"), $7, $8, $9);
+ $$.name = @$;
}
ECPG: ExecuteStmtCREATEOptTempTABLEIF_PNOTEXISTScreate_as_targetASEXECUTEprepared_nameexecute_param_clauseopt_with_dataexecute_rest block
{
- $$.name = cat_str(8, mm_strdup("create"), $2, mm_strdup("table if not exists"), $7, mm_strdup("as execute"), $10, $11, $12);
+ $$.name = @$;
}
ECPG: DeclareCursorStmtDECLAREcursor_namecursor_optionsCURSORopt_holdFORSelectStmt block
{
struct cursor *ptr,
*this;
- char *cursor_marker = $2[0] == ':' ? mm_strdup("$0") : mm_strdup($2);
+ char *cursor_marker = @2[0] == ':' ? mm_strdup("$0") : mm_strdup(@2);
char *comment,
*c1,
*c2;
- int (*strcmp_fn) (const char *, const char *) = (($2[0] == ':' || $2[0] == '"') ? strcmp : pg_strcasecmp);
+ int (*strcmp_fn) (const char *, const char *) = ((@2[0] == ':' || @2[0] == '"') ? strcmp : pg_strcasecmp);
- if (INFORMIX_MODE && pg_strcasecmp($2, "database") == 0)
+ if (INFORMIX_MODE && pg_strcasecmp(@2, "database") == 0)
mmfatal(PARSE_ERROR, "\"database\" cannot be used as cursor name in INFORMIX mode");
for (ptr = cur; ptr != NULL; ptr = ptr->next)
{
- if (strcmp_fn($2, ptr->name) == 0)
+ if (strcmp_fn(@2, ptr->name) == 0)
{
- if ($2[0] == ':')
- mmerror(PARSE_ERROR, ET_ERROR, "using variable \"%s\" in different declare statements is not supported", $2 + 1);
+ if (@2[0] == ':')
+ mmerror(PARSE_ERROR, ET_ERROR, "using variable \"%s\" in different declare statements is not supported", @2 + 1);
else
- mmerror(PARSE_ERROR, ET_ERROR, "cursor \"%s\" is already defined", $2);
+ mmerror(PARSE_ERROR, ET_ERROR, "cursor \"%s\" is already defined", @2);
}
}
this = (struct cursor *) mm_alloc(sizeof(struct cursor));
this->next = cur;
- this->name = $2;
+ this->name = mm_strdup(@2);
this->function = (current_function ? mm_strdup(current_function) : NULL);
this->connection = connection ? mm_strdup(connection) : NULL;
this->opened = false;
- this->command = cat_str(7, mm_strdup("declare"), cursor_marker, $3, mm_strdup("cursor"), $5, mm_strdup("for"), $7);
+ this->command = cat_str(7, mm_strdup("declare"), cursor_marker, @3, mm_strdup("cursor"), @5, mm_strdup("for"), @7);
this->argsinsert = argsinsert;
this->argsinsert_oos = NULL;
this->argsresult = argsresult;
@@ -435,47 +411,47 @@ ECPG: DeclareCursorStmtDECLAREcursor_namecursor_optionsCURSORopt_holdFORSelectSt
}
comment = cat_str(3, mm_strdup("/*"), c1, mm_strdup("*/"));
- $$ = cat2_str(adjust_outofscope_cursor_vars(this), comment);
+ @$ = cat2_str(adjust_outofscope_cursor_vars(this), comment);
}
ECPG: ClosePortalStmtCLOSEcursor_name block
{
- char *cursor_marker = $2[0] == ':' ? mm_strdup("$0") : $2;
+ char *cursor_marker = @2[0] == ':' ? mm_strdup("$0") : @2;
struct cursor *ptr = NULL;
for (ptr = cur; ptr != NULL; ptr = ptr->next)
{
- if (strcmp($2, ptr->name) == 0)
+ if (strcmp(@2, ptr->name) == 0)
{
if (ptr->connection)
connection = mm_strdup(ptr->connection);
break;
}
}
- $$ = cat2_str(mm_strdup("close"), cursor_marker);
+ @$ = cat2_str(mm_strdup("close"), cursor_marker);
}
ECPG: opt_hold block
{
if (compat == ECPG_COMPAT_INFORMIX_SE && autocommit)
- $$ = mm_strdup("with hold");
+ @$ = mm_strdup("with hold");
else
- $$ = EMPTY;
+ @$ = EMPTY;
}
ECPG: into_clauseINTOOptTempTableName block
{
FoundInto = 1;
- $$ = cat2_str(mm_strdup("into"), $2);
+ @$ = cat2_str(mm_strdup("into"), @2);
}
| ecpg_into
{
- $$ = EMPTY;
+ @$ = EMPTY;
}
ECPG: TypenameSimpleTypenameopt_array_bounds block
{
- $$ = cat2_str($1, $2.str);
+ @$ = cat2_str(@1, $2.str);
}
ECPG: TypenameSETOFSimpleTypenameopt_array_bounds block
{
- $$ = cat_str(3, mm_strdup("setof"), $2, $3.str);
+ @$ = cat_str(3, mm_strdup("setof"), @2, $3.str);
}
ECPG: opt_array_boundsopt_array_bounds'['']' block
{
@@ -492,10 +468,10 @@ ECPG: opt_array_boundsopt_array_bounds'['']' block
$$.index1 = $1.index1;
$$.index2 = $1.index2;
if (strcmp($1.index1, "-1") == 0)
- $$.index1 = mm_strdup($3);
+ $$.index1 = mm_strdup(@3);
else if (strcmp($1.index2, "-1") == 0)
- $$.index2 = mm_strdup($3);
- $$.str = cat_str(4, $1.str, mm_strdup("["), $3, mm_strdup("]"));
+ $$.index2 = mm_strdup(@3);
+ $$.str = cat_str(4, $1.str, mm_strdup("["), @3, mm_strdup("]"));
}
ECPG: opt_array_bounds block
{
@@ -505,108 +481,100 @@ ECPG: opt_array_bounds block
}
ECPG: IconstICONST block
{
- $$ = make_name();
+ @$ = make_name();
}
ECPG: AexprConstNULL_P rule
- | civar { $$ = $1; }
- | civarind { $$ = $1; }
+ | civar
+ | civarind
ECPG: VariableShowStmtSHOWALL block
{
mmerror(PARSE_ERROR, ET_ERROR, "SHOW ALL is not implemented");
- $$ = EMPTY;
}
ECPG: FetchStmtMOVEfetch_args rule
| FETCH fetch_args ecpg_fetch_into
- {
- $$ = cat2_str(mm_strdup("fetch"), $2);
- }
| FETCH FORWARD cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = $3[0] == ':' ? mm_strdup("$0") : $3;
- struct cursor *ptr = add_additional_variables($3, false);
+ char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("fetch forward"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("fetch forward"), cursor_marker);
}
| FETCH FORWARD from_in cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
- struct cursor *ptr = add_additional_variables($4, false);
+ char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("fetch forward from"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("fetch forward from"), cursor_marker);
}
| FETCH BACKWARD cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = $3[0] == ':' ? mm_strdup("$0") : $3;
- struct cursor *ptr = add_additional_variables($3, false);
+ char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("fetch backward"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("fetch backward"), cursor_marker);
}
| FETCH BACKWARD from_in cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
- struct cursor *ptr = add_additional_variables($4, false);
+ char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("fetch backward from"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("fetch backward from"), cursor_marker);
}
| MOVE FORWARD cursor_name
{
- char *cursor_marker = $3[0] == ':' ? mm_strdup("$0") : $3;
- struct cursor *ptr = add_additional_variables($3, false);
+ char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("move forward"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("move forward"), cursor_marker);
}
| MOVE FORWARD from_in cursor_name
{
- char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
- struct cursor *ptr = add_additional_variables($4, false);
+ char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("move forward from"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("move forward from"), cursor_marker);
}
| MOVE BACKWARD cursor_name
{
- char *cursor_marker = $3[0] == ':' ? mm_strdup("$0") : $3;
- struct cursor *ptr = add_additional_variables($3, false);
+ char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("move backward"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("move backward"), cursor_marker);
}
| MOVE BACKWARD from_in cursor_name
{
- char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
- struct cursor *ptr = add_additional_variables($4, false);
+ char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- $$ = cat_str(2, mm_strdup("move backward from"), cursor_marker);
+ @$ = cat_str(2, mm_strdup("move backward from"), cursor_marker);
}
ECPG: limit_clauseLIMITselect_limit_value','select_offset_value block
{
mmerror(PARSE_ERROR, ET_WARNING, "no longer supported LIMIT #,# syntax passed to server");
- $$ = cat_str(4, mm_strdup("limit"), $2, mm_strdup(","), $4);
}
ECPG: SignedIconstIconst rule
| civar
- {
- $$ = $1;
- }
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index 28e1b2aac4..8df6248c97 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -13,14 +13,6 @@
extern int base_yychar;
extern int base_yynerrs;
-/* Location tracking support --- simpler than bison's default */
-#define YYLLOC_DEFAULT(Current, Rhs, N) \
- do { \
- if (N) \
- (Current) = (Rhs)[1]; \
- else \
- (Current) = (Rhs)[0]; \
- } while (0)
/*
* The %name-prefix option below will make bison call base_yylex, but we
@@ -200,6 +192,61 @@ make3_str(char *str1, char *str2, char *str3)
return res_str;
}
+/*
+ * "Location tracking" support. We commandeer Bison's location tracking
+ * mechanism to manage the output string for productions that ordinarily would
+ * return a <str> result. This allows the majority of those productions to
+ * have default semantic actions, reducing the size of the parser, and also
+ * greatly reducing its compilation time on some versions of clang.
+ *
+ * To do this, we make YYLTYPE be a pointer to a malloc'd string, and then
+ * merge the location strings of the input tokens in the default YYLLOC
+ * computation. Productions that are okay with the standard merge need not
+ * do anything more; otherwise, they can override it by assigning to @$.
+ */
+#define YYLLOC_DEFAULT(Current, Rhs, N) yylloc_default(&(Current), Rhs, N)
+
+static void
+yylloc_default(YYLTYPE *target, YYLTYPE *rhs, int N)
+{
+ if (N > 1)
+ {
+ /* Concatenate non-empty inputs with one space between them */
+ char *result,
+ *ptr;
+ size_t needed = 0;
+
+ for (int i = 1; i <= N; i++)
+ {
+ size_t thislen = strlen(rhs[i]);
+
+ if (needed > 0 && thislen > 0)
+ needed++;
+ needed += thislen;
+ }
+ result = (char *) mm_alloc(needed + 1);
+ ptr = result;
+ for (int i = 1; i <= N; i++)
+ {
+ size_t thislen = strlen(rhs[i]);
+
+ if (ptr > result && thislen > 0)
+ *ptr++ = ' ';
+ memcpy(ptr, rhs[i], thislen);
+ ptr += thislen;
+ }
+ *ptr = '\0';
+ *target = result;
+ }
+ else if (N == 1)
+ {
+ /* Just re-use the single input */
+ *target = rhs[1];
+ }
+ else
+ *target = EMPTY;
+}
+
/* and the rest */
static char *
make_name(void)
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index f3ab73bed6..2a3949ca03 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -18,20 +18,17 @@ statement: ecpgstart at toplevel_stmt ';'
}
| ecpgstart ECPGVarDeclaration
{
- fprintf(base_yyout, "%s", $2);
- free($2);
+ fprintf(base_yyout, "%s", @$);
output_line_number();
}
| ECPGDeclaration
| c_thing
{
- fprintf(base_yyout, "%s", $1);
- free($1);
+ fprintf(base_yyout, "%s", @$);
}
| CPP_LINE
{
- fprintf(base_yyout, "%s", $1);
- free($1);
+ fprintf(base_yyout, "%s", @$);
}
| '{'
{
@@ -58,8 +55,6 @@ CreateAsStmt: CREATE OptTemp TABLE create_as_target AS
{
if (FoundInto == 1)
mmerror(PARSE_ERROR, ET_ERROR, "CREATE TABLE AS cannot specify INTO");
-
- $$ = cat_str(7, mm_strdup("create"), $2, mm_strdup("table"), $4, mm_strdup("as"), $7, $8);
}
| CREATE OptTemp TABLE IF_P NOT EXISTS create_as_target AS
{
@@ -68,14 +63,12 @@ CreateAsStmt: CREATE OptTemp TABLE create_as_target AS
{
if (FoundInto == 1)
mmerror(PARSE_ERROR, ET_ERROR, "CREATE TABLE AS cannot specify INTO");
-
- $$ = cat_str(7, mm_strdup("create"), $2, mm_strdup("table if not exists"), $7, mm_strdup("as"), $10, $11);
}
;
at: AT connection_object
{
- connection = $2;
+ connection = @2;
/*
* Do we have a variable as connection target? Remove the variable
@@ -91,55 +84,52 @@ at: AT connection_object
*/
ECPGConnect: SQL_CONNECT TO connection_target opt_connection_name opt_user
{
- $$ = cat_str(5, $3, mm_strdup(","), $5, mm_strdup(","), $4);
+ @$ = cat_str(5, @3, mm_strdup(","), @5, mm_strdup(","), @4);
}
| SQL_CONNECT TO DEFAULT
{
- $$ = mm_strdup("NULL, NULL, NULL, \"DEFAULT\"");
+ @$ = mm_strdup("NULL, NULL, NULL, \"DEFAULT\"");
}
/* also allow ORACLE syntax */
| SQL_CONNECT ora_user
{
- $$ = cat_str(3, mm_strdup("NULL,"), $2, mm_strdup(", NULL"));
+ @$ = cat_str(3, mm_strdup("NULL,"), @2, mm_strdup(", NULL"));
}
| DATABASE connection_target
{
- $$ = cat2_str($2, mm_strdup(", NULL, NULL, NULL"));
+ @$ = cat2_str(@2, mm_strdup(", NULL, NULL, NULL"));
}
;
connection_target: opt_database_name opt_server opt_port
{
/* old style: dbname[@server][:port] */
- if (strlen($2) > 0 && *($2) != '@')
- mmerror(PARSE_ERROR, ET_ERROR, "expected \"@\", found \"%s\"", $2);
+ if (strlen(@2) > 0 && *(@2) != '@')
+ mmerror(PARSE_ERROR, ET_ERROR, "expected \"@\", found \"%s\"", @2);
/* C strings need to be handled differently */
- if ($1[0] == '\"')
- $$ = $1;
+ if (@1[0] == '\"')
+ @$ = @1;
else
- $$ = make3_str(mm_strdup("\""), make3_str($1, $2, $3), mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), make3_str(@1, @2, @3), mm_strdup("\""));
}
| db_prefix ':' server opt_port '/' opt_database_name opt_options
{
/* new style: <tcp|unix>:postgresql://server[:port][/dbname] */
- if (strncmp($1, "unix:postgresql", strlen("unix:postgresql")) != 0 && strncmp($1, "tcp:postgresql", strlen("tcp:postgresql")) != 0)
+ if (strncmp(@1, "unix:postgresql", strlen("unix:postgresql")) != 0 && strncmp(@1, "tcp:postgresql", strlen("tcp:postgresql")) != 0)
mmerror(PARSE_ERROR, ET_ERROR, "only protocols \"tcp\" and \"unix\" and database type \"postgresql\" are supported");
- if (strncmp($3, "//", strlen("//")) != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "expected \"://\", found \"%s\"", $3);
+ if (strncmp(@3, "//", strlen("//")) != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "expected \"://\", found \"%s\"", @3);
- if (strncmp($1, "unix", strlen("unix")) == 0 &&
- strncmp($3 + strlen("//"), "localhost", strlen("localhost")) != 0 &&
- strncmp($3 + strlen("//"), "127.0.0.1", strlen("127.0.0.1")) != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "Unix-domain sockets only work on \"localhost\" but not on \"%s\"", $3 + strlen("//"));
+ if (strncmp(@1, "unix", strlen("unix")) == 0 &&
+ strncmp(@3 + strlen("//"), "localhost", strlen("localhost")) != 0 &&
+ strncmp(@3 + strlen("//"), "127.0.0.1", strlen("127.0.0.1")) != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "Unix-domain sockets only work on \"localhost\" but not on \"%s\"", @3 + strlen("//"));
- $$ = make3_str(make3_str(mm_strdup("\""), $1, mm_strdup(":")), $3, make3_str(make3_str($4, mm_strdup("/"), $6), $7, mm_strdup("\"")));
+ @$ = make3_str(make3_str(mm_strdup("\""), @1, mm_strdup(":")), @3, make3_str(make3_str(@4, mm_strdup("/"), @6), @7, mm_strdup("\"")));
}
| char_variable
- {
- $$ = $1;
- }
| ecpg_sconst
{
/*
@@ -147,128 +137,107 @@ connection_target: opt_database_name opt_server opt_port
* so we change the quotes. Note, that the rule for ecpg_sconst adds
* these single quotes.
*/
- $1[0] = '\"';
- $1[strlen($1) - 1] = '\"';
- $$ = $1;
+ @1[0] = '\"';
+ @1[strlen(@1) - 1] = '\"';
+ @$ = @1;
}
;
opt_database_name: name
- {
- $$ = $1;
- }
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
db_prefix: ecpg_ident cvariable
{
- if (strcmp($2, "postgresql") != 0 && strcmp($2, "postgres") != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "expected \"postgresql\", found \"%s\"", $2);
+ if (strcmp(@2, "postgresql") != 0 && strcmp(@2, "postgres") != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "expected \"postgresql\", found \"%s\"", @2);
- if (strcmp($1, "tcp") != 0 && strcmp($1, "unix") != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "invalid connection type: %s", $1);
+ if (strcmp(@1, "tcp") != 0 && strcmp(@1, "unix") != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "invalid connection type: %s", @1);
- $$ = make3_str($1, mm_strdup(":"), $2);
+ @$ = make3_str(@1, mm_strdup(":"), @2);
}
;
server: Op server_name
{
- if (strcmp($1, "@") != 0 && strcmp($1, "//") != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "expected \"@\" or \"://\", found \"%s\"", $1);
+ if (strcmp(@1, "@") != 0 && strcmp(@1, "//") != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "expected \"@\" or \"://\", found \"%s\"", @1);
- $$ = make2_str($1, $2);
+ @$ = make2_str(@1, @2);
}
;
opt_server: server
- {
- $$ = $1;
- }
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
server_name: ColId
- {
- $$ = $1;
- }
| ColId '.' server_name
- {
- $$ = make3_str($1, mm_strdup("."), $3);
- }
| IP
{
- $$ = make_name();
+ @$ = make_name();
}
;
opt_port: ':' Iconst
{
- $$ = make2_str(mm_strdup(":"), $2);
+ @$ = make2_str(mm_strdup(":"), @2);
}
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
opt_connection_name: AS connection_object
{
- $$ = $2;
+ @$ = @2;
}
| /* EMPTY */
{
- $$ = mm_strdup("NULL");
+ @$ = mm_strdup("NULL");
}
;
opt_user: USER ora_user
{
- $$ = $2;
+ @$ = @2;
}
| /* EMPTY */
{
- $$ = mm_strdup("NULL, NULL");
+ @$ = mm_strdup("NULL, NULL");
}
;
ora_user: user_name
{
- $$ = cat2_str($1, mm_strdup(", NULL"));
+ @$ = cat2_str(@1, mm_strdup(", NULL"));
}
| user_name '/' user_name
{
- $$ = cat_str(3, $1, mm_strdup(","), $3);
+ @$ = cat_str(3, @1, mm_strdup(","), @3);
}
| user_name SQL_IDENTIFIED BY user_name
{
- $$ = cat_str(3, $1, mm_strdup(","), $4);
+ @$ = cat_str(3, @1, mm_strdup(","), @4);
}
| user_name USING user_name
{
- $$ = cat_str(3, $1, mm_strdup(","), $3);
+ @$ = cat_str(3, @1, mm_strdup(","), @3);
}
;
user_name: RoleId
{
- if ($1[0] == '\"')
- $$ = $1;
+ if (@1[0] == '\"')
+ @$ = @1;
else
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
| ecpg_sconst
{
- if ($1[0] == '\"')
- $$ = $1;
+ if (@1[0] == '\"')
+ @$ = @1;
else
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
| civar
{
@@ -280,16 +249,16 @@ user_name: RoleId
/* handle varchars */
if (type == ECPGt_varchar)
- $$ = make2_str(mm_strdup(argsinsert->variable->name), mm_strdup(".arr"));
+ @$ = make2_str(mm_strdup(argsinsert->variable->name), mm_strdup(".arr"));
else
- $$ = mm_strdup(argsinsert->variable->name);
+ @$ = mm_strdup(argsinsert->variable->name);
}
;
char_variable: cvariable
{
/* check if we have a string variable */
- struct variable *p = find_variable($1);
+ struct variable *p = find_variable(@1);
enum ECPGttype type = p->type->type;
/* If we have just one character this is not a string */
@@ -306,14 +275,14 @@ char_variable: cvariable
case ECPGt_char:
case ECPGt_unsigned_char:
case ECPGt_string:
- $$ = $1;
+ @$ = @1;
break;
case ECPGt_varchar:
- $$ = make2_str($1, mm_strdup(".arr"));
+ @$ = make2_str(@1, mm_strdup(".arr"));
break;
default:
mmerror(PARSE_ERROR, ET_ERROR, "invalid data type");
- $$ = $1;
+ @$ = @1;
break;
}
}
@@ -322,72 +291,63 @@ char_variable: cvariable
opt_options: Op connect_options
{
- if (strlen($1) == 0)
+ if (strlen(@1) == 0)
mmerror(PARSE_ERROR, ET_ERROR, "incomplete statement");
- if (strcmp($1, "?") != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "unrecognized token \"%s\"", $1);
+ if (strcmp(@1, "?") != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "unrecognized token \"%s\"", @1);
- $$ = make2_str(mm_strdup("?"), $2);
+ @$ = make2_str(mm_strdup("?"), @2);
}
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
connect_options: ColId opt_opt_value
{
- $$ = make2_str($1, $2);
+ @$ = make2_str(@1, @2);
}
| ColId opt_opt_value Op connect_options
{
- if (strlen($3) == 0)
+ if (strlen(@3) == 0)
mmerror(PARSE_ERROR, ET_ERROR, "incomplete statement");
- if (strcmp($3, "&") != 0)
- mmerror(PARSE_ERROR, ET_ERROR, "unrecognized token \"%s\"", $3);
+ if (strcmp(@3, "&") != 0)
+ mmerror(PARSE_ERROR, ET_ERROR, "unrecognized token \"%s\"", @3);
- $$ = make3_str(make2_str($1, $2), $3, $4);
+ @$ = make3_str(make2_str(@1, @2), @3, @4);
}
;
opt_opt_value: /* EMPTY */
- {
- $$ = EMPTY;
- }
| '=' Iconst
{
- $$ = make2_str(mm_strdup("="), $2);
+ @$ = make2_str(mm_strdup("="), @2);
}
| '=' ecpg_ident
{
- $$ = make2_str(mm_strdup("="), $2);
+ @$ = make2_str(mm_strdup("="), @2);
}
| '=' civar
{
- $$ = make2_str(mm_strdup("="), $2);
+ @$ = make2_str(mm_strdup("="), @2);
}
;
prepared_name: name
{
- if ($1[0] == '\"' && $1[strlen($1) - 1] == '\"') /* already quoted? */
- $$ = $1;
+ if (@1[0] == '\"' && @1[strlen(@1) - 1] == '\"') /* already quoted? */
+ @$ = @1;
else /* not quoted => convert to lowercase */
{
size_t i;
- for (i = 0; i < strlen($1); i++)
- $1[i] = tolower((unsigned char) $1[i]);
+ for (i = 0; i < strlen(@1); i++)
+ @1[i] = tolower((unsigned char) @1[i]);
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
}
| char_variable
- {
- $$ = $1;
- }
;
/*
@@ -400,7 +360,7 @@ ECPGDeclareStmt: DECLARE prepared_name STATEMENT
/* Check whether the declared name has been defined or not */
for (ptr = g_declared_list; ptr != NULL; ptr = ptr->next)
{
- if (strcmp($2, ptr->name) == 0)
+ if (strcmp(@2, ptr->name) == 0)
{
/* re-definition is not allowed */
mmerror(PARSE_ERROR, ET_ERROR, "name \"%s\" is already declared", ptr->name);
@@ -413,7 +373,7 @@ ECPGDeclareStmt: DECLARE prepared_name STATEMENT
if (ptr)
{
/* initial definition */
- ptr->name = $2;
+ ptr->name = @2;
if (connection)
ptr->connection = mm_strdup(connection);
else
@@ -423,7 +383,7 @@ ECPGDeclareStmt: DECLARE prepared_name STATEMENT
g_declared_list = ptr;
}
- $$ = cat_str(3, mm_strdup("/* declare "), mm_strdup($2), mm_strdup(" as an SQL identifier */"));
+ @$ = cat_str(3, mm_strdup("/* declare "), mm_strdup(@2), mm_strdup(" as an SQL identifier */"));
}
;
@@ -435,26 +395,26 @@ ECPGCursorStmt: DECLARE cursor_name cursor_options CURSOR opt_hold FOR prepared_
{
struct cursor *ptr,
*this;
- char *cursor_marker = $2[0] == ':' ? mm_strdup("$0") : mm_strdup($2);
- int (*strcmp_fn) (const char *, const char *) = (($2[0] == ':' || $2[0] == '"') ? strcmp : pg_strcasecmp);
+ char *cursor_marker = @2[0] == ':' ? mm_strdup("$0") : mm_strdup(@2);
+ int (*strcmp_fn) (const char *, const char *) = ((@2[0] == ':' || @2[0] == '"') ? strcmp : pg_strcasecmp);
struct variable *thisquery = (struct variable *) mm_alloc(sizeof(struct variable));
char *comment;
char *con;
- if (INFORMIX_MODE && pg_strcasecmp($2, "database") == 0)
+ if (INFORMIX_MODE && pg_strcasecmp(@2, "database") == 0)
mmfatal(PARSE_ERROR, "\"database\" cannot be used as cursor name in INFORMIX mode");
- check_declared_list($7);
+ check_declared_list(@7);
con = connection ? connection : "NULL";
for (ptr = cur; ptr != NULL; ptr = ptr->next)
{
- if (strcmp_fn($2, ptr->name) == 0)
+ if (strcmp_fn(@2, ptr->name) == 0)
{
/* re-definition is a bug */
- if ($2[0] == ':')
- mmerror(PARSE_ERROR, ET_ERROR, "using variable \"%s\" in different declare statements is not supported", $2 + 1);
+ if (@2[0] == ':')
+ mmerror(PARSE_ERROR, ET_ERROR, "using variable \"%s\" in different declare statements is not supported", @2 + 1);
else
- mmerror(PARSE_ERROR, ET_ERROR, "cursor \"%s\" is already defined", $2);
+ mmerror(PARSE_ERROR, ET_ERROR, "cursor \"%s\" is already defined", @2);
}
}
@@ -462,24 +422,24 @@ ECPGCursorStmt: DECLARE cursor_name cursor_options CURSOR opt_hold FOR prepared_
/* initial definition */
this->next = cur;
- this->name = $2;
+ this->name = @2;
this->function = (current_function ? mm_strdup(current_function) : NULL);
this->connection = connection ? mm_strdup(connection) : NULL;
- this->command = cat_str(6, mm_strdup("declare"), cursor_marker, $3, mm_strdup("cursor"), $5, mm_strdup("for $1"));
+ this->command = cat_str(6, mm_strdup("declare"), cursor_marker, @3, mm_strdup("cursor"), @5, mm_strdup("for $1"));
this->argsresult = NULL;
this->argsresult_oos = NULL;
thisquery->type = &ecpg_query;
thisquery->brace_level = 0;
thisquery->next = NULL;
- thisquery->name = (char *) mm_alloc(sizeof("ECPGprepared_statement(, , __LINE__)") + strlen(con) + strlen($7));
- sprintf(thisquery->name, "ECPGprepared_statement(%s, %s, __LINE__)", con, $7);
+ thisquery->name = (char *) mm_alloc(sizeof("ECPGprepared_statement(, , __LINE__)") + strlen(con) + strlen(@7));
+ sprintf(thisquery->name, "ECPGprepared_statement(%s, %s, __LINE__)", con, @7);
this->argsinsert = NULL;
this->argsinsert_oos = NULL;
- if ($2[0] == ':')
+ if (@2[0] == ':')
{
- struct variable *var = find_variable($2 + 1);
+ struct variable *var = find_variable(@2 + 1);
remove_variable_from_list(&argsinsert, var);
add_variable_to_head(&(this->argsinsert), var, &no_indicator);
@@ -490,7 +450,7 @@ ECPGCursorStmt: DECLARE cursor_name cursor_options CURSOR opt_hold FOR prepared_
comment = cat_str(3, mm_strdup("/*"), mm_strdup(this->command), mm_strdup("*/"));
- $$ = cat_str(2, adjust_outofscope_cursor_vars(this),
+ @$ = cat_str(2, adjust_outofscope_cursor_vars(this),
comment);
}
;
@@ -501,7 +461,7 @@ ECPGExecuteImmediateStmt: EXECUTE IMMEDIATE execstring
* execute immediate means prepare the statement and immediately
* execute it
*/
- $$ = $3;
+ @$ = @3;
}
;
@@ -511,36 +471,24 @@ ECPGExecuteImmediateStmt: EXECUTE IMMEDIATE execstring
ECPGVarDeclaration: single_vt_declaration;
single_vt_declaration: type_declaration
- {
- $$ = $1;
- }
| var_declaration
- {
- $$ = $1;
- }
;
precision: NumericOnly
- {
- $$ = $1;
- }
;
opt_scale: ',' NumericOnly
{
- $$ = $2;
+ @$ = @2;
}
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
-ecpg_interval: opt_interval { $$ = $1; }
- | YEAR_P TO MINUTE_P { $$ = mm_strdup("year to minute"); }
- | YEAR_P TO SECOND_P { $$ = mm_strdup("year to second"); }
- | DAY_P TO DAY_P { $$ = mm_strdup("day to day"); }
- | MONTH_P TO MONTH_P { $$ = mm_strdup("month to month"); }
+ecpg_interval: opt_interval
+ | YEAR_P TO MINUTE_P
+ | YEAR_P TO SECOND_P
+ | DAY_P TO DAY_P
+ | MONTH_P TO MONTH_P
;
/*
@@ -552,8 +500,7 @@ ECPGDeclaration: sql_startdeclare
}
var_type_declarations sql_enddeclare
{
- fprintf(base_yyout, "%s/* exec sql end declare section */", $3);
- free($3);
+ fprintf(base_yyout, "%s/* exec sql end declare section */", @3);
output_line_number();
}
;
@@ -569,41 +516,17 @@ sql_enddeclare: ecpgstart END_P DECLARE SQL_SECTION ';'
;
var_type_declarations: /* EMPTY */
- {
- $$ = EMPTY;
- }
| vt_declarations
- {
- $$ = $1;
- }
;
vt_declarations: single_vt_declaration
- {
- $$ = $1;
- }
| CPP_LINE
- {
- $$ = $1;
- }
| vt_declarations single_vt_declaration
- {
- $$ = cat2_str($1, $2);
- }
| vt_declarations CPP_LINE
- {
- $$ = cat2_str($1, $2);
- }
;
variable_declarations: var_declaration
- {
- $$ = $1;
- }
| variable_declarations var_declaration
- {
- $$ = cat2_str($1, $2);
- }
;
type_declaration: S_TYPEDEF
@@ -614,18 +537,18 @@ type_declaration: S_TYPEDEF
}
var_type opt_pointer ECPGColLabel opt_array_bounds ';'
{
- add_typedef($5, $6.index1, $6.index2, $3.type_enum, $3.type_dimension, $3.type_index, initializer, *$4 ? 1 : 0);
+ add_typedef(@5, $6.index1, $6.index2, $3.type_enum, $3.type_dimension, $3.type_index, initializer, *@4 ? 1 : 0);
- fprintf(base_yyout, "typedef %s %s %s %s;\n", $3.type_str, *$4 ? "*" : "", $5, $6.str);
+ fprintf(base_yyout, "typedef %s %s %s %s;\n", $3.type_str, *@4 ? "*" : "", @5, $6.str);
output_line_number();
- $$ = mm_strdup("");
+ @$ = EMPTY;
}
;
var_declaration:
storage_declaration var_type
{
- actual_type[struct_level].type_storage = $1;
+ actual_type[struct_level].type_storage = @1;
actual_type[struct_level].type_enum = $2.type_enum;
actual_type[struct_level].type_str = $2.type_str;
actual_type[struct_level].type_dimension = $2.type_dimension;
@@ -636,7 +559,7 @@ var_declaration:
}
variable_list ';'
{
- $$ = cat_str(5, actual_startline[struct_level], $1, $2.type_str, $4, mm_strdup(";\n"));
+ @$ = cat_str(5, actual_startline[struct_level], @1, $2.type_str, @4, mm_strdup(";\n"));
}
| var_type
{
@@ -651,46 +574,31 @@ var_declaration:
}
variable_list ';'
{
- $$ = cat_str(4, actual_startline[struct_level], $1.type_str, $3, mm_strdup(";\n"));
+ @$ = cat_str(4, actual_startline[struct_level], $1.type_str, @3, mm_strdup(";\n"));
}
| struct_union_type_with_symbol ';'
{
- $$ = cat2_str($1, mm_strdup(";"));
+ @$ = cat2_str(@1, mm_strdup(";"));
}
;
opt_bit_field: ':' Iconst
- {
- $$ = cat2_str(mm_strdup(":"), $2);
- }
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
storage_declaration: storage_clause storage_modifier
- {
- $$ = cat2_str($1, $2);
- }
| storage_clause
- {
- $$ = $1;
- }
| storage_modifier
- {
- $$ = $1;
- }
;
-storage_clause: S_EXTERN { $$ = mm_strdup("extern"); }
- | S_STATIC { $$ = mm_strdup("static"); }
- | S_REGISTER { $$ = mm_strdup("register"); }
- | S_AUTO { $$ = mm_strdup("auto"); }
+storage_clause: S_EXTERN
+ | S_STATIC
+ | S_REGISTER
+ | S_AUTO
;
-storage_modifier: S_CONST { $$ = mm_strdup("const"); }
- | S_VOLATILE { $$ = mm_strdup("volatile"); }
+storage_modifier: S_CONST
+ | S_VOLATILE
;
var_type: simple_type
@@ -703,11 +611,11 @@ var_type: simple_type
}
| struct_union_type
{
- $$.type_str = $1;
+ $$.type_str = @1;
$$.type_dimension = mm_strdup("-1");
$$.type_index = mm_strdup("-1");
- if (strncmp($1, "struct", sizeof("struct") - 1) == 0)
+ if (strncmp(@1, "struct", sizeof("struct") - 1) == 0)
{
$$.type_enum = ECPGt_struct;
$$.type_sizeof = ECPGstruct_sizeof;
@@ -720,7 +628,7 @@ var_type: simple_type
}
| enum_type
{
- $$.type_str = $1;
+ $$.type_str = @1;
$$.type_enum = ECPGt_int;
$$.type_dimension = mm_strdup("-1");
$$.type_index = mm_strdup("-1");
@@ -749,12 +657,12 @@ var_type: simple_type
* will show up here as a plain identifier, and we need this duplicate
* code to recognize them.
*/
- if (strcmp($1, "numeric") == 0)
+ if (strcmp(@1, "numeric") == 0)
{
$$.type_enum = ECPGt_numeric;
$$.type_str = mm_strdup("numeric");
}
- else if (strcmp($1, "decimal") == 0)
+ else if (strcmp(@1, "decimal") == 0)
{
$$.type_enum = ECPGt_decimal;
$$.type_str = mm_strdup("decimal");
@@ -858,10 +766,10 @@ var_type: simple_type
* here, but not above because those are not currently SQL keywords.
* If they ever become so, they must gain duplicate productions above.
*/
- if (strlen($2) != 0 && strcmp($1, "datetime") != 0 && strcmp($1, "interval") != 0)
+ if (strlen(@2) != 0 && strcmp(@1, "datetime") != 0 && strcmp(@1, "interval") != 0)
mmerror(PARSE_ERROR, ET_ERROR, "interval specification not allowed here");
- if (strcmp($1, "varchar") == 0)
+ if (strcmp(@1, "varchar") == 0)
{
$$.type_enum = ECPGt_varchar;
$$.type_str = EMPTY; /* mm_strdup("varchar"); */
@@ -869,7 +777,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "bytea") == 0)
+ else if (strcmp(@1, "bytea") == 0)
{
$$.type_enum = ECPGt_bytea;
$$.type_str = EMPTY;
@@ -877,7 +785,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "float") == 0)
+ else if (strcmp(@1, "float") == 0)
{
$$.type_enum = ECPGt_float;
$$.type_str = mm_strdup("float");
@@ -885,7 +793,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "double") == 0)
+ else if (strcmp(@1, "double") == 0)
{
$$.type_enum = ECPGt_double;
$$.type_str = mm_strdup("double");
@@ -893,7 +801,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "numeric") == 0)
+ else if (strcmp(@1, "numeric") == 0)
{
$$.type_enum = ECPGt_numeric;
$$.type_str = mm_strdup("numeric");
@@ -901,7 +809,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "decimal") == 0)
+ else if (strcmp(@1, "decimal") == 0)
{
$$.type_enum = ECPGt_decimal;
$$.type_str = mm_strdup("decimal");
@@ -909,7 +817,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "date") == 0)
+ else if (strcmp(@1, "date") == 0)
{
$$.type_enum = ECPGt_date;
$$.type_str = mm_strdup("date");
@@ -917,7 +825,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "timestamp") == 0)
+ else if (strcmp(@1, "timestamp") == 0)
{
$$.type_enum = ECPGt_timestamp;
$$.type_str = mm_strdup("timestamp");
@@ -925,7 +833,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "interval") == 0)
+ else if (strcmp(@1, "interval") == 0)
{
$$.type_enum = ECPGt_interval;
$$.type_str = mm_strdup("interval");
@@ -933,7 +841,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if (strcmp($1, "datetime") == 0)
+ else if (strcmp(@1, "datetime") == 0)
{
$$.type_enum = ECPGt_timestamp;
$$.type_str = mm_strdup("timestamp");
@@ -941,7 +849,7 @@ var_type: simple_type
$$.type_index = mm_strdup("-1");
$$.type_sizeof = NULL;
}
- else if ((strcmp($1, "string") == 0) && INFORMIX_MODE)
+ else if ((strcmp(@1, "string") == 0) && INFORMIX_MODE)
{
$$.type_enum = ECPGt_string;
$$.type_str = mm_strdup("char");
@@ -952,7 +860,7 @@ var_type: simple_type
else
{
/* Otherwise, it must be a user-defined typedef name */
- struct typedefs *this = get_typedef($1, false);
+ struct typedefs *this = get_typedef(@1, false);
$$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? EMPTY : mm_strdup(this->name);
$$.type_enum = this->type->type_enum;
@@ -1001,23 +909,11 @@ var_type: simple_type
;
enum_type: ENUM_P symbol enum_definition
- {
- $$ = cat_str(3, mm_strdup("enum"), $2, $3);
- }
| ENUM_P enum_definition
- {
- $$ = cat2_str(mm_strdup("enum"), $2);
- }
| ENUM_P symbol
- {
- $$ = cat2_str(mm_strdup("enum"), $2);
- }
;
enum_definition: '{' c_list '}'
- {
- $$ = cat_str(3, mm_strdup("{"), $2, mm_strdup("}"));
- }
;
struct_union_type_with_symbol: s_struct_union_symbol
@@ -1071,14 +967,11 @@ struct_union_type_with_symbol: s_struct_union_symbol
this->struct_member_list = struct_member_list[struct_level];
types = this;
- $$ = cat_str(4, su_type.type_str, mm_strdup("{"), $4, mm_strdup("}"));
+ @$ = cat_str(4, su_type.type_str, mm_strdup("{"), @4, mm_strdup("}"));
}
;
struct_union_type: struct_union_type_with_symbol
- {
- $$ = $1;
- }
| s_struct_union
{
struct_member_list[struct_level++] = NULL;
@@ -1090,20 +983,20 @@ struct_union_type: struct_union_type_with_symbol
ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = NULL;
struct_level--;
- $$ = cat_str(4, $1, mm_strdup("{"), $4, mm_strdup("}"));
+ @$ = cat_str(4, @1, mm_strdup("{"), @4, mm_strdup("}"));
}
;
s_struct_union_symbol: SQL_STRUCT symbol
{
$$.su = mm_strdup("struct");
- $$.symbol = $2;
+ $$.symbol = @2;
ECPGstruct_sizeof = cat_str(3, mm_strdup("sizeof("), cat2_str(mm_strdup($$.su), mm_strdup($$.symbol)), mm_strdup(")"));
}
| UNION symbol
{
$$.su = mm_strdup("union");
- $$.symbol = $2;
+ $$.symbol = @2;
}
;
@@ -1111,15 +1004,15 @@ s_struct_union: SQL_STRUCT
{
ECPGstruct_sizeof = mm_strdup(""); /* This must not be NULL to
* distinguish from simple types. */
- $$ = mm_strdup("struct");
+ @$ = mm_strdup("struct");
}
| UNION
{
- $$ = mm_strdup("union");
+ @$ = mm_strdup("union");
}
;
-simple_type: unsigned_type { $$ = $1; }
+simple_type: unsigned_type
| opt_signed signed_type { $$ = $2; }
;
@@ -1151,15 +1044,12 @@ opt_signed: SQL_SIGNED
;
variable_list: variable
- {
- $$ = $1;
- }
| variable_list ',' variable
{
if (actual_type[struct_level].type_enum == ECPGt_varchar || actual_type[struct_level].type_enum == ECPGt_bytea)
- $$ = cat_str(4, $1, mm_strdup(";"), mm_strdup(actual_type[struct_level].type_storage), $3);
+ @$ = cat_str(4, @1, mm_strdup(";"), mm_strdup(actual_type[struct_level].type_storage), @3);
else
- $$ = cat_str(3, $1, mm_strdup(","), $3);
+ @$ = cat_str(3, @1, mm_strdup(","), @3);
}
;
@@ -1173,7 +1063,7 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
int *varlen_type_counter;
char *struct_name;
- adjust_array(actual_type[struct_level].type_enum, &dimension, &length, actual_type[struct_level].type_dimension, actual_type[struct_level].type_index, strlen($1), false);
+ adjust_array(actual_type[struct_level].type_enum, &dimension, &length, actual_type[struct_level].type_dimension, actual_type[struct_level].type_index, strlen(@1), false);
switch (actual_type[struct_level].type_enum)
{
case ECPGt_struct:
@@ -1183,7 +1073,7 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
else
type = ECPGmake_array_type(ECPGmake_struct_type(struct_member_list[struct_level], actual_type[struct_level].type_enum, actual_type[struct_level].type_str, actual_type[struct_level].type_sizeof), dimension);
- $$ = cat_str(5, $1, mm_strdup($2), $3.str, $4, $5);
+ @$ = cat_str(5, @1, mm_strdup(@2), $3.str, @4, @5);
break;
case ECPGt_varchar:
@@ -1222,9 +1112,9 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
vcn = (char *) mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
sprintf(vcn, "%d", *varlen_type_counter);
if (strcmp(dimension, "0") == 0)
- $$ = cat_str(7, make2_str(mm_strdup(struct_name), vcn), mm_strdup(" { int len; char arr["), mm_strdup(length), mm_strdup("]; } *"), mm_strdup($2), $4, $5);
+ @$ = cat_str(7, make2_str(mm_strdup(struct_name), vcn), mm_strdup(" { int len; char arr["), mm_strdup(length), mm_strdup("]; } *"), mm_strdup(@2), @4, @5);
else
- $$ = cat_str(8, make2_str(mm_strdup(struct_name), vcn), mm_strdup(" { int len; char arr["), mm_strdup(length), mm_strdup("]; } "), mm_strdup($2), dim_str, $4, $5);
+ @$ = cat_str(8, make2_str(mm_strdup(struct_name), vcn), mm_strdup(" { int len; char arr["), mm_strdup(length), mm_strdup("]; } "), mm_strdup(@2), dim_str, @4, @5);
(*varlen_type_counter)++;
break;
@@ -1233,7 +1123,7 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
case ECPGt_string:
if (atoi(dimension) == -1)
{
- int i = strlen($5);
+ int i = strlen(@5);
if (atoi(length) == -1 && i > 0) /* char <var>[] =
* "string" */
@@ -1244,14 +1134,14 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
*/
free(length);
length = mm_alloc(i + sizeof("sizeof()"));
- sprintf(length, "sizeof(%s)", $5 + 2);
+ sprintf(length, "sizeof(%s)", @5 + 2);
}
type = ECPGmake_simple_type(actual_type[struct_level].type_enum, length, 0);
}
else
type = ECPGmake_array_type(ECPGmake_simple_type(actual_type[struct_level].type_enum, length, 0), dimension);
- $$ = cat_str(5, $1, mm_strdup($2), $3.str, $4, $5);
+ @$ = cat_str(5, @1, mm_strdup(@2), $3.str, @4, @5);
break;
default:
@@ -1260,41 +1150,29 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
else
type = ECPGmake_array_type(ECPGmake_simple_type(actual_type[struct_level].type_enum, mm_strdup("1"), 0), dimension);
- $$ = cat_str(5, $1, mm_strdup($2), $3.str, $4, $5);
+ @$ = cat_str(5, @1, mm_strdup(@2), $3.str, @4, @5);
break;
}
if (struct_level == 0)
- new_variable($2, type, braces_open);
+ new_variable(@2, type, braces_open);
else
- ECPGmake_struct_member($2, type, &(struct_member_list[struct_level - 1]));
-
- free($2);
+ ECPGmake_struct_member(@2, type, &(struct_member_list[struct_level - 1]));
}
;
opt_initializer: /* EMPTY */
- {
- $$ = EMPTY;
- }
| '=' c_term
{
initializer = 1;
- $$ = cat2_str(mm_strdup("="), $2);
}
;
opt_pointer: /* EMPTY */
- {
- $$ = EMPTY;
- }
| '*'
- {
- $$ = mm_strdup("*");
- }
| '*' '*'
{
- $$ = mm_strdup("**");
+ @$ = mm_strdup("**");
}
;
@@ -1304,7 +1182,7 @@ opt_pointer: /* EMPTY */
ECPGDeclare: DECLARE STATEMENT ecpg_ident
{
/* this is only supported for compatibility */
- $$ = cat_str(3, mm_strdup("/* declare statement"), $3, mm_strdup("*/"));
+ @$ = cat_str(3, mm_strdup("/* declare statement"), @3, mm_strdup("*/"));
}
;
/*
@@ -1312,49 +1190,40 @@ ECPGDeclare: DECLARE STATEMENT ecpg_ident
*/
ECPGDisconnect: SQL_DISCONNECT dis_name
{
- $$ = $2;
+ @$ = @2;
}
;
dis_name: connection_object
- {
- $$ = $1;
- }
| CURRENT_P
{
- $$ = mm_strdup("\"CURRENT\"");
+ @$ = mm_strdup("\"CURRENT\"");
}
| ALL
{
- $$ = mm_strdup("\"ALL\"");
+ @$ = mm_strdup("\"ALL\"");
}
| /* EMPTY */
{
- $$ = mm_strdup("\"CURRENT\"");
+ @$ = mm_strdup("\"CURRENT\"");
}
;
connection_object: name
{
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
| DEFAULT
{
- $$ = mm_strdup("\"DEFAULT\"");
+ @$ = mm_strdup("\"DEFAULT\"");
}
| char_variable
- {
- $$ = $1;
- }
;
execstring: char_variable
- {
- $$ = $1;
- }
| CSTRING
{
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
;
@@ -1364,11 +1233,11 @@ execstring: char_variable
*/
ECPGFree: SQL_FREE cursor_name
{
- $$ = $2;
+ @$ = @2;
}
| SQL_FREE ALL
{
- $$ = mm_strdup("all");
+ @$ = mm_strdup("all");
}
;
@@ -1377,60 +1246,51 @@ ECPGFree: SQL_FREE cursor_name
*/
ECPGOpen: SQL_OPEN cursor_name opt_ecpg_using
{
- if ($2[0] == ':')
- remove_variable_from_list(&argsinsert, find_variable($2 + 1));
- $$ = $2;
+ if (@2[0] == ':')
+ remove_variable_from_list(&argsinsert, find_variable(@2 + 1));
+ @$ = @2;
}
;
opt_ecpg_using: /* EMPTY */
- {
- $$ = EMPTY;
- }
| ecpg_using
- {
- $$ = $1;
- }
;
ecpg_using: USING using_list
{
- $$ = EMPTY;
+ @$ = EMPTY;
}
| using_descriptor
- {
- $$ = $1;
- }
;
using_descriptor: USING SQL_P SQL_DESCRIPTOR quoted_ident_stringvar
{
- add_variable_to_head(&argsinsert, descriptor_variable($4, 0), &no_indicator);
- $$ = EMPTY;
+ add_variable_to_head(&argsinsert, descriptor_variable(@4, 0), &no_indicator);
+ @$ = EMPTY;
}
| USING SQL_DESCRIPTOR name
{
- add_variable_to_head(&argsinsert, sqlda_variable($3), &no_indicator);
- $$ = EMPTY;
+ add_variable_to_head(&argsinsert, sqlda_variable(@3), &no_indicator);
+ @$ = EMPTY;
}
;
into_descriptor: INTO SQL_P SQL_DESCRIPTOR quoted_ident_stringvar
{
- add_variable_to_head(&argsresult, descriptor_variable($4, 1), &no_indicator);
- $$ = EMPTY;
+ add_variable_to_head(&argsresult, descriptor_variable(@4, 1), &no_indicator);
+ @$ = EMPTY;
}
| INTO SQL_DESCRIPTOR name
{
- add_variable_to_head(&argsresult, sqlda_variable($3), &no_indicator);
- $$ = EMPTY;
+ add_variable_to_head(&argsresult, sqlda_variable(@3), &no_indicator);
+ @$ = EMPTY;
}
;
into_sqlda: INTO name
{
- add_variable_to_head(&argsresult, sqlda_variable($2), &no_indicator);
- $$ = EMPTY;
+ add_variable_to_head(&argsresult, sqlda_variable(@2), &no_indicator);
+ @$ = EMPTY;
}
;
@@ -1441,55 +1301,28 @@ UsingValue: UsingConst
{
char *length = mm_alloc(32);
- sprintf(length, "%zu", strlen($1));
- add_variable_to_head(&argsinsert, new_variable($1, ECPGmake_simple_type(ECPGt_const, length, 0), 0), &no_indicator);
+ sprintf(length, "%zu", strlen(@1));
+ add_variable_to_head(&argsinsert, new_variable(@1, ECPGmake_simple_type(ECPGt_const, length, 0), 0), &no_indicator);
}
| civar
{
- $$ = EMPTY;
+ @$ = EMPTY;
}
| civarind
{
- $$ = EMPTY;
+ @$ = EMPTY;
}
;
UsingConst: Iconst
- {
- $$ = $1;
- }
| '+' Iconst
- {
- $$ = cat_str(2, mm_strdup("+"), $2);
- }
| '-' Iconst
- {
- $$ = cat_str(2, mm_strdup("-"), $2);
- }
| ecpg_fconst
- {
- $$ = $1;
- }
| '+' ecpg_fconst
- {
- $$ = cat_str(2, mm_strdup("+"), $2);
- }
| '-' ecpg_fconst
- {
- $$ = cat_str(2, mm_strdup("-"), $2);
- }
| ecpg_sconst
- {
- $$ = $1;
- }
| ecpg_bconst
- {
- $$ = $1;
- }
| ecpg_xconst
- {
- $$ = $1;
- }
;
/*
@@ -1498,7 +1331,7 @@ UsingConst: Iconst
ECPGDescribe: SQL_DESCRIBE INPUT_P prepared_name using_descriptor
{
$$.input = 1;
- $$.stmt_name = $3;
+ $$.stmt_name = @3;
}
| SQL_DESCRIBE opt_output prepared_name using_descriptor
{
@@ -1509,33 +1342,27 @@ ECPGDescribe: SQL_DESCRIBE INPUT_P prepared_name using_descriptor
add_variable_to_head(&argsresult, var, &no_indicator);
$$.input = 0;
- $$.stmt_name = $3;
+ $$.stmt_name = @3;
}
| SQL_DESCRIBE opt_output prepared_name into_descriptor
{
$$.input = 0;
- $$.stmt_name = $3;
+ $$.stmt_name = @3;
}
| SQL_DESCRIBE INPUT_P prepared_name into_sqlda
{
$$.input = 1;
- $$.stmt_name = $3;
+ $$.stmt_name = @3;
}
| SQL_DESCRIBE opt_output prepared_name into_sqlda
{
$$.input = 0;
- $$.stmt_name = $3;
+ $$.stmt_name = @3;
}
;
opt_output: SQL_OUTPUT
- {
- $$ = mm_strdup("output");
- }
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
/*
@@ -1549,8 +1376,8 @@ opt_output: SQL_OUTPUT
*/
ECPGAllocateDescr: SQL_ALLOCATE SQL_DESCRIPTOR quoted_ident_stringvar
{
- add_descriptor($3, connection);
- $$ = $3;
+ add_descriptor(@3, connection);
+ @$ = @3;
}
;
@@ -1560,8 +1387,8 @@ ECPGAllocateDescr: SQL_ALLOCATE SQL_DESCRIPTOR quoted_ident_stringvar
*/
ECPGDeallocateDescr: DEALLOCATE SQL_DESCRIPTOR quoted_ident_stringvar
{
- drop_descriptor($3, connection);
- $$ = $3;
+ drop_descriptor(@3, connection);
+ @$ = @3;
}
;
@@ -1571,7 +1398,7 @@ ECPGDeallocateDescr: DEALLOCATE SQL_DESCRIPTOR quoted_ident_stringvar
ECPGGetDescriptorHeader: SQL_GET SQL_DESCRIPTOR quoted_ident_stringvar ECPGGetDescHeaderItems
{
- $$ = $3;
+ @$ = @3;
}
;
@@ -1581,13 +1408,13 @@ ECPGGetDescHeaderItems: ECPGGetDescHeaderItem
ECPGGetDescHeaderItem: cvariable '=' desc_header_item
{
- push_assignment($1, $3);
+ push_assignment(@1, $3);
}
;
ECPGSetDescriptorHeader: SET SQL_DESCRIPTOR quoted_ident_stringvar ECPGSetDescHeaderItems
{
- $$ = $3;
+ @$ = @3;
}
;
@@ -1597,7 +1424,7 @@ ECPGSetDescHeaderItems: ECPGSetDescHeaderItem
ECPGSetDescHeaderItem: desc_header_item '=' IntConstVar
{
- push_assignment($3, $1);
+ push_assignment(@3, $1);
}
;
@@ -1605,14 +1432,10 @@ IntConstVar: Iconst
{
char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- sprintf(length, "%zu", strlen($1));
- new_variable($1, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
- $$ = $1;
+ sprintf(length, "%zu", strlen(@1));
+ new_variable(@1, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
}
| cvariable
- {
- $$ = $1;
- }
;
desc_header_item: SQL_COUNT
@@ -1627,8 +1450,8 @@ desc_header_item: SQL_COUNT
ECPGGetDescriptor: SQL_GET SQL_DESCRIPTOR quoted_ident_stringvar VALUE_P IntConstVar ECPGGetDescItems
{
- $$.str = $5;
- $$.name = $3;
+ $$.str = @5;
+ $$.name = @3;
}
;
@@ -1638,14 +1461,14 @@ ECPGGetDescItems: ECPGGetDescItem
ECPGGetDescItem: cvariable '=' descriptor_item
{
- push_assignment($1, $3);
+ push_assignment(@1, $3);
}
;
ECPGSetDescriptor: SET SQL_DESCRIPTOR quoted_ident_stringvar VALUE_P IntConstVar ECPGSetDescItems
{
- $$.str = $5;
- $$.name = $3;
+ $$.str = @5;
+ $$.name = @3;
}
;
@@ -1655,7 +1478,7 @@ ECPGSetDescItems: ECPGSetDescItem
ECPGSetDescItem: descriptor_item '=' AllConstVar
{
- push_assignment($3, $1);
+ push_assignment(@3, $1);
}
;
@@ -1663,41 +1486,37 @@ AllConstVar: ecpg_fconst
{
char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- sprintf(length, "%zu", strlen($1));
- new_variable($1, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
- $$ = $1;
+ sprintf(length, "%zu", strlen(@1));
+ new_variable(@1, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
}
| IntConstVar
- {
- $$ = $1;
- }
| '-' ecpg_fconst
{
char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *var = cat2_str(mm_strdup("-"), $2);
+ char *var = cat2_str(mm_strdup("-"), @2);
sprintf(length, "%zu", strlen(var));
new_variable(var, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
- $$ = var;
+ @$ = var;
}
| '-' Iconst
{
char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *var = cat2_str(mm_strdup("-"), $2);
+ char *var = cat2_str(mm_strdup("-"), @2);
sprintf(length, "%zu", strlen(var));
new_variable(var, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
- $$ = var;
+ @$ = var;
}
| ecpg_sconst
{
char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *var = $1 + 1;
+ char *var = @1 + 1;
var[strlen(var) - 1] = '\0';
sprintf(length, "%zu", strlen(var));
new_variable(var, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
- $$ = var;
+ @$ = var;
}
;
@@ -1724,22 +1543,16 @@ descriptor_item: SQL_CARDINALITY { $$ = ECPGd_cardinality; }
*/
ECPGSetAutocommit: SET SQL_AUTOCOMMIT '=' on_off
{
- $$ = $4;
+ @$ = @4;
}
| SET SQL_AUTOCOMMIT TO on_off
{
- $$ = $4;
+ @$ = @4;
}
;
on_off: ON
- {
- $$ = mm_strdup("on");
- }
| OFF
- {
- $$ = mm_strdup("off");
- }
;
/*
@@ -1748,15 +1561,15 @@ on_off: ON
*/
ECPGSetConnection: SET CONNECTION TO connection_object
{
- $$ = $4;
+ @$ = @4;
}
| SET CONNECTION '=' connection_object
{
- $$ = $4;
+ @$ = @4;
}
| SET CONNECTION connection_object
{
- $$ = $3;
+ @$ = @3;
}
;
@@ -1771,23 +1584,17 @@ ECPGTypedef: TYPE_P
}
ECPGColLabel IS var_type opt_array_bounds opt_reference
{
- add_typedef($3, $6.index1, $6.index2, $5.type_enum, $5.type_dimension, $5.type_index, initializer, *$7 ? 1 : 0);
+ add_typedef(@3, $6.index1, $6.index2, $5.type_enum, $5.type_dimension, $5.type_index, initializer, *@7 ? 1 : 0);
if (auto_create_c == false)
- $$ = cat_str(7, mm_strdup("/* exec sql type"), mm_strdup($3), mm_strdup("is"), mm_strdup($5.type_str), mm_strdup($6.str), $7, mm_strdup("*/"));
+ @$ = cat_str(7, mm_strdup("/* exec sql type"), mm_strdup(@3), mm_strdup("is"), mm_strdup($5.type_str), mm_strdup($6.str), @7, mm_strdup("*/"));
else
- $$ = cat_str(6, mm_strdup("typedef "), mm_strdup($5.type_str), *$7 ? mm_strdup("*") : mm_strdup(""), mm_strdup($3), mm_strdup($6.str), mm_strdup(";"));
+ @$ = cat_str(6, mm_strdup("typedef "), mm_strdup($5.type_str), *@7 ? mm_strdup("*") : mm_strdup(""), mm_strdup(@3), mm_strdup($6.str), mm_strdup(";"));
}
;
opt_reference: SQL_REFERENCE
- {
- $$ = mm_strdup("reference");
- }
| /* EMPTY */
- {
- $$ = EMPTY;
- }
;
/*
@@ -1801,7 +1608,7 @@ ECPGVar: SQL_VAR
}
ColLabel IS var_type opt_array_bounds opt_reference
{
- struct variable *p = find_variable($3);
+ struct variable *p = find_variable(@3);
char *dimension = $6.index1;
char *length = $6.index2;
struct ECPGtype *type;
@@ -1812,7 +1619,7 @@ ECPGVar: SQL_VAR
mmerror(PARSE_ERROR, ET_ERROR, "initializer not allowed in EXEC SQL VAR command");
else
{
- adjust_array($5.type_enum, &dimension, &length, $5.type_dimension, $5.type_index, *$7 ? 1 : 0, false);
+ adjust_array($5.type_enum, &dimension, &length, $5.type_dimension, $5.type_index, *@7 ? 1 : 0, false);
switch ($5.type_enum)
{
@@ -1856,7 +1663,7 @@ ECPGVar: SQL_VAR
p->type = type;
}
- $$ = cat_str(7, mm_strdup("/* exec sql var"), mm_strdup($3), mm_strdup("is"), mm_strdup($5.type_str), mm_strdup($6.str), $7, mm_strdup("*/"));
+ @$ = cat_str(7, mm_strdup("/* exec sql var"), mm_strdup(@3), mm_strdup("is"), mm_strdup($5.type_str), mm_strdup($6.str), @7, mm_strdup("*/"));
}
;
@@ -1868,19 +1675,19 @@ ECPGWhenever: SQL_WHENEVER SQL_SQLERROR action
{
when_error.code = $<action>3.code;
when_error.command = $<action>3.command;
- $$ = cat_str(3, mm_strdup("/* exec sql whenever sqlerror "), $3.str, mm_strdup("; */"));
+ @$ = cat_str(3, mm_strdup("/* exec sql whenever sqlerror "), $3.str, mm_strdup("; */"));
}
| SQL_WHENEVER NOT SQL_FOUND action
{
when_nf.code = $<action>4.code;
when_nf.command = $<action>4.command;
- $$ = cat_str(3, mm_strdup("/* exec sql whenever not found "), $4.str, mm_strdup("; */"));
+ @$ = cat_str(3, mm_strdup("/* exec sql whenever not found "), $4.str, mm_strdup("; */"));
}
| SQL_WHENEVER SQL_SQLWARNING action
{
when_warn.code = $<action>3.code;
when_warn.command = $<action>3.command;
- $$ = cat_str(3, mm_strdup("/* exec sql whenever sql_warning "), $3.str, mm_strdup("; */"));
+ @$ = cat_str(3, mm_strdup("/* exec sql whenever sql_warning "), $3.str, mm_strdup("; */"));
}
;
@@ -1905,19 +1712,19 @@ action: CONTINUE_P
| SQL_GOTO name
{
$<action>$.code = W_GOTO;
- $<action>$.command = mm_strdup($2);
- $<action>$.str = cat2_str(mm_strdup("goto "), $2);
+ $<action>$.command = mm_strdup(@2);
+ $<action>$.str = cat2_str(mm_strdup("goto "), @2);
}
| SQL_GO TO name
{
$<action>$.code = W_GOTO;
- $<action>$.command = mm_strdup($3);
- $<action>$.str = cat2_str(mm_strdup("goto "), $3);
+ $<action>$.command = mm_strdup(@3);
+ $<action>$.str = cat2_str(mm_strdup("goto "), @3);
}
| DO name '(' c_args ')'
{
$<action>$.code = W_DO;
- $<action>$.command = cat_str(4, $2, mm_strdup("("), $4, mm_strdup(")"));
+ $<action>$.command = cat_str(4, @2, mm_strdup("("), @4, mm_strdup(")"));
$<action>$.str = cat2_str(mm_strdup("do"), mm_strdup($<action>$.command));
}
| DO SQL_BREAK
@@ -1935,13 +1742,13 @@ action: CONTINUE_P
| CALL name '(' c_args ')'
{
$<action>$.code = W_DO;
- $<action>$.command = cat_str(4, $2, mm_strdup("("), $4, mm_strdup(")"));
+ $<action>$.command = cat_str(4, @2, mm_strdup("("), @4, mm_strdup(")"));
$<action>$.str = cat2_str(mm_strdup("call"), mm_strdup($<action>$.command));
}
| CALL name
{
$<action>$.code = W_DO;
- $<action>$.command = cat2_str($2, mm_strdup("()"));
+ $<action>$.command = cat2_str(@2, mm_strdup("()"));
$<action>$.str = cat2_str(mm_strdup("call"), mm_strdup($<action>$.command));
}
;
@@ -1949,63 +1756,63 @@ action: CONTINUE_P
/* some other stuff for ecpg */
/* additional unreserved keywords */
-ECPGKeywords: ECPGKeywords_vanames { $$ = $1; }
- | ECPGKeywords_rest { $$ = $1; }
- ;
-
-ECPGKeywords_vanames: SQL_BREAK { $$ = mm_strdup("break"); }
- | SQL_CARDINALITY { $$ = mm_strdup("cardinality"); }
- | SQL_COUNT { $$ = mm_strdup("count"); }
- | SQL_DATETIME_INTERVAL_CODE { $$ = mm_strdup("datetime_interval_code"); }
- | SQL_DATETIME_INTERVAL_PRECISION { $$ = mm_strdup("datetime_interval_precision"); }
- | SQL_FOUND { $$ = mm_strdup("found"); }
- | SQL_GO { $$ = mm_strdup("go"); }
- | SQL_GOTO { $$ = mm_strdup("goto"); }
- | SQL_IDENTIFIED { $$ = mm_strdup("identified"); }
- | SQL_INDICATOR { $$ = mm_strdup("indicator"); }
- | SQL_KEY_MEMBER { $$ = mm_strdup("key_member"); }
- | SQL_LENGTH { $$ = mm_strdup("length"); }
- | SQL_NULLABLE { $$ = mm_strdup("nullable"); }
- | SQL_OCTET_LENGTH { $$ = mm_strdup("octet_length"); }
- | SQL_RETURNED_LENGTH { $$ = mm_strdup("returned_length"); }
- | SQL_RETURNED_OCTET_LENGTH { $$ = mm_strdup("returned_octet_length"); }
- | SQL_SCALE { $$ = mm_strdup("scale"); }
- | SQL_SECTION { $$ = mm_strdup("section"); }
- | SQL_SQLERROR { $$ = mm_strdup("sqlerror"); }
- | SQL_SQLPRINT { $$ = mm_strdup("sqlprint"); }
- | SQL_SQLWARNING { $$ = mm_strdup("sqlwarning"); }
- | SQL_STOP { $$ = mm_strdup("stop"); }
- ;
-
-ECPGKeywords_rest: SQL_CONNECT { $$ = mm_strdup("connect"); }
- | SQL_DESCRIBE { $$ = mm_strdup("describe"); }
- | SQL_DISCONNECT { $$ = mm_strdup("disconnect"); }
- | SQL_OPEN { $$ = mm_strdup("open"); }
- | SQL_VAR { $$ = mm_strdup("var"); }
- | SQL_WHENEVER { $$ = mm_strdup("whenever"); }
+ECPGKeywords: ECPGKeywords_vanames
+ | ECPGKeywords_rest
+ ;
+
+ECPGKeywords_vanames: SQL_BREAK
+ | SQL_CARDINALITY
+ | SQL_COUNT
+ | SQL_DATETIME_INTERVAL_CODE
+ | SQL_DATETIME_INTERVAL_PRECISION
+ | SQL_FOUND
+ | SQL_GO
+ | SQL_GOTO
+ | SQL_IDENTIFIED
+ | SQL_INDICATOR
+ | SQL_KEY_MEMBER
+ | SQL_LENGTH
+ | SQL_NULLABLE
+ | SQL_OCTET_LENGTH
+ | SQL_RETURNED_LENGTH
+ | SQL_RETURNED_OCTET_LENGTH
+ | SQL_SCALE
+ | SQL_SECTION
+ | SQL_SQLERROR
+ | SQL_SQLPRINT
+ | SQL_SQLWARNING
+ | SQL_STOP
+ ;
+
+ECPGKeywords_rest: SQL_CONNECT
+ | SQL_DESCRIBE
+ | SQL_DISCONNECT
+ | SQL_OPEN
+ | SQL_VAR
+ | SQL_WHENEVER
;
/* additional keywords that can be SQL type names (but not ECPGColLabels) */
-ECPGTypeName: SQL_BOOL { $$ = mm_strdup("bool"); }
- | SQL_LONG { $$ = mm_strdup("long"); }
- | SQL_OUTPUT { $$ = mm_strdup("output"); }
- | SQL_SHORT { $$ = mm_strdup("short"); }
- | SQL_STRUCT { $$ = mm_strdup("struct"); }
- | SQL_SIGNED { $$ = mm_strdup("signed"); }
- | SQL_UNSIGNED { $$ = mm_strdup("unsigned"); }
+ECPGTypeName: SQL_BOOL
+ | SQL_LONG
+ | SQL_OUTPUT
+ | SQL_SHORT
+ | SQL_STRUCT
+ | SQL_SIGNED
+ | SQL_UNSIGNED
;
-symbol: ColLabel { $$ = $1; }
+symbol: ColLabel
;
-ECPGColId: ecpg_ident { $$ = $1; }
- | unreserved_keyword { $$ = $1; }
- | col_name_keyword { $$ = $1; }
- | ECPGunreserved_interval { $$ = $1; }
- | ECPGKeywords { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
- | CHAR_P { $$ = mm_strdup("char"); }
- | VALUES { $$ = mm_strdup("values"); }
+ECPGColId: ecpg_ident
+ | unreserved_keyword
+ | col_name_keyword
+ | ECPGunreserved_interval
+ | ECPGKeywords
+ | ECPGCKeywords
+ | CHAR_P
+ | VALUES
;
/*
@@ -2018,58 +1825,58 @@ ECPGColId: ecpg_ident { $$ = $1; }
/* Column identifier --- names that can be column, table, etc names.
*/
-ColId: ecpg_ident { $$ = $1; }
- | all_unreserved_keyword { $$ = $1; }
- | col_name_keyword { $$ = $1; }
- | ECPGKeywords { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
- | CHAR_P { $$ = mm_strdup("char"); }
- | VALUES { $$ = mm_strdup("values"); }
+ColId: ecpg_ident
+ | all_unreserved_keyword
+ | col_name_keyword
+ | ECPGKeywords
+ | ECPGCKeywords
+ | CHAR_P
+ | VALUES
;
/* Type/function identifier --- names that can be type or function names.
*/
-type_function_name: ecpg_ident { $$ = $1; }
- | all_unreserved_keyword { $$ = $1; }
- | type_func_name_keyword { $$ = $1; }
- | ECPGKeywords { $$ = $1; }
- | ECPGCKeywords { $$ = $1; }
- | ECPGTypeName { $$ = $1; }
+type_function_name: ecpg_ident
+ | all_unreserved_keyword
+ | type_func_name_keyword
+ | ECPGKeywords
+ | ECPGCKeywords
+ | ECPGTypeName
;
/* Column label --- allowed labels in "AS" clauses.
* This presently includes *all* Postgres keywords.
*/
-ColLabel: ECPGColLabel { $$ = $1; }
- | ECPGTypeName { $$ = $1; }
- | CHAR_P { $$ = mm_strdup("char"); }
- | CURRENT_P { $$ = mm_strdup("current"); }
- | INPUT_P { $$ = mm_strdup("input"); }
- | INT_P { $$ = mm_strdup("int"); }
- | TO { $$ = mm_strdup("to"); }
- | UNION { $$ = mm_strdup("union"); }
- | VALUES { $$ = mm_strdup("values"); }
- | ECPGCKeywords { $$ = $1; }
- | ECPGunreserved_interval { $$ = $1; }
- ;
-
-ECPGColLabel: ecpg_ident { $$ = $1; }
- | unreserved_keyword { $$ = $1; }
- | col_name_keyword { $$ = $1; }
- | type_func_name_keyword { $$ = $1; }
- | reserved_keyword { $$ = $1; }
- | ECPGKeywords_vanames { $$ = $1; }
- | ECPGKeywords_rest { $$ = $1; }
- | CONNECTION { $$ = mm_strdup("connection"); }
- ;
-
-ECPGCKeywords: S_AUTO { $$ = mm_strdup("auto"); }
- | S_CONST { $$ = mm_strdup("const"); }
- | S_EXTERN { $$ = mm_strdup("extern"); }
- | S_REGISTER { $$ = mm_strdup("register"); }
- | S_STATIC { $$ = mm_strdup("static"); }
- | S_TYPEDEF { $$ = mm_strdup("typedef"); }
- | S_VOLATILE { $$ = mm_strdup("volatile"); }
+ColLabel: ECPGColLabel
+ | ECPGTypeName
+ | CHAR_P
+ | CURRENT_P
+ | INPUT_P
+ | INT_P
+ | TO
+ | UNION
+ | VALUES
+ | ECPGCKeywords
+ | ECPGunreserved_interval
+ ;
+
+ECPGColLabel: ecpg_ident
+ | unreserved_keyword
+ | col_name_keyword
+ | type_func_name_keyword
+ | reserved_keyword
+ | ECPGKeywords_vanames
+ | ECPGKeywords_rest
+ | CONNECTION
+ ;
+
+ECPGCKeywords: S_AUTO
+ | S_CONST
+ | S_EXTERN
+ | S_REGISTER
+ | S_STATIC
+ | S_TYPEDEF
+ | S_VOLATILE
;
/* "Unreserved" keywords --- available for use as any kind of name.
@@ -2086,17 +1893,17 @@ ECPGCKeywords: S_AUTO { $$ = mm_strdup("auto"); }
*
* The mentioned exclusions are done by $replace_line settings in parse.pl.
*/
-all_unreserved_keyword: unreserved_keyword { $$ = $1; }
- | ECPGunreserved_interval { $$ = $1; }
- | CONNECTION { $$ = mm_strdup("connection"); }
+all_unreserved_keyword: unreserved_keyword
+ | ECPGunreserved_interval
+ | CONNECTION
;
-ECPGunreserved_interval: DAY_P { $$ = mm_strdup("day"); }
- | HOUR_P { $$ = mm_strdup("hour"); }
- | MINUTE_P { $$ = mm_strdup("minute"); }
- | MONTH_P { $$ = mm_strdup("month"); }
- | SECOND_P { $$ = mm_strdup("second"); }
- | YEAR_P { $$ = mm_strdup("year"); }
+ECPGunreserved_interval: DAY_P
+ | HOUR_P
+ | MINUTE_P
+ | MONTH_P
+ | SECOND_P
+ | YEAR_P
;
into_list: coutputvariable | into_list ',' coutputvariable
@@ -2106,73 +1913,66 @@ ecpgstart: SQL_START
{
reset_variables();
pacounter = 1;
+ @$ = EMPTY;
}
;
c_args: /* EMPTY */
- {
- $$ = EMPTY;
- }
| c_list
- {
- $$ = $1;
- }
;
coutputvariable: cvariable indicator
{
- add_variable_to_head(&argsresult, find_variable($1), find_variable($2));
+ add_variable_to_head(&argsresult, find_variable(@1), find_variable(@2));
}
| cvariable
{
- add_variable_to_head(&argsresult, find_variable($1), &no_indicator);
+ add_variable_to_head(&argsresult, find_variable(@1), &no_indicator);
}
;
civarind: cvariable indicator
{
- if (find_variable($2)->type->type == ECPGt_array)
+ if (find_variable(@2)->type->type == ECPGt_array)
mmerror(PARSE_ERROR, ET_ERROR, "arrays of indicators are not allowed on input");
- add_variable_to_head(&argsinsert, find_variable($1), find_variable($2));
- $$ = create_questionmarks($1, false);
+ add_variable_to_head(&argsinsert, find_variable(@1), find_variable(@2));
+ @$ = create_questionmarks(@1, false);
}
;
char_civar: char_variable
{
- char *ptr = strstr($1, ".arr");
+ char *ptr = strstr(@1, ".arr");
if (ptr) /* varchar, we need the struct name here, not
* the struct element */
*ptr = '\0';
- add_variable_to_head(&argsinsert, find_variable($1), &no_indicator);
- $$ = $1;
+ add_variable_to_head(&argsinsert, find_variable(@1), &no_indicator);
}
;
civar: cvariable
{
- add_variable_to_head(&argsinsert, find_variable($1), &no_indicator);
- $$ = create_questionmarks($1, false);
+ add_variable_to_head(&argsinsert, find_variable(@1), &no_indicator);
+ @$ = create_questionmarks(@1, false);
}
;
indicator: cvariable
{
- check_indicator((find_variable($1))->type);
- $$ = $1;
+ check_indicator((find_variable(@1))->type);
}
| SQL_INDICATOR cvariable
{
- check_indicator((find_variable($2))->type);
- $$ = $2;
+ check_indicator((find_variable(@2))->type);
+ @$ = @2;
}
| SQL_INDICATOR name
{
- check_indicator((find_variable($2))->type);
- $$ = $2;
+ check_indicator((find_variable(@2))->type);
+ @$ = @2;
}
;
@@ -2182,7 +1982,7 @@ cvariable: CVARIABLE
* As long as multidimensional arrays are not implemented we have to
* check for those here
*/
- char *ptr = $1;
+ char *ptr = @1;
int brace_open = 0,
brace = false;
@@ -2209,57 +2009,44 @@ cvariable: CVARIABLE
break;
}
}
- $$ = $1;
}
;
ecpg_param: PARAM
{
- $$ = make_name();
+ @$ = make_name();
}
;
ecpg_bconst: BCONST
- {
- $$ = $1;
- }
;
ecpg_fconst: FCONST
{
- $$ = make_name();
+ @$ = make_name();
}
;
ecpg_sconst: SCONST
- {
- $$ = $1;
- }
;
ecpg_xconst: XCONST
- {
- $$ = $1;
- }
;
ecpg_ident: IDENT
- {
- $$ = $1;
- }
| CSTRING
{
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
;
quoted_ident_stringvar: name
{
- $$ = make3_str(mm_strdup("\""), $1, mm_strdup("\""));
+ @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
}
| char_variable
{
- $$ = make3_str(mm_strdup("("), $1, mm_strdup(")"));
+ @$ = make3_str(mm_strdup("("), @1, mm_strdup(")"));
}
;
@@ -2268,221 +2055,151 @@ quoted_ident_stringvar: name
*/
c_stuff_item: c_anything
- {
- $$ = $1;
- }
| '(' ')'
{
- $$ = mm_strdup("()");
+ @$ = mm_strdup("()");
}
| '(' c_stuff ')'
- {
- $$ = cat_str(3, mm_strdup("("), $2, mm_strdup(")"));
- }
;
c_stuff: c_stuff_item
- {
- $$ = $1;
- }
| c_stuff c_stuff_item
- {
- $$ = cat2_str($1, $2);
- }
;
c_list: c_term
- {
- $$ = $1;
- }
| c_list ',' c_term
- {
- $$ = cat_str(3, $1, mm_strdup(","), $3);
- }
;
c_term: c_stuff
- {
- $$ = $1;
- }
| '{' c_list '}'
- {
- $$ = cat_str(3, mm_strdup("{"), $2, mm_strdup("}"));
- }
- ;
-
-c_thing: c_anything { $$ = $1; }
- | '(' { $$ = mm_strdup("("); }
- | ')' { $$ = mm_strdup(")"); }
- | ',' { $$ = mm_strdup(","); }
- | ';' { $$ = mm_strdup(";"); }
- ;
-
-c_anything: ecpg_ident { $$ = $1; }
- | Iconst { $$ = $1; }
- | ecpg_fconst { $$ = $1; }
- | ecpg_sconst { $$ = $1; }
- | '*' { $$ = mm_strdup("*"); }
- | '+' { $$ = mm_strdup("+"); }
- | '-' { $$ = mm_strdup("-"); }
- | '/' { $$ = mm_strdup("/"); }
- | '%' { $$ = mm_strdup("%"); }
- | NULL_P { $$ = mm_strdup("NULL"); }
- | S_ADD { $$ = mm_strdup("+="); }
- | S_AND { $$ = mm_strdup("&&"); }
- | S_ANYTHING { $$ = make_name(); }
- | S_AUTO { $$ = mm_strdup("auto"); }
- | S_CONST { $$ = mm_strdup("const"); }
- | S_DEC { $$ = mm_strdup("--"); }
- | S_DIV { $$ = mm_strdup("/="); }
- | S_DOTPOINT { $$ = mm_strdup(".*"); }
- | S_EQUAL { $$ = mm_strdup("=="); }
- | S_EXTERN { $$ = mm_strdup("extern"); }
- | S_INC { $$ = mm_strdup("++"); }
- | S_LSHIFT { $$ = mm_strdup("<<"); }
- | S_MEMBER { $$ = mm_strdup("->"); }
- | S_MEMPOINT { $$ = mm_strdup("->*"); }
- | S_MOD { $$ = mm_strdup("%="); }
- | S_MUL { $$ = mm_strdup("*="); }
- | S_NEQUAL { $$ = mm_strdup("!="); }
- | S_OR { $$ = mm_strdup("||"); }
- | S_REGISTER { $$ = mm_strdup("register"); }
- | S_RSHIFT { $$ = mm_strdup(">>"); }
- | S_STATIC { $$ = mm_strdup("static"); }
- | S_SUB { $$ = mm_strdup("-="); }
- | S_TYPEDEF { $$ = mm_strdup("typedef"); }
- | S_VOLATILE { $$ = mm_strdup("volatile"); }
- | SQL_BOOL { $$ = mm_strdup("bool"); }
- | ENUM_P { $$ = mm_strdup("enum"); }
- | HOUR_P { $$ = mm_strdup("hour"); }
- | INT_P { $$ = mm_strdup("int"); }
- | SQL_LONG { $$ = mm_strdup("long"); }
- | MINUTE_P { $$ = mm_strdup("minute"); }
- | MONTH_P { $$ = mm_strdup("month"); }
- | SECOND_P { $$ = mm_strdup("second"); }
- | SQL_SHORT { $$ = mm_strdup("short"); }
- | SQL_SIGNED { $$ = mm_strdup("signed"); }
- | SQL_STRUCT { $$ = mm_strdup("struct"); }
- | SQL_UNSIGNED { $$ = mm_strdup("unsigned"); }
- | YEAR_P { $$ = mm_strdup("year"); }
- | CHAR_P { $$ = mm_strdup("char"); }
- | FLOAT_P { $$ = mm_strdup("float"); }
- | TO { $$ = mm_strdup("to"); }
- | UNION { $$ = mm_strdup("union"); }
- | VARCHAR { $$ = mm_strdup("varchar"); }
- | '[' { $$ = mm_strdup("["); }
- | ']' { $$ = mm_strdup("]"); }
- | '=' { $$ = mm_strdup("="); }
- | ':' { $$ = mm_strdup(":"); }
+ ;
+
+c_thing: c_anything
+ | '('
+ | ')'
+ | ','
+ | ';'
+ ;
+
+/*
+ * Note: NULL_P is treated specially to force it to be output in upper case,
+ * since it's likely meant as a reference to the standard C macro NULL.
+ */
+c_anything: ecpg_ident
+ | Iconst
+ | ecpg_fconst
+ | ecpg_sconst
+ | '*'
+ | '+'
+ | '-'
+ | '/'
+ | '%'
+ | NULL_P { @$ = mm_strdup("NULL"); }
+ | S_ADD
+ | S_AND
+ | S_ANYTHING
+ | S_AUTO
+ | S_CONST
+ | S_DEC
+ | S_DIV
+ | S_DOTPOINT
+ | S_EQUAL
+ | S_EXTERN
+ | S_INC
+ | S_LSHIFT
+ | S_MEMBER
+ | S_MEMPOINT
+ | S_MOD
+ | S_MUL
+ | S_NEQUAL
+ | S_OR
+ | S_REGISTER
+ | S_RSHIFT
+ | S_STATIC
+ | S_SUB
+ | S_TYPEDEF
+ | S_VOLATILE
+ | SQL_BOOL
+ | ENUM_P
+ | HOUR_P
+ | INT_P
+ | SQL_LONG
+ | MINUTE_P
+ | MONTH_P
+ | SECOND_P
+ | SQL_SHORT
+ | SQL_SIGNED
+ | SQL_STRUCT
+ | SQL_UNSIGNED
+ | YEAR_P
+ | CHAR_P
+ | FLOAT_P
+ | TO
+ | UNION
+ | VARCHAR
+ | '['
+ | ']'
+ | '='
+ | ':'
;
DeallocateStmt: DEALLOCATE prepared_name
{
- check_declared_list($2);
- $$ = $2;
+ check_declared_list(@2);
+ @$ = @2;
}
| DEALLOCATE PREPARE prepared_name
{
- check_declared_list($3);
- $$ = $3;
+ check_declared_list(@3);
+ @$ = @3;
}
| DEALLOCATE ALL
{
- $$ = mm_strdup("all");
+ @$ = mm_strdup("all");
}
| DEALLOCATE PREPARE ALL
{
- $$ = mm_strdup("all");
+ @$ = mm_strdup("all");
}
;
Iresult: Iconst
- {
- $$ = $1;
- }
| '(' Iresult ')'
- {
- $$ = cat_str(3, mm_strdup("("), $2, mm_strdup(")"));
- }
| Iresult '+' Iresult
- {
- $$ = cat_str(3, $1, mm_strdup("+"), $3);
- }
| Iresult '-' Iresult
- {
- $$ = cat_str(3, $1, mm_strdup("-"), $3);
- }
| Iresult '*' Iresult
- {
- $$ = cat_str(3, $1, mm_strdup("*"), $3);
- }
| Iresult '/' Iresult
- {
- $$ = cat_str(3, $1, mm_strdup("/"), $3);
- }
| Iresult '%' Iresult
- {
- $$ = cat_str(3, $1, mm_strdup("%"), $3);
- }
| ecpg_sconst
- {
- $$ = $1;
- }
| ColId
- {
- $$ = $1;
- }
| ColId '(' var_type ')'
{
- if (pg_strcasecmp($1, "sizeof") != 0)
+ if (pg_strcasecmp(@1, "sizeof") != 0)
mmerror(PARSE_ERROR, ET_ERROR, "operator not allowed in variable definition");
else
- $$ = cat_str(4, $1, mm_strdup("("), $3.type_str, mm_strdup(")"));
+ @$ = cat_str(4, @1, mm_strdup("("), $3.type_str, mm_strdup(")"));
}
;
execute_rest: /* EMPTY */
- {
- $$ = EMPTY;
- }
| ecpg_using opt_ecpg_into
- {
- $$ = EMPTY;
- }
| ecpg_into ecpg_using
- {
- $$ = EMPTY;
- }
| ecpg_into
- {
- $$ = EMPTY;
- }
;
ecpg_into: INTO into_list
{
- $$ = EMPTY;
+ /* always suppress this from the constructed string */
+ @$ = EMPTY;
}
| into_descriptor
- {
- $$ = $1;
- }
;
opt_ecpg_into: /* EMPTY */
- {
- $$ = EMPTY;
- }
| ecpg_into
- {
- $$ = $1;
- }
;
ecpg_fetch_into: ecpg_into
- {
- $$ = $1;
- }
| using_descriptor
{
struct variable *var;
@@ -2490,18 +2207,11 @@ ecpg_fetch_into: ecpg_into
var = argsinsert->variable;
remove_variable_from_list(&argsinsert, var);
add_variable_to_head(&argsresult, var, &no_indicator);
- $$ = $1;
}
;
opt_ecpg_fetch_into: /* EMPTY */
- {
- $$ = EMPTY;
- }
| ecpg_fetch_into
- {
- $$ = $1;
- }
;
%%
diff --git a/src/interfaces/ecpg/preproc/ecpg.type b/src/interfaces/ecpg/preproc/ecpg.type
index 4fe80a5a4b..2929f358ff 100644
--- a/src/interfaces/ecpg/preproc/ecpg.type
+++ b/src/interfaces/ecpg/preproc/ecpg.type
@@ -1,131 +1,4 @@
/* src/interfaces/ecpg/preproc/ecpg.type */
-%type <str> ECPGAllocateDescr
-%type <str> ECPGCKeywords
-%type <str> ECPGColId
-%type <str> ECPGColLabel
-%type <str> ECPGConnect
-%type <str> ECPGCursorStmt
-%type <str> ECPGDeallocateDescr
-%type <str> ECPGDeclaration
-%type <str> ECPGDeclare
-%type <str> ECPGDeclareStmt
-%type <str> ECPGDisconnect
-%type <str> ECPGExecuteImmediateStmt
-%type <str> ECPGFree
-%type <str> ECPGGetDescHeaderItem
-%type <str> ECPGGetDescItem
-%type <str> ECPGGetDescriptorHeader
-%type <str> ECPGKeywords
-%type <str> ECPGKeywords_rest
-%type <str> ECPGKeywords_vanames
-%type <str> ECPGOpen
-%type <str> ECPGSetAutocommit
-%type <str> ECPGSetConnection
-%type <str> ECPGSetDescHeaderItem
-%type <str> ECPGSetDescItem
-%type <str> ECPGSetDescriptorHeader
-%type <str> ECPGTypeName
-%type <str> ECPGTypedef
-%type <str> ECPGVar
-%type <str> ECPGVarDeclaration
-%type <str> ECPGWhenever
-%type <str> ECPGunreserved_interval
-%type <str> UsingConst
-%type <str> UsingValue
-%type <str> all_unreserved_keyword
-%type <str> c_anything
-%type <str> c_args
-%type <str> c_list
-%type <str> c_stuff
-%type <str> c_stuff_item
-%type <str> c_term
-%type <str> c_thing
-%type <str> char_variable
-%type <str> char_civar
-%type <str> civar
-%type <str> civarind
-%type <str> ColId
-%type <str> ColLabel
-%type <str> connect_options
-%type <str> connection_object
-%type <str> connection_target
-%type <str> coutputvariable
-%type <str> cvariable
-%type <str> db_prefix
-%type <str> CreateAsStmt
-%type <str> DeallocateStmt
-%type <str> dis_name
-%type <str> ecpg_bconst
-%type <str> ecpg_fconst
-%type <str> ecpg_ident
-%type <str> ecpg_interval
-%type <str> ecpg_into
-%type <str> ecpg_fetch_into
-%type <str> ecpg_param
-%type <str> ecpg_sconst
-%type <str> ecpg_using
-%type <str> ecpg_xconst
-%type <str> enum_definition
-%type <str> enum_type
-%type <str> execstring
-%type <str> execute_rest
-%type <str> indicator
-%type <str> into_descriptor
-%type <str> into_sqlda
-%type <str> Iresult
-%type <str> on_off
-%type <str> opt_bit_field
-%type <str> opt_connection_name
-%type <str> opt_database_name
-%type <str> opt_ecpg_into
-%type <str> opt_ecpg_fetch_into
-%type <str> opt_ecpg_using
-%type <str> opt_initializer
-%type <str> opt_options
-%type <str> opt_output
-%type <str> opt_pointer
-%type <str> opt_port
-%type <str> opt_reference
-%type <str> opt_scale
-%type <str> opt_server
-%type <str> opt_user
-%type <str> opt_opt_value
-%type <str> ora_user
-%type <str> precision
-%type <str> prepared_name
-%type <str> quoted_ident_stringvar
-%type <str> s_struct_union
-%type <str> server
-%type <str> server_name
-%type <str> single_vt_declaration
-%type <str> storage_clause
-%type <str> storage_declaration
-%type <str> storage_modifier
-%type <str> struct_union_type
-%type <str> struct_union_type_with_symbol
-%type <str> symbol
-%type <str> type_declaration
-%type <str> type_function_name
-%type <str> user_name
-%type <str> using_descriptor
-%type <str> var_declaration
-%type <str> var_type_declarations
-%type <str> variable
-%type <str> variable_declarations
-%type <str> variable_list
-%type <str> vt_declarations
-
-%type <str> Op
-%type <str> IntConstVar
-%type <str> AllConstVar
-%type <str> CSTRING
-%type <str> CPP_LINE
-%type <str> CVARIABLE
-%type <str> BCONST
-%type <str> SCONST
-%type <str> XCONST
-%type <str> IDENT
-
%type <struct_union> s_struct_union_symbol
%type <descriptor> ECPGGetDescriptor
diff --git a/src/interfaces/ecpg/preproc/output.c b/src/interfaces/ecpg/preproc/output.c
index 6c0b8a27b1..8d2b6e7cb8 100644
--- a/src/interfaces/ecpg/preproc/output.c
+++ b/src/interfaces/ecpg/preproc/output.c
@@ -4,7 +4,7 @@
#include "preproc_extern.h"
-static void output_escaped_str(char *str, bool quoted);
+static void output_escaped_str(const char *str, bool quoted);
void
output_line_number(void)
@@ -16,13 +16,12 @@ output_line_number(void)
}
void
-output_simple_statement(char *stmt, int whenever_mode)
+output_simple_statement(const char *stmt, int whenever_mode)
{
output_escaped_str(stmt, false);
if (whenever_mode)
whenever_action(whenever_mode);
output_line_number();
- free(stmt);
}
@@ -133,7 +132,7 @@ static char *ecpg_statement_type_name[] = {
};
void
-output_statement(char *stmt, int whenever_mode, enum ECPG_statement_type st)
+output_statement(const char *stmt, int whenever_mode, enum ECPG_statement_type st)
{
fprintf(base_yyout, "{ ECPGdo(__LINE__, %d, %d, %s, %d, ", compat, force_indicator, connection ? connection : "NULL", questionmarks);
@@ -163,11 +162,10 @@ output_statement(char *stmt, int whenever_mode, enum ECPG_statement_type st)
reset_variables();
whenever_action(whenever_mode | 2);
- free(stmt);
}
void
-output_prepare_statement(char *name, char *stmt)
+output_prepare_statement(const char *name, const char *stmt)
{
fprintf(base_yyout, "{ ECPGprepare(__LINE__, %s, %d, ", connection ? connection : "NULL", questionmarks);
output_escaped_str(name, true);
@@ -175,11 +173,10 @@ output_prepare_statement(char *name, char *stmt)
output_escaped_str(stmt, true);
fputs(");", base_yyout);
whenever_action(2);
- free(name);
}
void
-output_deallocate_prepare_statement(char *name)
+output_deallocate_prepare_statement(const char *name)
{
const char *con = connection ? connection : "NULL";
@@ -193,11 +190,10 @@ output_deallocate_prepare_statement(char *name)
fprintf(base_yyout, "{ ECPGdeallocate_all(__LINE__, %d, %s);", compat, con);
whenever_action(2);
- free(name);
}
static void
-output_escaped_str(char *str, bool quoted)
+output_escaped_str(const char *str, bool quoted)
{
int i = 0;
int len = strlen(str);
diff --git a/src/interfaces/ecpg/preproc/parse.pl b/src/interfaces/ecpg/preproc/parse.pl
index 5a00271468..98d44d4bf2 100644
--- a/src/interfaces/ecpg/preproc/parse.pl
+++ b/src/interfaces/ecpg/preproc/parse.pl
@@ -44,27 +44,10 @@ my %replace_token = (
'IDENT' => 'ecpg_ident',
'PARAM' => 'ecpg_param',);
-# Substitutions to apply to terminal token names to reconstruct the
-# literal form of the token. (There is also a hard-wired substitution
-# rule that strips trailing '_P'.)
-my %replace_string = (
- 'FORMAT_LA' => 'format',
- 'NOT_LA' => 'not',
- 'NULLS_LA' => 'nulls',
- 'WITH_LA' => 'with',
- 'WITHOUT_LA' => 'without',
- 'TYPECAST' => '::',
- 'DOT_DOT' => '..',
- 'COLON_EQUALS' => ':=',
- 'EQUALS_GREATER' => '=>',
- 'LESS_EQUALS' => '<=',
- 'GREATER_EQUALS' => '>=',
- 'NOT_EQUALS' => '<>',);
-
-# This hash can provide a result type to override '<str>' for nonterminals
+# This hash can provide a result type to override "void" for nonterminals
# that need that, or it can specify 'ignore' to cause us to skip the rule
-# for that nonterminal. (In that case, ecpg.trailer had better provide
-# a substitute rule.)
+# for that nonterminal. (In either case, ecpg.trailer had better provide
+# a substitute rule, since the default won't do.)
my %replace_types = (
'PrepareStmt' => '<prep>',
'ExecuteStmt' => '<exec>',
@@ -175,11 +158,8 @@ my $non_term_id;
# we plan to emit for the current rule.
my $line = '';
-# @fields holds the items to be emitted in the token-concatenation action
-# for the current rule (assuming we emit one). "$N" refers to the N'th
-# input token of the rule; anything else is a string to emit literally.
-# (We assume no such string can need to start with '$'.)
-my @fields;
+# count of tokens included in $line.
+my $line_count = 0;
# Open parser / output file early, to raise errors early.
@@ -244,10 +224,6 @@ sub main
$has_if_command = 1 if /^\s*if/;
}
- # We track %prec per-line, not per-rule, which is not quite right
- # but there are no counterexamples in gram.y at present.
- my $prec = 0;
-
# Make sure any braces are split into separate fields
s/{/ { /g;
s/}/ } /g;
@@ -296,7 +272,7 @@ sub main
}
# If it's "<something>", it's a type in a %token declaration,
- # which we can just drop.
+ # which we should just drop so that the tokens have void type.
if (substr($a, 0, 1) eq '<')
{
next;
@@ -376,7 +352,7 @@ sub main
if ($copymode)
{
# Print the accumulated rule.
- emit_rule(\@fields);
+ emit_rule();
add_to_buffer('rules', ";\n\n");
}
else
@@ -386,8 +362,8 @@ sub main
}
# Reset for the next rule.
- @fields = ();
$line = '';
+ $line_count = 0;
$in_rule = 0;
$alt_count = 0;
$has_feature_not_supported = 0;
@@ -401,11 +377,10 @@ sub main
{
# Print the accumulated alternative.
# Increment $alt_count for each non-ignored alternative.
- $alt_count += emit_rule(\@fields);
+ $alt_count += emit_rule();
}
# Reset for the next alternative.
- @fields = ();
# Start the next line with '|' if we've printed at least one
# alternative.
if ($alt_count > 1)
@@ -416,6 +391,7 @@ sub main
{
$line = '';
}
+ $line_count = 0;
$has_feature_not_supported = 0;
$has_if_command = 0;
next;
@@ -444,13 +420,9 @@ sub main
$fieldIndexer++;
}
- # Check for %replace_types override of nonterminal's type
- if (not defined $replace_types{$non_term_id})
- {
- # By default, the type is <str>
- $replace_types{$non_term_id} = '<str>';
- }
- elsif ($replace_types{$non_term_id} eq 'ignore')
+ # Check for %replace_types entry indicating to ignore it.
+ if (defined $replace_types{$non_term_id}
+ && $replace_types{$non_term_id} eq 'ignore')
{
# We'll ignore this nonterminal and rule altogether.
$copymode = 0;
@@ -470,22 +442,26 @@ sub main
$stmt_mode = 0;
}
- # Emit appropriate %type declaration for this nonterminal.
- my $tstr =
- '%type '
- . $replace_types{$non_term_id} . ' '
- . $non_term_id;
- add_to_buffer('types', $tstr);
+ # Emit appropriate %type declaration for this nonterminal,
+ # if it has a type; otherwise omit that.
+ if (defined $replace_types{$non_term_id})
+ {
+ my $tstr =
+ '%type '
+ . $replace_types{$non_term_id} . ' '
+ . $non_term_id;
+ add_to_buffer('types', $tstr);
+ }
# Emit the target part of the rule.
# Note: the leading space is just to match
# the old, rather weird output logic.
- $tstr = ' ' . $non_term_id . ':';
+ my $tstr = ' ' . $non_term_id . ':';
add_to_buffer('rules', $tstr);
- # Prepare for reading the fields (tokens) of the rule.
+ # Prepare for reading the tokens of the rule.
$line = '';
- @fields = ();
+ $line_count = 0;
die "unterminated rule at grammar line $.\n"
if $in_rule;
$in_rule = 1;
@@ -496,48 +472,7 @@ sub main
{
# Not a nonterminal declaration, so just add it to $line.
$line = $line . ' ' . $arr[$fieldIndexer];
- }
-
- # %prec and whatever follows it should get added to $line,
- # but not to @fields.
- if ($arr[$fieldIndexer] eq '%prec')
- {
- $prec = 1;
- next;
- }
-
- # Emit transformed version of token to @fields if appropriate.
- if ( $copymode
- && !$prec
- && !$comment
- && $in_rule)
- {
- my $S = $arr[$fieldIndexer];
-
- # If it's a known terminal token (other than Op) or a literal
- # character, we need to emit the equivalent string, which'll
- # later get wrapped into a C string literal, perhaps after
- # merging with adjacent strings.
- if ($S ne 'Op'
- && (defined $tokens{$S}
- || $S =~ /^'.+'$/))
- {
- # Apply replace_string substitution if any.
- $S = $replace_string{$S} if (exists $replace_string{$S});
- # Automatically strip _P if present.
- $S =~ s/_P$//;
- # And get rid of quotes if it's a literal character.
- $S =~ tr/'//d;
- # Finally, downcase and push into @fields.
- push(@fields, lc($S));
- }
- else
- {
- # Otherwise, push a $N reference to this input token.
- # (We assume this cannot be confused with anything the
- # above code would produce.)
- push(@fields, '$' . (scalar(@fields) + 1));
- }
+ $line_count++;
}
}
}
@@ -568,13 +503,13 @@ sub include_file
# by an ecpg.addons entry.
sub emit_rule_action
{
- my ($tag, $fields) = @_;
+ my ($tag) = @_;
# See if we have an addons entry; if not, just emit default action
my $rec = $addons{$tag};
if (!$rec)
{
- emit_default_action($fields, 0);
+ emit_default_action(0);
return;
}
@@ -585,7 +520,7 @@ sub emit_rule_action
if ($rectype eq 'rule')
{
# Emit default action and then the code block.
- emit_default_action($fields, 0);
+ emit_default_action(0);
}
elsif ($rectype eq 'addon')
{
@@ -600,7 +535,7 @@ sub emit_rule_action
if ($rectype eq 'addon')
{
- emit_default_action($fields, 1);
+ emit_default_action(1);
}
return;
}
@@ -626,12 +561,11 @@ sub dump_buffer
}
# Emit the default action (usually token concatenation) for the current rule.
-# Pass: fields array, brace_printed boolean
+# Pass: brace_printed boolean
# brace_printed should be true if caller already printed action's open brace.
sub emit_default_action
{
- my ($flds, $brace_printed) = @_;
- my $len = scalar(@$flds);
+ my ($brace_printed) = @_;
if ($stmt_mode == 0)
{
@@ -651,91 +585,21 @@ sub emit_default_action
);
}
- if ($len == 0)
- {
- # Empty rule
- if (!$brace_printed)
- {
- add_to_buffer('rules', ' { ');
- $brace_printed = 1;
- }
- add_to_buffer('rules', ' $$=EMPTY; }');
- }
- else
- {
- # Go through each field and aggregate consecutive literal tokens
- # into a single 'mm_strdup' call.
- my @flds_new;
- my $str;
- for (my $z = 0; $z < $len; $z++)
- {
- if (substr($flds->[$z], 0, 1) eq '$')
- {
- push(@flds_new, $flds->[$z]);
- next;
- }
-
- $str = $flds->[$z];
-
- while (1)
- {
- if ($z >= $len - 1
- || substr($flds->[ $z + 1 ], 0, 1) eq '$')
- {
- # Can't combine any more literals; push to @flds_new.
- # This code would need work if any literals contain
- # backslash or double quote, but right now that never
- # happens.
- push(@flds_new, "mm_strdup(\"$str\")");
- last;
- }
- $z++;
- $str = $str . ' ' . $flds->[$z];
- }
- }
-
- # So - how many fields did we end up with ?
- $len = scalar(@flds_new);
- if ($len == 1)
- {
- # Single field can be handled by straight assignment
- if (!$brace_printed)
- {
- add_to_buffer('rules', ' { ');
- $brace_printed = 1;
- }
- $str = ' $$ = ' . $flds_new[0] . ';';
- add_to_buffer('rules', $str);
- }
- else
- {
- # Need to concatenate the results to form our final string
- if (!$brace_printed)
- {
- add_to_buffer('rules', ' { ');
- $brace_printed = 1;
- }
- $str =
- ' $$ = cat_str(' . $len . ',' . join(',', @flds_new) . ');';
- add_to_buffer('rules', $str);
- }
- add_to_buffer('rules', '}') if ($brace_printed);
- }
+ add_to_buffer('rules', '}') if ($brace_printed);
}
else
{
# We're in the "stmt:" rule, where we need to output special actions.
# This code assumes that no ecpg.addons entry applies.
- if ($len)
+ if ($line_count)
{
# Any regular kind of statement calls output_statement
add_to_buffer('rules',
- ' { output_statement($1, 0, ECPGst_normal); }');
+ ' { output_statement(@1, 0, ECPGst_normal); }');
}
else
{
# The empty production for stmt: do nothing
- add_to_buffer('rules', ' { $$ = NULL; }');
}
}
return;
@@ -746,8 +610,6 @@ sub emit_default_action
# entry in %replace_line, then do nothing and return 0.
sub emit_rule
{
- my ($fields) = @_;
-
# compute tag to be used as lookup key in %replace_line and %addons
my $tag = $non_term_id . $line;
$tag =~ tr/ |//d;
@@ -761,7 +623,8 @@ sub emit_rule
return 0;
}
- # non-ignore entries replace the line, but we'd better keep any '|'
+ # non-ignore entries replace the line, but we'd better keep any '|';
+ # we don't bother to update $line_count here.
if (index($line, '|') != -1)
{
$line = '| ' . $rep;
@@ -778,7 +641,7 @@ sub emit_rule
# Emit $line, then print the appropriate action.
add_to_buffer('rules', $line);
- emit_rule_action($tag, $fields);
+ emit_rule_action($tag);
return 1;
}
diff --git a/src/interfaces/ecpg/preproc/parser.c b/src/interfaces/ecpg/preproc/parser.c
index 9daeee3303..8807c22cb6 100644
--- a/src/interfaces/ecpg/preproc/parser.c
+++ b/src/interfaces/ecpg/preproc/parser.c
@@ -31,6 +31,7 @@ static YYSTYPE lookahead_yylval; /* yylval for lookahead token */
static YYLTYPE lookahead_yylloc; /* yylloc for lookahead token */
static char *lookahead_yytext; /* start current token */
+static int base_yylex_location(void);
static bool check_uescapechar(unsigned char escape);
static bool ecpg_isspace(char ch);
@@ -71,7 +72,7 @@ filtered_base_yylex(void)
have_lookahead = false;
}
else
- cur_token = base_yylex();
+ cur_token = base_yylex_location();
/*
* If this token isn't one that requires lookahead, just return it.
@@ -96,7 +97,7 @@ filtered_base_yylex(void)
cur_yytext = base_yytext;
/* Get next token, saving outputs into lookahead variables */
- next_token = base_yylex();
+ next_token = base_yylex_location();
lookahead_token = next_token;
lookahead_yylval = base_yylval;
@@ -184,7 +185,7 @@ filtered_base_yylex(void)
cur_yytext = base_yytext;
/* Get third token */
- next_token = base_yylex();
+ next_token = base_yylex_location();
if (next_token != SCONST)
mmerror(PARSE_ERROR, ET_ERROR, "UESCAPE must be followed by a simple string literal");
@@ -203,6 +204,7 @@ filtered_base_yylex(void)
/* Combine 3 tokens into 1 */
base_yylval.str = psprintf("%s UESCAPE %s", base_yylval.str, escstr);
+ base_yylloc = mm_strdup(base_yylval.str);
/* Clear have_lookahead, thereby consuming all three tokens */
have_lookahead = false;
@@ -218,6 +220,56 @@ filtered_base_yylex(void)
return cur_token;
}
+/*
+ * Call base_yylex() and fill in base_yylloc.
+ *
+ * pgc.l does not worry about setting yylloc, and given what we want for
+ * that, trying to set it there would be pretty inconvenient. What we
+ * want is: if the returned token has type <str>, then duplicate its
+ * string value as yylloc; otherwise, make a downcased copy of yytext.
+ * The downcasing is ASCII-only because all that we care about there
+ * is producing uniformly-cased output of keywords. (That's mostly
+ * cosmetic, but there are places in ecpglib that expect to receive
+ * downcased keywords, plus it keeps us regression-test-compatible
+ * with the old implementation of ecpg.)
+ */
+static int
+base_yylex_location(void)
+{
+ int token = base_yylex();
+
+ switch (token)
+ {
+ /* List a token here if pgc.l assigns to base_yylval.str for it */
+ case Op:
+ case CSTRING:
+ case CPP_LINE:
+ case CVARIABLE:
+ case BCONST:
+ case SCONST:
+ case USCONST:
+ case XCONST:
+ case FCONST:
+ case IDENT:
+ case UIDENT:
+ case IP:
+ /* Duplicate the <str> value */
+ base_yylloc = mm_strdup(base_yylval.str);
+ break;
+ default:
+ /* Else just use the input, i.e., yytext */
+ base_yylloc = mm_strdup(base_yytext);
+ /* Apply an ASCII-only downcasing */
+ for (unsigned char *ptr = (unsigned char *) base_yylloc; *ptr; ptr++)
+ {
+ if (*ptr >= 'A' && *ptr <= 'Z')
+ *ptr += 'a' - 'A';
+ }
+ break;
+ }
+ return token;
+}
+
/*
* check_uescapechar() and ecpg_isspace() should match their equivalents
* in pgc.l.
diff --git a/src/interfaces/ecpg/preproc/preproc_extern.h b/src/interfaces/ecpg/preproc/preproc_extern.h
index c5fd07fbd8..da93967462 100644
--- a/src/interfaces/ecpg/preproc/preproc_extern.h
+++ b/src/interfaces/ecpg/preproc/preproc_extern.h
@@ -15,6 +15,13 @@
#define STRUCT_DEPTH 128
#define EMPTY mm_strdup("")
+/*
+ * "Location tracking" support --- see ecpg.header for more comments.
+ */
+typedef char *YYLTYPE;
+
+#define YYLTYPE_IS_DECLARED 1
+
/* variables */
extern bool autocommit,
@@ -65,10 +72,10 @@ extern const uint16 SQLScanKeywordTokens[];
extern const char *get_dtype(enum ECPGdtype);
extern void lex_init(void);
extern void output_line_number(void);
-extern void output_statement(char *stmt, int whenever_mode, enum ECPG_statement_type st);
-extern void output_prepare_statement(char *name, char *stmt);
-extern void output_deallocate_prepare_statement(char *name);
-extern void output_simple_statement(char *stmt, int whenever_mode);
+extern void output_statement(const char *stmt, int whenever_mode, enum ECPG_statement_type st);
+extern void output_prepare_statement(const char *name, const char *stmt);
+extern void output_deallocate_prepare_statement(const char *name);
+extern void output_simple_statement(const char *stmt, int whenever_mode);
extern char *hashline_number(void);
extern int base_yyparse(void);
extern int base_yylex(void);
--
2.43.5
[text/x-diff] v5-0004-Move-some-functions-into-a-new-file-ecpg-preproc-.patch (10.1K, ../../[email protected]/5-v5-0004-Move-some-functions-into-a-new-file-ecpg-preproc-.patch)
download | inline diff:
From e8210db09837a2fc3d141dcc5c832230dc3a16f9 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:13:54 -0400
Subject: [PATCH v5 4/9] Move some functions into a new file
ecpg/preproc/util.c.
mm_alloc and mm_strdup were in type.c, which seems a completely
random choice. No doubt the original author thought two small
functions didn't deserve their own file. But I'm about to add
some more memory-management stuff beside them, so let's put them
in a less surprising place. This seems like a better home for
mmerror, mmfatal, and the cat_str/make_str family, too.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/Makefile | 1 +
src/interfaces/ecpg/preproc/ecpg.header | 129 -------------
src/interfaces/ecpg/preproc/meson.build | 1 +
src/interfaces/ecpg/preproc/preproc_extern.h | 4 +
src/interfaces/ecpg/preproc/type.c | 24 ---
src/interfaces/ecpg/preproc/util.c | 189 +++++++++++++++++++
6 files changed, 195 insertions(+), 153 deletions(-)
create mode 100644 src/interfaces/ecpg/preproc/util.c
diff --git a/src/interfaces/ecpg/preproc/Makefile b/src/interfaces/ecpg/preproc/Makefile
index 934b7cef1b..7866037cbb 100644
--- a/src/interfaces/ecpg/preproc/Makefile
+++ b/src/interfaces/ecpg/preproc/Makefile
@@ -36,6 +36,7 @@ OBJS = \
preproc.o \
type.o \
typename.o \
+ util.o \
variable.o
# where to find gen_keywordlist.pl and subsidiary files
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index 8df6248c97..929ffa97aa 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -60,137 +60,8 @@ struct variable no_indicator = {"no_indicator", &ecpg_no_indicator, 0, NULL};
static struct ECPGtype ecpg_query = {ECPGt_char_variable, NULL, NULL, NULL, {NULL}, 0};
-static void vmmerror(int error_code, enum errortype type, const char *error, va_list ap) pg_attribute_printf(3, 0);
-
static bool check_declared_list(const char *name);
-/*
- * Handle parsing errors and warnings
- */
-static void
-vmmerror(int error_code, enum errortype type, const char *error, va_list ap)
-{
- /* localize the error message string */
- error = _(error);
-
- fprintf(stderr, "%s:%d: ", input_filename, base_yylineno);
-
- switch (type)
- {
- case ET_WARNING:
- fprintf(stderr, _("WARNING: "));
- break;
- case ET_ERROR:
- fprintf(stderr, _("ERROR: "));
- break;
- }
-
- vfprintf(stderr, error, ap);
-
- fprintf(stderr, "\n");
-
- switch (type)
- {
- case ET_WARNING:
- break;
- case ET_ERROR:
- ret_value = error_code;
- break;
- }
-}
-
-void
-mmerror(int error_code, enum errortype type, const char *error,...)
-{
- va_list ap;
-
- va_start(ap, error);
- vmmerror(error_code, type, error, ap);
- va_end(ap);
-}
-
-void
-mmfatal(int error_code, const char *error,...)
-{
- va_list ap;
-
- va_start(ap, error);
- vmmerror(error_code, ET_ERROR, error, ap);
- va_end(ap);
-
- if (base_yyin)
- fclose(base_yyin);
- if (base_yyout)
- fclose(base_yyout);
-
- if (strcmp(output_filename, "-") != 0 && unlink(output_filename) != 0)
- fprintf(stderr, _("could not remove output file \"%s\"\n"), output_filename);
- exit(error_code);
-}
-
-/*
- * string concatenation
- */
-
-static char *
-cat2_str(char *str1, char *str2)
-{
- char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + 2);
-
- strcpy(res_str, str1);
- if (strlen(str1) != 0 && strlen(str2) != 0)
- strcat(res_str, " ");
- strcat(res_str, str2);
- free(str1);
- free(str2);
- return res_str;
-}
-
-static char *
-cat_str(int count,...)
-{
- va_list args;
- int i;
- char *res_str;
-
- va_start(args, count);
-
- res_str = va_arg(args, char *);
-
- /* now add all other strings */
- for (i = 1; i < count; i++)
- res_str = cat2_str(res_str, va_arg(args, char *));
-
- va_end(args);
-
- return res_str;
-}
-
-static char *
-make2_str(char *str1, char *str2)
-{
- char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + 1);
-
- strcpy(res_str, str1);
- strcat(res_str, str2);
- free(str1);
- free(str2);
- return res_str;
-}
-
-static char *
-make3_str(char *str1, char *str2, char *str3)
-{
- char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + strlen(str3) + 1);
-
- strcpy(res_str, str1);
- strcat(res_str, str2);
- strcat(res_str, str3);
- free(str1);
- free(str2);
- free(str3);
- return res_str;
-}
/*
* "Location tracking" support. We commandeer Bison's location tracking
diff --git a/src/interfaces/ecpg/preproc/meson.build b/src/interfaces/ecpg/preproc/meson.build
index ddd7a66547..f680e5d59e 100644
--- a/src/interfaces/ecpg/preproc/meson.build
+++ b/src/interfaces/ecpg/preproc/meson.build
@@ -10,6 +10,7 @@ ecpg_sources = files(
'output.c',
'parser.c',
'type.c',
+ 'util.c',
'variable.c',
)
diff --git a/src/interfaces/ecpg/preproc/preproc_extern.h b/src/interfaces/ecpg/preproc/preproc_extern.h
index da93967462..29329ccd89 100644
--- a/src/interfaces/ecpg/preproc/preproc_extern.h
+++ b/src/interfaces/ecpg/preproc/preproc_extern.h
@@ -82,6 +82,10 @@ extern int base_yylex(void);
extern void base_yyerror(const char *error);
extern void *mm_alloc(size_t size);
extern char *mm_strdup(const char *string);
+extern char *cat2_str(char *str1, char *str2);
+extern char *cat_str(int count,...);
+extern char *make2_str(char *str1, char *str2);
+extern char *make3_str(char *str1, char *str2, char *str3);
extern void mmerror(int error_code, enum errortype type, const char *error,...) pg_attribute_printf(3, 4);
extern void mmfatal(int error_code, const char *error,...) pg_attribute_printf(2, 3) pg_attribute_noreturn();
extern void output_get_descr_header(char *desc_name);
diff --git a/src/interfaces/ecpg/preproc/type.c b/src/interfaces/ecpg/preproc/type.c
index a842bb6a1f..5610a8dc76 100644
--- a/src/interfaces/ecpg/preproc/type.c
+++ b/src/interfaces/ecpg/preproc/type.c
@@ -8,30 +8,6 @@
static struct ECPGstruct_member struct_no_indicator = {"no_indicator", &ecpg_no_indicator, NULL};
-/* malloc + error check */
-void *
-mm_alloc(size_t size)
-{
- void *ptr = malloc(size);
-
- if (ptr == NULL)
- mmfatal(OUT_OF_MEMORY, "out of memory");
-
- return ptr;
-}
-
-/* strdup + error check */
-char *
-mm_strdup(const char *string)
-{
- char *new = strdup(string);
-
- if (new == NULL)
- mmfatal(OUT_OF_MEMORY, "out of memory");
-
- return new;
-}
-
/* duplicate memberlist */
struct ECPGstruct_member *
ECPGstruct_member_dup(struct ECPGstruct_member *rm)
diff --git a/src/interfaces/ecpg/preproc/util.c b/src/interfaces/ecpg/preproc/util.c
new file mode 100644
index 0000000000..cb1eca7f3c
--- /dev/null
+++ b/src/interfaces/ecpg/preproc/util.c
@@ -0,0 +1,189 @@
+/* src/interfaces/ecpg/preproc/util.c */
+
+#include "postgres_fe.h"
+
+#include <unistd.h>
+
+#include "preproc_extern.h"
+
+static void vmmerror(int error_code, enum errortype type, const char *error, va_list ap) pg_attribute_printf(3, 0);
+
+
+/*
+ * Handle preprocessor errors and warnings
+ */
+static void
+vmmerror(int error_code, enum errortype type, const char *error, va_list ap)
+{
+ /* localize the error message string */
+ error = _(error);
+
+ fprintf(stderr, "%s:%d: ", input_filename, base_yylineno);
+
+ switch (type)
+ {
+ case ET_WARNING:
+ fprintf(stderr, _("WARNING: "));
+ break;
+ case ET_ERROR:
+ fprintf(stderr, _("ERROR: "));
+ break;
+ }
+
+ vfprintf(stderr, error, ap);
+
+ fprintf(stderr, "\n");
+
+ /* If appropriate, set error code to be inspected by ecpg.c */
+ switch (type)
+ {
+ case ET_WARNING:
+ break;
+ case ET_ERROR:
+ ret_value = error_code;
+ break;
+ }
+}
+
+/* Report an error or warning */
+void
+mmerror(int error_code, enum errortype type, const char *error,...)
+{
+ va_list ap;
+
+ va_start(ap, error);
+ vmmerror(error_code, type, error, ap);
+ va_end(ap);
+}
+
+/* Report an error and abandon execution */
+void
+mmfatal(int error_code, const char *error,...)
+{
+ va_list ap;
+
+ va_start(ap, error);
+ vmmerror(error_code, ET_ERROR, error, ap);
+ va_end(ap);
+
+ if (base_yyin)
+ fclose(base_yyin);
+ if (base_yyout)
+ fclose(base_yyout);
+
+ if (strcmp(output_filename, "-") != 0 && unlink(output_filename) != 0)
+ fprintf(stderr, _("could not remove output file \"%s\"\n"), output_filename);
+ exit(error_code);
+}
+
+/*
+ * Basic memory management support
+ */
+
+/* malloc + error check */
+void *
+mm_alloc(size_t size)
+{
+ void *ptr = malloc(size);
+
+ if (ptr == NULL)
+ mmfatal(OUT_OF_MEMORY, "out of memory");
+
+ return ptr;
+}
+
+/* strdup + error check */
+char *
+mm_strdup(const char *string)
+{
+ char *new = strdup(string);
+
+ if (new == NULL)
+ mmfatal(OUT_OF_MEMORY, "out of memory");
+
+ return new;
+}
+
+/*
+ * String concatenation
+ */
+
+/*
+ * Concatenate 2 strings, inserting a space between them unless either is empty
+ *
+ * The input strings are freed.
+ */
+char *
+cat2_str(char *str1, char *str2)
+{
+ char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + 2);
+
+ strcpy(res_str, str1);
+ if (strlen(str1) != 0 && strlen(str2) != 0)
+ strcat(res_str, " ");
+ strcat(res_str, str2);
+ free(str1);
+ free(str2);
+ return res_str;
+}
+
+/*
+ * Concatenate N strings, inserting spaces between them unless they are empty
+ *
+ * The input strings are freed.
+ */
+char *
+cat_str(int count,...)
+{
+ va_list args;
+ int i;
+ char *res_str;
+
+ va_start(args, count);
+
+ res_str = va_arg(args, char *);
+
+ /* now add all other strings */
+ for (i = 1; i < count; i++)
+ res_str = cat2_str(res_str, va_arg(args, char *));
+
+ va_end(args);
+
+ return res_str;
+}
+
+/*
+ * Concatenate 2 strings, with no space between
+ *
+ * The input strings are freed.
+ */
+char *
+make2_str(char *str1, char *str2)
+{
+ char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + 1);
+
+ strcpy(res_str, str1);
+ strcat(res_str, str2);
+ free(str1);
+ free(str2);
+ return res_str;
+}
+
+/*
+ * Concatenate 3 strings, with no space between
+ *
+ * The input strings are freed.
+ */
+char *
+make3_str(char *str1, char *str2, char *str3)
+{
+ char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + strlen(str3) + 1);
+
+ strcpy(res_str, str1);
+ strcat(res_str, str2);
+ strcat(res_str, str3);
+ free(str1);
+ free(str2);
+ free(str3);
+ return res_str;
+}
--
2.43.5
[text/x-diff] v5-0005-Improve-ecpg-preprocessor-s-memory-management.patch (88.9K, ../../[email protected]/6-v5-0005-Improve-ecpg-preprocessor-s-memory-management.patch)
download | inline diff:
From df2ad066914216b422ae0f4f90969cc6f0334636 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:17:19 -0400
Subject: [PATCH v5 5/9] Improve ecpg preprocessor's memory management.
Invent a notion of "local" storage that will automatically be
reclaimed at the end of each statement. Use this for location
strings as well as other visibly short-lived data within the parser.
Also, make cat_str and make_str return local storage and not free
their inputs, which allows dispensing with a whole lot of retail
mm_strdup calls. We do have to add some new ones in places where
a local-lifetime string needs to be added to a longer-lived data
structure, but on balance there are a lot less mm_strdup calls than
before.
In hopes of flushing out places where changes were necessary,
I changed YYLTYPE from "char *" to "const char *", which forced
const-ification of various function arguments that probably
should've been like that all along.
This still leaks memory to some extent, but that will be cleaned up
in the next step.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/descriptor.c | 32 +-
src/interfaces/ecpg/preproc/ecpg.addons | 144 +++--
src/interfaces/ecpg/preproc/ecpg.header | 167 +++---
src/interfaces/ecpg/preproc/ecpg.trailer | 549 ++++++++++---------
src/interfaces/ecpg/preproc/output.c | 5 +-
src/interfaces/ecpg/preproc/parser.c | 6 +-
src/interfaces/ecpg/preproc/preproc_extern.h | 36 +-
src/interfaces/ecpg/preproc/type.c | 8 +-
src/interfaces/ecpg/preproc/type.h | 30 +-
src/interfaces/ecpg/preproc/util.c | 119 +++-
src/interfaces/ecpg/preproc/variable.c | 31 +-
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 599 insertions(+), 529 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/descriptor.c b/src/interfaces/ecpg/preproc/descriptor.c
index f4b1878289..9b87d07d09 100644
--- a/src/interfaces/ecpg/preproc/descriptor.c
+++ b/src/interfaces/ecpg/preproc/descriptor.c
@@ -18,13 +18,12 @@
static struct assignment *assignments;
void
-push_assignment(char *var, enum ECPGdtype value)
+push_assignment(const char *var, enum ECPGdtype value)
{
struct assignment *new = (struct assignment *) mm_alloc(sizeof(struct assignment));
new->next = assignments;
- new->variable = mm_alloc(strlen(var) + 1);
- strcpy(new->variable, var);
+ new->variable = mm_strdup(var);
new->value = value;
assignments = new;
}
@@ -73,7 +72,7 @@ ECPGnumeric_lvalue(char *name)
static struct descriptor *descriptors;
void
-add_descriptor(char *name, char *connection)
+add_descriptor(const char *name, const char *connection)
{
struct descriptor *new;
@@ -83,20 +82,16 @@ add_descriptor(char *name, char *connection)
new = (struct descriptor *) mm_alloc(sizeof(struct descriptor));
new->next = descriptors;
- new->name = mm_alloc(strlen(name) + 1);
- strcpy(new->name, name);
+ new->name = mm_strdup(name);
if (connection)
- {
- new->connection = mm_alloc(strlen(connection) + 1);
- strcpy(new->connection, connection);
- }
+ new->connection = mm_strdup(connection);
else
- new->connection = connection;
+ new->connection = NULL;
descriptors = new;
}
void
-drop_descriptor(char *name, char *connection)
+drop_descriptor(const char *name, const char *connection)
{
struct descriptor *i;
struct descriptor **lastptr = &descriptors;
@@ -126,9 +121,8 @@ drop_descriptor(char *name, char *connection)
mmerror(PARSE_ERROR, ET_WARNING, "descriptor %s bound to the default connection does not exist", name);
}
-struct descriptor
- *
-lookup_descriptor(char *name, char *connection)
+struct descriptor *
+lookup_descriptor(const char *name, const char *connection)
{
struct descriptor *i;
@@ -159,7 +153,7 @@ lookup_descriptor(char *name, char *connection)
}
void
-output_get_descr_header(char *desc_name)
+output_get_descr_header(const char *desc_name)
{
struct assignment *results;
@@ -178,7 +172,7 @@ output_get_descr_header(char *desc_name)
}
void
-output_get_descr(char *desc_name, char *index)
+output_get_descr(const char *desc_name, const char *index)
{
struct assignment *results;
@@ -211,7 +205,7 @@ output_get_descr(char *desc_name, char *index)
}
void
-output_set_descr_header(char *desc_name)
+output_set_descr_header(const char *desc_name)
{
struct assignment *results;
@@ -272,7 +266,7 @@ descriptor_item_name(enum ECPGdtype itemcode)
}
void
-output_set_descr(char *desc_name, char *index)
+output_set_descr(const char *desc_name, const char *index)
{
struct assignment *results;
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index 24ee54554e..9c120fead2 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -45,18 +45,16 @@ ECPG: stmtExecuteStmt block
else
{
/* case of ecpg_ident or CSTRING */
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *str = mm_strdup($1.name + 1);
+ char length[32];
+ char *str;
- /*
- * It must be cut off double quotation because new_variable()
- * double-quotes.
- */
+ /* Remove double quotes from name */
+ str = loc_strdup($1.name + 1);
str[strlen(str) - 1] = '\0';
- sprintf(length, "%zu", strlen(str));
+ snprintf(length, sizeof(length), "%zu", strlen(str));
add_variable_to_tail(&argsinsert, new_variable(str, ECPGmake_simple_type(ECPGt_const, length, 0), 0), &no_indicator);
}
- output_statement(cat_str(3, mm_strdup("execute"), mm_strdup("$0"), $1.type), 0, ECPGst_exec_with_exprlist);
+ output_statement(cat_str(3, "execute", "$0", $1.type), 0, ECPGst_exec_with_exprlist);
}
}
ECPG: stmtPrepareStmt block
@@ -66,7 +64,7 @@ ECPG: stmtPrepareStmt block
output_prepare_statement($1.name, $1.stmt);
else if (strlen($1.type) == 0)
{
- char *stmt = cat_str(3, mm_strdup("\""), $1.stmt, mm_strdup("\""));
+ char *stmt = cat_str(3, "\"", $1.stmt, "\"");
output_prepare_statement($1.name, stmt);
}
@@ -77,18 +75,16 @@ ECPG: stmtPrepareStmt block
add_variable_to_tail(&argsinsert, find_variable($1.name), &no_indicator);
else
{
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *str = mm_strdup($1.name + 1);
+ char length[32];
+ char *str;
- /*
- * It must be cut off double quotation because new_variable()
- * double-quotes.
- */
+ /* Remove double quotes from name */
+ str = loc_strdup($1.name + 1);
str[strlen(str) - 1] = '\0';
- sprintf(length, "%zu", strlen(str));
+ snprintf(length, sizeof(length), "%zu", strlen(str));
add_variable_to_tail(&argsinsert, new_variable(str, ECPGmake_simple_type(ECPGt_const, length, 0), 0), &no_indicator);
}
- output_statement(cat_str(5, mm_strdup("prepare"), mm_strdup("$0"), $1.type, mm_strdup("as"), $1.stmt), 0, ECPGst_prepare);
+ output_statement(cat_str(5, "prepare", "$0", $1.type, "as", $1.stmt), 0, ECPGst_prepare);
}
}
ECPG: stmtTransactionStmt block
@@ -142,8 +138,6 @@ ECPG: stmtViewStmt rule
fputs("ECPGt_EORT);", base_yyout);
fprintf(base_yyout, "}");
output_line_number();
-
- free($1.stmt_name);
}
| ECPGDisconnect
{
@@ -175,8 +169,6 @@ ECPG: stmtViewStmt rule
{
lookup_descriptor($1.name, connection);
output_get_descr($1.name, $1.str);
- free($1.name);
- free($1.str);
}
| ECPGGetDescriptorHeader
{
@@ -190,7 +182,7 @@ ECPG: stmtViewStmt rule
if ((ptr = add_additional_variables(@1, true)) != NULL)
{
connection = ptr->connection ? mm_strdup(ptr->connection) : NULL;
- output_statement(mm_strdup(ptr->command), 0, ECPGst_normal);
+ output_statement(ptr->command, 0, ECPGst_normal);
ptr->opened = true;
}
}
@@ -211,8 +203,6 @@ ECPG: stmtViewStmt rule
{
lookup_descriptor($1.name, connection);
output_set_descr($1.name, $1.str);
- free($1.name);
- free($1.str);
}
| ECPGSetDescriptorHeader
{
@@ -243,9 +233,9 @@ ECPG: stmtViewStmt rule
}
ECPG: where_or_current_clauseWHERECURRENT_POFcursor_name block
{
- char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
- @$ = cat_str(2, mm_strdup("where current of"), cursor_marker);
+ @$ = cat_str(2, "where current of", cursor_marker);
}
ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listcopy_fromopt_programcopy_file_namecopy_delimiteropt_withcopy_optionswhere_clause addon
if (strcmp(@6, "from") == 0 &&
@@ -253,21 +243,21 @@ ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listcopy_fromopt_programcop
mmerror(PARSE_ERROR, ET_WARNING, "COPY FROM STDIN is not implemented");
ECPG: var_valueNumericOnly addon
if (@1[0] == '$')
- @$ = mm_strdup("$0");
+ @$ = "$0";
ECPG: fetch_argscursor_name addon
struct cursor *ptr = add_additional_variables(@1, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
if (@1[0] == ':')
- @$ = mm_strdup("$0");
+ @$ = "$0";
ECPG: fetch_argsfrom_incursor_name addon
struct cursor *ptr = add_additional_variables(@2, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
if (@2[0] == ':')
- @$ = cat2_str(mm_strdup(@1), mm_strdup("$0"));
+ @$ = cat2_str(@1, "$0");
ECPG: fetch_argsNEXTopt_from_incursor_name addon
ECPG: fetch_argsPRIORopt_from_incursor_name addon
ECPG: fetch_argsFIRST_Popt_from_incursor_name addon
@@ -278,7 +268,7 @@ ECPG: fetch_argsALLopt_from_incursor_name addon
if (ptr->connection)
connection = mm_strdup(ptr->connection);
if (@3[0] == ':')
- @$ = cat_str(3, mm_strdup(@1), mm_strdup(@2), mm_strdup("$0"));
+ @$ = cat_str(3, @1, @2, "$0");
ECPG: fetch_argsSignedIconstopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@3, false);
bool replace = false;
@@ -287,16 +277,16 @@ ECPG: fetch_argsSignedIconstopt_from_incursor_name addon
connection = mm_strdup(ptr->connection);
if (@3[0] == ':')
{
- @3 = mm_strdup("$0");
+ @3 = "$0";
replace = true;
}
if (@1[0] == '$')
{
- @1 = mm_strdup("$0");
+ @1 = "$0";
replace = true;
}
if (replace)
- @$ = cat_str(3, mm_strdup(@1), mm_strdup(@2), mm_strdup(@3));
+ @$ = cat_str(3, @1, @2, @3);
ECPG: fetch_argsFORWARDALLopt_from_incursor_name addon
ECPG: fetch_argsBACKWARDALLopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@4, false);
@@ -304,7 +294,7 @@ ECPG: fetch_argsBACKWARDALLopt_from_incursor_name addon
if (ptr->connection)
connection = mm_strdup(ptr->connection);
if (@4[0] == ':')
- @$ = cat_str(4, mm_strdup(@1), mm_strdup(@2), mm_strdup(@3), mm_strdup("$0"));
+ @$ = cat_str(4, @1, @2, @3, "$0");
ECPG: fetch_argsABSOLUTE_PSignedIconstopt_from_incursor_name addon
ECPG: fetch_argsRELATIVE_PSignedIconstopt_from_incursor_name addon
ECPG: fetch_argsFORWARDSignedIconstopt_from_incursor_name addon
@@ -316,20 +306,20 @@ ECPG: fetch_argsBACKWARDSignedIconstopt_from_incursor_name addon
connection = mm_strdup(ptr->connection);
if (@4[0] == ':')
{
- @4 = mm_strdup("$0");
+ @4 = "$0";
replace = true;
}
if (@2[0] == '$')
{
- @2 = mm_strdup("$0");
+ @2 = "$0";
replace = true;
}
if (replace)
- @$ = cat_str(4, mm_strdup(@1), mm_strdup(@2), mm_strdup(@3), mm_strdup(@4));
+ @$ = cat_str(4, @1, @2, @3, @4);
ECPG: cursor_namename block
| char_civar
{
- char *curname = mm_alloc(strlen(@1) + 2);
+ char *curname = loc_alloc(strlen(@1) + 2);
sprintf(curname, ":%s", @1);
@$ = curname;
@@ -367,7 +357,7 @@ ECPG: DeclareCursorStmtDECLAREcursor_namecursor_optionsCURSORopt_holdFORSelectSt
{
struct cursor *ptr,
*this;
- char *cursor_marker = @2[0] == ':' ? mm_strdup("$0") : mm_strdup(@2);
+ const char *cursor_marker = @2[0] == ':' ? "$0" : @2;
char *comment,
*c1,
*c2;
@@ -394,7 +384,7 @@ ECPG: DeclareCursorStmtDECLAREcursor_namecursor_optionsCURSORopt_holdFORSelectSt
this->function = (current_function ? mm_strdup(current_function) : NULL);
this->connection = connection ? mm_strdup(connection) : NULL;
this->opened = false;
- this->command = cat_str(7, mm_strdup("declare"), cursor_marker, @3, mm_strdup("cursor"), @5, mm_strdup("for"), @7);
+ this->command = mm_strdup(cat_str(7, "declare", cursor_marker, @3, "cursor", @5, "for", @7));
this->argsinsert = argsinsert;
this->argsinsert_oos = NULL;
this->argsresult = argsresult;
@@ -402,20 +392,20 @@ ECPG: DeclareCursorStmtDECLAREcursor_namecursor_optionsCURSORopt_holdFORSelectSt
argsinsert = argsresult = NULL;
cur = this;
- c1 = mm_strdup(this->command);
- if ((c2 = strstr(c1, "*/")) != NULL)
+ c1 = loc_strdup(this->command);
+ while ((c2 = strstr(c1, "*/")) != NULL)
{
/* We put this text into a comment, so we better remove [*][/]. */
c2[0] = '.';
c2[1] = '.';
}
- comment = cat_str(3, mm_strdup("/*"), c1, mm_strdup("*/"));
+ comment = cat_str(3, "/*", c1, "*/");
@$ = cat2_str(adjust_outofscope_cursor_vars(this), comment);
}
ECPG: ClosePortalStmtCLOSEcursor_name block
{
- char *cursor_marker = @2[0] == ':' ? mm_strdup("$0") : @2;
+ const char *cursor_marker = @2[0] == ':' ? "$0" : @2;
struct cursor *ptr = NULL;
for (ptr = cur; ptr != NULL; ptr = ptr->next)
@@ -427,23 +417,23 @@ ECPG: ClosePortalStmtCLOSEcursor_name block
break;
}
}
- @$ = cat2_str(mm_strdup("close"), cursor_marker);
+ @$ = cat2_str("close", cursor_marker);
}
ECPG: opt_hold block
{
if (compat == ECPG_COMPAT_INFORMIX_SE && autocommit)
- @$ = mm_strdup("with hold");
+ @$ = "with hold";
else
- @$ = EMPTY;
+ @$ = "";
}
ECPG: into_clauseINTOOptTempTableName block
{
FoundInto = 1;
- @$ = cat2_str(mm_strdup("into"), @2);
+ @$ = cat2_str("into", @2);
}
| ecpg_into
{
- @$ = EMPTY;
+ @$ = "";
}
ECPG: TypenameSimpleTypenameopt_array_bounds block
{
@@ -451,37 +441,33 @@ ECPG: TypenameSimpleTypenameopt_array_bounds block
}
ECPG: TypenameSETOFSimpleTypenameopt_array_bounds block
{
- @$ = cat_str(3, mm_strdup("setof"), @2, $3.str);
+ @$ = cat_str(3, "setof", @2, $3.str);
}
ECPG: opt_array_boundsopt_array_bounds'['']' block
{
$$.index1 = $1.index1;
$$.index2 = $1.index2;
if (strcmp($$.index1, "-1") == 0)
- $$.index1 = mm_strdup("0");
+ $$.index1 = "0";
else if (strcmp($1.index2, "-1") == 0)
- $$.index2 = mm_strdup("0");
- $$.str = cat_str(2, $1.str, mm_strdup("[]"));
+ $$.index2 = "0";
+ $$.str = cat_str(2, $1.str, "[]");
}
| opt_array_bounds '[' Iresult ']'
{
$$.index1 = $1.index1;
$$.index2 = $1.index2;
if (strcmp($1.index1, "-1") == 0)
- $$.index1 = mm_strdup(@3);
+ $$.index1 = @3;
else if (strcmp($1.index2, "-1") == 0)
- $$.index2 = mm_strdup(@3);
- $$.str = cat_str(4, $1.str, mm_strdup("["), @3, mm_strdup("]"));
+ $$.index2 = @3;
+ $$.str = cat_str(4, $1.str, "[", @3, "]");
}
ECPG: opt_array_bounds block
{
- $$.index1 = mm_strdup("-1");
- $$.index2 = mm_strdup("-1");
- $$.str = EMPTY;
- }
-ECPG: IconstICONST block
- {
- @$ = make_name();
+ $$.index1 = "-1";
+ $$.index2 = "-1";
+ $$.str = "";
}
ECPG: AexprConstNULL_P rule
| civar
@@ -494,83 +480,83 @@ ECPG: FetchStmtMOVEfetch_args rule
| FETCH fetch_args ecpg_fetch_into
| FETCH FORWARD cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("fetch forward"), cursor_marker);
+ @$ = cat_str(2, "fetch forward", cursor_marker);
}
| FETCH FORWARD from_in cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("fetch forward from"), cursor_marker);
+ @$ = cat_str(2, "fetch forward from", cursor_marker);
}
| FETCH BACKWARD cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("fetch backward"), cursor_marker);
+ @$ = cat_str(2, "fetch backward", cursor_marker);
}
| FETCH BACKWARD from_in cursor_name opt_ecpg_fetch_into
{
- char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("fetch backward from"), cursor_marker);
+ @$ = cat_str(2, "fetch backward from", cursor_marker);
}
| MOVE FORWARD cursor_name
{
- char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("move forward"), cursor_marker);
+ @$ = cat_str(2, "move forward", cursor_marker);
}
| MOVE FORWARD from_in cursor_name
{
- char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("move forward from"), cursor_marker);
+ @$ = cat_str(2, "move forward from", cursor_marker);
}
| MOVE BACKWARD cursor_name
{
- char *cursor_marker = @3[0] == ':' ? mm_strdup("$0") : @3;
+ const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("move backward"), cursor_marker);
+ @$ = cat_str(2, "move backward", cursor_marker);
}
| MOVE BACKWARD from_in cursor_name
{
- char *cursor_marker = @4[0] == ':' ? mm_strdup("$0") : @4;
+ const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
if (ptr->connection)
connection = mm_strdup(ptr->connection);
- @$ = cat_str(2, mm_strdup("move backward from"), cursor_marker);
+ @$ = cat_str(2, "move backward from", cursor_marker);
}
ECPG: limit_clauseLIMITselect_limit_value','select_offset_value block
{
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index 929ffa97aa..d3df8eabbb 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -39,8 +39,6 @@ char *input_filename = NULL;
static int FoundInto = 0;
static int initializer = 0;
static int pacounter = 1;
-static char pacounter_buffer[sizeof(int) * CHAR_BIT * 10 / 3]; /* a rough guess at the
- * size we need */
static struct this_type actual_type[STRUCT_DEPTH];
static char *actual_startline[STRUCT_DEPTH];
static int varchar_counter = 1;
@@ -95,7 +93,7 @@ yylloc_default(YYLTYPE *target, YYLTYPE *rhs, int N)
needed++;
needed += thislen;
}
- result = (char *) mm_alloc(needed + 1);
+ result = (char *) loc_alloc(needed + 1);
ptr = result;
for (int i = 1; i <= N; i++)
{
@@ -115,22 +113,19 @@ yylloc_default(YYLTYPE *target, YYLTYPE *rhs, int N)
*target = rhs[1];
}
else
- *target = EMPTY;
+ {
+ /* No need to allocate any space */
+ *target = "";
+ }
}
/* and the rest */
static char *
-make_name(void)
-{
- return mm_strdup(base_yytext);
-}
-
-static char *
-create_questionmarks(char *name, bool array)
+create_questionmarks(const char *name, bool array)
{
struct variable *p = find_variable(name);
int count;
- char *result = EMPTY;
+ char *result = "";
/*
* In case we have a struct, we have to print as many "?" as there are
@@ -158,12 +153,13 @@ create_questionmarks(char *name, bool array)
for (; count > 0; count--)
{
- sprintf(pacounter_buffer, "$%d", pacounter++);
- result = cat_str(3, result, mm_strdup(pacounter_buffer), mm_strdup(" , "));
- }
+ char buf[32];
- /* removed the trailing " ," */
+ snprintf(buf, sizeof(buf), "$%d", pacounter++);
+ result = cat_str(3, result, buf, " , ");
+ }
+ /* remove the trailing " ," */
result[strlen(result) - 3] = '\0';
return result;
}
@@ -183,8 +179,7 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
* pointer instead of the variable. Do it only for local variables, not
* for globals.
*/
-
- char *result = EMPTY;
+ char *result = "";
int insert;
for (insert = 1; insert >= 0; insert--)
@@ -206,7 +201,7 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
/* change variable name to "ECPGget_var(<counter>)" */
original_var = ptr->variable->name;
- sprintf(var_text, "%d))", ecpg_internal_var);
+ snprintf(var_text, sizeof(var_text), "%d))", ecpg_internal_var);
/* Don't emit ECPGset_var() calls for global variables */
if (ptr->variable->brace_level == 0)
@@ -227,12 +222,12 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
&& ptr->variable->type->type != ECPGt_bytea)
&& atoi(ptr->variable->type->size) > 1)
{
- newvar = new_variable(cat_str(4, mm_strdup("("),
- mm_strdup(ecpg_type_name(ptr->variable->type->u.element->type)),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text)),
+ newvar = new_variable(cat_str(4, "(",
+ ecpg_type_name(ptr->variable->type->u.element->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_array_type(ECPGmake_simple_type(ptr->variable->type->u.element->type,
- mm_strdup("1"),
+ "1",
ptr->variable->type->u.element->counter),
ptr->variable->type->size),
0);
@@ -244,10 +239,10 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
|| ptr->variable->type->type == ECPGt_bytea)
&& atoi(ptr->variable->type->size) > 1)
{
- newvar = new_variable(cat_str(4, mm_strdup("("),
- mm_strdup(ecpg_type_name(ptr->variable->type->type)),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text)),
+ newvar = new_variable(cat_str(4, "(",
+ ecpg_type_name(ptr->variable->type->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_simple_type(ptr->variable->type->type,
ptr->variable->type->size,
ptr->variable->type->counter),
@@ -259,11 +254,11 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
else if (ptr->variable->type->type == ECPGt_struct
|| ptr->variable->type->type == ECPGt_union)
{
- newvar = new_variable(cat_str(5, mm_strdup("(*("),
- mm_strdup(ptr->variable->type->type_name),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text),
- mm_strdup(")")),
+ newvar = new_variable(cat_str(5, "(*(",
+ ptr->variable->type->type_name,
+ " *)(ECPGget_var(",
+ var_text,
+ ")"),
ECPGmake_struct_type(ptr->variable->type->u.members,
ptr->variable->type->type,
ptr->variable->type->type_name,
@@ -276,11 +271,11 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
if (ptr->variable->type->u.element->type == ECPGt_struct
|| ptr->variable->type->u.element->type == ECPGt_union)
{
- newvar = new_variable(cat_str(5, mm_strdup("(*("),
- mm_strdup(ptr->variable->type->u.element->type_name),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text),
- mm_strdup(")")),
+ newvar = new_variable(cat_str(5, "(*(",
+ ptr->variable->type->u.element->type_name,
+ " *)(ECPGget_var(",
+ var_text,
+ ")"),
ECPGmake_struct_type(ptr->variable->type->u.element->u.members,
ptr->variable->type->u.element->type,
ptr->variable->type->u.element->type_name,
@@ -289,10 +284,10 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
}
else
{
- newvar = new_variable(cat_str(4, mm_strdup("("),
- mm_strdup(ecpg_type_name(ptr->variable->type->u.element->type)),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text)),
+ newvar = new_variable(cat_str(4, "(",
+ ecpg_type_name(ptr->variable->type->u.element->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_array_type(ECPGmake_simple_type(ptr->variable->type->u.element->type,
ptr->variable->type->u.element->size,
ptr->variable->type->u.element->counter),
@@ -303,10 +298,10 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
}
else
{
- newvar = new_variable(cat_str(4, mm_strdup("*("),
- mm_strdup(ecpg_type_name(ptr->variable->type->type)),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text)),
+ newvar = new_variable(cat_str(4, "*(",
+ ecpg_type_name(ptr->variable->type->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_simple_type(ptr->variable->type->type,
ptr->variable->type->size,
ptr->variable->type->counter),
@@ -320,10 +315,11 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
*/
if (!skip_set_var)
{
- sprintf(var_text, "%d, %s", ecpg_internal_var++, var_ptr ? "&(" : "(");
- result = cat_str(5, result, mm_strdup("ECPGset_var("),
- mm_strdup(var_text), mm_strdup(original_var),
- mm_strdup("), __LINE__);\n"));
+ snprintf(var_text, sizeof(var_text), "%d, %s",
+ ecpg_internal_var++, var_ptr ? "&(" : "(");
+ result = cat_str(5, result, "ECPGset_var(",
+ var_text, original_var,
+ "), __LINE__);\n");
}
/*
@@ -338,17 +334,17 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
{
/* change variable name to "ECPGget_var(<counter>)" */
original_var = ptr->indicator->name;
- sprintf(var_text, "%d))", ecpg_internal_var);
+ snprintf(var_text, sizeof(var_text), "%d))", ecpg_internal_var);
var_ptr = false;
if (ptr->indicator->type->type == ECPGt_struct
|| ptr->indicator->type->type == ECPGt_union)
{
- newind = new_variable(cat_str(5, mm_strdup("(*("),
- mm_strdup(ptr->indicator->type->type_name),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text),
- mm_strdup(")")),
+ newind = new_variable(cat_str(5, "(*(",
+ ptr->indicator->type->type_name,
+ " *)(ECPGget_var(",
+ var_text,
+ ")"),
ECPGmake_struct_type(ptr->indicator->type->u.members,
ptr->indicator->type->type,
ptr->indicator->type->type_name,
@@ -361,11 +357,11 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
if (ptr->indicator->type->u.element->type == ECPGt_struct
|| ptr->indicator->type->u.element->type == ECPGt_union)
{
- newind = new_variable(cat_str(5, mm_strdup("(*("),
- mm_strdup(ptr->indicator->type->u.element->type_name),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text),
- mm_strdup(")")),
+ newind = new_variable(cat_str(5, "(*(",
+ ptr->indicator->type->u.element->type_name,
+ " *)(ECPGget_var(",
+ var_text,
+ ")"),
ECPGmake_struct_type(ptr->indicator->type->u.element->u.members,
ptr->indicator->type->u.element->type,
ptr->indicator->type->u.element->type_name,
@@ -374,9 +370,10 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
}
else
{
- newind = new_variable(cat_str(4, mm_strdup("("),
- mm_strdup(ecpg_type_name(ptr->indicator->type->u.element->type)),
- mm_strdup(" *)(ECPGget_var("), mm_strdup(var_text)),
+ newind = new_variable(cat_str(4, "(",
+ ecpg_type_name(ptr->indicator->type->u.element->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_array_type(ECPGmake_simple_type(ptr->indicator->type->u.element->type,
ptr->indicator->type->u.element->size,
ptr->indicator->type->u.element->counter),
@@ -387,10 +384,10 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
}
else if (atoi(ptr->indicator->type->size) > 1)
{
- newind = new_variable(cat_str(4, mm_strdup("("),
- mm_strdup(ecpg_type_name(ptr->indicator->type->type)),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text)),
+ newind = new_variable(cat_str(4, "(",
+ ecpg_type_name(ptr->indicator->type->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_simple_type(ptr->indicator->type->type,
ptr->indicator->type->size,
ptr->variable->type->counter),
@@ -398,10 +395,10 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
}
else
{
- newind = new_variable(cat_str(4, mm_strdup("*("),
- mm_strdup(ecpg_type_name(ptr->indicator->type->type)),
- mm_strdup(" *)(ECPGget_var("),
- mm_strdup(var_text)),
+ newind = new_variable(cat_str(4, "*(",
+ ecpg_type_name(ptr->indicator->type->type),
+ " *)(ECPGget_var(",
+ var_text),
ECPGmake_simple_type(ptr->indicator->type->type,
ptr->indicator->type->size,
ptr->variable->type->counter),
@@ -413,10 +410,11 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
* create call to "ECPGset_var(<counter>, <pointer>. <line
* number>)"
*/
- sprintf(var_text, "%d, %s", ecpg_internal_var++, var_ptr ? "&(" : "(");
- result = cat_str(5, result, mm_strdup("ECPGset_var("),
- mm_strdup(var_text), mm_strdup(original_var),
- mm_strdup("), __LINE__);\n"));
+ snprintf(var_text, sizeof(var_text), "%d, %s",
+ ecpg_internal_var++, var_ptr ? "&(" : "(");
+ result = cat_str(5, result, "ECPGset_var(",
+ var_text, original_var,
+ "), __LINE__);\n");
}
add_variable_to_tail(&newlist, newvar, newind);
@@ -437,7 +435,7 @@ adjust_outofscope_cursor_vars(struct cursor *cur)
(cur->function != NULL && strcmp(cur->function, current_function) == 0))
static struct cursor *
-add_additional_variables(char *name, bool insert)
+add_additional_variables(const char *name, bool insert)
{
struct cursor *ptr;
struct arguments *p;
@@ -475,8 +473,10 @@ add_additional_variables(char *name, bool insert)
}
static void
-add_typedef(char *name, char *dimension, char *length, enum ECPGttype type_enum,
- char *type_dimension, char *type_index, int initializer, int array)
+add_typedef(const char *name, const char *dimension, const char *length,
+ enum ECPGttype type_enum,
+ const char *type_dimension, const char *type_index,
+ int initializer, int array)
{
/* add entry to list */
struct typedefs *ptr,
@@ -496,19 +496,20 @@ add_typedef(char *name, char *dimension, char *length, enum ECPGttype type_enum,
/* re-definition is a bug */
mmerror(PARSE_ERROR, ET_ERROR, "type \"%s\" is already defined", name);
}
- adjust_array(type_enum, &dimension, &length, type_dimension, type_index, array, true);
+ adjust_array(type_enum, &dimension, &length,
+ type_dimension, type_index, array, true);
this = (struct typedefs *) mm_alloc(sizeof(struct typedefs));
/* initial definition */
this->next = types;
- this->name = name;
+ this->name = mm_strdup(name);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
this->type->type_enum = type_enum;
this->type->type_str = mm_strdup(name);
- this->type->type_dimension = dimension; /* dimension of array */
- this->type->type_index = length; /* length of string */
+ this->type->type_dimension = mm_strdup(dimension); /* dimension of array */
+ this->type->type_index = mm_strdup(length); /* length of string */
this->type->type_sizeof = ECPGstruct_sizeof;
this->struct_member_list = (type_enum == ECPGt_struct || type_enum == ECPGt_union) ?
ECPGstruct_member_dup(struct_member_list[struct_level]) : NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index 2a3949ca03..0a77559e83 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -2,6 +2,12 @@
statements: /* EMPTY */
| statements statement
+ {
+ /* Reclaim local storage used while processing statement */
+ reclaim_local_storage();
+ /* Clean up now-dangling location pointer */
+ @$ = "";
+ }
;
statement: ecpgstart at toplevel_stmt ';'
@@ -68,7 +74,7 @@ CreateAsStmt: CREATE OptTemp TABLE create_as_target AS
at: AT connection_object
{
- connection = @2;
+ connection = mm_strdup(@2);
/*
* Do we have a variable as connection target? Remove the variable
@@ -84,20 +90,20 @@ at: AT connection_object
*/
ECPGConnect: SQL_CONNECT TO connection_target opt_connection_name opt_user
{
- @$ = cat_str(5, @3, mm_strdup(","), @5, mm_strdup(","), @4);
+ @$ = cat_str(5, @3, ",", @5, ",", @4);
}
| SQL_CONNECT TO DEFAULT
{
- @$ = mm_strdup("NULL, NULL, NULL, \"DEFAULT\"");
+ @$ = "NULL, NULL, NULL, \"DEFAULT\"";
}
/* also allow ORACLE syntax */
| SQL_CONNECT ora_user
{
- @$ = cat_str(3, mm_strdup("NULL,"), @2, mm_strdup(", NULL"));
+ @$ = cat_str(3, "NULL,", @2, ", NULL");
}
| DATABASE connection_target
{
- @$ = cat2_str(@2, mm_strdup(", NULL, NULL, NULL"));
+ @$ = cat2_str(@2, ", NULL, NULL, NULL");
}
;
@@ -111,7 +117,7 @@ connection_target: opt_database_name opt_server opt_port
if (@1[0] == '\"')
@$ = @1;
else
- @$ = make3_str(mm_strdup("\""), make3_str(@1, @2, @3), mm_strdup("\""));
+ @$ = make3_str("\"", make3_str(@1, @2, @3), "\"");
}
| db_prefix ':' server opt_port '/' opt_database_name opt_options
{
@@ -127,19 +133,21 @@ connection_target: opt_database_name opt_server opt_port
strncmp(@3 + strlen("//"), "127.0.0.1", strlen("127.0.0.1")) != 0)
mmerror(PARSE_ERROR, ET_ERROR, "Unix-domain sockets only work on \"localhost\" but not on \"%s\"", @3 + strlen("//"));
- @$ = make3_str(make3_str(mm_strdup("\""), @1, mm_strdup(":")), @3, make3_str(make3_str(@4, mm_strdup("/"), @6), @7, mm_strdup("\"")));
+ @$ = make3_str(make3_str("\"", @1, ":"), @3, make3_str(make3_str(@4, "/", @6), @7, "\""));
}
| char_variable
| ecpg_sconst
{
/*
- * We can only process double quoted strings not single quotes ones,
- * so we change the quotes. Note, that the rule for ecpg_sconst adds
+ * We can only process double quoted strings not single quoted ones,
+ * so we change the quotes. Note that the rule for ecpg_sconst adds
* these single quotes.
*/
- @1[0] = '\"';
- @1[strlen(@1) - 1] = '\"';
- @$ = @1;
+ char *str = loc_strdup(@1);
+
+ str[0] = '\"';
+ str[strlen(str) - 1] = '\"';
+ @$ = str;
}
;
@@ -155,7 +163,7 @@ db_prefix: ecpg_ident cvariable
if (strcmp(@1, "tcp") != 0 && strcmp(@1, "unix") != 0)
mmerror(PARSE_ERROR, ET_ERROR, "invalid connection type: %s", @1);
- @$ = make3_str(@1, mm_strdup(":"), @2);
+ @$ = make3_str(@1, ":", @2);
}
;
@@ -175,14 +183,11 @@ opt_server: server
server_name: ColId
| ColId '.' server_name
| IP
- {
- @$ = make_name();
- }
;
opt_port: ':' Iconst
{
- @$ = make2_str(mm_strdup(":"), @2);
+ @$ = make2_str(":", @2);
}
| /* EMPTY */
;
@@ -193,7 +198,7 @@ opt_connection_name: AS connection_object
}
| /* EMPTY */
{
- @$ = mm_strdup("NULL");
+ @$ = "NULL";
}
;
@@ -203,25 +208,25 @@ opt_user: USER ora_user
}
| /* EMPTY */
{
- @$ = mm_strdup("NULL, NULL");
+ @$ = "NULL, NULL";
}
;
ora_user: user_name
{
- @$ = cat2_str(@1, mm_strdup(", NULL"));
+ @$ = cat2_str(@1, ", NULL");
}
| user_name '/' user_name
{
- @$ = cat_str(3, @1, mm_strdup(","), @3);
+ @$ = cat_str(3, @1, ",", @3);
}
| user_name SQL_IDENTIFIED BY user_name
{
- @$ = cat_str(3, @1, mm_strdup(","), @4);
+ @$ = cat_str(3, @1, ",", @4);
}
| user_name USING user_name
{
- @$ = cat_str(3, @1, mm_strdup(","), @3);
+ @$ = cat_str(3, @1, ",", @3);
}
;
@@ -230,14 +235,14 @@ user_name: RoleId
if (@1[0] == '\"')
@$ = @1;
else
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ @$ = make3_str("\"", @1, "\"");
}
| ecpg_sconst
{
if (@1[0] == '\"')
@$ = @1;
else
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ @$ = make3_str("\"", @1, "\"");
}
| civar
{
@@ -249,9 +254,9 @@ user_name: RoleId
/* handle varchars */
if (type == ECPGt_varchar)
- @$ = make2_str(mm_strdup(argsinsert->variable->name), mm_strdup(".arr"));
+ @$ = make2_str(argsinsert->variable->name, ".arr");
else
- @$ = mm_strdup(argsinsert->variable->name);
+ @$ = argsinsert->variable->name;
}
;
@@ -278,7 +283,7 @@ char_variable: cvariable
@$ = @1;
break;
case ECPGt_varchar:
- @$ = make2_str(@1, mm_strdup(".arr"));
+ @$ = make2_str(@1, ".arr");
break;
default:
mmerror(PARSE_ERROR, ET_ERROR, "invalid data type");
@@ -297,7 +302,7 @@ opt_options: Op connect_options
if (strcmp(@1, "?") != 0)
mmerror(PARSE_ERROR, ET_ERROR, "unrecognized token \"%s\"", @1);
- @$ = make2_str(mm_strdup("?"), @2);
+ @$ = make2_str("?", @2);
}
| /* EMPTY */
;
@@ -321,30 +326,34 @@ connect_options: ColId opt_opt_value
opt_opt_value: /* EMPTY */
| '=' Iconst
{
- @$ = make2_str(mm_strdup("="), @2);
+ @$ = make2_str("=", @2);
}
| '=' ecpg_ident
{
- @$ = make2_str(mm_strdup("="), @2);
+ @$ = make2_str("=", @2);
}
| '=' civar
{
- @$ = make2_str(mm_strdup("="), @2);
+ @$ = make2_str("=", @2);
}
;
prepared_name: name
{
- if (@1[0] == '\"' && @1[strlen(@1) - 1] == '\"') /* already quoted? */
+ size_t slen = strlen(@1);
+
+ if (@1[0] == '\"' && @1[slen - 1] == '\"') /* already quoted? */
@$ = @1;
else /* not quoted => convert to lowercase */
{
- size_t i;
-
- for (i = 0; i < strlen(@1); i++)
- @1[i] = tolower((unsigned char) @1[i]);
-
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ char *str = loc_alloc(slen + 3);
+
+ str[0] = '\"';
+ for (size_t i = 0; i < slen; i++)
+ str[i + 1] = tolower((unsigned char) @1[i]);
+ str[slen + 1] = '\"';
+ str[slen + 2] = '\0';
+ @$ = str;
}
}
| char_variable
@@ -355,7 +364,7 @@ prepared_name: name
*/
ECPGDeclareStmt: DECLARE prepared_name STATEMENT
{
- struct declared_list *ptr = NULL;
+ struct declared_list *ptr;
/* Check whether the declared name has been defined or not */
for (ptr = g_declared_list; ptr != NULL; ptr = ptr->next)
@@ -368,12 +377,11 @@ ECPGDeclareStmt: DECLARE prepared_name STATEMENT
}
/* Add a new declared name into the g_declared_list */
- ptr = NULL;
ptr = (struct declared_list *) mm_alloc(sizeof(struct declared_list));
if (ptr)
{
/* initial definition */
- ptr->name = @2;
+ ptr->name = mm_strdup(@2);
if (connection)
ptr->connection = mm_strdup(connection);
else
@@ -383,7 +391,7 @@ ECPGDeclareStmt: DECLARE prepared_name STATEMENT
g_declared_list = ptr;
}
- @$ = cat_str(3, mm_strdup("/* declare "), mm_strdup(@2), mm_strdup(" as an SQL identifier */"));
+ @$ = cat_str(3, "/* declare ", @2, " as an SQL identifier */");
}
;
@@ -395,7 +403,7 @@ ECPGCursorStmt: DECLARE cursor_name cursor_options CURSOR opt_hold FOR prepared_
{
struct cursor *ptr,
*this;
- char *cursor_marker = @2[0] == ':' ? mm_strdup("$0") : mm_strdup(@2);
+ const char *cursor_marker = @2[0] == ':' ? "$0" : @2;
int (*strcmp_fn) (const char *, const char *) = ((@2[0] == ':' || @2[0] == '"') ? strcmp : pg_strcasecmp);
struct variable *thisquery = (struct variable *) mm_alloc(sizeof(struct variable));
char *comment;
@@ -422,10 +430,10 @@ ECPGCursorStmt: DECLARE cursor_name cursor_options CURSOR opt_hold FOR prepared_
/* initial definition */
this->next = cur;
- this->name = @2;
+ this->name = mm_strdup(@2);
this->function = (current_function ? mm_strdup(current_function) : NULL);
this->connection = connection ? mm_strdup(connection) : NULL;
- this->command = cat_str(6, mm_strdup("declare"), cursor_marker, @3, mm_strdup("cursor"), @5, mm_strdup("for $1"));
+ this->command = mm_strdup(cat_str(6, "declare", cursor_marker, @3, "cursor", @5, "for $1"));
this->argsresult = NULL;
this->argsresult_oos = NULL;
@@ -448,7 +456,7 @@ ECPGCursorStmt: DECLARE cursor_name cursor_options CURSOR opt_hold FOR prepared_
cur = this;
- comment = cat_str(3, mm_strdup("/*"), mm_strdup(this->command), mm_strdup("*/"));
+ comment = cat_str(3, "/*", this->command, "*/");
@$ = cat_str(2, adjust_outofscope_cursor_vars(this),
comment);
@@ -541,45 +549,44 @@ type_declaration: S_TYPEDEF
fprintf(base_yyout, "typedef %s %s %s %s;\n", $3.type_str, *@4 ? "*" : "", @5, $6.str);
output_line_number();
- @$ = EMPTY;
+ @$ = "";
}
;
var_declaration:
storage_declaration var_type
{
- actual_type[struct_level].type_storage = @1;
+ actual_type[struct_level].type_storage = mm_strdup(@1);
actual_type[struct_level].type_enum = $2.type_enum;
- actual_type[struct_level].type_str = $2.type_str;
- actual_type[struct_level].type_dimension = $2.type_dimension;
- actual_type[struct_level].type_index = $2.type_index;
- actual_type[struct_level].type_sizeof = $2.type_sizeof;
+ actual_type[struct_level].type_str = mm_strdup($2.type_str);
+ actual_type[struct_level].type_dimension = mm_strdup($2.type_dimension);
+ actual_type[struct_level].type_index = mm_strdup($2.type_index);
+ actual_type[struct_level].type_sizeof =
+ $2.type_sizeof ? mm_strdup($2.type_sizeof) : NULL;
actual_startline[struct_level] = hashline_number();
}
variable_list ';'
{
- @$ = cat_str(5, actual_startline[struct_level], @1, $2.type_str, @4, mm_strdup(";\n"));
+ @$ = cat_str(5, actual_startline[struct_level], @1, $2.type_str, @4, ";\n");
}
| var_type
{
- actual_type[struct_level].type_storage = EMPTY;
+ actual_type[struct_level].type_storage = mm_strdup("");
actual_type[struct_level].type_enum = $1.type_enum;
- actual_type[struct_level].type_str = $1.type_str;
- actual_type[struct_level].type_dimension = $1.type_dimension;
- actual_type[struct_level].type_index = $1.type_index;
- actual_type[struct_level].type_sizeof = $1.type_sizeof;
+ actual_type[struct_level].type_str = mm_strdup($1.type_str);
+ actual_type[struct_level].type_dimension = mm_strdup($1.type_dimension);
+ actual_type[struct_level].type_index = mm_strdup($1.type_index);
+ actual_type[struct_level].type_sizeof =
+ $1.type_sizeof ? mm_strdup($1.type_sizeof) : NULL;
actual_startline[struct_level] = hashline_number();
}
variable_list ';'
{
- @$ = cat_str(4, actual_startline[struct_level], $1.type_str, @3, mm_strdup(";\n"));
+ @$ = cat_str(4, actual_startline[struct_level], $1.type_str, @3, ";\n");
}
| struct_union_type_with_symbol ';'
- {
- @$ = cat2_str(@1, mm_strdup(";"));
- }
;
opt_bit_field: ':' Iconst
@@ -604,16 +611,16 @@ storage_modifier: S_CONST
var_type: simple_type
{
$$.type_enum = $1;
- $$.type_str = mm_strdup(ecpg_type_name($1));
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = loc_strdup(ecpg_type_name($1));
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| struct_union_type
{
- $$.type_str = @1;
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = loc_strdup(@1);
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
if (strncmp(@1, "struct", sizeof("struct") - 1) == 0)
{
@@ -628,26 +635,26 @@ var_type: simple_type
}
| enum_type
{
- $$.type_str = @1;
+ $$.type_str = loc_strdup(@1);
$$.type_enum = ECPGt_int;
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| NUMERIC '(' precision opt_scale ')'
{
$$.type_enum = ECPGt_numeric;
- $$.type_str = mm_strdup("numeric");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "numeric";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| DECIMAL_P '(' precision opt_scale ')'
{
$$.type_enum = ECPGt_decimal;
- $$.type_str = mm_strdup("decimal");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "decimal";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| IDENT '(' precision opt_scale ')'
@@ -660,63 +667,63 @@ var_type: simple_type
if (strcmp(@1, "numeric") == 0)
{
$$.type_enum = ECPGt_numeric;
- $$.type_str = mm_strdup("numeric");
+ $$.type_str = "numeric";
}
else if (strcmp(@1, "decimal") == 0)
{
$$.type_enum = ECPGt_decimal;
- $$.type_str = mm_strdup("decimal");
+ $$.type_str = "decimal";
}
else
{
mmerror(PARSE_ERROR, ET_ERROR, "only data types numeric and decimal have precision/scale argument");
$$.type_enum = ECPGt_numeric;
- $$.type_str = mm_strdup("numeric");
+ $$.type_str = "numeric";
}
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| VARCHAR
{
$$.type_enum = ECPGt_varchar;
- $$.type_str = EMPTY; /* mm_strdup("varchar"); */
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = ""; /* "varchar"; */
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| FLOAT_P
{
/* Note: DOUBLE is handled in simple_type */
$$.type_enum = ECPGt_float;
- $$.type_str = mm_strdup("float");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "float";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| NUMERIC
{
$$.type_enum = ECPGt_numeric;
- $$.type_str = mm_strdup("numeric");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "numeric";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| DECIMAL_P
{
$$.type_enum = ECPGt_decimal;
- $$.type_str = mm_strdup("decimal");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "decimal";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| TIMESTAMP
{
$$.type_enum = ECPGt_timestamp;
- $$.type_str = mm_strdup("timestamp");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "timestamp";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| STRING_P
@@ -725,9 +732,9 @@ var_type: simple_type
{
/* In Informix mode, "string" is automatically a typedef */
$$.type_enum = ECPGt_string;
- $$.type_str = mm_strdup("char");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "char";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else
@@ -735,14 +742,14 @@ var_type: simple_type
/* Otherwise, legal only if user typedef'ed it */
struct typedefs *this = get_typedef("string", false);
- $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? EMPTY : mm_strdup(this->name);
+ $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? mm_strdup("") : mm_strdup(this->name);
$$.type_enum = this->type->type_enum;
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
if (this->type->type_sizeof && strlen(this->type->type_sizeof) != 0)
$$.type_sizeof = this->type->type_sizeof;
else
- $$.type_sizeof = cat_str(3, mm_strdup("sizeof("), mm_strdup(this->name), mm_strdup(")"));
+ $$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
@@ -750,9 +757,9 @@ var_type: simple_type
| INTERVAL ecpg_interval
{
$$.type_enum = ECPGt_interval;
- $$.type_str = mm_strdup("interval");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "interval";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
| IDENT ecpg_interval
@@ -772,89 +779,89 @@ var_type: simple_type
if (strcmp(@1, "varchar") == 0)
{
$$.type_enum = ECPGt_varchar;
- $$.type_str = EMPTY; /* mm_strdup("varchar"); */
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = ""; /* "varchar"; */
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "bytea") == 0)
{
$$.type_enum = ECPGt_bytea;
- $$.type_str = EMPTY;
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "float") == 0)
{
$$.type_enum = ECPGt_float;
- $$.type_str = mm_strdup("float");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "float";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "double") == 0)
{
$$.type_enum = ECPGt_double;
- $$.type_str = mm_strdup("double");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "double";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "numeric") == 0)
{
$$.type_enum = ECPGt_numeric;
- $$.type_str = mm_strdup("numeric");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "numeric";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "decimal") == 0)
{
$$.type_enum = ECPGt_decimal;
- $$.type_str = mm_strdup("decimal");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "decimal";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "date") == 0)
{
$$.type_enum = ECPGt_date;
- $$.type_str = mm_strdup("date");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "date";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "timestamp") == 0)
{
$$.type_enum = ECPGt_timestamp;
- $$.type_str = mm_strdup("timestamp");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "timestamp";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "interval") == 0)
{
$$.type_enum = ECPGt_interval;
- $$.type_str = mm_strdup("interval");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "interval";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if (strcmp(@1, "datetime") == 0)
{
$$.type_enum = ECPGt_timestamp;
- $$.type_str = mm_strdup("timestamp");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "timestamp";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else if ((strcmp(@1, "string") == 0) && INFORMIX_MODE)
{
$$.type_enum = ECPGt_string;
- $$.type_str = mm_strdup("char");
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
+ $$.type_str = "char";
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
$$.type_sizeof = NULL;
}
else
@@ -862,14 +869,14 @@ var_type: simple_type
/* Otherwise, it must be a user-defined typedef name */
struct typedefs *this = get_typedef(@1, false);
- $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? EMPTY : mm_strdup(this->name);
+ $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? mm_strdup("") : mm_strdup(this->name);
$$.type_enum = this->type->type_enum;
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
if (this->type->type_sizeof && strlen(this->type->type_sizeof) != 0)
$$.type_sizeof = this->type->type_sizeof;
else
- $$.type_sizeof = cat_str(3, mm_strdup("sizeof("), mm_strdup(this->name), mm_strdup(")"));
+ $$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
@@ -888,21 +895,20 @@ var_type: simple_type
/* No */
this = get_typedef(name, false);
- $$.type_str = mm_strdup(this->name);
+ $$.type_str = this->name;
$$.type_enum = this->type->type_enum;
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
$$.type_sizeof = this->type->type_sizeof;
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
- free(name);
}
else
{
$$.type_str = name;
$$.type_enum = ECPGt_long;
- $$.type_dimension = mm_strdup("-1");
- $$.type_index = mm_strdup("-1");
- $$.type_sizeof = mm_strdup("");
+ $$.type_dimension = "-1";
+ $$.type_index = "-1";
+ $$.type_sizeof = "";
struct_member_list[struct_level] = NULL;
}
}
@@ -932,7 +938,7 @@ struct_union_type_with_symbol: s_struct_union_symbol
ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = NULL;
struct_level--;
- if (strncmp($1.su, "struct", sizeof("struct") - 1) == 0)
+ if (strcmp($1.su, "struct") == 0)
su_type.type_enum = ECPGt_struct;
else
su_type.type_enum = ECPGt_union;
@@ -967,7 +973,7 @@ struct_union_type_with_symbol: s_struct_union_symbol
this->struct_member_list = struct_member_list[struct_level];
types = this;
- @$ = cat_str(4, su_type.type_str, mm_strdup("{"), @4, mm_strdup("}"));
+ @$ = cat_str(4, su_type.type_str, "{", @4, "}");
}
;
@@ -983,19 +989,21 @@ struct_union_type: struct_union_type_with_symbol
ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = NULL;
struct_level--;
- @$ = cat_str(4, @1, mm_strdup("{"), @4, mm_strdup("}"));
+ @$ = cat_str(4, @1, "{", @4, "}");
}
;
s_struct_union_symbol: SQL_STRUCT symbol
{
- $$.su = mm_strdup("struct");
+ $$.su = "struct";
$$.symbol = @2;
- ECPGstruct_sizeof = cat_str(3, mm_strdup("sizeof("), cat2_str(mm_strdup($$.su), mm_strdup($$.symbol)), mm_strdup(")"));
+ ECPGstruct_sizeof = mm_strdup(cat_str(3, "sizeof(",
+ cat2_str($$.su, $$.symbol),
+ ")"));
}
| UNION symbol
{
- $$.su = mm_strdup("union");
+ $$.su = "union";
$$.symbol = @2;
}
;
@@ -1004,11 +1012,11 @@ s_struct_union: SQL_STRUCT
{
ECPGstruct_sizeof = mm_strdup(""); /* This must not be NULL to
* distinguish from simple types. */
- @$ = mm_strdup("struct");
+ @$ = "struct";
}
| UNION
{
- @$ = mm_strdup("union");
+ @$ = "union";
}
;
@@ -1047,23 +1055,27 @@ variable_list: variable
| variable_list ',' variable
{
if (actual_type[struct_level].type_enum == ECPGt_varchar || actual_type[struct_level].type_enum == ECPGt_bytea)
- @$ = cat_str(4, @1, mm_strdup(";"), mm_strdup(actual_type[struct_level].type_storage), @3);
+ @$ = cat_str(4, @1, ";", actual_type[struct_level].type_storage, @3);
else
- @$ = cat_str(3, @1, mm_strdup(","), @3);
+ @$ = cat_str(3, @1, ",", @3);
}
;
variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initializer
{
struct ECPGtype *type;
- char *dimension = $3.index1; /* dimension of array */
- char *length = $3.index2; /* length of string */
+ const char *dimension = $3.index1; /* dimension of array */
+ const char *length = $3.index2; /* length of string */
char *dim_str;
- char *vcn;
+ char vcn[32];
int *varlen_type_counter;
char *struct_name;
- adjust_array(actual_type[struct_level].type_enum, &dimension, &length, actual_type[struct_level].type_dimension, actual_type[struct_level].type_index, strlen(@1), false);
+ adjust_array(actual_type[struct_level].type_enum,
+ &dimension, &length,
+ actual_type[struct_level].type_dimension,
+ actual_type[struct_level].type_index,
+ strlen(@1), false);
switch (actual_type[struct_level].type_enum)
{
case ECPGt_struct:
@@ -1073,7 +1085,7 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
else
type = ECPGmake_array_type(ECPGmake_struct_type(struct_member_list[struct_level], actual_type[struct_level].type_enum, actual_type[struct_level].type_str, actual_type[struct_level].type_sizeof), dimension);
- @$ = cat_str(5, @1, mm_strdup(@2), $3.str, @4, @5);
+ @$ = cat_str(5, @1, @2, $3.str, @4, @5);
break;
case ECPGt_varchar:
@@ -1094,9 +1106,9 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
type = ECPGmake_array_type(ECPGmake_simple_type(actual_type[struct_level].type_enum, length, *varlen_type_counter), dimension);
if (strcmp(dimension, "0") == 0 || abs(atoi(dimension)) == 1)
- dim_str = mm_strdup("");
+ dim_str = "";
else
- dim_str = cat_str(3, mm_strdup("["), mm_strdup(dimension), mm_strdup("]"));
+ dim_str = cat_str(3, "[", dimension, "]");
/*
* cannot check for atoi <= 0 because a defined constant will
@@ -1109,12 +1121,11 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
* make sure varchar struct name is unique by adding a unique
* counter to its definition
*/
- vcn = (char *) mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- sprintf(vcn, "%d", *varlen_type_counter);
+ snprintf(vcn, sizeof(vcn), "%d", *varlen_type_counter);
if (strcmp(dimension, "0") == 0)
- @$ = cat_str(7, make2_str(mm_strdup(struct_name), vcn), mm_strdup(" { int len; char arr["), mm_strdup(length), mm_strdup("]; } *"), mm_strdup(@2), @4, @5);
+ @$ = cat_str(7, make2_str(struct_name, vcn), " { int len; char arr[", length, "]; } *", @2, @4, @5);
else
- @$ = cat_str(8, make2_str(mm_strdup(struct_name), vcn), mm_strdup(" { int len; char arr["), mm_strdup(length), mm_strdup("]; } "), mm_strdup(@2), dim_str, @4, @5);
+ @$ = cat_str(8, make2_str(struct_name, vcn), " { int len; char arr[", length, "]; } ", @2, dim_str, @4, @5);
(*varlen_type_counter)++;
break;
@@ -1132,25 +1143,26 @@ variable: opt_pointer ECPGColLabel opt_array_bounds opt_bit_field opt_initialize
* if we have an initializer but no string size set,
* let's use the initializer's length
*/
- free(length);
- length = mm_alloc(i + sizeof("sizeof()"));
- sprintf(length, "sizeof(%s)", @5 + 2);
+ char *buf = loc_alloc(32);
+
+ snprintf(buf, 32, "sizeof(%s)", @5 + 2);
+ length = buf;
}
type = ECPGmake_simple_type(actual_type[struct_level].type_enum, length, 0);
}
else
type = ECPGmake_array_type(ECPGmake_simple_type(actual_type[struct_level].type_enum, length, 0), dimension);
- @$ = cat_str(5, @1, mm_strdup(@2), $3.str, @4, @5);
+ @$ = cat_str(5, @1, @2, $3.str, @4, @5);
break;
default:
if (atoi(dimension) < 0)
- type = ECPGmake_simple_type(actual_type[struct_level].type_enum, mm_strdup("1"), 0);
+ type = ECPGmake_simple_type(actual_type[struct_level].type_enum, "1", 0);
else
- type = ECPGmake_array_type(ECPGmake_simple_type(actual_type[struct_level].type_enum, mm_strdup("1"), 0), dimension);
+ type = ECPGmake_array_type(ECPGmake_simple_type(actual_type[struct_level].type_enum, "1", 0), dimension);
- @$ = cat_str(5, @1, mm_strdup(@2), $3.str, @4, @5);
+ @$ = cat_str(5, @1, @2, $3.str, @4, @5);
break;
}
@@ -1172,7 +1184,7 @@ opt_pointer: /* EMPTY */
| '*'
| '*' '*'
{
- @$ = mm_strdup("**");
+ @$ = "**";
}
;
@@ -1182,7 +1194,7 @@ opt_pointer: /* EMPTY */
ECPGDeclare: DECLARE STATEMENT ecpg_ident
{
/* this is only supported for compatibility */
- @$ = cat_str(3, mm_strdup("/* declare statement"), @3, mm_strdup("*/"));
+ @$ = cat_str(3, "/* declare statement", @3, "*/");
}
;
/*
@@ -1197,25 +1209,25 @@ ECPGDisconnect: SQL_DISCONNECT dis_name
dis_name: connection_object
| CURRENT_P
{
- @$ = mm_strdup("\"CURRENT\"");
+ @$ = "\"CURRENT\"";
}
| ALL
{
- @$ = mm_strdup("\"ALL\"");
+ @$ = "\"ALL\"";
}
| /* EMPTY */
{
- @$ = mm_strdup("\"CURRENT\"");
+ @$ = "\"CURRENT\"";
}
;
connection_object: name
{
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ @$ = make3_str("\"", @1, "\"");
}
| DEFAULT
{
- @$ = mm_strdup("\"DEFAULT\"");
+ @$ = "\"DEFAULT\"";
}
| char_variable
;
@@ -1223,7 +1235,7 @@ connection_object: name
execstring: char_variable
| CSTRING
{
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ @$ = make3_str("\"", @1, "\"");
}
;
@@ -1237,7 +1249,7 @@ ECPGFree: SQL_FREE cursor_name
}
| SQL_FREE ALL
{
- @$ = mm_strdup("all");
+ @$ = "all";
}
;
@@ -1258,7 +1270,7 @@ opt_ecpg_using: /* EMPTY */
ecpg_using: USING using_list
{
- @$ = EMPTY;
+ @$ = "";
}
| using_descriptor
;
@@ -1266,31 +1278,31 @@ ecpg_using: USING using_list
using_descriptor: USING SQL_P SQL_DESCRIPTOR quoted_ident_stringvar
{
add_variable_to_head(&argsinsert, descriptor_variable(@4, 0), &no_indicator);
- @$ = EMPTY;
+ @$ = "";
}
| USING SQL_DESCRIPTOR name
{
add_variable_to_head(&argsinsert, sqlda_variable(@3), &no_indicator);
- @$ = EMPTY;
+ @$ = "";
}
;
into_descriptor: INTO SQL_P SQL_DESCRIPTOR quoted_ident_stringvar
{
add_variable_to_head(&argsresult, descriptor_variable(@4, 1), &no_indicator);
- @$ = EMPTY;
+ @$ = "";
}
| INTO SQL_DESCRIPTOR name
{
add_variable_to_head(&argsresult, sqlda_variable(@3), &no_indicator);
- @$ = EMPTY;
+ @$ = "";
}
;
into_sqlda: INTO name
{
add_variable_to_head(&argsresult, sqlda_variable(@2), &no_indicator);
- @$ = EMPTY;
+ @$ = "";
}
;
@@ -1299,18 +1311,18 @@ using_list: UsingValue | UsingValue ',' using_list
UsingValue: UsingConst
{
- char *length = mm_alloc(32);
+ char length[32];
- sprintf(length, "%zu", strlen(@1));
+ snprintf(length, sizeof(length), "%zu", strlen(@1));
add_variable_to_head(&argsinsert, new_variable(@1, ECPGmake_simple_type(ECPGt_const, length, 0), 0), &no_indicator);
}
| civar
{
- @$ = EMPTY;
+ @$ = "";
}
| civarind
{
- @$ = EMPTY;
+ @$ = "";
}
;
@@ -1430,9 +1442,9 @@ ECPGSetDescHeaderItem: desc_header_item '=' IntConstVar
IntConstVar: Iconst
{
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
+ char length[32];
- sprintf(length, "%zu", strlen(@1));
+ snprintf(length, sizeof(length), "%zu", strlen(@1));
new_variable(@1, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
}
| cvariable
@@ -1484,37 +1496,39 @@ ECPGSetDescItem: descriptor_item '=' AllConstVar
AllConstVar: ecpg_fconst
{
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
+ char length[32];
- sprintf(length, "%zu", strlen(@1));
+ snprintf(length, sizeof(length), "%zu", strlen(@1));
new_variable(@1, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
}
| IntConstVar
| '-' ecpg_fconst
{
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *var = cat2_str(mm_strdup("-"), @2);
+ char length[32];
+ char *var = cat2_str("-", @2);
- sprintf(length, "%zu", strlen(var));
+ snprintf(length, sizeof(length), "%zu", strlen(var));
new_variable(var, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
@$ = var;
}
| '-' Iconst
{
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *var = cat2_str(mm_strdup("-"), @2);
+ char length[32];
+ char *var = cat2_str("-", @2);
- sprintf(length, "%zu", strlen(var));
+ snprintf(length, sizeof(length), "%zu", strlen(var));
new_variable(var, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
@$ = var;
}
| ecpg_sconst
{
- char *length = mm_alloc(sizeof(int) * CHAR_BIT * 10 / 3);
- char *var = @1 + 1;
+ char length[32];
+ char *var;
+ /* Strip single quotes from ecpg_sconst */
+ var = loc_strdup(@1 + 1);
var[strlen(var) - 1] = '\0';
- sprintf(length, "%zu", strlen(var));
+ snprintf(length, sizeof(length), "%zu", strlen(var));
new_variable(var, ECPGmake_simple_type(ECPGt_const, length, 0), 0);
@$ = var;
}
@@ -1587,9 +1601,9 @@ ECPGTypedef: TYPE_P
add_typedef(@3, $6.index1, $6.index2, $5.type_enum, $5.type_dimension, $5.type_index, initializer, *@7 ? 1 : 0);
if (auto_create_c == false)
- @$ = cat_str(7, mm_strdup("/* exec sql type"), mm_strdup(@3), mm_strdup("is"), mm_strdup($5.type_str), mm_strdup($6.str), @7, mm_strdup("*/"));
+ @$ = cat_str(7, "/* exec sql type", @3, "is", $5.type_str, $6.str, @7, "*/");
else
- @$ = cat_str(6, mm_strdup("typedef "), mm_strdup($5.type_str), *@7 ? mm_strdup("*") : mm_strdup(""), mm_strdup(@3), mm_strdup($6.str), mm_strdup(";"));
+ @$ = cat_str(6, "typedef ", $5.type_str, *@7 ? "*" : "", @3, $6.str, ";");
}
;
@@ -1609,8 +1623,8 @@ ECPGVar: SQL_VAR
ColLabel IS var_type opt_array_bounds opt_reference
{
struct variable *p = find_variable(@3);
- char *dimension = $6.index1;
- char *length = $6.index2;
+ const char *dimension = $6.index1;
+ const char *length = $6.index2;
struct ECPGtype *type;
if (($5.type_enum == ECPGt_struct ||
@@ -1619,7 +1633,8 @@ ECPGVar: SQL_VAR
mmerror(PARSE_ERROR, ET_ERROR, "initializer not allowed in EXEC SQL VAR command");
else
{
- adjust_array($5.type_enum, &dimension, &length, $5.type_dimension, $5.type_index, *@7 ? 1 : 0, false);
+ adjust_array($5.type_enum, &dimension, &length,
+ $5.type_dimension, $5.type_index, *@7 ? 1 : 0, false);
switch ($5.type_enum)
{
@@ -1653,9 +1668,9 @@ ECPGVar: SQL_VAR
mmerror(PARSE_ERROR, ET_ERROR, "multidimensional arrays for simple data types are not supported");
if (atoi(dimension) < 0)
- type = ECPGmake_simple_type($5.type_enum, mm_strdup("1"), 0);
+ type = ECPGmake_simple_type($5.type_enum, "1", 0);
else
- type = ECPGmake_array_type(ECPGmake_simple_type($5.type_enum, mm_strdup("1"), 0), dimension);
+ type = ECPGmake_array_type(ECPGmake_simple_type($5.type_enum, "1", 0), dimension);
break;
}
@@ -1663,7 +1678,7 @@ ECPGVar: SQL_VAR
p->type = type;
}
- @$ = cat_str(7, mm_strdup("/* exec sql var"), mm_strdup(@3), mm_strdup("is"), mm_strdup($5.type_str), mm_strdup($6.str), @7, mm_strdup("*/"));
+ @$ = cat_str(7, "/* exec sql var", @3, "is", $5.type_str, $6.str, @7, "*/");
}
;
@@ -1673,83 +1688,83 @@ ECPGVar: SQL_VAR
*/
ECPGWhenever: SQL_WHENEVER SQL_SQLERROR action
{
- when_error.code = $<action>3.code;
- when_error.command = $<action>3.command;
- @$ = cat_str(3, mm_strdup("/* exec sql whenever sqlerror "), $3.str, mm_strdup("; */"));
+ when_error.code = $3.code;
+ when_error.command = $3.command ? mm_strdup($3.command) : NULL;
+ @$ = cat_str(3, "/* exec sql whenever sqlerror ", $3.str, "; */");
}
| SQL_WHENEVER NOT SQL_FOUND action
{
- when_nf.code = $<action>4.code;
- when_nf.command = $<action>4.command;
- @$ = cat_str(3, mm_strdup("/* exec sql whenever not found "), $4.str, mm_strdup("; */"));
+ when_nf.code = $4.code;
+ when_nf.command = $4.command ? mm_strdup($4.command) : NULL;
+ @$ = cat_str(3, "/* exec sql whenever not found ", $4.str, "; */");
}
| SQL_WHENEVER SQL_SQLWARNING action
{
- when_warn.code = $<action>3.code;
- when_warn.command = $<action>3.command;
- @$ = cat_str(3, mm_strdup("/* exec sql whenever sql_warning "), $3.str, mm_strdup("; */"));
+ when_warn.code = $3.code;
+ when_warn.command = $3.command ? mm_strdup($3.command) : NULL;
+ @$ = cat_str(3, "/* exec sql whenever sql_warning ", $3.str, "; */");
}
;
action: CONTINUE_P
{
- $<action>$.code = W_NOTHING;
- $<action>$.command = NULL;
- $<action>$.str = mm_strdup("continue");
+ $$.code = W_NOTHING;
+ $$.command = NULL;
+ $$.str = "continue";
}
| SQL_SQLPRINT
{
- $<action>$.code = W_SQLPRINT;
- $<action>$.command = NULL;
- $<action>$.str = mm_strdup("sqlprint");
+ $$.code = W_SQLPRINT;
+ $$.command = NULL;
+ $$.str = "sqlprint";
}
| SQL_STOP
{
- $<action>$.code = W_STOP;
- $<action>$.command = NULL;
- $<action>$.str = mm_strdup("stop");
+ $$.code = W_STOP;
+ $$.command = NULL;
+ $$.str = "stop";
}
| SQL_GOTO name
{
- $<action>$.code = W_GOTO;
- $<action>$.command = mm_strdup(@2);
- $<action>$.str = cat2_str(mm_strdup("goto "), @2);
+ $$.code = W_GOTO;
+ $$.command = loc_strdup(@2);
+ $$.str = cat2_str("goto ", @2);
}
| SQL_GO TO name
{
- $<action>$.code = W_GOTO;
- $<action>$.command = mm_strdup(@3);
- $<action>$.str = cat2_str(mm_strdup("goto "), @3);
+ $$.code = W_GOTO;
+ $$.command = loc_strdup(@3);
+ $$.str = cat2_str("goto ", @3);
}
| DO name '(' c_args ')'
{
- $<action>$.code = W_DO;
- $<action>$.command = cat_str(4, @2, mm_strdup("("), @4, mm_strdup(")"));
- $<action>$.str = cat2_str(mm_strdup("do"), mm_strdup($<action>$.command));
+ $$.code = W_DO;
+ $$.command = cat_str(4, @2, "(", @4, ")");
+ $$.str = cat2_str("do", $$.command);
}
| DO SQL_BREAK
{
- $<action>$.code = W_BREAK;
- $<action>$.command = NULL;
- $<action>$.str = mm_strdup("break");
+ $$.code = W_BREAK;
+ $$.command = NULL;
+ $$.str = "break";
}
| DO CONTINUE_P
{
- $<action>$.code = W_CONTINUE;
- $<action>$.command = NULL;
- $<action>$.str = mm_strdup("continue");
+ $$.code = W_CONTINUE;
+ $$.command = NULL;
+ $$.str = "continue";
}
| CALL name '(' c_args ')'
{
- $<action>$.code = W_DO;
- $<action>$.command = cat_str(4, @2, mm_strdup("("), @4, mm_strdup(")"));
- $<action>$.str = cat2_str(mm_strdup("call"), mm_strdup($<action>$.command));
+ $$.code = W_DO;
+ $$.command = cat_str(4, @2, "(", @4, ")");
+ $$.str = cat2_str("call", $$.command);
}
| CALL name
{
- $<action>$.code = W_DO;
- $<action>$.command = cat2_str(@2, mm_strdup("()"));
- $<action>$.str = cat2_str(mm_strdup("call"), mm_strdup($<action>$.command));
+ $$.code = W_DO;
+ $$.command = cat2_str(@2, "()");
+ $$.str = cat2_str("call", $$.command);
}
;
@@ -1913,7 +1928,7 @@ ecpgstart: SQL_START
{
reset_variables();
pacounter = 1;
- @$ = EMPTY;
+ @$ = "";
}
;
@@ -1982,7 +1997,7 @@ cvariable: CVARIABLE
* As long as multidimensional arrays are not implemented we have to
* check for those here
*/
- char *ptr = @1;
+ const char *ptr = @1;
int brace_open = 0,
brace = false;
@@ -2013,18 +2028,12 @@ cvariable: CVARIABLE
;
ecpg_param: PARAM
- {
- @$ = make_name();
- }
;
ecpg_bconst: BCONST
;
ecpg_fconst: FCONST
- {
- @$ = make_name();
- }
;
ecpg_sconst: SCONST
@@ -2036,17 +2045,17 @@ ecpg_xconst: XCONST
ecpg_ident: IDENT
| CSTRING
{
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ @$ = make3_str("\"", @1, "\"");
}
;
quoted_ident_stringvar: name
{
- @$ = make3_str(mm_strdup("\""), @1, mm_strdup("\""));
+ @$ = make3_str("\"", @1, "\"");
}
| char_variable
{
- @$ = make3_str(mm_strdup("("), @1, mm_strdup(")"));
+ @$ = make3_str("(", @1, ")");
}
;
@@ -2057,7 +2066,7 @@ quoted_ident_stringvar: name
c_stuff_item: c_anything
| '(' ')'
{
- @$ = mm_strdup("()");
+ @$ = "()";
}
| '(' c_stuff ')'
;
@@ -2094,7 +2103,7 @@ c_anything: ecpg_ident
| '-'
| '/'
| '%'
- | NULL_P { @$ = mm_strdup("NULL"); }
+ | NULL_P { @$ = "NULL"; }
| S_ADD
| S_AND
| S_ANYTHING
@@ -2155,11 +2164,11 @@ DeallocateStmt: DEALLOCATE prepared_name
}
| DEALLOCATE ALL
{
- @$ = mm_strdup("all");
+ @$ = "all";
}
| DEALLOCATE PREPARE ALL
{
- @$ = mm_strdup("all");
+ @$ = "all";
}
;
@@ -2177,7 +2186,7 @@ Iresult: Iconst
if (pg_strcasecmp(@1, "sizeof") != 0)
mmerror(PARSE_ERROR, ET_ERROR, "operator not allowed in variable definition");
else
- @$ = cat_str(4, @1, mm_strdup("("), $3.type_str, mm_strdup(")"));
+ @$ = cat_str(4, @1, "(", $3.type_str, ")");
}
;
@@ -2190,7 +2199,7 @@ execute_rest: /* EMPTY */
ecpg_into: INTO into_list
{
/* always suppress this from the constructed string */
- @$ = EMPTY;
+ @$ = "";
}
| into_descriptor
;
diff --git a/src/interfaces/ecpg/preproc/output.c b/src/interfaces/ecpg/preproc/output.c
index 8d2b6e7cb8..a18904f88b 100644
--- a/src/interfaces/ecpg/preproc/output.c
+++ b/src/interfaces/ecpg/preproc/output.c
@@ -12,7 +12,6 @@ output_line_number(void)
char *line = hashline_number();
fprintf(base_yyout, "%s", line);
- free(line);
}
void
@@ -100,7 +99,7 @@ hashline_number(void)
)
{
/* "* 2" here is for escaping '\' and '"' below */
- char *line = mm_alloc(strlen("\n#line %d \"%s\"\n") + sizeof(int) * CHAR_BIT * 10 / 3 + strlen(input_filename) * 2);
+ char *line = loc_alloc(strlen("\n#line %d \"%s\"\n") + sizeof(int) * CHAR_BIT * 10 / 3 + strlen(input_filename) * 2);
char *src,
*dest;
@@ -119,7 +118,7 @@ hashline_number(void)
return line;
}
- return EMPTY;
+ return "";
}
static char *ecpg_statement_type_name[] = {
diff --git a/src/interfaces/ecpg/preproc/parser.c b/src/interfaces/ecpg/preproc/parser.c
index 8807c22cb6..78eeb78466 100644
--- a/src/interfaces/ecpg/preproc/parser.c
+++ b/src/interfaces/ecpg/preproc/parser.c
@@ -204,7 +204,7 @@ filtered_base_yylex(void)
/* Combine 3 tokens into 1 */
base_yylval.str = psprintf("%s UESCAPE %s", base_yylval.str, escstr);
- base_yylloc = mm_strdup(base_yylval.str);
+ base_yylloc = loc_strdup(base_yylval.str);
/* Clear have_lookahead, thereby consuming all three tokens */
have_lookahead = false;
@@ -254,11 +254,11 @@ base_yylex_location(void)
case UIDENT:
case IP:
/* Duplicate the <str> value */
- base_yylloc = mm_strdup(base_yylval.str);
+ base_yylloc = loc_strdup(base_yylval.str);
break;
default:
/* Else just use the input, i.e., yytext */
- base_yylloc = mm_strdup(base_yytext);
+ base_yylloc = loc_strdup(base_yytext);
/* Apply an ASCII-only downcasing */
for (unsigned char *ptr = (unsigned char *) base_yylloc; *ptr; ptr++)
{
diff --git a/src/interfaces/ecpg/preproc/preproc_extern.h b/src/interfaces/ecpg/preproc/preproc_extern.h
index 29329ccd89..a60b0381fb 100644
--- a/src/interfaces/ecpg/preproc/preproc_extern.h
+++ b/src/interfaces/ecpg/preproc/preproc_extern.h
@@ -13,12 +13,11 @@
/* defines */
#define STRUCT_DEPTH 128
-#define EMPTY mm_strdup("")
/*
* "Location tracking" support --- see ecpg.header for more comments.
*/
-typedef char *YYLTYPE;
+typedef const char *YYLTYPE;
#define YYLTYPE_IS_DECLARED 1
@@ -82,22 +81,25 @@ extern int base_yylex(void);
extern void base_yyerror(const char *error);
extern void *mm_alloc(size_t size);
extern char *mm_strdup(const char *string);
-extern char *cat2_str(char *str1, char *str2);
+extern void *loc_alloc(size_t size);
+extern char *loc_strdup(const char *string);
+extern void reclaim_local_storage(void);
+extern char *cat2_str(const char *str1, const char *str2);
extern char *cat_str(int count,...);
-extern char *make2_str(char *str1, char *str2);
-extern char *make3_str(char *str1, char *str2, char *str3);
+extern char *make2_str(const char *str1, const char *str2);
+extern char *make3_str(const char *str1, const char *str2, const char *str3);
extern void mmerror(int error_code, enum errortype type, const char *error,...) pg_attribute_printf(3, 4);
extern void mmfatal(int error_code, const char *error,...) pg_attribute_printf(2, 3) pg_attribute_noreturn();
-extern void output_get_descr_header(char *desc_name);
-extern void output_get_descr(char *desc_name, char *index);
-extern void output_set_descr_header(char *desc_name);
-extern void output_set_descr(char *desc_name, char *index);
-extern void push_assignment(char *var, enum ECPGdtype value);
-extern struct variable *find_variable(char *name);
+extern void output_get_descr_header(const char *desc_name);
+extern void output_get_descr(const char *desc_name, const char *index);
+extern void output_set_descr_header(const char *desc_name);
+extern void output_set_descr(const char *desc_name, const char *index);
+extern void push_assignment(const char *var, enum ECPGdtype value);
+extern struct variable *find_variable(const char *name);
extern void whenever_action(int mode);
-extern void add_descriptor(char *name, char *connection);
-extern void drop_descriptor(char *name, char *connection);
-extern struct descriptor *lookup_descriptor(char *name, char *connection);
+extern void add_descriptor(const char *name, const char *connection);
+extern void drop_descriptor(const char *name, const char *connection);
+extern struct descriptor *lookup_descriptor(const char *name, const char *connection);
extern struct variable *descriptor_variable(const char *name, int input);
extern struct variable *sqlda_variable(const char *name);
extern void add_variable_to_head(struct arguments **list,
@@ -109,9 +111,9 @@ extern void add_variable_to_tail(struct arguments **list,
extern void remove_variable_from_list(struct arguments **list, struct variable *var);
extern void dump_variables(struct arguments *list, int mode);
extern struct typedefs *get_typedef(const char *name, bool noerror);
-extern void adjust_array(enum ECPGttype type_enum, char **dimension,
- char **length, char *type_dimension,
- char *type_index, int pointer_len,
+extern void adjust_array(enum ECPGttype type_enum, const char **dimension,
+ const char **length, const char *type_dimension,
+ const char *type_index, int pointer_len,
bool type_definition);
extern void reset_variables(void);
extern void check_indicator(struct ECPGtype *var);
diff --git a/src/interfaces/ecpg/preproc/type.c b/src/interfaces/ecpg/preproc/type.c
index 5610a8dc76..7f52521dbf 100644
--- a/src/interfaces/ecpg/preproc/type.c
+++ b/src/interfaces/ecpg/preproc/type.c
@@ -69,13 +69,13 @@ ECPGmake_struct_member(const char *name, struct ECPGtype *type, struct ECPGstruc
}
struct ECPGtype *
-ECPGmake_simple_type(enum ECPGttype type, char *size, int counter)
+ECPGmake_simple_type(enum ECPGttype type, const char *size, int counter)
{
struct ECPGtype *ne = (struct ECPGtype *) mm_alloc(sizeof(struct ECPGtype));
ne->type = type;
ne->type_name = NULL;
- ne->size = size;
+ ne->size = mm_strdup(size);
ne->u.element = NULL;
ne->struct_sizeof = NULL;
ne->counter = counter; /* only needed for varchar and bytea */
@@ -84,7 +84,7 @@ ECPGmake_simple_type(enum ECPGttype type, char *size, int counter)
}
struct ECPGtype *
-ECPGmake_array_type(struct ECPGtype *type, char *size)
+ECPGmake_array_type(struct ECPGtype *type, const char *size)
{
struct ECPGtype *ne = ECPGmake_simple_type(ECPGt_array, size, 0);
@@ -96,7 +96,7 @@ ECPGmake_array_type(struct ECPGtype *type, char *size)
struct ECPGtype *
ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type, char *type_name, char *struct_sizeof)
{
- struct ECPGtype *ne = ECPGmake_simple_type(type, mm_strdup("1"), 0);
+ struct ECPGtype *ne = ECPGmake_simple_type(type, "1", 0);
ne->type_name = mm_strdup(type_name);
ne->u.members = ECPGstruct_member_dup(rm);
diff --git a/src/interfaces/ecpg/preproc/type.h b/src/interfaces/ecpg/preproc/type.h
index ce2124361f..90126551d1 100644
--- a/src/interfaces/ecpg/preproc/type.h
+++ b/src/interfaces/ecpg/preproc/type.h
@@ -35,8 +35,8 @@ struct ECPGtype
/* Everything is malloced. */
void ECPGmake_struct_member(const char *name, struct ECPGtype *type,
struct ECPGstruct_member **start);
-struct ECPGtype *ECPGmake_simple_type(enum ECPGttype type, char *size, int counter);
-struct ECPGtype *ECPGmake_array_type(struct ECPGtype *type, char *size);
+struct ECPGtype *ECPGmake_simple_type(enum ECPGttype type, const char *size, int counter);
+struct ECPGtype *ECPGmake_array_type(struct ECPGtype *type, const char *size);
struct ECPGtype *ECPGmake_struct_type(struct ECPGstruct_member *rm,
enum ECPGttype type, char *type_name,
char *struct_sizeof);
@@ -93,28 +93,28 @@ struct when
struct index
{
- char *index1;
- char *index2;
- char *str;
+ const char *index1;
+ const char *index2;
+ const char *str;
};
struct su_symbol
{
- char *su;
- char *symbol;
+ const char *su;
+ const char *symbol;
};
struct prep
{
- char *name;
- char *stmt;
- char *type;
+ const char *name;
+ const char *stmt;
+ const char *type;
};
struct exec
{
- char *name;
- char *type;
+ const char *name;
+ const char *type;
};
struct this_type
@@ -221,14 +221,14 @@ enum errortype
struct fetch_desc
{
- char *str;
- char *name;
+ const char *str;
+ const char *name;
};
struct describe
{
int input;
- char *stmt_name;
+ const char *stmt_name;
};
#endif /* _ECPG_PREPROC_TYPE_H */
diff --git a/src/interfaces/ecpg/preproc/util.c b/src/interfaces/ecpg/preproc/util.c
index cb1eca7f3c..9672c12b29 100644
--- a/src/interfaces/ecpg/preproc/util.c
+++ b/src/interfaces/ecpg/preproc/util.c
@@ -104,33 +104,117 @@ mm_strdup(const char *string)
return new;
}
+
/*
- * String concatenation
+ * "Local" (or "location"?) memory management support
+ *
+ * These functions manage memory that is only needed for a short time
+ * (processing of one input statement) within the ecpg grammar.
+ * Data allocated with these is not meant to be freed separately;
+ * rather it's freed by calling reclaim_local_storage() at the end
+ * of each statement cycle.
*/
+typedef struct loc_chunk
+{
+ struct loc_chunk *next; /* list link */
+ unsigned int chunk_used; /* index of first unused byte in data[] */
+ unsigned int chunk_avail; /* # bytes still available in data[] */
+ char data[FLEXIBLE_ARRAY_MEMBER]; /* actual storage */
+} loc_chunk;
+
+#define LOC_CHUNK_OVERHEAD MAXALIGN(offsetof(loc_chunk, data))
+#define LOC_CHUNK_MIN_SIZE 8192
+
+/* Head of list of loc_chunks */
+static loc_chunk *loc_chunks = NULL;
+
/*
- * Concatenate 2 strings, inserting a space between them unless either is empty
+ * Allocate local space of the requested size.
*
- * The input strings are freed.
+ * Exits on OOM.
+ */
+void *
+loc_alloc(size_t size)
+{
+ void *result;
+ loc_chunk *cur_chunk = loc_chunks;
+
+ /* Ensure all allocations are adequately aligned */
+ size = MAXALIGN(size);
+
+ /* Need a new chunk? */
+ if (cur_chunk == NULL || size > cur_chunk->chunk_avail)
+ {
+ size_t chunk_size = Max(size, LOC_CHUNK_MIN_SIZE);
+
+ cur_chunk = mm_alloc(chunk_size + LOC_CHUNK_OVERHEAD);
+ /* Depending on alignment rules, we could waste a bit here */
+ cur_chunk->chunk_used = LOC_CHUNK_OVERHEAD - offsetof(loc_chunk, data);
+ cur_chunk->chunk_avail = chunk_size;
+ /* New chunk becomes the head of the list */
+ cur_chunk->next = loc_chunks;
+ loc_chunks = cur_chunk;
+ }
+
+ result = cur_chunk->data + cur_chunk->chunk_used;
+ cur_chunk->chunk_used += size;
+ cur_chunk->chunk_avail -= size;
+ return result;
+}
+
+/*
+ * Copy given string into local storage
+ */
+char *
+loc_strdup(const char *string)
+{
+ char *result = loc_alloc(strlen(string) + 1);
+
+ strcpy(result, string);
+ return result;
+}
+
+/*
+ * Reclaim local storage when appropriate
+ */
+void
+reclaim_local_storage(void)
+{
+ loc_chunk *cur_chunk,
+ *next_chunk;
+
+ for (cur_chunk = loc_chunks; cur_chunk; cur_chunk = next_chunk)
+ {
+ next_chunk = cur_chunk->next;
+ free(cur_chunk);
+ }
+ loc_chunks = NULL;
+}
+
+
+/*
+ * String concatenation support routines. These return "local" (transient)
+ * storage.
+ */
+
+/*
+ * Concatenate 2 strings, inserting a space between them unless either is empty
*/
char *
-cat2_str(char *str1, char *str2)
+cat2_str(const char *str1, const char *str2)
{
- char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + 2);
+ char *res_str = (char *) loc_alloc(strlen(str1) + strlen(str2) + 2);
strcpy(res_str, str1);
if (strlen(str1) != 0 && strlen(str2) != 0)
strcat(res_str, " ");
strcat(res_str, str2);
- free(str1);
- free(str2);
return res_str;
}
/*
* Concatenate N strings, inserting spaces between them unless they are empty
- *
- * The input strings are freed.
*/
char *
cat_str(int count,...)
@@ -154,36 +238,27 @@ cat_str(int count,...)
/*
* Concatenate 2 strings, with no space between
- *
- * The input strings are freed.
*/
char *
-make2_str(char *str1, char *str2)
+make2_str(const char *str1, const char *str2)
{
- char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + 1);
+ char *res_str = (char *) loc_alloc(strlen(str1) + strlen(str2) + 1);
strcpy(res_str, str1);
strcat(res_str, str2);
- free(str1);
- free(str2);
return res_str;
}
/*
* Concatenate 3 strings, with no space between
- *
- * The input strings are freed.
*/
char *
-make3_str(char *str1, char *str2, char *str3)
+make3_str(const char *str1, const char *str2, const char *str3)
{
- char *res_str = (char *) mm_alloc(strlen(str1) + strlen(str2) + strlen(str3) + 1);
+ char *res_str = (char *) loc_alloc(strlen(str1) + strlen(str2) + strlen(str3) + 1);
strcpy(res_str, str1);
strcat(res_str, str2);
strcat(res_str, str3);
- free(str1);
- free(str2);
- free(str3);
return res_str;
}
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index b23ed5edf4..6b87d5ff3d 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -22,7 +22,7 @@ new_variable(const char *name, struct ECPGtype *type, int brace_level)
}
static struct variable *
-find_struct_member(char *name, char *str, struct ECPGstruct_member *members, int brace_level)
+find_struct_member(const char *name, char *str, struct ECPGstruct_member *members, int brace_level)
{
char *next = strpbrk(++str, ".-["),
*end,
@@ -123,7 +123,7 @@ find_struct_member(char *name, char *str, struct ECPGstruct_member *members, int
}
static struct variable *
-find_struct(char *name, char *next, char *end)
+find_struct(const char *name, char *next, char *end)
{
struct variable *p;
char c = *next;
@@ -174,7 +174,7 @@ find_struct(char *name, char *next, char *end)
}
static struct variable *
-find_simple(char *name)
+find_simple(const char *name)
{
struct variable *p;
@@ -190,7 +190,7 @@ find_simple(char *name)
/* Note that this function will end the program in case of an unknown */
/* variable */
struct variable *
-find_variable(char *name)
+find_variable(const char *name)
{
char *next,
*end;
@@ -513,7 +513,10 @@ get_typedef(const char *name, bool noerror)
}
void
-adjust_array(enum ECPGttype type_enum, char **dimension, char **length, char *type_dimension, char *type_index, int pointer_len, bool type_definition)
+adjust_array(enum ECPGttype type_enum,
+ const char **dimension, const char **length,
+ const char *type_dimension, const char *type_index,
+ int pointer_len, bool type_definition)
{
if (atoi(type_index) >= 0)
{
@@ -556,7 +559,7 @@ adjust_array(enum ECPGttype type_enum, char **dimension, char **length, char *ty
if (pointer_len)
{
*length = *dimension;
- *dimension = mm_strdup("0");
+ *dimension = "0";
}
if (atoi(*length) >= 0)
@@ -567,13 +570,13 @@ adjust_array(enum ECPGttype type_enum, char **dimension, char **length, char *ty
case ECPGt_bytea:
/* pointer has to get dimension 0 */
if (pointer_len)
- *dimension = mm_strdup("0");
+ *dimension = "0";
/* one index is the string length */
if (atoi(*length) < 0)
{
*length = *dimension;
- *dimension = mm_strdup("-1");
+ *dimension = "-1";
}
break;
@@ -583,13 +586,13 @@ adjust_array(enum ECPGttype type_enum, char **dimension, char **length, char *ty
/* char ** */
if (pointer_len == 2)
{
- *length = *dimension = mm_strdup("0");
+ *length = *dimension = "0";
break;
}
/* pointer has to get length 0 */
if (pointer_len == 1)
- *length = mm_strdup("0");
+ *length = "0";
/* one index is the string length */
if (atoi(*length) < 0)
@@ -604,13 +607,13 @@ adjust_array(enum ECPGttype type_enum, char **dimension, char **length, char *ty
* do not change this for typedefs since it will be
* changed later on when the variable is defined
*/
- *length = mm_strdup("1");
+ *length = "1";
else if (strcmp(*dimension, "0") == 0)
- *length = mm_strdup("-1");
+ *length = "-1";
else
*length = *dimension;
- *dimension = mm_strdup("-1");
+ *dimension = "-1";
}
break;
default:
@@ -618,7 +621,7 @@ adjust_array(enum ECPGttype type_enum, char **dimension, char **length, char *ty
if (pointer_len)
{
*length = *dimension;
- *dimension = mm_strdup("0");
+ *dimension = "0";
}
if (atoi(*length) >= 0)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c4de597b1f..c6b06c0ef9 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3601,6 +3601,7 @@ libpq_source
line_t
lineno_t
list_sort_comparator
+loc_chunk
local_relopt
local_relopts
local_source
--
2.43.5
[text/x-diff] v5-0006-Fix-some-memory-leakage-of-data-type-related-stru.patch (8.6K, ../../[email protected]/7-v5-0006-Fix-some-memory-leakage-of-data-type-related-stru.patch)
download | inline diff:
From 7962cd2e4585d7446a4bc81140bd65874cd7d581 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:19:53 -0400
Subject: [PATCH v5 6/9] Fix some memory leakage of data-type-related
structures.
ECPGfree_type() and related functions were quite incomplete
about removing subsidiary data structures. Possibly this is
because ecpg wasn't careful to make sure said data structures
always had their own storage. Previous patches in this series
cleaned up a lot of that, and I had to add a couple more
mm_strdup's here.
Also, ecpg.trailer tended to overwrite struct_member_list[struct_level]
without bothering to free up its previous contents, thus potentially
leaking a lot of struct-member-related storage. Add
ECPGfree_struct_member() calls at appropriate points. (Note: the
lifetime of those lists is not obvious. They are still live after
initial construction, in order to handle cases like
struct foo { ... } foovar1, foovar2;
We can't delete the list immediately after parsing the right brace,
because it needs to be copied for each of the variables. Instead,
it's kept around until the next struct declaration.)
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/ecpg.header | 3 ++-
src/interfaces/ecpg/preproc/ecpg.trailer | 11 +++++++++--
src/interfaces/ecpg/preproc/type.c | 15 +++++++++------
src/interfaces/ecpg/preproc/type.h | 5 +++--
src/interfaces/ecpg/preproc/variable.c | 7 ++++++-
5 files changed, 29 insertions(+), 12 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index d3df8eabbb..d54eca918d 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -506,11 +506,12 @@ add_typedef(const char *name, const char *dimension, const char *length,
this->name = mm_strdup(name);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
+ this->type->type_storage = NULL;
this->type->type_enum = type_enum;
this->type->type_str = mm_strdup(name);
this->type->type_dimension = mm_strdup(dimension); /* dimension of array */
this->type->type_index = mm_strdup(length); /* length of string */
- this->type->type_sizeof = ECPGstruct_sizeof;
+ this->type->type_sizeof = ECPGstruct_sizeof ? mm_strdup(ECPGstruct_sizeof) : NULL;
this->struct_member_list = (type_enum == ECPGt_struct || type_enum == ECPGt_union) ?
ECPGstruct_member_dup(struct_member_list[struct_level]) : NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index 0a77559e83..41029701fc 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -751,6 +751,7 @@ var_type: simple_type
else
$$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
}
@@ -878,6 +879,7 @@ var_type: simple_type
else
$$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
}
@@ -900,6 +902,7 @@ var_type: simple_type
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
$$.type_sizeof = this->type->type_sizeof;
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
else
@@ -909,6 +912,7 @@ var_type: simple_type
$$.type_dimension = "-1";
$$.type_index = "-1";
$$.type_sizeof = "";
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = NULL;
}
}
@@ -924,6 +928,7 @@ enum_definition: '{' c_list '}'
struct_union_type_with_symbol: s_struct_union_symbol
{
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level++] = NULL;
if (struct_level >= STRUCT_DEPTH)
mmerror(PARSE_ERROR, ET_ERROR, "too many levels in nested structure/union definition");
@@ -965,12 +970,13 @@ struct_union_type_with_symbol: s_struct_union_symbol
this->name = mm_strdup(su_type.type_str);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
+ this->type->type_storage = NULL;
this->type->type_enum = su_type.type_enum;
this->type->type_str = mm_strdup(su_type.type_str);
this->type->type_dimension = mm_strdup("-1"); /* dimension of array */
this->type->type_index = mm_strdup("-1"); /* length of string */
- this->type->type_sizeof = ECPGstruct_sizeof;
- this->struct_member_list = struct_member_list[struct_level];
+ this->type->type_sizeof = ECPGstruct_sizeof ? mm_strdup(ECPGstruct_sizeof) : NULL;
+ this->struct_member_list = ECPGstruct_member_dup(struct_member_list[struct_level]);
types = this;
@$ = cat_str(4, su_type.type_str, "{", @4, "}");
@@ -980,6 +986,7 @@ struct_union_type_with_symbol: s_struct_union_symbol
struct_union_type: struct_union_type_with_symbol
| s_struct_union
{
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level++] = NULL;
if (struct_level >= STRUCT_DEPTH)
mmerror(PARSE_ERROR, ET_ERROR, "too many levels in nested structure/union definition");
diff --git a/src/interfaces/ecpg/preproc/type.c b/src/interfaces/ecpg/preproc/type.c
index 7f52521dbf..9f6dacd2ae 100644
--- a/src/interfaces/ecpg/preproc/type.c
+++ b/src/interfaces/ecpg/preproc/type.c
@@ -94,13 +94,14 @@ ECPGmake_array_type(struct ECPGtype *type, const char *size)
}
struct ECPGtype *
-ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type, char *type_name, char *struct_sizeof)
+ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type,
+ const char *type_name, const char *struct_sizeof)
{
struct ECPGtype *ne = ECPGmake_simple_type(type, "1", 0);
ne->type_name = mm_strdup(type_name);
ne->u.members = ECPGstruct_member_dup(rm);
- ne->struct_sizeof = struct_sizeof;
+ ne->struct_sizeof = mm_strdup(struct_sizeof);
return ne;
}
@@ -622,7 +623,7 @@ ECPGfree_struct_member(struct ECPGstruct_member *rm)
rm = rm->next;
free(p->name);
- free(p->type);
+ ECPGfree_type(p->type);
free(p);
}
}
@@ -643,14 +644,13 @@ ECPGfree_type(struct ECPGtype *type)
case ECPGt_struct:
case ECPGt_union:
/* Array of structs. */
- ECPGfree_struct_member(type->u.element->u.members);
- free(type->u.element);
+ ECPGfree_type(type->u.element);
break;
default:
if (!IS_SIMPLE_TYPE(type->u.element->type))
base_yyerror("internal error: unknown datatype, please report this to <" PACKAGE_BUGREPORT ">");
- free(type->u.element);
+ ECPGfree_type(type->u.element);
}
break;
case ECPGt_struct:
@@ -662,6 +662,9 @@ ECPGfree_type(struct ECPGtype *type)
break;
}
}
+ free(type->type_name);
+ free(type->size);
+ free(type->struct_sizeof);
free(type);
}
diff --git a/src/interfaces/ecpg/preproc/type.h b/src/interfaces/ecpg/preproc/type.h
index 90126551d1..3d99e1703d 100644
--- a/src/interfaces/ecpg/preproc/type.h
+++ b/src/interfaces/ecpg/preproc/type.h
@@ -38,8 +38,9 @@ void ECPGmake_struct_member(const char *name, struct ECPGtype *type,
struct ECPGtype *ECPGmake_simple_type(enum ECPGttype type, const char *size, int counter);
struct ECPGtype *ECPGmake_array_type(struct ECPGtype *type, const char *size);
struct ECPGtype *ECPGmake_struct_type(struct ECPGstruct_member *rm,
- enum ECPGttype type, char *type_name,
- char *struct_sizeof);
+ enum ECPGttype type,
+ const char *type_name,
+ const char *struct_sizeof);
struct ECPGstruct_member *ECPGstruct_member_dup(struct ECPGstruct_member *rm);
/* Frees a type. */
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index 6b87d5ff3d..4831f56cba 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -273,7 +273,12 @@ remove_typedefs(int brace_level)
prev->next = p->next;
if (p->type->type_enum == ECPGt_struct || p->type->type_enum == ECPGt_union)
- free(p->struct_member_list);
+ ECPGfree_struct_member(p->struct_member_list);
+ free(p->type->type_storage);
+ free(p->type->type_str);
+ free(p->type->type_dimension);
+ free(p->type->type_index);
+ free(p->type->type_sizeof);
free(p->type);
free(p->name);
free(p);
--
2.43.5
[text/x-diff] v5-0007-Make-all-string-valued-tokens-returned-by-pgc.l-b.patch (7.1K, ../../[email protected]/8-v5-0007-Make-all-string-valued-tokens-returned-by-pgc.l-b.patch)
download | inline diff:
From c44995cb40feda171bf2debc65f16095d158e44f Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:22:44 -0400
Subject: [PATCH v5 7/9] Make all string-valued tokens returned by pgc.l be
local storage.
This didn't work earlier in the patch series (I think some of
the strings were ending up in data-type-related structures),
but apparently we're now clean enough for it. This considerably
reduces process-lifespan memory leakage.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/parser.c | 4 ++-
src/interfaces/ecpg/preproc/pgc.l | 42 ++++++++++++++--------------
2 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/parser.c b/src/interfaces/ecpg/preproc/parser.c
index 78eeb78466..0a14e3714a 100644
--- a/src/interfaces/ecpg/preproc/parser.c
+++ b/src/interfaces/ecpg/preproc/parser.c
@@ -203,7 +203,9 @@ filtered_base_yylex(void)
base_yytext = cur_yytext;
/* Combine 3 tokens into 1 */
- base_yylval.str = psprintf("%s UESCAPE %s", base_yylval.str, escstr);
+ base_yylval.str = make3_str(base_yylval.str,
+ " UESCAPE ",
+ escstr);
base_yylloc = loc_strdup(base_yylval.str);
/* Clear have_lookahead, thereby consuming all three tokens */
diff --git a/src/interfaces/ecpg/preproc/pgc.l b/src/interfaces/ecpg/preproc/pgc.l
index f3c03482ae..82708013ee 100644
--- a/src/interfaces/ecpg/preproc/pgc.l
+++ b/src/interfaces/ecpg/preproc/pgc.l
@@ -641,26 +641,26 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
case xb:
if (literalbuf[strspn(literalbuf, "01")] != '\0')
mmerror(PARSE_ERROR, ET_ERROR, "invalid bit string literal");
- base_yylval.str = psprintf("b'%s'", literalbuf);
+ base_yylval.str = make3_str("b'", literalbuf, "'");
return BCONST;
case xh:
if (literalbuf[strspn(literalbuf, "0123456789abcdefABCDEF")] != '\0')
mmerror(PARSE_ERROR, ET_ERROR, "invalid hexadecimal string literal");
- base_yylval.str = psprintf("x'%s'", literalbuf);
+ base_yylval.str = make3_str("x'", literalbuf, "'");
return XCONST;
case xq:
/* fallthrough */
case xqc:
- base_yylval.str = psprintf("'%s'", literalbuf);
+ base_yylval.str = make3_str("'", literalbuf, "'");
return SCONST;
case xe:
- base_yylval.str = psprintf("E'%s'", literalbuf);
+ base_yylval.str = make3_str("E'", literalbuf, "'");
return SCONST;
case xn:
- base_yylval.str = psprintf("N'%s'", literalbuf);
+ base_yylval.str = make3_str("N'", literalbuf, "'");
return SCONST;
case xus:
- base_yylval.str = psprintf("U&'%s'", literalbuf);
+ base_yylval.str = make3_str("U&'", literalbuf, "'");
return USCONST;
default:
mmfatal(PARSE_ERROR, "unhandled previous state in xqs\n");
@@ -724,7 +724,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
free(dolqstart);
dolqstart = NULL;
BEGIN(SQL);
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return SCONST;
}
else
@@ -778,12 +778,12 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* PREPARE and EXECUTE IMMEDIATE, which can certainly be
* longer than NAMEDATALEN.
*/
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return CSTRING;
}
<xdc>{xdstop} {
BEGIN(state_before_str_start);
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return CSTRING;
}
<xui>{dquote} {
@@ -795,7 +795,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* The backend will truncate the identifier here. We do
* not as it does not change the result.
*/
- base_yylval.str = psprintf("U&\"%s\"", literalbuf);
+ base_yylval.str = make3_str("U&\"", literalbuf, "\"");
return UIDENT;
}
<xd,xui>{xddouble} {
@@ -971,7 +971,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
}
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return Op;
}
@@ -990,7 +990,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
{ip} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IP;
}
} /* <SQL> */
@@ -1003,7 +1003,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return process_integer_literal(yytext, &base_yylval, 16);
}
{numeric} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return FCONST;
}
{numericfail} {
@@ -1012,7 +1012,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return process_integer_literal(yytext, &base_yylval, 10);
}
{real} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return FCONST;
}
{realfail} {
@@ -1048,7 +1048,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
:{identifier}((("->"|\.){identifier})|(\[{array}\]))* {
- base_yylval.str = mm_strdup(yytext + 1);
+ base_yylval.str = loc_strdup(yytext + 1);
return CVARIABLE;
}
@@ -1085,7 +1085,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* to do so; that's just another way that ecpg could
* get out of step with the backend.
*/
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IDENT;
}
}
@@ -1124,7 +1124,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
}
@@ -1136,12 +1136,12 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
}
<C,SQL>{cppline} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
<C>{identifier} {
@@ -1167,7 +1167,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return kwvalue;
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IDENT;
}
}
@@ -1685,7 +1685,7 @@ process_integer_literal(const char *token, YYSTYPE *lval, int base)
if (*endptr != '\0' || errno == ERANGE)
{
/* integer too large (or contains decimal pt), treat it as a float */
- lval->str = mm_strdup(token);
+ lval->str = loc_strdup(token);
return FCONST;
}
lval->ival = val;
--
2.43.5
[text/x-diff] v5-0008-Clean-up-some-other-assorted-ecpg-memory-leaks.patch (16.1K, ../../[email protected]/9-v5-0008-Clean-up-some-other-assorted-ecpg-memory-leaks.patch)
download | inline diff:
From 65480c05d2d62706fdcfe3015e29b0148ae86fe9 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:26:07 -0400
Subject: [PATCH v5 8/9] Clean up some other assorted ecpg memory leaks.
Avoid leaking the prior value when updating the "connection"
state variable.
Ditto for ECPGstruct_sizeof. (It seems like this one ought to
be statement-local, but testing says it isn't, and I didn't
feel like diving deeper.)
The actual_type[] entries are statement-local, though, so
no need to mm_strdup() strings stored in them.
Likewise, sqlda variables are statement-local, so we can
loc_alloc them.
Also clean up sloppiness around management of the argsinsert and
argsresult lists.
progname changes are strictly to prevent valgrind from complaining
about leaked allocations.
With this, valgrind reports zero leakage in ecpg for all of our
ecpg regression test cases.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/descriptor.c | 14 +++++--
src/interfaces/ecpg/preproc/ecpg.addons | 47 +++++++++---------------
src/interfaces/ecpg/preproc/ecpg.c | 2 +-
src/interfaces/ecpg/preproc/ecpg.header | 17 ++++++++-
src/interfaces/ecpg/preproc/ecpg.trailer | 31 +++++++++-------
src/interfaces/ecpg/preproc/output.c | 3 +-
src/interfaces/ecpg/preproc/variable.c | 25 +++++++++++--
7 files changed, 86 insertions(+), 53 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/descriptor.c b/src/interfaces/ecpg/preproc/descriptor.c
index 9b87d07d09..e8c7016bdc 100644
--- a/src/interfaces/ecpg/preproc/descriptor.c
+++ b/src/interfaces/ecpg/preproc/descriptor.c
@@ -344,11 +344,17 @@ descriptor_variable(const char *name, int input)
struct variable *
sqlda_variable(const char *name)
{
- struct variable *p = (struct variable *) mm_alloc(sizeof(struct variable));
-
- p->name = mm_strdup(name);
- p->type = (struct ECPGtype *) mm_alloc(sizeof(struct ECPGtype));
+ /*
+ * Presently, sqlda variables are only needed for the duration of the
+ * current statement. Rather than add infrastructure to manage them,
+ * let's just loc_alloc them.
+ */
+ struct variable *p = (struct variable *) loc_alloc(sizeof(struct variable));
+
+ p->name = loc_strdup(name);
+ p->type = (struct ECPGtype *) loc_alloc(sizeof(struct ECPGtype));
p->type->type = ECPGt_sqlda;
+ p->type->type_name = NULL;
p->type->size = NULL;
p->type->struct_sizeof = NULL;
p->type->u.element = NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index 9c120fead2..05de4ff1f1 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -135,6 +135,7 @@ ECPG: stmtViewStmt rule
fprintf(base_yyout, "{ ECPGdescribe(__LINE__, %d, %d, %s, %s,", compat, $1.input, connection ? connection : "NULL", $1.stmt_name);
dump_variables(argsresult, 1);
+ argsresult = NULL;
fputs("ECPGt_EORT);", base_yyout);
fprintf(base_yyout, "}");
output_line_number();
@@ -181,6 +182,7 @@ ECPG: stmtViewStmt rule
if ((ptr = add_additional_variables(@1, true)) != NULL)
{
+ free(connection);
connection = ptr->connection ? mm_strdup(ptr->connection) : NULL;
output_statement(ptr->command, 0, ECPGst_normal);
ptr->opened = true;
@@ -247,15 +249,13 @@ ECPG: var_valueNumericOnly addon
ECPG: fetch_argscursor_name addon
struct cursor *ptr = add_additional_variables(@1, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@1[0] == ':')
@$ = "$0";
ECPG: fetch_argsfrom_incursor_name addon
struct cursor *ptr = add_additional_variables(@2, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@2[0] == ':')
@$ = cat2_str(@1, "$0");
ECPG: fetch_argsNEXTopt_from_incursor_name addon
@@ -265,16 +265,14 @@ ECPG: fetch_argsLAST_Popt_from_incursor_name addon
ECPG: fetch_argsALLopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@3[0] == ':')
@$ = cat_str(3, @1, @2, "$0");
ECPG: fetch_argsSignedIconstopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@3, false);
bool replace = false;
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@3[0] == ':')
{
@3 = "$0";
@@ -291,8 +289,7 @@ ECPG: fetch_argsFORWARDALLopt_from_incursor_name addon
ECPG: fetch_argsBACKWARDALLopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@4[0] == ':')
@$ = cat_str(4, @1, @2, @3, "$0");
ECPG: fetch_argsABSOLUTE_PSignedIconstopt_from_incursor_name addon
@@ -302,8 +299,7 @@ ECPG: fetch_argsBACKWARDSignedIconstopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@4, false);
bool replace = false;
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@4[0] == ':')
{
@4 = "$0";
@@ -412,8 +408,7 @@ ECPG: ClosePortalStmtCLOSEcursor_name block
{
if (strcmp(@2, ptr->name) == 0)
{
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
break;
}
}
@@ -483,8 +478,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch forward", cursor_marker);
}
@@ -493,8 +487,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch forward from", cursor_marker);
}
@@ -503,8 +496,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch backward", cursor_marker);
}
@@ -513,8 +505,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch backward from", cursor_marker);
}
@@ -523,8 +514,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move forward", cursor_marker);
}
@@ -533,8 +523,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move forward from", cursor_marker);
}
@@ -543,8 +532,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move backward", cursor_marker);
}
@@ -553,8 +541,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move backward from", cursor_marker);
}
diff --git a/src/interfaces/ecpg/preproc/ecpg.c b/src/interfaces/ecpg/preproc/ecpg.c
index 73c37631ac..2fcc6f8f99 100644
--- a/src/interfaces/ecpg/preproc/ecpg.c
+++ b/src/interfaces/ecpg/preproc/ecpg.c
@@ -20,6 +20,7 @@ bool autocommit = false,
regression_mode = false,
auto_prepare = false;
+static const char *progname;
char *output_filename;
enum COMPAT_MODE compat = ECPG_COMPAT_PGSQL;
@@ -139,7 +140,6 @@ main(int argc, char *const argv[])
bool verbose = false,
header_mode = false;
struct _include_path *ip;
- const char *progname;
char my_exec_path[MAXPGPATH];
char include_path[MAXPGPATH];
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index d54eca918d..a9a0ef9847 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -59,6 +59,7 @@ struct variable no_indicator = {"no_indicator", &ecpg_no_indicator, 0, NULL};
static struct ECPGtype ecpg_query = {ECPGt_char_variable, NULL, NULL, NULL, {NULL}, 0};
static bool check_declared_list(const char *name);
+static void update_connection(const char *newconn);
/*
@@ -545,12 +546,26 @@ check_declared_list(const char *name)
{
if (connection && strcmp(ptr->connection, connection) != 0)
mmerror(PARSE_ERROR, ET_WARNING, "connection %s is overwritten with %s by DECLARE statement %s", connection, ptr->connection, name);
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
return true;
}
}
return false;
}
+
+/*
+ * If newconn isn't NULL, update the global "connection" variable to that;
+ * otherwise do nothing.
+ */
+static void
+update_connection(const char *newconn)
+{
+ if (newconn)
+ {
+ free(connection);
+ connection = mm_strdup(newconn);
+ }
+}
%}
%expect 0
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index 41029701fc..a90b1771fc 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -74,6 +74,8 @@ CreateAsStmt: CREATE OptTemp TABLE create_as_target AS
at: AT connection_object
{
+ if (connection)
+ free(connection);
connection = mm_strdup(@2);
/*
@@ -556,13 +558,12 @@ type_declaration: S_TYPEDEF
var_declaration:
storage_declaration var_type
{
- actual_type[struct_level].type_storage = mm_strdup(@1);
+ actual_type[struct_level].type_storage = loc_strdup(@1);
actual_type[struct_level].type_enum = $2.type_enum;
- actual_type[struct_level].type_str = mm_strdup($2.type_str);
- actual_type[struct_level].type_dimension = mm_strdup($2.type_dimension);
- actual_type[struct_level].type_index = mm_strdup($2.type_index);
- actual_type[struct_level].type_sizeof =
- $2.type_sizeof ? mm_strdup($2.type_sizeof) : NULL;
+ actual_type[struct_level].type_str = $2.type_str;
+ actual_type[struct_level].type_dimension = $2.type_dimension;
+ actual_type[struct_level].type_index = $2.type_index;
+ actual_type[struct_level].type_sizeof = $2.type_sizeof;
actual_startline[struct_level] = hashline_number();
}
@@ -572,13 +573,12 @@ var_declaration:
}
| var_type
{
- actual_type[struct_level].type_storage = mm_strdup("");
+ actual_type[struct_level].type_storage = loc_strdup("");
actual_type[struct_level].type_enum = $1.type_enum;
- actual_type[struct_level].type_str = mm_strdup($1.type_str);
- actual_type[struct_level].type_dimension = mm_strdup($1.type_dimension);
- actual_type[struct_level].type_index = mm_strdup($1.type_index);
- actual_type[struct_level].type_sizeof =
- $1.type_sizeof ? mm_strdup($1.type_sizeof) : NULL;
+ actual_type[struct_level].type_str = $1.type_str;
+ actual_type[struct_level].type_dimension = $1.type_dimension;
+ actual_type[struct_level].type_index = $1.type_index;
+ actual_type[struct_level].type_sizeof = $1.type_sizeof;
actual_startline[struct_level] = hashline_number();
}
@@ -870,7 +870,7 @@ var_type: simple_type
/* Otherwise, it must be a user-defined typedef name */
struct typedefs *this = get_typedef(@1, false);
- $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? mm_strdup("") : mm_strdup(this->name);
+ $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? "" : this->name;
$$.type_enum = this->type->type_enum;
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
@@ -1004,6 +1004,7 @@ s_struct_union_symbol: SQL_STRUCT symbol
{
$$.su = "struct";
$$.symbol = @2;
+ free(ECPGstruct_sizeof);
ECPGstruct_sizeof = mm_strdup(cat_str(3, "sizeof(",
cat2_str($$.su, $$.symbol),
")"));
@@ -1017,6 +1018,7 @@ s_struct_union_symbol: SQL_STRUCT symbol
s_struct_union: SQL_STRUCT
{
+ free(ECPGstruct_sizeof);
ECPGstruct_sizeof = mm_strdup(""); /* This must not be NULL to
* distinguish from simple types. */
@$ = "struct";
@@ -1696,18 +1698,21 @@ ECPGVar: SQL_VAR
ECPGWhenever: SQL_WHENEVER SQL_SQLERROR action
{
when_error.code = $3.code;
+ free(when_error.command);
when_error.command = $3.command ? mm_strdup($3.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever sqlerror ", $3.str, "; */");
}
| SQL_WHENEVER NOT SQL_FOUND action
{
when_nf.code = $4.code;
+ free(when_nf.command);
when_nf.command = $4.command ? mm_strdup($4.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever not found ", $4.str, "; */");
}
| SQL_WHENEVER SQL_SQLWARNING action
{
when_warn.code = $3.code;
+ free(when_warn.command);
when_warn.command = $3.command ? mm_strdup($3.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever sql_warning ", $3.str, "; */");
}
diff --git a/src/interfaces/ecpg/preproc/output.c b/src/interfaces/ecpg/preproc/output.c
index a18904f88b..b190e9f0ce 100644
--- a/src/interfaces/ecpg/preproc/output.c
+++ b/src/interfaces/ecpg/preproc/output.c
@@ -155,10 +155,11 @@ output_statement(const char *stmt, int whenever_mode, enum ECPG_statement_type s
/* dump variables to C file */
dump_variables(argsinsert, 1);
+ argsinsert = NULL;
fputs("ECPGt_EOIT, ", base_yyout);
dump_variables(argsresult, 1);
+ argsresult = NULL;
fputs("ECPGt_EORT);", base_yyout);
- reset_variables();
whenever_action(whenever_mode | 2);
}
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index 4831f56cba..8a2d0414ae 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -311,10 +311,12 @@ remove_variables(int brace_level)
for (ptr = cur; ptr != NULL; ptr = ptr->next)
{
struct arguments *varptr,
- *prevvar;
+ *prevvar,
+ *nextvar;
- for (varptr = prevvar = ptr->argsinsert; varptr != NULL; varptr = varptr->next)
+ for (varptr = prevvar = ptr->argsinsert; varptr != NULL; varptr = nextvar)
{
+ nextvar = varptr->next;
if (p == varptr->variable)
{
/* remove from list */
@@ -322,10 +324,12 @@ remove_variables(int brace_level)
ptr->argsinsert = varptr->next;
else
prevvar->next = varptr->next;
+ free(varptr);
}
}
- for (varptr = prevvar = ptr->argsresult; varptr != NULL; varptr = varptr->next)
+ for (varptr = prevvar = ptr->argsresult; varptr != NULL; varptr = nextvar)
{
+ nextvar = varptr->next;
if (p == varptr->variable)
{
/* remove from list */
@@ -333,6 +337,7 @@ remove_variables(int brace_level)
ptr->argsresult = varptr->next;
else
prevvar->next = varptr->next;
+ free(varptr);
}
}
}
@@ -372,7 +377,20 @@ struct arguments *argsresult = NULL;
void
reset_variables(void)
{
+ struct arguments *p,
+ *next;
+
+ for (p = argsinsert; p; p = next)
+ {
+ next = p->next;
+ free(p);
+ }
argsinsert = NULL;
+ for (p = argsresult; p; p = next)
+ {
+ next = p->next;
+ free(p);
+ }
argsresult = NULL;
}
@@ -431,6 +449,7 @@ remove_variable_from_list(struct arguments **list, struct variable *var)
prev->next = p->next;
else
*list = p->next;
+ free(p);
}
}
--
2.43.5
[text/x-diff] v5-0009-Remove-ecpg-s-check_rules.pl.patch (6.9K, ../../[email protected]/10-v5-0009-Remove-ecpg-s-check_rules.pl.patch)
download | inline diff:
From ea033cd69c6759e3a02459e04ea3630ab69f5e67 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Fri, 4 Oct 2024 16:31:14 -0400
Subject: [PATCH v5 9/9] Remove ecpg's check_rules.pl.
As noted in a previous commit, check_rules.pl is now entirely
redundant with checks made by parse.pl, or would be if it weren't
for the places where it's wrong. It's a waste of build cycles
and maintenance effort, so remove it.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/Makefile | 3 +-
src/interfaces/ecpg/preproc/check_rules.pl | 202 ---------------------
src/interfaces/ecpg/preproc/meson.build | 15 --
3 files changed, 1 insertion(+), 219 deletions(-)
delete mode 100644 src/interfaces/ecpg/preproc/check_rules.pl
diff --git a/src/interfaces/ecpg/preproc/Makefile b/src/interfaces/ecpg/preproc/Makefile
index 7866037cbb..4f403da935 100644
--- a/src/interfaces/ecpg/preproc/Makefile
+++ b/src/interfaces/ecpg/preproc/Makefile
@@ -65,8 +65,7 @@ preproc.h: preproc.c
preproc.c: BISONFLAGS += -d
-preproc.y: ../../../backend/parser/gram.y parse.pl check_rules.pl ecpg.addons ecpg.header ecpg.tokens ecpg.trailer ecpg.type
- $(PERL) $(srcdir)/check_rules.pl --srcdir $(srcdir) --parser $<
+preproc.y: ../../../backend/parser/gram.y parse.pl ecpg.addons ecpg.header ecpg.tokens ecpg.trailer ecpg.type
$(PERL) $(srcdir)/parse.pl --srcdir $(srcdir) --parser $< --output $@
# generate keyword headers
diff --git a/src/interfaces/ecpg/preproc/check_rules.pl b/src/interfaces/ecpg/preproc/check_rules.pl
deleted file mode 100644
index 1ee1aed2f6..0000000000
--- a/src/interfaces/ecpg/preproc/check_rules.pl
+++ /dev/null
@@ -1,202 +0,0 @@
-#!/usr/bin/perl
-# src/interfaces/ecpg/preproc/check_rules.pl
-# test parser generator for ecpg
-# call with backend grammar as stdin
-#
-# Copyright (c) 2009-2024, PostgreSQL Global Development Group
-#
-# Written by Michael Meskes <[email protected]>
-# Andy Colson <[email protected]>
-#
-# Placed under the same license as PostgreSQL.
-#
-# Command line: [-v] [path only to ecpg.addons] [full filename of gram.y]
-# -v enables verbose mode... show's some stats... thought it might be interesting
-#
-# This script loads rule names from gram.y and sets $found{rule} = 1 for each.
-# Then it checks to make sure each rule in ecpg.addons was found in gram.y
-
-use strict;
-use warnings FATAL => 'all';
-use Getopt::Long;
-
-my $srcdir = '.';
-my $parser = '../../../backend/parser/gram.y';
-my $stamp = '';
-my $verbose = 0;
-
-GetOptions(
- 'srcdir=s' => \$srcdir,
- 'parser=s' => \$parser,
- 'stamp=s' => \$stamp,
- 'verbose' => \$verbose,) or die "wrong arguments";
-
-my $filename = "$srcdir/ecpg.addons";
-if ($verbose)
-{
- print "parser: $parser\n";
- print "addons: $filename\n";
-}
-
-my %replace_line = (
- 'ExecuteStmtEXECUTEnameexecute_param_clause' =>
- 'EXECUTE prepared_name execute_param_clause execute_rest',
-
- 'ExecuteStmtCREATEOptTempTABLEcreate_as_targetASEXECUTEnameexecute_param_clauseopt_with_data'
- => 'CREATE OptTemp TABLE create_as_target AS EXECUTE prepared_name execute_param_clause opt_with_data execute_rest',
-
- 'ExecuteStmtCREATEOptTempTABLEIF_PNOTEXISTScreate_as_targetASEXECUTEnameexecute_param_clauseopt_with_data'
- => 'CREATE OptTemp TABLE IF_P NOT EXISTS create_as_target AS EXECUTE prepared_name execute_param_clause opt_with_data execute_rest',
-
- 'PrepareStmtPREPAREnameprep_type_clauseASPreparableStmt' =>
- 'PREPARE prepared_name prep_type_clause AS PreparableStmt');
-
-my $block = '';
-my $yaccmode = 0;
-my $in_rule = 0;
-my $brace_indent = 0;
-my (@arr, %found);
-my $comment = 0;
-my $non_term_id = '';
-my $cc = 0;
-
-open my $parser_fh, '<', $parser or die $!;
-while (<$parser_fh>)
-{
- if (/^%%/)
- {
- $yaccmode++;
- }
-
- if ($yaccmode != 1)
- {
- next;
- }
-
- chomp; # strip record separator
-
- next if ($_ eq '');
-
- # Make sure any braces are split
- s/{/ { /g;
- s/}/ } /g;
-
- # Any comments are split
- s|\/\*| /* |g;
- s|\*\/| */ |g;
-
- # Now split the line into individual fields
- my $n = (@arr = split(' '));
-
- # Go through each field in turn
- for (my $fieldIndexer = 0; $fieldIndexer < $n; $fieldIndexer++)
- {
- if ($arr[$fieldIndexer] eq '*/' && $comment)
- {
- $comment = 0;
- next;
- }
- elsif ($comment)
- {
- next;
- }
- elsif ($arr[$fieldIndexer] eq '/*')
- {
-
- # start of a multiline comment
- $comment = 1;
- next;
- }
- elsif ($arr[$fieldIndexer] eq '//')
- {
- next;
- }
- elsif ($arr[$fieldIndexer] eq '}')
- {
- $brace_indent--;
- next;
- }
- elsif ($arr[$fieldIndexer] eq '{')
- {
- $brace_indent++;
- next;
- }
-
- if ($brace_indent > 0)
- {
- next;
- }
-
- if ($arr[$fieldIndexer] eq ';' || $arr[$fieldIndexer] eq '|')
- {
- $block = $non_term_id . $block;
- if ($replace_line{$block})
- {
- $block = $non_term_id . $replace_line{$block};
- $block =~ tr/ |//d;
- }
- $found{$block} = 1;
- $cc++;
- $block = '';
- $in_rule = 0 if $arr[$fieldIndexer] eq ';';
- }
- elsif (
- ($arr[$fieldIndexer] =~ '[A-Za-z0-9]+:')
- || ( $fieldIndexer + 1 < $n
- && $arr[ $fieldIndexer + 1 ] eq ':'))
- {
- die "unterminated rule at grammar line $.\n"
- if $in_rule;
- $in_rule = 1;
- $non_term_id = $arr[$fieldIndexer];
- $non_term_id =~ tr/://d;
- }
- else
- {
- $block = $block . $arr[$fieldIndexer];
- }
- }
-}
-
-die "unterminated rule at end of grammar\n"
- if $in_rule;
-
-close $parser_fh;
-if ($verbose)
-{
- print "$cc rules loaded\n";
-}
-
-my $ret = 0;
-$cc = 0;
-
-open my $ecpg_fh, '<', $filename or die $!;
-while (<$ecpg_fh>)
-{
- if (!/^ECPG:/)
- {
- next;
- }
-
- my @Fld = split(' ', $_, 3);
- $cc++;
- if (not exists $found{ $Fld[1] })
- {
- print $Fld[1], " is not used for building parser!\n";
- $ret = 1;
- }
-}
-close $ecpg_fh;
-
-if ($verbose)
-{
- print "$cc rules checked\n";
-}
-
-if ($stamp)
-{
- open my $stampfh, '>', $stamp or die $!;
- close $stampfh;
-}
-
-exit $ret;
diff --git a/src/interfaces/ecpg/preproc/meson.build b/src/interfaces/ecpg/preproc/meson.build
index f680e5d59e..2fb1402c70 100644
--- a/src/interfaces/ecpg/preproc/meson.build
+++ b/src/interfaces/ecpg/preproc/meson.build
@@ -45,20 +45,6 @@ preproc_y = custom_target('preproc.y',
)
generated_sources += preproc_y
-check_rules = custom_target('preproc.y.check_rules',
- input: [
- '../../../backend/parser/gram.y',
- ecpg_files,
- ],
- output: 'preproc.y.check_rules',
- command: [
- perl, files('check_rules.pl'),
- '--srcdir', '@CURRENT_SOURCE_DIR@',
- '--parser', '@INPUT0@',
- '--stamp', '@OUTPUT0@',
- ],
-)
-
preproc = custom_target('preproc.c',
input: preproc_y,
kwargs: bison_kw,
@@ -69,7 +55,6 @@ ecpg_sources += preproc
c_kwlist = custom_target('c_kwlist_d.h',
input: ['c_kwlist.h'],
output: ['c_kwlist_d.h'],
- depends: check_rules,
depend_files: gen_kwlist_deps,
command: [gen_kwlist_cmd, '--varname', 'ScanCKeywords', '--no-case-fold'],
)
--
2.43.5
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
@ 2024-10-07 09:52 ` John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: John Naylor @ 2024-10-07 09:52 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
On Saturday, October 5, 2024, Tom Lane <[email protected]> wrote:
>
>
> Rebase needed after f22e84df1, so here's an update that rebases
> up to HEAD and adds the missing "static". No other changes.
>
> (Anybody want to review this? I'm getting tired of rebasing it,
> and we're missing out on the clang build time savings.)
Sorry for the delay, I'll respond in a couple days.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
@ 2024-10-11 11:43 ` John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: John Naylor @ 2024-10-11 11:43 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
> [v5]
0001 - LGTM, maybe can be squashed with 0009?
0002 - I went through this again and don't see anything that should
raise eyebrows.
+ # HACK: insert our own %nonassoc line after IDENT.
+ # XXX: this seems pretty wrong, IDENT is not last on its line!
We can come back to this afterwards, as mentioned elsewhere in the thread.
0003
Clang is what motivated this, but gcc also shows a speedup -- from
5.9s to 1.6s on this old machine, which is great. The giant switch
statement in preproc.c has about 1/10 the labels as before.
+In the original implementation of ecpg, the strings constructed
+by grammar rules were returned as the Bison result of each rule.
+This led to a large number of effectively-identical rule actions,
+which caused compilation-time problems with some versions of clang.
+Now, rules that need to return a string are declared as having
"Original" is going to be a mystery in a few years -- I'd describe
this in terms of "as of PG18" or some such.
+ * is producing uniformly-cased output of keywords. (That's mostly
+ * cosmetic, but there are places in ecpglib that expect to receive
+ * downcased keywords, plus it keeps us regression-test-compatible
+ * with the old implementation of ecpg.)
Ditto with "old".
+ /* List a token here if pgc.l assigns to base_yylval.str for it */
Does pgc.l need to have a similar comment?
0004 seems like a sensible reorg.
0005 - I wondered if the change of YYLTYPE to "const char *" can be
done in a separate commit to make the other changes more legible, but
that might not be worth the effort.
+ * "Local" (or "location"?) memory management support
"Local" seems to fit well enough. Tying the arena to the statement
level seems sound.
I haven't looked closely at 0006 through 0009. One possible concern is
that the regression tests might not cover very well, but if you can
get valgrind silent for memory leaks for what they do cover, that's
certainly a good step.
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
@ 2024-10-14 18:25 ` Tom Lane <[email protected]>
2024-10-16 03:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
2024-10-16 17:54 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-11-27 07:02 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
0 siblings, 3 replies; 27+ messages in thread
From: Tom Lane @ 2024-10-14 18:25 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
John Naylor <[email protected]> writes:
>> [v5]
Thanks for reviewing! I pushed 0001-0005 and 0009, adopting
your suggestions except for
> + /* List a token here if pgc.l assigns to base_yylval.str for it */
> Does pgc.l need to have a similar comment?
That's not a bad suggestion, but I couldn't see any very useful place
to put such a comment.
> I haven't looked closely at 0006 through 0009. One possible concern is
> that the regression tests might not cover very well, but if you can
> get valgrind silent for memory leaks for what they do cover, that's
> certainly a good step.
Attached are rebased and renumbered 0006-0008, mostly to keep the
cfbot happy. We could actually stop here, if we were feeling lazy,
but now that I've done the work I'm inclined to push forward with
the rest.
The rest is just memory leak removal, and I suspect that nobody really
cares that much about small leakage in the preprocessor: you'd have to
be running some darn big files through it to notice. FTR, here are
the total leaks reported by valgrind for running the ecpg regression
tests, using code like
$ grep lost: *log | tr -d ',' | awk '{sum += $5}
END {print sum}'
Before these patches: 25743
after 0003: 59049363
after 0005: 141556 (this is master now)
after 0006(0001): 132633
after 0007(0002): 9087
after 0008(0003): 0
So clearly, 0003 by itself wasn't good enough, but arguably no
real users will notice the extra inefficiency as of HEAD.
Still, I'd kind of like to get 0007 (now 0002) in there, and
I believe 0006 (0001) is a necessary prerequisite to that.
regards, tom lane
Attachments:
[text/x-diff] v6-0001-ecpg-fix-some-memory-leakage-of-data-type-related.patch (8.6K, ../../[email protected]/2-v6-0001-ecpg-fix-some-memory-leakage-of-data-type-related.patch)
download | inline diff:
From 43f3f423db776695172f74d35322f99b23706b86 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Mon, 14 Oct 2024 14:00:40 -0400
Subject: [PATCH v6 1/3] ecpg: fix some memory leakage of data-type-related
structures.
ECPGfree_type() and related functions were quite incomplete
about removing subsidiary data structures. Possibly this is
because ecpg wasn't careful to make sure said data structures
always had their own storage. Previous patches in this series
cleaned up a lot of that, and I had to add a couple more
mm_strdup's here.
Also, ecpg.trailer tended to overwrite struct_member_list[struct_level]
without bothering to free up its previous contents, thus potentially
leaking a lot of struct-member-related storage. Add
ECPGfree_struct_member() calls at appropriate points. (Note: the
lifetime of those lists is not obvious. They are still live after
initial construction, in order to handle cases like
struct foo { ... } foovar1, foovar2;
We can't delete the list immediately after parsing the right brace,
because it needs to be copied for each of the variables. Instead,
it's kept around until the next struct declaration.)
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/ecpg.header | 3 ++-
src/interfaces/ecpg/preproc/ecpg.trailer | 11 +++++++++--
src/interfaces/ecpg/preproc/type.c | 15 +++++++++------
src/interfaces/ecpg/preproc/type.h | 5 +++--
src/interfaces/ecpg/preproc/variable.c | 7 ++++++-
5 files changed, 29 insertions(+), 12 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index d3df8eabbb..d54eca918d 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -506,11 +506,12 @@ add_typedef(const char *name, const char *dimension, const char *length,
this->name = mm_strdup(name);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
+ this->type->type_storage = NULL;
this->type->type_enum = type_enum;
this->type->type_str = mm_strdup(name);
this->type->type_dimension = mm_strdup(dimension); /* dimension of array */
this->type->type_index = mm_strdup(length); /* length of string */
- this->type->type_sizeof = ECPGstruct_sizeof;
+ this->type->type_sizeof = ECPGstruct_sizeof ? mm_strdup(ECPGstruct_sizeof) : NULL;
this->struct_member_list = (type_enum == ECPGt_struct || type_enum == ECPGt_union) ?
ECPGstruct_member_dup(struct_member_list[struct_level]) : NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index 0a77559e83..41029701fc 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -751,6 +751,7 @@ var_type: simple_type
else
$$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
}
@@ -878,6 +879,7 @@ var_type: simple_type
else
$$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
}
@@ -900,6 +902,7 @@ var_type: simple_type
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
$$.type_sizeof = this->type->type_sizeof;
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
else
@@ -909,6 +912,7 @@ var_type: simple_type
$$.type_dimension = "-1";
$$.type_index = "-1";
$$.type_sizeof = "";
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = NULL;
}
}
@@ -924,6 +928,7 @@ enum_definition: '{' c_list '}'
struct_union_type_with_symbol: s_struct_union_symbol
{
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level++] = NULL;
if (struct_level >= STRUCT_DEPTH)
mmerror(PARSE_ERROR, ET_ERROR, "too many levels in nested structure/union definition");
@@ -965,12 +970,13 @@ struct_union_type_with_symbol: s_struct_union_symbol
this->name = mm_strdup(su_type.type_str);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
+ this->type->type_storage = NULL;
this->type->type_enum = su_type.type_enum;
this->type->type_str = mm_strdup(su_type.type_str);
this->type->type_dimension = mm_strdup("-1"); /* dimension of array */
this->type->type_index = mm_strdup("-1"); /* length of string */
- this->type->type_sizeof = ECPGstruct_sizeof;
- this->struct_member_list = struct_member_list[struct_level];
+ this->type->type_sizeof = ECPGstruct_sizeof ? mm_strdup(ECPGstruct_sizeof) : NULL;
+ this->struct_member_list = ECPGstruct_member_dup(struct_member_list[struct_level]);
types = this;
@$ = cat_str(4, su_type.type_str, "{", @4, "}");
@@ -980,6 +986,7 @@ struct_union_type_with_symbol: s_struct_union_symbol
struct_union_type: struct_union_type_with_symbol
| s_struct_union
{
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level++] = NULL;
if (struct_level >= STRUCT_DEPTH)
mmerror(PARSE_ERROR, ET_ERROR, "too many levels in nested structure/union definition");
diff --git a/src/interfaces/ecpg/preproc/type.c b/src/interfaces/ecpg/preproc/type.c
index 7f52521dbf..9f6dacd2ae 100644
--- a/src/interfaces/ecpg/preproc/type.c
+++ b/src/interfaces/ecpg/preproc/type.c
@@ -94,13 +94,14 @@ ECPGmake_array_type(struct ECPGtype *type, const char *size)
}
struct ECPGtype *
-ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type, char *type_name, char *struct_sizeof)
+ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type,
+ const char *type_name, const char *struct_sizeof)
{
struct ECPGtype *ne = ECPGmake_simple_type(type, "1", 0);
ne->type_name = mm_strdup(type_name);
ne->u.members = ECPGstruct_member_dup(rm);
- ne->struct_sizeof = struct_sizeof;
+ ne->struct_sizeof = mm_strdup(struct_sizeof);
return ne;
}
@@ -622,7 +623,7 @@ ECPGfree_struct_member(struct ECPGstruct_member *rm)
rm = rm->next;
free(p->name);
- free(p->type);
+ ECPGfree_type(p->type);
free(p);
}
}
@@ -643,14 +644,13 @@ ECPGfree_type(struct ECPGtype *type)
case ECPGt_struct:
case ECPGt_union:
/* Array of structs. */
- ECPGfree_struct_member(type->u.element->u.members);
- free(type->u.element);
+ ECPGfree_type(type->u.element);
break;
default:
if (!IS_SIMPLE_TYPE(type->u.element->type))
base_yyerror("internal error: unknown datatype, please report this to <" PACKAGE_BUGREPORT ">");
- free(type->u.element);
+ ECPGfree_type(type->u.element);
}
break;
case ECPGt_struct:
@@ -662,6 +662,9 @@ ECPGfree_type(struct ECPGtype *type)
break;
}
}
+ free(type->type_name);
+ free(type->size);
+ free(type->struct_sizeof);
free(type);
}
diff --git a/src/interfaces/ecpg/preproc/type.h b/src/interfaces/ecpg/preproc/type.h
index 90126551d1..3d99e1703d 100644
--- a/src/interfaces/ecpg/preproc/type.h
+++ b/src/interfaces/ecpg/preproc/type.h
@@ -38,8 +38,9 @@ void ECPGmake_struct_member(const char *name, struct ECPGtype *type,
struct ECPGtype *ECPGmake_simple_type(enum ECPGttype type, const char *size, int counter);
struct ECPGtype *ECPGmake_array_type(struct ECPGtype *type, const char *size);
struct ECPGtype *ECPGmake_struct_type(struct ECPGstruct_member *rm,
- enum ECPGttype type, char *type_name,
- char *struct_sizeof);
+ enum ECPGttype type,
+ const char *type_name,
+ const char *struct_sizeof);
struct ECPGstruct_member *ECPGstruct_member_dup(struct ECPGstruct_member *rm);
/* Frees a type. */
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index 6b87d5ff3d..4831f56cba 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -273,7 +273,12 @@ remove_typedefs(int brace_level)
prev->next = p->next;
if (p->type->type_enum == ECPGt_struct || p->type->type_enum == ECPGt_union)
- free(p->struct_member_list);
+ ECPGfree_struct_member(p->struct_member_list);
+ free(p->type->type_storage);
+ free(p->type->type_str);
+ free(p->type->type_dimension);
+ free(p->type->type_index);
+ free(p->type->type_sizeof);
free(p->type);
free(p->name);
free(p);
--
2.43.5
[text/x-diff] v6-0002-ecpg-put-all-string-valued-tokens-returned-by-pgc.patch (7.1K, ../../[email protected]/3-v6-0002-ecpg-put-all-string-valued-tokens-returned-by-pgc.patch)
download | inline diff:
From 82b12c83c5c9479f18b1f1f9b41cb0a897627b47 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Mon, 14 Oct 2024 14:08:10 -0400
Subject: [PATCH v6 2/3] ecpg: put all string-valued tokens returned by pgc.l
in local storage.
This didn't work earlier in the patch series (I think some of
the strings were ending up in data-type-related structures),
but apparently we're now clean enough for it. This considerably
reduces process-lifespan memory leakage.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/parser.c | 4 ++-
src/interfaces/ecpg/preproc/pgc.l | 42 ++++++++++++++--------------
2 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/parser.c b/src/interfaces/ecpg/preproc/parser.c
index 373c93fc04..181417fb39 100644
--- a/src/interfaces/ecpg/preproc/parser.c
+++ b/src/interfaces/ecpg/preproc/parser.c
@@ -203,7 +203,9 @@ filtered_base_yylex(void)
base_yytext = cur_yytext;
/* Combine 3 tokens into 1 */
- base_yylval.str = psprintf("%s UESCAPE %s", base_yylval.str, escstr);
+ base_yylval.str = make3_str(base_yylval.str,
+ " UESCAPE ",
+ escstr);
base_yylloc = loc_strdup(base_yylval.str);
/* Clear have_lookahead, thereby consuming all three tokens */
diff --git a/src/interfaces/ecpg/preproc/pgc.l b/src/interfaces/ecpg/preproc/pgc.l
index f3c03482ae..82708013ee 100644
--- a/src/interfaces/ecpg/preproc/pgc.l
+++ b/src/interfaces/ecpg/preproc/pgc.l
@@ -641,26 +641,26 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
case xb:
if (literalbuf[strspn(literalbuf, "01")] != '\0')
mmerror(PARSE_ERROR, ET_ERROR, "invalid bit string literal");
- base_yylval.str = psprintf("b'%s'", literalbuf);
+ base_yylval.str = make3_str("b'", literalbuf, "'");
return BCONST;
case xh:
if (literalbuf[strspn(literalbuf, "0123456789abcdefABCDEF")] != '\0')
mmerror(PARSE_ERROR, ET_ERROR, "invalid hexadecimal string literal");
- base_yylval.str = psprintf("x'%s'", literalbuf);
+ base_yylval.str = make3_str("x'", literalbuf, "'");
return XCONST;
case xq:
/* fallthrough */
case xqc:
- base_yylval.str = psprintf("'%s'", literalbuf);
+ base_yylval.str = make3_str("'", literalbuf, "'");
return SCONST;
case xe:
- base_yylval.str = psprintf("E'%s'", literalbuf);
+ base_yylval.str = make3_str("E'", literalbuf, "'");
return SCONST;
case xn:
- base_yylval.str = psprintf("N'%s'", literalbuf);
+ base_yylval.str = make3_str("N'", literalbuf, "'");
return SCONST;
case xus:
- base_yylval.str = psprintf("U&'%s'", literalbuf);
+ base_yylval.str = make3_str("U&'", literalbuf, "'");
return USCONST;
default:
mmfatal(PARSE_ERROR, "unhandled previous state in xqs\n");
@@ -724,7 +724,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
free(dolqstart);
dolqstart = NULL;
BEGIN(SQL);
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return SCONST;
}
else
@@ -778,12 +778,12 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* PREPARE and EXECUTE IMMEDIATE, which can certainly be
* longer than NAMEDATALEN.
*/
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return CSTRING;
}
<xdc>{xdstop} {
BEGIN(state_before_str_start);
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return CSTRING;
}
<xui>{dquote} {
@@ -795,7 +795,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* The backend will truncate the identifier here. We do
* not as it does not change the result.
*/
- base_yylval.str = psprintf("U&\"%s\"", literalbuf);
+ base_yylval.str = make3_str("U&\"", literalbuf, "\"");
return UIDENT;
}
<xd,xui>{xddouble} {
@@ -971,7 +971,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
}
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return Op;
}
@@ -990,7 +990,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
{ip} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IP;
}
} /* <SQL> */
@@ -1003,7 +1003,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return process_integer_literal(yytext, &base_yylval, 16);
}
{numeric} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return FCONST;
}
{numericfail} {
@@ -1012,7 +1012,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return process_integer_literal(yytext, &base_yylval, 10);
}
{real} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return FCONST;
}
{realfail} {
@@ -1048,7 +1048,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
:{identifier}((("->"|\.){identifier})|(\[{array}\]))* {
- base_yylval.str = mm_strdup(yytext + 1);
+ base_yylval.str = loc_strdup(yytext + 1);
return CVARIABLE;
}
@@ -1085,7 +1085,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* to do so; that's just another way that ecpg could
* get out of step with the backend.
*/
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IDENT;
}
}
@@ -1124,7 +1124,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
}
@@ -1136,12 +1136,12 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
}
<C,SQL>{cppline} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
<C>{identifier} {
@@ -1167,7 +1167,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return kwvalue;
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IDENT;
}
}
@@ -1685,7 +1685,7 @@ process_integer_literal(const char *token, YYSTYPE *lval, int base)
if (*endptr != '\0' || errno == ERANGE)
{
/* integer too large (or contains decimal pt), treat it as a float */
- lval->str = mm_strdup(token);
+ lval->str = loc_strdup(token);
return FCONST;
}
lval->ival = val;
--
2.43.5
[text/x-diff] v6-0003-ecpg-clean-up-some-other-assorted-memory-leaks.patch (16.2K, ../../[email protected]/4-v6-0003-ecpg-clean-up-some-other-assorted-memory-leaks.patch)
download | inline diff:
From bd5ea1b8ccf31920016b59cad0d37eea63a702f4 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Mon, 14 Oct 2024 14:09:37 -0400
Subject: [PATCH v6 3/3] ecpg: clean up some other assorted memory leaks.
Avoid leaking the prior value when updating the "connection"
state variable.
Ditto for ECPGstruct_sizeof. (It seems like this one ought to
be statement-local, but testing says it isn't, and I didn't
feel like diving deeper.)
The actual_type[] entries are statement-local, though, so
no need to mm_strdup() strings stored in them.
Likewise, sqlda variables are statement-local, so we can
loc_alloc them.
Also clean up sloppiness around management of the argsinsert and
argsresult lists.
progname changes are strictly to prevent valgrind from complaining
about leaked allocations.
With this, valgrind reports zero leakage in the ecpg preprocessor
for all of our ecpg regression test cases.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/descriptor.c | 14 +++++--
src/interfaces/ecpg/preproc/ecpg.addons | 47 +++++++++---------------
src/interfaces/ecpg/preproc/ecpg.c | 2 +-
src/interfaces/ecpg/preproc/ecpg.header | 17 ++++++++-
src/interfaces/ecpg/preproc/ecpg.trailer | 31 +++++++++-------
src/interfaces/ecpg/preproc/output.c | 3 +-
src/interfaces/ecpg/preproc/variable.c | 25 +++++++++++--
7 files changed, 86 insertions(+), 53 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/descriptor.c b/src/interfaces/ecpg/preproc/descriptor.c
index 9b87d07d09..e8c7016bdc 100644
--- a/src/interfaces/ecpg/preproc/descriptor.c
+++ b/src/interfaces/ecpg/preproc/descriptor.c
@@ -344,11 +344,17 @@ descriptor_variable(const char *name, int input)
struct variable *
sqlda_variable(const char *name)
{
- struct variable *p = (struct variable *) mm_alloc(sizeof(struct variable));
-
- p->name = mm_strdup(name);
- p->type = (struct ECPGtype *) mm_alloc(sizeof(struct ECPGtype));
+ /*
+ * Presently, sqlda variables are only needed for the duration of the
+ * current statement. Rather than add infrastructure to manage them,
+ * let's just loc_alloc them.
+ */
+ struct variable *p = (struct variable *) loc_alloc(sizeof(struct variable));
+
+ p->name = loc_strdup(name);
+ p->type = (struct ECPGtype *) loc_alloc(sizeof(struct ECPGtype));
p->type->type = ECPGt_sqlda;
+ p->type->type_name = NULL;
p->type->size = NULL;
p->type->struct_sizeof = NULL;
p->type->u.element = NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index 9c120fead2..05de4ff1f1 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -135,6 +135,7 @@ ECPG: stmtViewStmt rule
fprintf(base_yyout, "{ ECPGdescribe(__LINE__, %d, %d, %s, %s,", compat, $1.input, connection ? connection : "NULL", $1.stmt_name);
dump_variables(argsresult, 1);
+ argsresult = NULL;
fputs("ECPGt_EORT);", base_yyout);
fprintf(base_yyout, "}");
output_line_number();
@@ -181,6 +182,7 @@ ECPG: stmtViewStmt rule
if ((ptr = add_additional_variables(@1, true)) != NULL)
{
+ free(connection);
connection = ptr->connection ? mm_strdup(ptr->connection) : NULL;
output_statement(ptr->command, 0, ECPGst_normal);
ptr->opened = true;
@@ -247,15 +249,13 @@ ECPG: var_valueNumericOnly addon
ECPG: fetch_argscursor_name addon
struct cursor *ptr = add_additional_variables(@1, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@1[0] == ':')
@$ = "$0";
ECPG: fetch_argsfrom_incursor_name addon
struct cursor *ptr = add_additional_variables(@2, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@2[0] == ':')
@$ = cat2_str(@1, "$0");
ECPG: fetch_argsNEXTopt_from_incursor_name addon
@@ -265,16 +265,14 @@ ECPG: fetch_argsLAST_Popt_from_incursor_name addon
ECPG: fetch_argsALLopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@3[0] == ':')
@$ = cat_str(3, @1, @2, "$0");
ECPG: fetch_argsSignedIconstopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@3, false);
bool replace = false;
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@3[0] == ':')
{
@3 = "$0";
@@ -291,8 +289,7 @@ ECPG: fetch_argsFORWARDALLopt_from_incursor_name addon
ECPG: fetch_argsBACKWARDALLopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@4[0] == ':')
@$ = cat_str(4, @1, @2, @3, "$0");
ECPG: fetch_argsABSOLUTE_PSignedIconstopt_from_incursor_name addon
@@ -302,8 +299,7 @@ ECPG: fetch_argsBACKWARDSignedIconstopt_from_incursor_name addon
struct cursor *ptr = add_additional_variables(@4, false);
bool replace = false;
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@4[0] == ':')
{
@4 = "$0";
@@ -412,8 +408,7 @@ ECPG: ClosePortalStmtCLOSEcursor_name block
{
if (strcmp(@2, ptr->name) == 0)
{
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
break;
}
}
@@ -483,8 +478,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch forward", cursor_marker);
}
@@ -493,8 +487,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch forward from", cursor_marker);
}
@@ -503,8 +496,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch backward", cursor_marker);
}
@@ -513,8 +505,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch backward from", cursor_marker);
}
@@ -523,8 +514,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move forward", cursor_marker);
}
@@ -533,8 +523,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move forward from", cursor_marker);
}
@@ -543,8 +532,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move backward", cursor_marker);
}
@@ -553,8 +541,7 @@ ECPG: FetchStmtMOVEfetch_args rule
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move backward from", cursor_marker);
}
diff --git a/src/interfaces/ecpg/preproc/ecpg.c b/src/interfaces/ecpg/preproc/ecpg.c
index 73c37631ac..2fcc6f8f99 100644
--- a/src/interfaces/ecpg/preproc/ecpg.c
+++ b/src/interfaces/ecpg/preproc/ecpg.c
@@ -20,6 +20,7 @@ bool autocommit = false,
regression_mode = false,
auto_prepare = false;
+static const char *progname;
char *output_filename;
enum COMPAT_MODE compat = ECPG_COMPAT_PGSQL;
@@ -139,7 +140,6 @@ main(int argc, char *const argv[])
bool verbose = false,
header_mode = false;
struct _include_path *ip;
- const char *progname;
char my_exec_path[MAXPGPATH];
char include_path[MAXPGPATH];
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index d54eca918d..a9a0ef9847 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -59,6 +59,7 @@ struct variable no_indicator = {"no_indicator", &ecpg_no_indicator, 0, NULL};
static struct ECPGtype ecpg_query = {ECPGt_char_variable, NULL, NULL, NULL, {NULL}, 0};
static bool check_declared_list(const char *name);
+static void update_connection(const char *newconn);
/*
@@ -545,12 +546,26 @@ check_declared_list(const char *name)
{
if (connection && strcmp(ptr->connection, connection) != 0)
mmerror(PARSE_ERROR, ET_WARNING, "connection %s is overwritten with %s by DECLARE statement %s", connection, ptr->connection, name);
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
return true;
}
}
return false;
}
+
+/*
+ * If newconn isn't NULL, update the global "connection" variable to that;
+ * otherwise do nothing.
+ */
+static void
+update_connection(const char *newconn)
+{
+ if (newconn)
+ {
+ free(connection);
+ connection = mm_strdup(newconn);
+ }
+}
%}
%expect 0
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index 41029701fc..a90b1771fc 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -74,6 +74,8 @@ CreateAsStmt: CREATE OptTemp TABLE create_as_target AS
at: AT connection_object
{
+ if (connection)
+ free(connection);
connection = mm_strdup(@2);
/*
@@ -556,13 +558,12 @@ type_declaration: S_TYPEDEF
var_declaration:
storage_declaration var_type
{
- actual_type[struct_level].type_storage = mm_strdup(@1);
+ actual_type[struct_level].type_storage = loc_strdup(@1);
actual_type[struct_level].type_enum = $2.type_enum;
- actual_type[struct_level].type_str = mm_strdup($2.type_str);
- actual_type[struct_level].type_dimension = mm_strdup($2.type_dimension);
- actual_type[struct_level].type_index = mm_strdup($2.type_index);
- actual_type[struct_level].type_sizeof =
- $2.type_sizeof ? mm_strdup($2.type_sizeof) : NULL;
+ actual_type[struct_level].type_str = $2.type_str;
+ actual_type[struct_level].type_dimension = $2.type_dimension;
+ actual_type[struct_level].type_index = $2.type_index;
+ actual_type[struct_level].type_sizeof = $2.type_sizeof;
actual_startline[struct_level] = hashline_number();
}
@@ -572,13 +573,12 @@ var_declaration:
}
| var_type
{
- actual_type[struct_level].type_storage = mm_strdup("");
+ actual_type[struct_level].type_storage = loc_strdup("");
actual_type[struct_level].type_enum = $1.type_enum;
- actual_type[struct_level].type_str = mm_strdup($1.type_str);
- actual_type[struct_level].type_dimension = mm_strdup($1.type_dimension);
- actual_type[struct_level].type_index = mm_strdup($1.type_index);
- actual_type[struct_level].type_sizeof =
- $1.type_sizeof ? mm_strdup($1.type_sizeof) : NULL;
+ actual_type[struct_level].type_str = $1.type_str;
+ actual_type[struct_level].type_dimension = $1.type_dimension;
+ actual_type[struct_level].type_index = $1.type_index;
+ actual_type[struct_level].type_sizeof = $1.type_sizeof;
actual_startline[struct_level] = hashline_number();
}
@@ -870,7 +870,7 @@ var_type: simple_type
/* Otherwise, it must be a user-defined typedef name */
struct typedefs *this = get_typedef(@1, false);
- $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? mm_strdup("") : mm_strdup(this->name);
+ $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? "" : this->name;
$$.type_enum = this->type->type_enum;
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
@@ -1004,6 +1004,7 @@ s_struct_union_symbol: SQL_STRUCT symbol
{
$$.su = "struct";
$$.symbol = @2;
+ free(ECPGstruct_sizeof);
ECPGstruct_sizeof = mm_strdup(cat_str(3, "sizeof(",
cat2_str($$.su, $$.symbol),
")"));
@@ -1017,6 +1018,7 @@ s_struct_union_symbol: SQL_STRUCT symbol
s_struct_union: SQL_STRUCT
{
+ free(ECPGstruct_sizeof);
ECPGstruct_sizeof = mm_strdup(""); /* This must not be NULL to
* distinguish from simple types. */
@$ = "struct";
@@ -1696,18 +1698,21 @@ ECPGVar: SQL_VAR
ECPGWhenever: SQL_WHENEVER SQL_SQLERROR action
{
when_error.code = $3.code;
+ free(when_error.command);
when_error.command = $3.command ? mm_strdup($3.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever sqlerror ", $3.str, "; */");
}
| SQL_WHENEVER NOT SQL_FOUND action
{
when_nf.code = $4.code;
+ free(when_nf.command);
when_nf.command = $4.command ? mm_strdup($4.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever not found ", $4.str, "; */");
}
| SQL_WHENEVER SQL_SQLWARNING action
{
when_warn.code = $3.code;
+ free(when_warn.command);
when_warn.command = $3.command ? mm_strdup($3.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever sql_warning ", $3.str, "; */");
}
diff --git a/src/interfaces/ecpg/preproc/output.c b/src/interfaces/ecpg/preproc/output.c
index a18904f88b..b190e9f0ce 100644
--- a/src/interfaces/ecpg/preproc/output.c
+++ b/src/interfaces/ecpg/preproc/output.c
@@ -155,10 +155,11 @@ output_statement(const char *stmt, int whenever_mode, enum ECPG_statement_type s
/* dump variables to C file */
dump_variables(argsinsert, 1);
+ argsinsert = NULL;
fputs("ECPGt_EOIT, ", base_yyout);
dump_variables(argsresult, 1);
+ argsresult = NULL;
fputs("ECPGt_EORT);", base_yyout);
- reset_variables();
whenever_action(whenever_mode | 2);
}
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index 4831f56cba..8a2d0414ae 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -311,10 +311,12 @@ remove_variables(int brace_level)
for (ptr = cur; ptr != NULL; ptr = ptr->next)
{
struct arguments *varptr,
- *prevvar;
+ *prevvar,
+ *nextvar;
- for (varptr = prevvar = ptr->argsinsert; varptr != NULL; varptr = varptr->next)
+ for (varptr = prevvar = ptr->argsinsert; varptr != NULL; varptr = nextvar)
{
+ nextvar = varptr->next;
if (p == varptr->variable)
{
/* remove from list */
@@ -322,10 +324,12 @@ remove_variables(int brace_level)
ptr->argsinsert = varptr->next;
else
prevvar->next = varptr->next;
+ free(varptr);
}
}
- for (varptr = prevvar = ptr->argsresult; varptr != NULL; varptr = varptr->next)
+ for (varptr = prevvar = ptr->argsresult; varptr != NULL; varptr = nextvar)
{
+ nextvar = varptr->next;
if (p == varptr->variable)
{
/* remove from list */
@@ -333,6 +337,7 @@ remove_variables(int brace_level)
ptr->argsresult = varptr->next;
else
prevvar->next = varptr->next;
+ free(varptr);
}
}
}
@@ -372,7 +377,20 @@ struct arguments *argsresult = NULL;
void
reset_variables(void)
{
+ struct arguments *p,
+ *next;
+
+ for (p = argsinsert; p; p = next)
+ {
+ next = p->next;
+ free(p);
+ }
argsinsert = NULL;
+ for (p = argsresult; p; p = next)
+ {
+ next = p->next;
+ free(p);
+ }
argsresult = NULL;
}
@@ -431,6 +449,7 @@ remove_variable_from_list(struct arguments **list, struct variable *var)
prev->next = p->next;
else
*list = p->next;
+ free(p);
}
}
--
2.43.5
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
@ 2024-10-16 03:00 ` Alexander Lakhin <[email protected]>
2024-10-16 16:26 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2 siblings, 1 reply; 27+ messages in thread
From: Alexander Lakhin @ 2024-10-16 03:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; John Naylor <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
Hello Tom,
14.10.2024 21:25, Tom Lane wrote:
> Attached are rebased and renumbered 0006-0008, mostly to keep the
> cfbot happy. We could actually stop here, if we were feeling lazy,
> but now that I've done the work I'm inclined to push forward with
> the rest.
>
> The rest is just memory leak removal, and I suspect that nobody really
> cares that much about small leakage in the preprocessor: you'd have to
> be running some darn big files through it to notice. FTR, here are
> the total leaks reported by valgrind for running the ecpg regression
> tests, using code like
Maybe you would like to fix in passing several (not new) defects, I've
found while playing with ecpg under Valgrind:
echo "
EXEC SQL DECLARE cur1 CURSOR FOR stmt1;
" > t.pgc
valgrind .../preproc/ecpg ... t.pgc
==831888== Conditional jump or move depends on uninitialised value(s)
==831888== at 0x10C7B0: main (ecpg.c:490)
==831888==
char_array.pgc:2: WARNING: cursor "cur1" has been declared but not opened
Another case:
EXEC SQL DECLARE cur_1 CURSOR FOR stmt_1;
EXEC SQL FETCH cur_1 INTO :f1[[i];
==1335775==
==1335775== Conditional jump or move depends on uninitialised value(s)
==1335775== at 0x121294: find_variable (variable.c:211)
==1335775== by 0x11D661: base_yyparse (preproc.y:9749)
==1335775== by 0x10C78F: main (ecpg.c:483)
==1335775==
==1335775== Conditional jump or move depends on uninitialised value(s)
==1335775== at 0x121299: find_variable (variable.c:211)
==1335775== by 0x11D661: base_yyparse (preproc.y:9749)
==1335775== by 0x10C78F: main (ecpg.c:483)
==1335775==
==1335775== Invalid read of size 1
==1335775== at 0x12128B: find_variable (variable.c:211)
==1335775== by 0x11D661: base_yyparse (preproc.y:9749)
==1335775== by 0x10C78F: main (ecpg.c:483)
==1335775== Address 0x4e3bc80 is 0 bytes after a block of size 8,208 alloc'd
==1335775== at 0x4848899: malloc (vg_replace_malloc.c:381)
==1335775== by 0x120585: mm_alloc (util.c:87)
==1335775== by 0x12065A: loc_alloc (util.c:151)
==1335775== by 0x120701: loc_strdup (util.c:172)
==1335775== by 0x10D9EC: base_yylex_location (parser.c:261)
==1335775== by 0x10D4A1: filtered_base_yylex (parser.c:75)
==1335775== by 0x114CA4: base_yyparse (preproc.c:39316)
==1335775== by 0x10C78F: main (ecpg.c:483)
==1335775==
declare.pgc:2: ERROR: variable "f1" is not declared
One more case:
EXEC SQL BEGIN DECLARE SECTION;
int i;
EXEC SQL END DECLARE SECTION;
EXEC SQL DECLARE C CURSOR FOR SELECT 1;
{
EXEC SQL FETCH 1 IN C INTO :i;
}
EXEC SQL MOVE BACKWARD 1 IN C;
==961441== Invalid read of size 1
==961441== at 0x484FBD7: strcmp (vg_replace_strmem.c:924)
==961441== by 0x11442F: add_additional_variables (preproc.y:470)
==961441== by 0x117DEF: base_yyparse (preproc.y:3548)
==961441== by 0x10C78F: main (ecpg.c:483)
==961441== Address 0x0 is not stack'd, malloc'd or (recently) free'd
Best regards,
Alexander
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-16 03:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
@ 2024-10-16 16:26 ` Tom Lane <[email protected]>
2024-10-17 18:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Tom Lane @ 2024-10-16 16:26 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: John Naylor <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
Alexander Lakhin <[email protected]> writes:
> Maybe you would like to fix in passing several (not new) defects, I've
> found while playing with ecpg under Valgrind:
Done. After evaluation I concluded that none of these were worth the
trouble to back-patch, but by all means let's fix such things in HEAD.
regards, tom lane
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-16 03:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
2024-10-16 16:26 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
@ 2024-10-17 18:00 ` Alexander Lakhin <[email protected]>
2024-10-17 19:42 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
0 siblings, 1 reply; 27+ messages in thread
From: Alexander Lakhin @ 2024-10-17 18:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: John Naylor <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
Hello Tom,
16.10.2024 19:26, Tom Lane wrote:
> Alexander Lakhin <[email protected]> writes:
>> Maybe you would like to fix in passing several (not new) defects, I've
>> found while playing with ecpg under Valgrind:
> Done. After evaluation I concluded that none of these were worth the
> trouble to back-patch, but by all means let's fix such things in HEAD.
Thank you for fixing these defects!
I've spent a day testing ecpg preprocessor and found another couple of
bugs:
1)
EXEC SQL BEGIN DECLARE SECTION;
int i = 1;
EXEC SQL END DECLARE SECTION;
EXEC SQL DECLARE c CURSOR FOR SELECT :i;
{;}
}
EXEC SQL OPEN c;
==1247560==
==1247560== Invalid read of size 4
==1247560== at 0x121C13: dump_variables (variable.c:462)
==1247560== by 0x10CF75: output_statement (output.c:157)
==1247560== by 0x116B6B: base_yyparse (preproc.y:1233)
==1247560== by 0x10C78F: main (ecpg.c:483)
==1247560== Address 0x4e39bc0 is 16 bytes inside a block of size 32 free'd
==1247560== at 0x484B27F: free (vg_replace_malloc.c:872)
==1247560== by 0x1219AE: remove_variables (variable.c:351)
==1247560== by 0x11899F: base_yyparse (preproc.y:7853)
==1247560== by 0x10C78F: main (ecpg.c:483)
==1247560== Block was alloc'd at
==1247560== at 0x4848899: malloc (vg_replace_malloc.c:381)
==1247560== by 0x120703: mm_alloc (util.c:87)
==1247560== by 0x120C3C: new_variable (variable.c:12)
==1247560== by 0x11C27C: base_yyparse (preproc.y:8984)
==1247560== by 0x10C78F: main (ecpg.c:483)
...
---
2)
EXEC SQL BEGIN DECLARE SECTION;
char s[100];
EXEC SQL END DECLARE SECTION;
EXEC SQL DECLARE cur_1 CURSOR FOR SELECT 1;
EXEC SQL FETCH cur_1 INTO :s[0];
==1247848== Invalid read of size 4
==1247848== at 0x121388: find_variable (variable.c:238)
==1247848== by 0x11D684: base_yyparse (preproc.y:9751)
==1247848== by 0x10C78F: main (ecpg.c:483)
==1247848== Address 0x0 is not stack'd, malloc'd or (recently) free'd
---
Also, processing of .../ecpg/test/sql/include.pgc, containing only:
EXEC SQL INCLUDE ../sql;
emits merely:
input in flex scanner failed
I think that's all that can be found here without extra efforts.
Best regards,
Alexander
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-16 03:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
2024-10-16 16:26 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-17 18:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
@ 2024-10-17 19:42 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Tom Lane @ 2024-10-17 19:42 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: John Naylor <[email protected]>; Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
Alexander Lakhin <[email protected]> writes:
> I've spent a day testing ecpg preprocessor and found another couple of
> bugs:
Thanks for the report! The first couple of these seem simple
enough to fix, so I've done so. As for
> Also, processing of .../ecpg/test/sql/include.pgc, containing only:
> EXEC SQL INCLUDE ../sql;
> emits merely:
> input in flex scanner failed
what we have here is an attempt to read a directory. On Linux
it seems that fopen() is okay with that but then fread() fails
with EISDIR. The fread() occurs in code emitted by flex that
is totally failing to produce a useful error report:
while ( (result = (int) fread(buf, 1, (yy_size_t) max_size, base_yyin)) == 0 && ferror(base_yyin)) \
{ \
if( errno != EINTR) \
{ \
YY_FATAL_ERROR( "input in flex scanner failed" ); \
break; \
} \
It looks like YY_FATAL_ERROR can only accept a literal string,
so I can see why this isn't including strerror(errno), but still
I'd say that this is their poor error reporting not ours. The
only thing we could really do about it is fstat the fopen's FD
and see if it's a directory, but that would only improve matters
for EISDIR not any other error cause. I don't believe we've done
that anywhere else we use flex, so I'm not inclined to do it here.
> I think that's all that can be found here without extra efforts.
Thanks for poking at it! I don't feel a huge need to search out
ecpg bugs in advance of field reports, but we might as well fix
things we do stumble across.
regards, tom lane
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
@ 2024-10-16 17:54 ` Tom Lane <[email protected]>
2 siblings, 0 replies; 27+ messages in thread
From: Tom Lane @ 2024-10-16 17:54 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
I wrote:
> Attached are rebased and renumbered 0006-0008, mostly to keep the
> cfbot happy.
For some reason I thought the stuff I pushed later on Monday
didn't interact with these patches, but the cfbot disabused me
of that folly. Here's a rebased v7 --- no substantive change.
regards, tom lane
Attachments:
[text/x-diff] v7-0001-ecpg-fix-some-memory-leakage-of-data-type-related.patch (8.6K, ../../[email protected]/2-v7-0001-ecpg-fix-some-memory-leakage-of-data-type-related.patch)
download | inline diff:
From 52e50770ca723278cf04da52fee0ffaddfae8235 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Wed, 16 Oct 2024 13:38:13 -0400
Subject: [PATCH v7 1/3] ecpg: fix some memory leakage of data-type-related
structures.
ECPGfree_type() and related functions were quite incomplete
about removing subsidiary data structures. Possibly this is
because ecpg wasn't careful to make sure said data structures
always had their own storage. Previous patches in this series
cleaned up a lot of that, and I had to add a couple more
mm_strdup's here.
Also, ecpg.trailer tended to overwrite struct_member_list[struct_level]
without bothering to free up its previous contents, thus potentially
leaking a lot of struct-member-related storage. Add
ECPGfree_struct_member() calls at appropriate points. (Note: the
lifetime of those lists is not obvious. They are still live after
initial construction, in order to handle cases like
struct foo { ... } foovar1, foovar2;
We can't delete the list immediately after parsing the right brace,
because it needs to be copied for each of the variables. Instead,
it's kept around until the next struct declaration.)
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/ecpg.header | 3 ++-
src/interfaces/ecpg/preproc/ecpg.trailer | 11 +++++++++--
src/interfaces/ecpg/preproc/type.c | 15 +++++++++------
src/interfaces/ecpg/preproc/type.h | 5 +++--
src/interfaces/ecpg/preproc/variable.c | 7 ++++++-
5 files changed, 29 insertions(+), 12 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index a415780faf..9554e2b02e 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -507,11 +507,12 @@ add_typedef(const char *name, const char *dimension, const char *length,
this->name = mm_strdup(name);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
+ this->type->type_storage = NULL;
this->type->type_enum = type_enum;
this->type->type_str = mm_strdup(name);
this->type->type_dimension = mm_strdup(dimension); /* dimension of array */
this->type->type_index = mm_strdup(length); /* length of string */
- this->type->type_sizeof = ECPGstruct_sizeof;
+ this->type->type_sizeof = ECPGstruct_sizeof ? mm_strdup(ECPGstruct_sizeof) : NULL;
this->struct_member_list = (type_enum == ECPGt_struct || type_enum == ECPGt_union) ?
ECPGstruct_member_dup(struct_member_list[struct_level]) : NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index e466668ea2..0b15252433 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -752,6 +752,7 @@ var_type: simple_type
else
$$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
}
@@ -879,6 +880,7 @@ var_type: simple_type
else
$$.type_sizeof = cat_str(3, "sizeof(", this->name, ")");
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
}
@@ -901,6 +903,7 @@ var_type: simple_type
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
$$.type_sizeof = this->type->type_sizeof;
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = ECPGstruct_member_dup(this->struct_member_list);
}
else
@@ -910,6 +913,7 @@ var_type: simple_type
$$.type_dimension = "-1";
$$.type_index = "-1";
$$.type_sizeof = "";
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level] = NULL;
}
}
@@ -925,6 +929,7 @@ enum_definition: '{' c_list '}'
struct_union_type_with_symbol: s_struct_union_symbol
{
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level++] = NULL;
if (struct_level >= STRUCT_DEPTH)
mmerror(PARSE_ERROR, ET_ERROR, "too many levels in nested structure/union definition");
@@ -966,12 +971,13 @@ struct_union_type_with_symbol: s_struct_union_symbol
this->name = mm_strdup(su_type.type_str);
this->brace_level = braces_open;
this->type = (struct this_type *) mm_alloc(sizeof(struct this_type));
+ this->type->type_storage = NULL;
this->type->type_enum = su_type.type_enum;
this->type->type_str = mm_strdup(su_type.type_str);
this->type->type_dimension = mm_strdup("-1"); /* dimension of array */
this->type->type_index = mm_strdup("-1"); /* length of string */
- this->type->type_sizeof = ECPGstruct_sizeof;
- this->struct_member_list = struct_member_list[struct_level];
+ this->type->type_sizeof = ECPGstruct_sizeof ? mm_strdup(ECPGstruct_sizeof) : NULL;
+ this->struct_member_list = ECPGstruct_member_dup(struct_member_list[struct_level]);
types = this;
@$ = cat_str(4, su_type.type_str, "{", @4, "}");
@@ -981,6 +987,7 @@ struct_union_type_with_symbol: s_struct_union_symbol
struct_union_type: struct_union_type_with_symbol
| s_struct_union
{
+ ECPGfree_struct_member(struct_member_list[struct_level]);
struct_member_list[struct_level++] = NULL;
if (struct_level >= STRUCT_DEPTH)
mmerror(PARSE_ERROR, ET_ERROR, "too many levels in nested structure/union definition");
diff --git a/src/interfaces/ecpg/preproc/type.c b/src/interfaces/ecpg/preproc/type.c
index 7f52521dbf..9f6dacd2ae 100644
--- a/src/interfaces/ecpg/preproc/type.c
+++ b/src/interfaces/ecpg/preproc/type.c
@@ -94,13 +94,14 @@ ECPGmake_array_type(struct ECPGtype *type, const char *size)
}
struct ECPGtype *
-ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type, char *type_name, char *struct_sizeof)
+ECPGmake_struct_type(struct ECPGstruct_member *rm, enum ECPGttype type,
+ const char *type_name, const char *struct_sizeof)
{
struct ECPGtype *ne = ECPGmake_simple_type(type, "1", 0);
ne->type_name = mm_strdup(type_name);
ne->u.members = ECPGstruct_member_dup(rm);
- ne->struct_sizeof = struct_sizeof;
+ ne->struct_sizeof = mm_strdup(struct_sizeof);
return ne;
}
@@ -622,7 +623,7 @@ ECPGfree_struct_member(struct ECPGstruct_member *rm)
rm = rm->next;
free(p->name);
- free(p->type);
+ ECPGfree_type(p->type);
free(p);
}
}
@@ -643,14 +644,13 @@ ECPGfree_type(struct ECPGtype *type)
case ECPGt_struct:
case ECPGt_union:
/* Array of structs. */
- ECPGfree_struct_member(type->u.element->u.members);
- free(type->u.element);
+ ECPGfree_type(type->u.element);
break;
default:
if (!IS_SIMPLE_TYPE(type->u.element->type))
base_yyerror("internal error: unknown datatype, please report this to <" PACKAGE_BUGREPORT ">");
- free(type->u.element);
+ ECPGfree_type(type->u.element);
}
break;
case ECPGt_struct:
@@ -662,6 +662,9 @@ ECPGfree_type(struct ECPGtype *type)
break;
}
}
+ free(type->type_name);
+ free(type->size);
+ free(type->struct_sizeof);
free(type);
}
diff --git a/src/interfaces/ecpg/preproc/type.h b/src/interfaces/ecpg/preproc/type.h
index 90126551d1..3d99e1703d 100644
--- a/src/interfaces/ecpg/preproc/type.h
+++ b/src/interfaces/ecpg/preproc/type.h
@@ -38,8 +38,9 @@ void ECPGmake_struct_member(const char *name, struct ECPGtype *type,
struct ECPGtype *ECPGmake_simple_type(enum ECPGttype type, const char *size, int counter);
struct ECPGtype *ECPGmake_array_type(struct ECPGtype *type, const char *size);
struct ECPGtype *ECPGmake_struct_type(struct ECPGstruct_member *rm,
- enum ECPGttype type, char *type_name,
- char *struct_sizeof);
+ enum ECPGttype type,
+ const char *type_name,
+ const char *struct_sizeof);
struct ECPGstruct_member *ECPGstruct_member_dup(struct ECPGstruct_member *rm);
/* Frees a type. */
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index a4294b8f0f..2c129646d2 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -276,7 +276,12 @@ remove_typedefs(int brace_level)
prev->next = p->next;
if (p->type->type_enum == ECPGt_struct || p->type->type_enum == ECPGt_union)
- free(p->struct_member_list);
+ ECPGfree_struct_member(p->struct_member_list);
+ free(p->type->type_storage);
+ free(p->type->type_str);
+ free(p->type->type_dimension);
+ free(p->type->type_index);
+ free(p->type->type_sizeof);
free(p->type);
free(p->name);
free(p);
--
2.43.5
[text/x-diff] v7-0002-ecpg-put-all-string-valued-tokens-returned-by-pgc.patch (7.1K, ../../[email protected]/3-v7-0002-ecpg-put-all-string-valued-tokens-returned-by-pgc.patch)
download | inline diff:
From 1a2904611f307ec2b0dc31f04094d90e0ce178ea Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Wed, 16 Oct 2024 13:39:55 -0400
Subject: [PATCH v7 2/3] ecpg: put all string-valued tokens returned by pgc.l
in local storage.
This didn't work earlier in the patch series (I think some of
the strings were ending up in data-type-related structures),
but apparently we're now clean enough for it. This considerably
reduces process-lifespan memory leakage.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/parser.c | 4 ++-
src/interfaces/ecpg/preproc/pgc.l | 42 ++++++++++++++--------------
2 files changed, 24 insertions(+), 22 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/parser.c b/src/interfaces/ecpg/preproc/parser.c
index 373c93fc04..181417fb39 100644
--- a/src/interfaces/ecpg/preproc/parser.c
+++ b/src/interfaces/ecpg/preproc/parser.c
@@ -203,7 +203,9 @@ filtered_base_yylex(void)
base_yytext = cur_yytext;
/* Combine 3 tokens into 1 */
- base_yylval.str = psprintf("%s UESCAPE %s", base_yylval.str, escstr);
+ base_yylval.str = make3_str(base_yylval.str,
+ " UESCAPE ",
+ escstr);
base_yylloc = loc_strdup(base_yylval.str);
/* Clear have_lookahead, thereby consuming all three tokens */
diff --git a/src/interfaces/ecpg/preproc/pgc.l b/src/interfaces/ecpg/preproc/pgc.l
index f3c03482ae..82708013ee 100644
--- a/src/interfaces/ecpg/preproc/pgc.l
+++ b/src/interfaces/ecpg/preproc/pgc.l
@@ -641,26 +641,26 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
case xb:
if (literalbuf[strspn(literalbuf, "01")] != '\0')
mmerror(PARSE_ERROR, ET_ERROR, "invalid bit string literal");
- base_yylval.str = psprintf("b'%s'", literalbuf);
+ base_yylval.str = make3_str("b'", literalbuf, "'");
return BCONST;
case xh:
if (literalbuf[strspn(literalbuf, "0123456789abcdefABCDEF")] != '\0')
mmerror(PARSE_ERROR, ET_ERROR, "invalid hexadecimal string literal");
- base_yylval.str = psprintf("x'%s'", literalbuf);
+ base_yylval.str = make3_str("x'", literalbuf, "'");
return XCONST;
case xq:
/* fallthrough */
case xqc:
- base_yylval.str = psprintf("'%s'", literalbuf);
+ base_yylval.str = make3_str("'", literalbuf, "'");
return SCONST;
case xe:
- base_yylval.str = psprintf("E'%s'", literalbuf);
+ base_yylval.str = make3_str("E'", literalbuf, "'");
return SCONST;
case xn:
- base_yylval.str = psprintf("N'%s'", literalbuf);
+ base_yylval.str = make3_str("N'", literalbuf, "'");
return SCONST;
case xus:
- base_yylval.str = psprintf("U&'%s'", literalbuf);
+ base_yylval.str = make3_str("U&'", literalbuf, "'");
return USCONST;
default:
mmfatal(PARSE_ERROR, "unhandled previous state in xqs\n");
@@ -724,7 +724,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
free(dolqstart);
dolqstart = NULL;
BEGIN(SQL);
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return SCONST;
}
else
@@ -778,12 +778,12 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* PREPARE and EXECUTE IMMEDIATE, which can certainly be
* longer than NAMEDATALEN.
*/
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return CSTRING;
}
<xdc>{xdstop} {
BEGIN(state_before_str_start);
- base_yylval.str = mm_strdup(literalbuf);
+ base_yylval.str = loc_strdup(literalbuf);
return CSTRING;
}
<xui>{dquote} {
@@ -795,7 +795,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* The backend will truncate the identifier here. We do
* not as it does not change the result.
*/
- base_yylval.str = psprintf("U&\"%s\"", literalbuf);
+ base_yylval.str = make3_str("U&\"", literalbuf, "\"");
return UIDENT;
}
<xd,xui>{xddouble} {
@@ -971,7 +971,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
}
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return Op;
}
@@ -990,7 +990,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
{ip} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IP;
}
} /* <SQL> */
@@ -1003,7 +1003,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return process_integer_literal(yytext, &base_yylval, 16);
}
{numeric} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return FCONST;
}
{numericfail} {
@@ -1012,7 +1012,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return process_integer_literal(yytext, &base_yylval, 10);
}
{real} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return FCONST;
}
{realfail} {
@@ -1048,7 +1048,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
:{identifier}((("->"|\.){identifier})|(\[{array}\]))* {
- base_yylval.str = mm_strdup(yytext + 1);
+ base_yylval.str = loc_strdup(yytext + 1);
return CVARIABLE;
}
@@ -1085,7 +1085,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
* to do so; that's just another way that ecpg could
* get out of step with the backend.
*/
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IDENT;
}
}
@@ -1124,7 +1124,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
}
@@ -1136,12 +1136,12 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
}
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
}
<C,SQL>{cppline} {
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return CPP_LINE;
}
<C>{identifier} {
@@ -1167,7 +1167,7 @@ cppline {space}*#([^i][A-Za-z]*|{if}|{ifdef}|{ifndef}|{import})((\/\*[^*/]*\*+
return kwvalue;
else
{
- base_yylval.str = mm_strdup(yytext);
+ base_yylval.str = loc_strdup(yytext);
return IDENT;
}
}
@@ -1685,7 +1685,7 @@ process_integer_literal(const char *token, YYSTYPE *lval, int base)
if (*endptr != '\0' || errno == ERANGE)
{
/* integer too large (or contains decimal pt), treat it as a float */
- lval->str = mm_strdup(token);
+ lval->str = loc_strdup(token);
return FCONST;
}
lval->ival = val;
--
2.43.5
[text/x-diff] v7-0003-ecpg-clean-up-some-other-assorted-memory-leaks.patch (16.2K, ../../[email protected]/4-v7-0003-ecpg-clean-up-some-other-assorted-memory-leaks.patch)
download | inline diff:
From f4454425adbcf57a3537cd3a00c131e8754f8154 Mon Sep 17 00:00:00 2001
From: Tom Lane <[email protected]>
Date: Wed, 16 Oct 2024 13:48:33 -0400
Subject: [PATCH v7 3/3] ecpg: clean up some other assorted memory leaks.
Avoid leaking the prior value when updating the "connection"
state variable.
Ditto for ECPGstruct_sizeof. (It seems like this one ought to
be statement-local, but testing says it isn't, and I didn't
feel like diving deeper.)
The actual_type[] entries are statement-local, though, so
no need to mm_strdup() strings stored in them.
Likewise, sqlda variables are statement-local, so we can
loc_alloc them.
Also clean up sloppiness around management of the argsinsert and
argsresult lists.
progname changes are strictly to prevent valgrind from complaining
about leaked allocations.
With this, valgrind reports zero leakage in the ecpg preprocessor
for all of our ecpg regression test cases.
Discussion: https://postgr.es/m/[email protected]
---
src/interfaces/ecpg/preproc/descriptor.c | 14 +++++--
src/interfaces/ecpg/preproc/ecpg.addons | 47 +++++++++---------------
src/interfaces/ecpg/preproc/ecpg.c | 2 +-
src/interfaces/ecpg/preproc/ecpg.header | 17 ++++++++-
src/interfaces/ecpg/preproc/ecpg.trailer | 31 +++++++++-------
src/interfaces/ecpg/preproc/output.c | 3 +-
src/interfaces/ecpg/preproc/variable.c | 25 +++++++++++--
7 files changed, 86 insertions(+), 53 deletions(-)
diff --git a/src/interfaces/ecpg/preproc/descriptor.c b/src/interfaces/ecpg/preproc/descriptor.c
index 9b87d07d09..e8c7016bdc 100644
--- a/src/interfaces/ecpg/preproc/descriptor.c
+++ b/src/interfaces/ecpg/preproc/descriptor.c
@@ -344,11 +344,17 @@ descriptor_variable(const char *name, int input)
struct variable *
sqlda_variable(const char *name)
{
- struct variable *p = (struct variable *) mm_alloc(sizeof(struct variable));
-
- p->name = mm_strdup(name);
- p->type = (struct ECPGtype *) mm_alloc(sizeof(struct ECPGtype));
+ /*
+ * Presently, sqlda variables are only needed for the duration of the
+ * current statement. Rather than add infrastructure to manage them,
+ * let's just loc_alloc them.
+ */
+ struct variable *p = (struct variable *) loc_alloc(sizeof(struct variable));
+
+ p->name = loc_strdup(name);
+ p->type = (struct ECPGtype *) loc_alloc(sizeof(struct ECPGtype));
p->type->type = ECPGt_sqlda;
+ p->type->type_name = NULL;
p->type->size = NULL;
p->type->struct_sizeof = NULL;
p->type->u.element = NULL;
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
index 935820c8d4..71f7ad26ad 100644
--- a/src/interfaces/ecpg/preproc/ecpg.addons
+++ b/src/interfaces/ecpg/preproc/ecpg.addons
@@ -135,6 +135,7 @@ ECPG: rule stmt ViewStmt
fprintf(base_yyout, "{ ECPGdescribe(__LINE__, %d, %d, %s, %s,", compat, $1.input, connection ? connection : "NULL", $1.stmt_name);
dump_variables(argsresult, 1);
+ argsresult = NULL;
fputs("ECPGt_EORT);", base_yyout);
fprintf(base_yyout, "}");
output_line_number();
@@ -181,6 +182,7 @@ ECPG: rule stmt ViewStmt
if ((ptr = add_additional_variables(@1, true)) != NULL)
{
+ free(connection);
connection = ptr->connection ? mm_strdup(ptr->connection) : NULL;
output_statement(ptr->command, 0, ECPGst_normal);
ptr->opened = true;
@@ -247,15 +249,13 @@ ECPG: addon var_value NumericOnly
ECPG: addon fetch_args cursor_name
struct cursor *ptr = add_additional_variables(@1, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@1[0] == ':')
@$ = "$0";
ECPG: addon fetch_args from_in cursor_name
struct cursor *ptr = add_additional_variables(@2, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@2[0] == ':')
@$ = cat2_str(@1, "$0");
ECPG: addon fetch_args NEXT opt_from_in cursor_name
@@ -265,16 +265,14 @@ ECPG: addon fetch_args LAST_P opt_from_in cursor_name
ECPG: addon fetch_args ALL opt_from_in cursor_name
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@3[0] == ':')
@$ = cat_str(3, @1, @2, "$0");
ECPG: addon fetch_args SignedIconst opt_from_in cursor_name
struct cursor *ptr = add_additional_variables(@3, false);
bool replace = false;
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@3[0] == ':')
{
@3 = "$0";
@@ -291,8 +289,7 @@ ECPG: addon fetch_args FORWARD ALL opt_from_in cursor_name
ECPG: addon fetch_args BACKWARD ALL opt_from_in cursor_name
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@4[0] == ':')
@$ = cat_str(4, @1, @2, @3, "$0");
ECPG: addon fetch_args ABSOLUTE_P SignedIconst opt_from_in cursor_name
@@ -302,8 +299,7 @@ ECPG: addon fetch_args BACKWARD SignedIconst opt_from_in cursor_name
struct cursor *ptr = add_additional_variables(@4, false);
bool replace = false;
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
if (@4[0] == ':')
{
@4 = "$0";
@@ -412,8 +408,7 @@ ECPG: block ClosePortalStmt CLOSE cursor_name
{
if (strcmp(@2, ptr->name) == 0)
{
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
break;
}
}
@@ -483,8 +478,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch forward", cursor_marker);
}
@@ -493,8 +487,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch forward from", cursor_marker);
}
@@ -503,8 +496,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch backward", cursor_marker);
}
@@ -513,8 +505,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "fetch backward from", cursor_marker);
}
@@ -523,8 +514,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move forward", cursor_marker);
}
@@ -533,8 +523,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move forward from", cursor_marker);
}
@@ -543,8 +532,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @3[0] == ':' ? "$0" : @3;
struct cursor *ptr = add_additional_variables(@3, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move backward", cursor_marker);
}
@@ -553,8 +541,7 @@ ECPG: rule FetchStmt MOVE fetch_args
const char *cursor_marker = @4[0] == ':' ? "$0" : @4;
struct cursor *ptr = add_additional_variables(@4, false);
- if (ptr->connection)
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
@$ = cat_str(2, "move backward from", cursor_marker);
}
diff --git a/src/interfaces/ecpg/preproc/ecpg.c b/src/interfaces/ecpg/preproc/ecpg.c
index 73c37631ac..2fcc6f8f99 100644
--- a/src/interfaces/ecpg/preproc/ecpg.c
+++ b/src/interfaces/ecpg/preproc/ecpg.c
@@ -20,6 +20,7 @@ bool autocommit = false,
regression_mode = false,
auto_prepare = false;
+static const char *progname;
char *output_filename;
enum COMPAT_MODE compat = ECPG_COMPAT_PGSQL;
@@ -139,7 +140,6 @@ main(int argc, char *const argv[])
bool verbose = false,
header_mode = false;
struct _include_path *ip;
- const char *progname;
char my_exec_path[MAXPGPATH];
char include_path[MAXPGPATH];
diff --git a/src/interfaces/ecpg/preproc/ecpg.header b/src/interfaces/ecpg/preproc/ecpg.header
index 9554e2b02e..e002e0d835 100644
--- a/src/interfaces/ecpg/preproc/ecpg.header
+++ b/src/interfaces/ecpg/preproc/ecpg.header
@@ -59,6 +59,7 @@ struct variable no_indicator = {"no_indicator", &ecpg_no_indicator, 0, NULL};
static struct ECPGtype ecpg_query = {ECPGt_char_variable, NULL, NULL, NULL, {NULL}, 0};
static bool check_declared_list(const char *name);
+static void update_connection(const char *newconn);
/*
@@ -546,12 +547,26 @@ check_declared_list(const char *name)
{
if (connection && strcmp(ptr->connection, connection) != 0)
mmerror(PARSE_ERROR, ET_WARNING, "connection %s is overwritten with %s by DECLARE statement %s", connection, ptr->connection, name);
- connection = mm_strdup(ptr->connection);
+ update_connection(ptr->connection);
return true;
}
}
return false;
}
+
+/*
+ * If newconn isn't NULL, update the global "connection" variable to that;
+ * otherwise do nothing.
+ */
+static void
+update_connection(const char *newconn)
+{
+ if (newconn)
+ {
+ free(connection);
+ connection = mm_strdup(newconn);
+ }
+}
%}
%expect 0
diff --git a/src/interfaces/ecpg/preproc/ecpg.trailer b/src/interfaces/ecpg/preproc/ecpg.trailer
index 0b15252433..95630c1e66 100644
--- a/src/interfaces/ecpg/preproc/ecpg.trailer
+++ b/src/interfaces/ecpg/preproc/ecpg.trailer
@@ -74,6 +74,8 @@ CreateAsStmt: CREATE OptTemp TABLE create_as_target AS
at: AT connection_object
{
+ if (connection)
+ free(connection);
connection = mm_strdup(@2);
/*
@@ -557,13 +559,12 @@ type_declaration: S_TYPEDEF
var_declaration:
storage_declaration var_type
{
- actual_type[struct_level].type_storage = mm_strdup(@1);
+ actual_type[struct_level].type_storage = loc_strdup(@1);
actual_type[struct_level].type_enum = $2.type_enum;
- actual_type[struct_level].type_str = mm_strdup($2.type_str);
- actual_type[struct_level].type_dimension = mm_strdup($2.type_dimension);
- actual_type[struct_level].type_index = mm_strdup($2.type_index);
- actual_type[struct_level].type_sizeof =
- $2.type_sizeof ? mm_strdup($2.type_sizeof) : NULL;
+ actual_type[struct_level].type_str = $2.type_str;
+ actual_type[struct_level].type_dimension = $2.type_dimension;
+ actual_type[struct_level].type_index = $2.type_index;
+ actual_type[struct_level].type_sizeof = $2.type_sizeof;
actual_startline[struct_level] = hashline_number();
}
@@ -573,13 +574,12 @@ var_declaration:
}
| var_type
{
- actual_type[struct_level].type_storage = mm_strdup("");
+ actual_type[struct_level].type_storage = loc_strdup("");
actual_type[struct_level].type_enum = $1.type_enum;
- actual_type[struct_level].type_str = mm_strdup($1.type_str);
- actual_type[struct_level].type_dimension = mm_strdup($1.type_dimension);
- actual_type[struct_level].type_index = mm_strdup($1.type_index);
- actual_type[struct_level].type_sizeof =
- $1.type_sizeof ? mm_strdup($1.type_sizeof) : NULL;
+ actual_type[struct_level].type_str = $1.type_str;
+ actual_type[struct_level].type_dimension = $1.type_dimension;
+ actual_type[struct_level].type_index = $1.type_index;
+ actual_type[struct_level].type_sizeof = $1.type_sizeof;
actual_startline[struct_level] = hashline_number();
}
@@ -871,7 +871,7 @@ var_type: simple_type
/* Otherwise, it must be a user-defined typedef name */
struct typedefs *this = get_typedef(@1, false);
- $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? mm_strdup("") : mm_strdup(this->name);
+ $$.type_str = (this->type->type_enum == ECPGt_varchar || this->type->type_enum == ECPGt_bytea) ? "" : this->name;
$$.type_enum = this->type->type_enum;
$$.type_dimension = this->type->type_dimension;
$$.type_index = this->type->type_index;
@@ -1005,6 +1005,7 @@ s_struct_union_symbol: SQL_STRUCT symbol
{
$$.su = "struct";
$$.symbol = @2;
+ free(ECPGstruct_sizeof);
ECPGstruct_sizeof = mm_strdup(cat_str(3, "sizeof(",
cat2_str($$.su, $$.symbol),
")"));
@@ -1018,6 +1019,7 @@ s_struct_union_symbol: SQL_STRUCT symbol
s_struct_union: SQL_STRUCT
{
+ free(ECPGstruct_sizeof);
ECPGstruct_sizeof = mm_strdup(""); /* This must not be NULL to
* distinguish from simple types. */
@$ = "struct";
@@ -1697,18 +1699,21 @@ ECPGVar: SQL_VAR
ECPGWhenever: SQL_WHENEVER SQL_SQLERROR action
{
when_error.code = $3.code;
+ free(when_error.command);
when_error.command = $3.command ? mm_strdup($3.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever sqlerror ", $3.str, "; */");
}
| SQL_WHENEVER NOT SQL_FOUND action
{
when_nf.code = $4.code;
+ free(when_nf.command);
when_nf.command = $4.command ? mm_strdup($4.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever not found ", $4.str, "; */");
}
| SQL_WHENEVER SQL_SQLWARNING action
{
when_warn.code = $3.code;
+ free(when_warn.command);
when_warn.command = $3.command ? mm_strdup($3.command) : NULL;
@$ = cat_str(3, "/* exec sql whenever sql_warning ", $3.str, "; */");
}
diff --git a/src/interfaces/ecpg/preproc/output.c b/src/interfaces/ecpg/preproc/output.c
index a18904f88b..b190e9f0ce 100644
--- a/src/interfaces/ecpg/preproc/output.c
+++ b/src/interfaces/ecpg/preproc/output.c
@@ -155,10 +155,11 @@ output_statement(const char *stmt, int whenever_mode, enum ECPG_statement_type s
/* dump variables to C file */
dump_variables(argsinsert, 1);
+ argsinsert = NULL;
fputs("ECPGt_EOIT, ", base_yyout);
dump_variables(argsresult, 1);
+ argsresult = NULL;
fputs("ECPGt_EORT);", base_yyout);
- reset_variables();
whenever_action(whenever_mode | 2);
}
diff --git a/src/interfaces/ecpg/preproc/variable.c b/src/interfaces/ecpg/preproc/variable.c
index 2c129646d2..3bed096d95 100644
--- a/src/interfaces/ecpg/preproc/variable.c
+++ b/src/interfaces/ecpg/preproc/variable.c
@@ -314,10 +314,12 @@ remove_variables(int brace_level)
for (ptr = cur; ptr != NULL; ptr = ptr->next)
{
struct arguments *varptr,
- *prevvar;
+ *prevvar,
+ *nextvar;
- for (varptr = prevvar = ptr->argsinsert; varptr != NULL; varptr = varptr->next)
+ for (varptr = prevvar = ptr->argsinsert; varptr != NULL; varptr = nextvar)
{
+ nextvar = varptr->next;
if (p == varptr->variable)
{
/* remove from list */
@@ -325,10 +327,12 @@ remove_variables(int brace_level)
ptr->argsinsert = varptr->next;
else
prevvar->next = varptr->next;
+ free(varptr);
}
}
- for (varptr = prevvar = ptr->argsresult; varptr != NULL; varptr = varptr->next)
+ for (varptr = prevvar = ptr->argsresult; varptr != NULL; varptr = nextvar)
{
+ nextvar = varptr->next;
if (p == varptr->variable)
{
/* remove from list */
@@ -336,6 +340,7 @@ remove_variables(int brace_level)
ptr->argsresult = varptr->next;
else
prevvar->next = varptr->next;
+ free(varptr);
}
}
}
@@ -375,7 +380,20 @@ struct arguments *argsresult = NULL;
void
reset_variables(void)
{
+ struct arguments *p,
+ *next;
+
+ for (p = argsinsert; p; p = next)
+ {
+ next = p->next;
+ free(p);
+ }
argsinsert = NULL;
+ for (p = argsresult; p; p = next)
+ {
+ next = p->next;
+ free(p);
+ }
argsresult = NULL;
}
@@ -434,6 +452,7 @@ remove_variable_from_list(struct arguments **list, struct variable *var)
prev->next = p->next;
else
*list = p->next;
+ free(p);
}
}
--
2.43.5
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
@ 2024-11-27 07:02 ` John Naylor <[email protected]>
2024-11-27 17:59 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2 siblings, 1 reply; 27+ messages in thread
From: John Naylor @ 2024-11-27 07:02 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
On Tue, Oct 15, 2024 at 1:25 AM Tom Lane <[email protected]> wrote:
>
> The rest is just memory leak removal, and I suspect that nobody really
> cares that much about small leakage in the preprocessor: you'd have to
> be running some darn big files through it to notice. FTR, here are
> the total leaks reported by valgrind for running the ecpg regression
> tests, using code like
>
> $ grep lost: *log | tr -d ',' | awk '{sum += $5}
> END {print sum}'
>
> Before these patches: 25743
> after 0003: 59049363
> after 0005: 141556 (this is master now)
> after 0006(0001): 132633
> after 0007(0002): 9087
> after 0008(0003): 0
>
> So clearly, 0003 by itself wasn't good enough, but arguably no
> real users will notice the extra inefficiency as of HEAD.
> Still, I'd kind of like to get 0007 (now 0002) in there, and
> I believe 0006 (0001) is a necessary prerequisite to that.
Hi Tom,
I think you can go ahead and commit 0001-0003. For 0003 I do admit
being confused why valgrind had a problem with progname...
FWIW, I also took a quick eyeball check of the coverage output for
preproc.y looking for untested non-error branches that do things that
are not covered elsewhere, and I didn't notice any. Some places
allocating new connections are not covered, which should be handled by
0003.
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 27+ messages in thread
* Re: ECPG cleanup and fix for clang compile-time problem
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-11-27 07:02 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
@ 2024-11-27 17:59 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 27+ messages in thread
From: Tom Lane @ 2024-11-27 17:59 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; [email protected] <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>
John Naylor <[email protected]> writes:
> I think you can go ahead and commit 0001-0003. For 0003 I do admit
> being confused why valgrind had a problem with progname...
Pushed, thanks for looking at it!
As for progname, valgrind was unhappy because the local in main()
went out of scope at return, but the malloc block was still there.
But this was weird coding anyway: pretty much everywhere else,
we store progname in a globally-accessible variable so that it can
be used in error messages. I figured making it static was a down
payment on someday making ecpg's messages honor that convention.
> FWIW, I also took a quick eyeball check of the coverage output for
> preproc.y looking for untested non-error branches that do things that
> are not covered elsewhere, and I didn't notice any. Some places
> allocating new connections are not covered, which should be handled by
> 0003.
Yeah, ecpg's coverage report is fairly sad overall, but a lot of the
uncovered stuff is autogenerated grammar productions, which I think
we can have high confidence in. I'm not currently feeling motivated
to try to improve that number.
regards, tom lane
^ permalink raw reply [nested|flat] 27+ messages in thread
end of thread, other threads:[~2024-11-27 17:59 UTC | newest]
Thread overview: 27+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-07-05 14:24 [PATCH 06/17] Allow user to use password instead of encryption key. Antonin Houska <[email protected]>
2020-03-08 22:27 [PATCH 4/4] Support clauses of the form Var op Var Tomas Vondra <[email protected]>
2023-07-20 17:19 [PATCH v9 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v10 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v10 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v6 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v9 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v9 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v10 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-07-20 17:19 [PATCH v8 3/4] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-09-04 22:35 [PATCH v7 4/5] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-09-04 22:35 [PATCH v7 4/5] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2023-09-04 22:35 [PATCH v7 4/5] Convert pg_restore's ready_list to a priority queue. Nathan Bossart <[email protected]>
2024-10-04 20:46 Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-07 09:52 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-11 11:43 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-10-14 18:25 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-16 03:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
2024-10-16 16:26 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-17 18:00 ` Re: ECPG cleanup and fix for clang compile-time problem Alexander Lakhin <[email protected]>
2024-10-17 19:42 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-10-16 17:54 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
2024-11-27 07:02 ` Re: ECPG cleanup and fix for clang compile-time problem John Naylor <[email protected]>
2024-11-27 17:59 ` Re: ECPG cleanup and fix for clang compile-time problem Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox