agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH 03/17] Teach postmaster to accept encryption key 13+ messages / 4 participants [nested] [flat]
* [PATCH 03/17] Teach postmaster to accept encryption key @ 2019-07-05 14:24 Antonin Houska <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: Antonin Houska @ 2019-07-05 14:24 UTC (permalink / raw) This patch teaches postmaster to accept encryption key (as a special kind of startup packet) before recovery starts. In fact the key is received by a backend that postmaster launches if a client tries to initiate connection while the postmaster is in PM_INIT state. At this stage, only the key is received, whereas regular connection attempt ends up with FATAL: the database system is starting up If postmaster receives no key for some time, it raises this error and exits: FATAL: Encryption key not received. With this patch, user can start postmaster as usual: postgres -D data and use pg_keytool utility in another session to send the key: echo 71916b8b93063c280e4e0ee646395f450bda4f1bac85c57bc3a6afd168f3ab47 | pg_keytool -s --- src/backend/postmaster/postmaster.c | 211 +++++++++++++++++++++++++++++++++- src/backend/storage/file/encryption.c | 34 ++++++ src/backend/storage/ipc/ipci.c | 2 + src/bin/Makefile | 1 + src/bin/pg_keytool/.gitignore | 1 + src/bin/pg_keytool/Makefile | 31 +++++ src/bin/pg_keytool/pg_keytool.c | 199 ++++++++++++++++++++++++++++++++ src/fe_utils/encryption.c | 101 ++++++++++++++++ src/include/fe_utils/encryption.h | 2 + src/include/libpq/pqcomm.h | 24 ++++ src/include/storage/encryption.h | 26 +++++ 11 files changed, 628 insertions(+), 4 deletions(-) create mode 100644 src/bin/pg_keytool/.gitignore create mode 100644 src/bin/pg_keytool/Makefile create mode 100644 src/bin/pg_keytool/pg_keytool.c diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index e8e3f8a4d3..47f7bc93f7 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -117,6 +117,7 @@ #include "postmaster/syslogger.h" #include "replication/logicallauncher.h" #include "replication/walsender.h" +#include "storage/encryption.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/pg_shmem.h" @@ -402,12 +403,13 @@ static void PostmasterStateMachine(void); static void BackendInitialize(Port *port); static void BackendRun(Port *port) pg_attribute_noreturn(); static void ExitPostmaster(int status) pg_attribute_noreturn(); -static int ServerLoop(void); +static int ServerLoop(bool receive_encryption_key); static void ServerLoopCheckTimeouts(void); static int BackendStartup(Port *port); static int ProcessStartupPacket(Port *port, bool secure_done); static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options); static void processCancelRequest(Port *port, void *pkt); +static void processEncryptionKey(void *pkt); static int initMasks(fd_set *rmask); static void report_fork_failure_to_client(Port *port, int errnum); static CAC_state canAcceptConnections(void); @@ -1365,6 +1367,41 @@ PostmasterMain(int argc, char *argv[]) AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING); /* + * If encryption key is needed and we don't have it yet, call ServerLoop() + * in a restricted mode which allows a client application to send us the + * key. Regular client connections are not allowed yet. + */ + if (data_encrypted && !encryption_setup_done) + { + char sample[ENCRYPTION_SAMPLE_SIZE]; + + status = ServerLoop(true); + + /* No other return code should be seen here. */ + Assert(status == STATUS_OK); + + /* ServerLoop() shouldn't have exited otherwise. */ + Assert(encryption_key_shmem->initialized); + + /* + * Take a local copy of the key so that we don't have to receive it + * again if restarting after a crash. + */ + memcpy(encryption_key, encryption_key_shmem, ENCRYPTION_KEY_LENGTH); + + /* Finalize the setup. */ + setup_encryption(); + + /* Verify the key. */ + sample_encryption(sample); + if (memcmp(encryption_verification, sample, ENCRYPTION_SAMPLE_SIZE)) + ereport(FATAL, + (errmsg("invalid encryption key"), + errdetail("The passed encryption key does not match" + " database encryption key."))); + } + + /* * We're ready to rock and roll... */ StartupPID = StartupDataBase(); @@ -1375,7 +1412,7 @@ PostmasterMain(int argc, char *argv[]) /* Some workers may be scheduled to start now */ maybe_start_bgworkers(); - status = ServerLoop(); + status = ServerLoop(false); /* * ServerLoop probably shouldn't ever return, but if it does, close down. @@ -1615,18 +1652,37 @@ DetermineSleepTime(struct timeval *timeout) static time_t last_lockfile_recheck_time, last_touch_time; /* + * The maximum amount of time postmaster can wait for encryption key if the + * cluster is encrypted. It should be shorter than the time pg_ctl waits for + * postgres to start (60 seconds) and should not be too long so that failure + * to specify encryption key command is recognized soon. + */ +#define MAX_WAIT_FOR_KEY_SECS 20 + +/* When should waiting for encryption key end. */ +static TimestampTz wait_for_keys_until = 0; + +/* * Main idle loop of postmaster * + * If receive_encryption_key is true, only launch backend(s) to receive + * cluster encryption key and stop as soon as the key is in shared memory. + * * NB: Needs to be called with signals blocked */ static int -ServerLoop(void) +ServerLoop(bool receive_encryption_key) { fd_set readmask; int nSockets; last_lockfile_recheck_time = last_touch_time = time(NULL); + /* Compute wait_for_keys_until if needed and not done yet. */ + if (receive_encryption_key && wait_for_keys_until == 0) + wait_for_keys_until = PgStartTime + + MAX_WAIT_FOR_KEY_SECS * USECS_PER_SEC; + nSockets = initMasks(&readmask); for (;;) @@ -1663,6 +1719,27 @@ ServerLoop(void) /* Needs to run with blocked signals! */ DetermineSleepTime(&timeout); + /* + * If waiting for the encryption key, make sure + * MAX_WAIT_FOR_KEY_SECS is not exceeded. + */ + if (receive_encryption_key) + { + TimestampTz timeout_tz, timeout_end;; + + timeout_tz = timeout.tv_sec * USECS_PER_SEC + + timeout.tv_usec; + timeout_end = GetCurrentTimestamp() + timeout_tz; + + if (timeout_end > wait_for_keys_until) + { + timeout_tz -= (timeout_end - wait_for_keys_until); + + timeout.tv_sec = timeout_tz / USECS_PER_SEC; + timeout.tv_usec = timeout_tz % USECS_PER_SEC; + } + } + PG_SETMASK(&UnBlockSig); selres = select(nSockets, &rmask, NULL, NULL, &timeout); @@ -1714,6 +1791,33 @@ ServerLoop(void) } } + /* + * If only waiting for the encryption key, it's too early to launch + * the other processes. + */ + if (receive_encryption_key) + { + /* The regular checks should take place though. */ + ServerLoopCheckTimeouts(); + + /* Done if the key has been delivered. */ + if (encryption_key_shmem->initialized) + return STATUS_OK; + + /* + * Do not wait for the key forever. One problem we solve here is + * that pg_ctl (or custom script that starts postgres) does not + * have to check whether the cluster is encrypted. If DBA forgets + * to pass the encryption key command, he'll simply see the + * startup to fail and the appropriate message to appear in the + * server log. + */ + if (GetCurrentTimestamp() >= wait_for_keys_until) + ereport(FATAL, (errmsg("Encryption key not received."))); + + continue; + } + /* If we have lost the log collector, try to start a new one */ if (SysLoggerPID == 0 && Logging_collector) SysLoggerPID = SysLogger_Start(); @@ -1990,6 +2094,13 @@ ProcessStartupPacket(Port *port, bool secure_done) return STATUS_ERROR; } + if (proto == ENCRYPTION_KEY_MSG_CODE && data_encrypted) + { + processEncryptionKey(buf); + /* Not really an error, but we don't want to proceed further */ + return STATUS_ERROR; + } + if (proto == NEGOTIATE_SSL_CODE && !secure_done) { char SSLok; @@ -2394,6 +2505,77 @@ processCancelRequest(Port *port, void *pkt) } /* + * Process a message from backend that contains the encryption key. + * + * Currently there's no guarantee that only a single backend tries to deliver + * the key, but it's not worth trying to synchronize the access. If multiple + * callers try to process the same message, nothing should get broken. If some + * caller is delivering a wrong key (e.g. due to misconfiguration of another + * cluster), it does not help if it waits until processing of the correct key + * is finished. + * + * If the key gets messed up, the worst case is that the server fails to start + * up. (No data corruption is expected.) + * + * Client should not expect any response to this request: failure on client + * side indicates either broken connection, wrong key, bug in + * send_key_to_postmaster() or repeated call of the client - nothing to be + * handled easily by client code. Most of these failures need attention of the + * DBA. + */ +void +processEncryptionKey(void *pkt) +{ + EncryptionKeyMsg *msg = (EncryptionKeyMsg *) pkt; + + /* Backend to receive the key should not be launched otherwise. */ + Assert(data_encrypted); + + /* + * The checks below would fire in this case too, but this error message is + * clearer for the case like this. + */ + if (pmState != PM_INIT) + { + ereport(COMMERROR, + (errmsg("encryption key can only be setup during startup"), + errhint("Check if the cluster is encrypted."))); + return; + } + + /* + * Someone else already initialized the key. This is not fatal but seems + * to indicate misconfiguration which DBA should be aware of. Check also + * encryption_setup_done because shared memory could have been reset. As + * postmaster has a local copy of the key, the new key should not be used, + * but it'd be inconsistent if we didn't complain in this special case. + */ + if (encryption_key_shmem->initialized || encryption_setup_done) + { + ereport(COMMERROR, + (errmsg("received encryption key more than once"))); + return; + } + + if (msg->version != 1) + { + ereport(COMMERROR, + (errmsg("unexpected version of encryption key message %d", + msg->version))); + return; + } + + memcpy(encryption_key_shmem->data, msg->data, ENCRYPTION_KEY_LENGTH); + /* + * XXX Is memory barrier needed here to make sure that the copying is + * done? + */ + encryption_key_shmem->initialized = true; + + ereport(DEBUG1, (errmsg_internal("encryption key received"))); +} + +/* * canAcceptConnections --- check to see if database state allows connections. */ static CAC_state @@ -2418,7 +2600,8 @@ canAcceptConnections(void) else if (Shutdown > NoShutdown) return CAC_SHUTDOWN; /* shutdown is pending */ else if (!FatalError && - (pmState == PM_STARTUP || + ((data_encrypted && pmState == PM_INIT) || + pmState == PM_STARTUP || pmState == PM_RECOVERY)) return CAC_STARTUP; /* normal startup */ else if (!FatalError && @@ -2761,6 +2944,16 @@ pmdie(SIGNAL_ARGS) pmState = (pmState == PM_RUN) ? PM_WAIT_BACKUP : PM_WAIT_READONLY; } + else if (pmState == PM_INIT && data_encrypted && + !encryption_setup_done) + { + /* + * Only backends to receive encryption key may be active now, + * and these should not be involved in any transactions. + */ + SignalSomeChildren(SIGTERM, BACKEND_TYPE_NORMAL); + pmState = PM_WAIT_BACKENDS; + } /* * Now wait for online backup mode to end and backends to exit. If @@ -2828,6 +3021,16 @@ pmdie(SIGNAL_ARGS) signal_child(WalWriterPID, SIGTERM); pmState = PM_WAIT_BACKENDS; } + else if (pmState == PM_INIT && data_encrypted && + !encryption_setup_done) + { + /* + * Only backends to receive encryption key may be active now, + * and these should not be involved in any transactions. + */ + SignalSomeChildren(SIGTERM, BACKEND_TYPE_NORMAL); + pmState = PM_WAIT_BACKENDS; + } /* * Now wait for backends to exit. If there are none, diff --git a/src/backend/storage/file/encryption.c b/src/backend/storage/file/encryption.c index 629138fae7..1eda334762 100644 --- a/src/backend/storage/file/encryption.c +++ b/src/backend/storage/file/encryption.c @@ -53,6 +53,10 @@ EVP_CIPHER_CTX *ctx_encrypt, unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; +#ifndef FRONTEND +ShmemEncryptionKey *encryption_key_shmem = NULL; +#endif /* FRONTEND */ + bool data_encrypted = false; char encryption_verification[ENCRYPTION_SAMPLE_SIZE]; @@ -66,6 +70,36 @@ static void evp_error(void); #ifndef FRONTEND /* + * Report space needed for our shared memory area + */ +Size +EncryptionShmemSize(void) +{ + return sizeof(ShmemEncryptionKey); +} + +/* + * Initialize our shared memory area + */ +void +EncryptionShmemInit(void) +{ + bool found; + + encryption_key_shmem = ShmemInitStruct("Cluster Encryption Key", + EncryptionShmemSize(), + &found); + if (!IsUnderPostmaster) + { + Assert(!found); + + encryption_key_shmem->initialized = false; + } + else + Assert(found); +} + +/* * Read encryption key in hexadecimal form from stdin and store it in * encryption_key variable. */ diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index d7d733530f..771bf30c63 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -147,6 +147,7 @@ CreateSharedMemoryAndSemaphores(int port) size = add_size(size, BTreeShmemSize()); size = add_size(size, SyncScanShmemSize()); size = add_size(size, AsyncShmemSize()); + size = add_size(size, EncryptionShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -263,6 +264,7 @@ CreateSharedMemoryAndSemaphores(int port) BTreeShmemInit(); SyncScanShmemInit(); AsyncShmemInit(); + EncryptionShmemInit(); #ifdef EXEC_BACKEND diff --git a/src/bin/Makefile b/src/bin/Makefile index 903e58121f..b05a8bbb4e 100644 --- a/src/bin/Makefile +++ b/src/bin/Makefile @@ -22,6 +22,7 @@ SUBDIRS = \ pg_controldata \ pg_ctl \ pg_dump \ + pg_keytool \ pg_resetwal \ pg_rewind \ pg_test_fsync \ diff --git a/src/bin/pg_keytool/.gitignore b/src/bin/pg_keytool/.gitignore new file mode 100644 index 0000000000..249876a6ed --- /dev/null +++ b/src/bin/pg_keytool/.gitignore @@ -0,0 +1 @@ +/pg_keytool diff --git a/src/bin/pg_keytool/Makefile b/src/bin/pg_keytool/Makefile new file mode 100644 index 0000000000..6da0631f32 --- /dev/null +++ b/src/bin/pg_keytool/Makefile @@ -0,0 +1,31 @@ +# src/bin/pg_keysetup/Makefile + +PGFILEDESC = "pg_keytool - handle cluster encryption key" +PGAPPICON=win32 + +subdir = src/bin/pg_keytool +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = pg_keytool.o $(RMGRDESCOBJS) $(WIN32RES) + +override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS) +override CPPFLAGS := -DFRONTEND $(CPPFLAGS) +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) + +all: pg_keytool + +pg_keytool: $(OBJS) | submake-libpgport + $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) + +install: all installdirs + $(INSTALL_PROGRAM) pg_keytool$(X) '$(DESTDIR)$(bindir)/pg_keytool$(X)' + +installdirs: + $(MKDIR_P) '$(DESTDIR)$(bindir)' + +uninstall: + rm -f '$(DESTDIR)$(bindir)/pg_keytool$(X)' + +clean distclean maintainer-clean: + rm -f pg_keytool$(X) $(OBJS) encryption.c diff --git a/src/bin/pg_keytool/pg_keytool.c b/src/bin/pg_keytool/pg_keytool.c new file mode 100644 index 0000000000..322625da41 --- /dev/null +++ b/src/bin/pg_keytool/pg_keytool.c @@ -0,0 +1,199 @@ +/*------------------------------------------------------------------------- + * + * pg_keytool.c - Handle cluster encryption key. + * + * Copyright (c) 2013-2019, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/bin/pg_keytool/pg_keytool.c + *------------------------------------------------------------------------- + */ +/* + * TODO Adopt the new frontend logging API, after some things are clarified: + * https://www.postgresql.org/message-id/1939.1560773970%40localhost + */ +#define FRONTEND 1 +#include "postgres.h" + +#include <dirent.h> +#include <unistd.h> + +#include "common/fe_memutils.h" +#include "common/logging.h" +#include "fe_utils/encryption.h" +#include "libpq-fe.h" +#include "libpq-int.h" +#include "libpq/pqcomm.h" +#include "port/pg_crc32c.h" +#include "storage/encryption.h" +#include "getopt_long.h" + +#ifdef USE_ENCRYPTION +static const char *progname; + +unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; + +static void +usage(const char *progname) +{ + const char *env; + + printf(_("%s is a tool to handle cluster encryption key.\n\n"), + progname); + printf(_("Usage:\n")); + printf(_(" %s [OPTION]...\n"), progname); + printf(_("\nOptions:\n")); + /* Display default host */ + env = getenv("PGHOST"); + printf(_(" -h, --host=HOSTNAME database server host or socket directory (default: \"%s\")\n"), + env ? env : _("local socket")); + /* Display default port */ + env = getenv("PGPORT"); + printf(_(" -p, --port=PORT database server port (default: \"%s\")\n"), + env ? env : DEF_PGPORT_STR); + printf(_(" -s, send output to database server\n")); + printf(_(" -?, --help show this help, then exit\n\n")); + printf(_("Key is read from stdin and sent either to stdout or to PostgreSQL server being started\n")); +} +#endif /* USE_ENCRYPTION */ + +int +main(int argc, char **argv) +{ +/* + * If no encryption library is linked, let the utility fail immediately. It'd + * be weird if we reported incorrect usage just to say later that no useful + * work can be done anyway. + */ +#ifdef USE_ENCRYPTION + int c; + char *host = NULL; + char *port_str = NULL; + bool to_server = false; + int i, n; + int optindex; + char key_chars[ENCRYPTION_KEY_CHARS]; + + static struct option long_options[] = + { + {"pgdata", required_argument, NULL, 'D'}, + {"host", required_argument, NULL, 'h'}, + {"port", required_argument, NULL, 'p'}, + {NULL, 0, NULL, 0} + }; + + pg_logging_init(argv[0]); + progname = get_progname(argv[0]); + + if (argc > 1) + { + if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0) + { + usage(progname); + exit(0); + } + if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0) + { + puts("pg_keytool (PostgreSQL) " PG_VERSION); + exit(0); + } + } + + while ((c = getopt_long(argc, argv, "h:p:s", + long_options, &optindex)) != -1) + { + switch (c) + { + case 'h': + host = pg_strdup(optarg); + break; + + case 'p': + port_str = pg_strdup(optarg); + break; + + case 's': + to_server = true; + break; + + case '?': + /* Actual help option given */ + if (strcmp(argv[optind - 1], "-?") == 0) + { + usage(progname); + exit(EXIT_SUCCESS); + } + + default: + pg_log_error("Try \"%s --help\" for more information.", progname); + exit(1); + } + } + + /* Complain if any arguments remain */ + if (optind < argc) + { + fprintf(stderr, _("%s: too many command-line arguments (first is \"%s\")\n"), + progname, argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + if ((host || port_str) && !to_server) + { + pg_log_error("host and port can only be passed along with the -s option"); + exit(1); + } + + /* Read the key. */ + n = 0; + /* Key length in characters (two characters per hexadecimal digit) */ + while ((c = getchar()) != EOF && c != '\n') + { + if (n >= ENCRYPTION_KEY_CHARS) + { + pg_log_error("The key is too long"); + exit(EXIT_FAILURE); + } + + key_chars[n++] = c; + } + + if (n < ENCRYPTION_KEY_CHARS) + { + pg_log_error("The key is too short"); + exit(EXIT_FAILURE); + } + + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + { + if (sscanf(key_chars + 2 * i, "%2hhx", encryption_key + i) == 0) + { + pg_log_error("Invalid character in encryption key at position %d", + 2 * i); + exit(EXIT_FAILURE); + } + } + + /* + * Send the encryption key either to stdout or to server. + */ + if (!to_server) + { + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + printf("%.2x", encryption_key[i]); + printf("\n"); + } + else + { + if (!send_key_to_postmaster(host, port_str, encryption_key)) + pg_log_error("could not send encryption key to server"); + } + +#else + pg_log_fatal(ENCRYPTION_NOT_SUPPORTED_MSG); + exit(EXIT_FAILURE); +#endif /* USE_ENCRYPTION */ + return 0; +} diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c index dab1435014..ca37c9f373 100644 --- a/src/fe_utils/encryption.c +++ b/src/fe_utils/encryption.c @@ -88,3 +88,104 @@ run_encryption_key_command(unsigned char *encryption_key) pfree(buf); pclose(fp); } + +/* + * Send the contents of encryption_key in the form of special startup packet + * to a server that is being started. + * + * Returns true if we could send the message and false if not, however even + * success does not guarantee that server started up - caller should + * eventually test server connection himself. + * + * TODO Windows stuff? + */ +bool +send_key_to_postmaster(const char *host, const char *port, + const unsigned char *encryption_Key) +{ + const char **keywords = pg_malloc0(3 * sizeof(*keywords)); + const char **values = pg_malloc0(3 * sizeof(*values)); + int i; + PGconn *conn = NULL; + int sock = -1; + EncryptionKeyMsg message; + int msg_size, packet_size; + char *packet = NULL; + bool res = true; + +/* How many seconds we can wait for the postmaster to receive the key. */ +#define SEND_ENCRYPT_KEY_TIMEOUT 60 + + if (host) + { + keywords[0] = "host"; + values[0] = host; + } + keywords[1] = "port"; + values[1] = port; + + /* Compose the message. */ + message.encryptionKeyCode = pg_hton32(ENCRYPTION_KEY_MSG_CODE); + message.version = 1; + memcpy(message.data, encryption_key, ENCRYPTION_KEY_LENGTH); + msg_size = offsetof(EncryptionKeyMsg, data) + ENCRYPTION_KEY_LENGTH; + + packet_size = msg_size + 4; + packet = (char *) palloc(packet_size); + *((int32 *) packet) = pg_hton32(packet_size); + memcpy(packet + 4, &message, msg_size); + + /* + * Although we don't expect the server to accept regular libpq messages, + * we try to get at least a valid socket. + */ + for (i = 0; i < SEND_ENCRYPT_KEY_TIMEOUT + 1; i++) + { + if (i > 0) + /* Sleep for 1 second. */ + pg_usleep(1000000L); + + if (conn) + { + PQfinish(conn); + conn = NULL; + } + + conn = PQconnectStartParams(keywords, values, false); + if (conn == NULL) + continue; + + sock = PQsocket(conn); + /* Cannot send the key if there's no valid socket. */ + if (sock == -1) + continue; + + /* Non-blocking write would only make this simple case tricky. */ + if (!pg_set_block(sock)) + continue; + + retry: + /* + * Send the packet. Here we need to use low level API because the server + * does is not fully up so libpq cannot be used properly. + */ + if (send(sock, (char *) packet, packet_size, 0) != packet_size) + { + if (SOCK_ERRNO == EINTR) + /* Interrupted system call - we'll just try again */ + goto retry; + continue; + } + else + /* Success */ + break; + } + + pg_free(keywords); + pg_free(values); + if (conn) + PQfinish(conn); + pfree(packet); + + return res; +} diff --git a/src/include/fe_utils/encryption.h b/src/include/fe_utils/encryption.h index 69e3b2f756..7307557ed6 100644 --- a/src/include/fe_utils/encryption.h +++ b/src/include/fe_utils/encryption.h @@ -16,3 +16,5 @@ extern char *encryption_key_command; extern void run_encryption_key_command(unsigned char *encryption_key); +extern bool send_key_to_postmaster(const char *host, const char *port, + const unsigned char *encryption_Key); diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h index baf6a4b6c0..09822801c7 100644 --- a/src/include/libpq/pqcomm.h +++ b/src/include/libpq/pqcomm.h @@ -23,6 +23,8 @@ #endif #include <netinet/in.h> +#include "storage/encryption.h" + #ifdef HAVE_STRUCT_SOCKADDR_STORAGE #ifndef HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY @@ -205,4 +207,26 @@ typedef struct CancelRequestPacket #define NEGOTIATE_SSL_CODE PG_PROTOCOL(1234,5679) #define NEGOTIATE_GSS_CODE PG_PROTOCOL(1234,5680) +/* Client can also send cluster encryption key to the postmaster. */ +#define ENCRYPTION_KEY_MSG_CODE PG_PROTOCOL(1234,5681) + +/* + * Message containing the encryption key, received by postmaster during + * startup. + * + * TODO Consider adding extension identifier and key_length field so that + * postmaster can also receove extension-specific keys. Extension that needs a + * key would call a function from _PG_init() that tells postmaster where to + * store the key and how long the key is. (Should also postgres in standalone + * mode accept keys for extensions?) + */ +typedef struct EncryptionKeyMsg +{ + /* Note this integer field is stored in network byte order! */ + MsgType encryptionKeyCode; /* code to identify a key message */ + + unsigned char version; /* message format version. */ + char data[ENCRYPTION_KEY_LENGTH]; /* the key itself. */ +} EncryptionKeyMsg; + #endif /* PQCOMM_H */ diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h index 8479b76a34..4f7b96d4f3 100644 --- a/src/include/storage/encryption.h +++ b/src/include/storage/encryption.h @@ -60,6 +60,29 @@ typedef enum CipherKind /* Key to encrypt / decrypt data. */ extern unsigned char encryption_key[]; +#ifndef FRONTEND +/* + * Space for the encryption key in shared memory. Backend that receives the + * key during startup stores it here so postmaster can eventually take a local + * copy. + * + * We cannot protect the "initialized" field with a spinlock because the data + * will be accessed by postmaster, but it should not be necessary for bool + * type. Furthermore, synchronization is not really critical here, see + * processEncryptionKey(). + */ +typedef struct ShmemEncryptionKey +{ + char data[ENCRYPTION_KEY_LENGTH]; /* the key */ + bool initialized; /* received the key? */ +} ShmemEncryptionKey; + +/* + * Encryption key in the shared memory. + */ +extern ShmemEncryptionKey *encryption_key_shmem; +#endif /* FRONTEND */ + /* * The encrypted data is a series of blocks of size * ENCRYPTION_BLOCK. Currently we use the EVP_aes_256_xts implementation. Make @@ -86,6 +109,9 @@ extern char encryption_verification[]; extern bool encryption_setup_done; #ifndef FRONTEND +extern Size EncryptionShmemSize(void); +extern void EncryptionShmemInit(void); + typedef int (*read_encryption_key_cb) (void); extern void read_encryption_key(read_encryption_key_cb read_char); #endif /* FRONTEND */ -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0004-Minor-refactoring-of-pg_ctl.c.patch ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH v7 1/5] Allow binaryheap to use any type. @ 2023-09-04 22:04 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: Nathan Bossart @ 2023-09-04 22:04 UTC (permalink / raw) We would like to make binaryheap available to frontend code in a future commit, but presently the implementation uses Datum for the node type, which is inaccessible in the frontend. This commit modifies the implementation to allow using any type for the nodes. binaryheap_allocate() now requires callers to specify the size of the node, and several of the other functions now use void pointers. --- src/backend/executor/nodeGatherMerge.c | 19 ++-- src/backend/executor/nodeMergeAppend.c | 22 ++--- src/backend/lib/binaryheap.c | 99 +++++++++++-------- src/backend/postmaster/pgarch.c | 36 ++++--- .../replication/logical/reorderbuffer.c | 21 ++-- src/backend/storage/buffer/bufmgr.c | 20 ++-- src/include/lib/binaryheap.h | 21 ++-- 7 files changed, 137 insertions(+), 101 deletions(-) diff --git a/src/backend/executor/nodeGatherMerge.c b/src/backend/executor/nodeGatherMerge.c index 9d5e1a46e9..f76406b575 100644 --- a/src/backend/executor/nodeGatherMerge.c +++ b/src/backend/executor/nodeGatherMerge.c @@ -52,7 +52,7 @@ typedef struct GMReaderTupleBuffer } GMReaderTupleBuffer; static TupleTableSlot *ExecGatherMerge(PlanState *pstate); -static int32 heap_compare_slots(Datum a, Datum b, void *arg); +static int32 heap_compare_slots(const void *a, const void *b, void *arg); static TupleTableSlot *gather_merge_getnext(GatherMergeState *gm_state); static MinimalTuple gm_readnext_tuple(GatherMergeState *gm_state, int nreader, bool nowait, bool *done); @@ -429,6 +429,7 @@ gather_merge_setup(GatherMergeState *gm_state) /* Allocate the resources for the merge */ gm_state->gm_heap = binaryheap_allocate(nreaders + 1, + sizeof(int), heap_compare_slots, gm_state); } @@ -489,7 +490,7 @@ reread: /* Don't have a tuple yet, try to get one */ if (gather_merge_readnext(gm_state, i, nowait)) binaryheap_add_unordered(gm_state->gm_heap, - Int32GetDatum(i)); + &i); } else { @@ -565,14 +566,14 @@ gather_merge_getnext(GatherMergeState *gm_state) * the heap, because it might now compare differently against the * other elements of the heap. */ - i = DatumGetInt32(binaryheap_first(gm_state->gm_heap)); + binaryheap_first(gm_state->gm_heap, &i); if (gather_merge_readnext(gm_state, i, false)) - binaryheap_replace_first(gm_state->gm_heap, Int32GetDatum(i)); + binaryheap_replace_first(gm_state->gm_heap, &i); else { /* reader exhausted, remove it from heap */ - (void) binaryheap_remove_first(gm_state->gm_heap); + binaryheap_remove_first(gm_state->gm_heap, NULL); } } @@ -585,7 +586,7 @@ gather_merge_getnext(GatherMergeState *gm_state) else { /* Return next tuple from whichever participant has the leading one */ - i = DatumGetInt32(binaryheap_first(gm_state->gm_heap)); + binaryheap_first(gm_state->gm_heap, &i); return gm_state->gm_slots[i]; } } @@ -750,11 +751,11 @@ typedef int32 SlotNumber; * Compare the tuples in the two given slots. */ static int32 -heap_compare_slots(Datum a, Datum b, void *arg) +heap_compare_slots(const void *a, const void *b, void *arg) { GatherMergeState *node = (GatherMergeState *) arg; - SlotNumber slot1 = DatumGetInt32(a); - SlotNumber slot2 = DatumGetInt32(b); + SlotNumber slot1 = *((const int *) a); + SlotNumber slot2 = *((const int *) b); TupleTableSlot *s1 = node->gm_slots[slot1]; TupleTableSlot *s2 = node->gm_slots[slot2]; diff --git a/src/backend/executor/nodeMergeAppend.c b/src/backend/executor/nodeMergeAppend.c index 21b5726e6e..e0ace294a0 100644 --- a/src/backend/executor/nodeMergeAppend.c +++ b/src/backend/executor/nodeMergeAppend.c @@ -52,7 +52,7 @@ typedef int32 SlotNumber; static TupleTableSlot *ExecMergeAppend(PlanState *pstate); -static int heap_compare_slots(Datum a, Datum b, void *arg); +static int heap_compare_slots(const void *a, const void *b, void *arg); /* ---------------------------------------------------------------- @@ -125,8 +125,8 @@ ExecInitMergeAppend(MergeAppend *node, EState *estate, int eflags) mergestate->ms_nplans = nplans; mergestate->ms_slots = (TupleTableSlot **) palloc0(sizeof(TupleTableSlot *) * nplans); - mergestate->ms_heap = binaryheap_allocate(nplans, heap_compare_slots, - mergestate); + mergestate->ms_heap = binaryheap_allocate(nplans, sizeof(int), + heap_compare_slots, mergestate); /* * Miscellaneous initialization @@ -229,7 +229,7 @@ ExecMergeAppend(PlanState *pstate) { node->ms_slots[i] = ExecProcNode(node->mergeplans[i]); if (!TupIsNull(node->ms_slots[i])) - binaryheap_add_unordered(node->ms_heap, Int32GetDatum(i)); + binaryheap_add_unordered(node->ms_heap, &i); } binaryheap_build(node->ms_heap); node->ms_initialized = true; @@ -244,12 +244,12 @@ ExecMergeAppend(PlanState *pstate) * by doing this before returning from the prior call, but it's better * to not pull tuples until necessary.) */ - i = DatumGetInt32(binaryheap_first(node->ms_heap)); + binaryheap_first(node->ms_heap, &i); node->ms_slots[i] = ExecProcNode(node->mergeplans[i]); if (!TupIsNull(node->ms_slots[i])) - binaryheap_replace_first(node->ms_heap, Int32GetDatum(i)); + binaryheap_replace_first(node->ms_heap, &i); else - (void) binaryheap_remove_first(node->ms_heap); + binaryheap_remove_first(node->ms_heap, NULL); } if (binaryheap_empty(node->ms_heap)) @@ -259,7 +259,7 @@ ExecMergeAppend(PlanState *pstate) } else { - i = DatumGetInt32(binaryheap_first(node->ms_heap)); + binaryheap_first(node->ms_heap, &i); result = node->ms_slots[i]; } @@ -270,11 +270,11 @@ ExecMergeAppend(PlanState *pstate) * Compare the tuples in the two given slots. */ static int32 -heap_compare_slots(Datum a, Datum b, void *arg) +heap_compare_slots(const void *a, const void *b, void *arg) { MergeAppendState *node = (MergeAppendState *) arg; - SlotNumber slot1 = DatumGetInt32(a); - SlotNumber slot2 = DatumGetInt32(b); + SlotNumber slot1 = *((const int *) a); + SlotNumber slot2 = *((const int *) b); TupleTableSlot *s1 = node->ms_slots[slot1]; TupleTableSlot *s2 = node->ms_slots[slot2]; diff --git a/src/backend/lib/binaryheap.c b/src/backend/lib/binaryheap.c index 1737546757..6f63181c4c 100644 --- a/src/backend/lib/binaryheap.c +++ b/src/backend/lib/binaryheap.c @@ -29,16 +29,19 @@ static void sift_up(binaryheap *heap, int node_off); * argument specified by 'arg'. */ binaryheap * -binaryheap_allocate(int capacity, binaryheap_comparator compare, void *arg) +binaryheap_allocate(int capacity, size_t elem_size, + binaryheap_comparator compare, void *arg) { int sz; binaryheap *heap; - sz = offsetof(binaryheap, bh_nodes) + sizeof(Datum) * capacity; + sz = offsetof(binaryheap, bh_nodes) + elem_size * (capacity + 1); heap = (binaryheap *) palloc(sz); heap->bh_space = capacity; + heap->bh_elem_size = elem_size; heap->bh_compare = compare; heap->bh_arg = arg; + heap->bh_hole = &heap->bh_nodes[capacity * elem_size]; heap->bh_size = 0; heap->bh_has_heap_property = true; @@ -97,6 +100,27 @@ parent_offset(int i) return (i - 1) / 2; } +/* + * This utility function returns a pointer to the nth node of the binary heap. + */ +static inline void * +bh_node(binaryheap *heap, int n) +{ + return &heap->bh_nodes[n * heap->bh_elem_size]; +} + +/* + * This utility function sets the nth node of the binary heap to the value that + * d points to. + */ +static inline void +bh_set(binaryheap *heap, int n, const void *d) +{ + void *node = bh_node(heap, n); + + memmove(node, d, heap->bh_elem_size); +} + /* * binaryheap_add_unordered * @@ -106,12 +130,12 @@ parent_offset(int i) * afterwards. */ void -binaryheap_add_unordered(binaryheap *heap, Datum d) +binaryheap_add_unordered(binaryheap *heap, void *d) { if (heap->bh_size >= heap->bh_space) elog(ERROR, "out of binary heap slots"); heap->bh_has_heap_property = false; - heap->bh_nodes[heap->bh_size] = d; + bh_set(heap, heap->bh_size, d); heap->bh_size++; } @@ -138,11 +162,11 @@ binaryheap_build(binaryheap *heap) * the heap property. */ void -binaryheap_add(binaryheap *heap, Datum d) +binaryheap_add(binaryheap *heap, void *d) { if (heap->bh_size >= heap->bh_space) elog(ERROR, "out of binary heap slots"); - heap->bh_nodes[heap->bh_size] = d; + bh_set(heap, heap->bh_size, d); heap->bh_size++; sift_up(heap, heap->bh_size - 1); } @@ -154,11 +178,11 @@ binaryheap_add(binaryheap *heap, Datum d) * without modifying the heap. The caller must ensure that this * routine is not used on an empty heap. Always O(1). */ -Datum -binaryheap_first(binaryheap *heap) +void +binaryheap_first(binaryheap *heap, void *result) { Assert(!binaryheap_empty(heap) && heap->bh_has_heap_property); - return heap->bh_nodes[0]; + memmove(result, heap->bh_nodes, heap->bh_elem_size); } /* @@ -169,31 +193,28 @@ binaryheap_first(binaryheap *heap) * that this routine is not used on an empty heap. O(log n) worst * case. */ -Datum -binaryheap_remove_first(binaryheap *heap) +void +binaryheap_remove_first(binaryheap *heap, void *result) { - Datum result; - Assert(!binaryheap_empty(heap) && heap->bh_has_heap_property); /* extract the root node, which will be the result */ - result = heap->bh_nodes[0]; + if (result) + memmove(result, heap->bh_nodes, heap->bh_elem_size); /* easy if heap contains one element */ if (heap->bh_size == 1) { heap->bh_size--; - return result; + return; } /* * Remove the last node, placing it in the vacated root entry, and sift * the new root node down to its correct position. */ - heap->bh_nodes[0] = heap->bh_nodes[--heap->bh_size]; + bh_set(heap, 0, bh_node(heap, --heap->bh_size)); sift_down(heap, 0); - - return result; } /* @@ -204,11 +225,11 @@ binaryheap_remove_first(binaryheap *heap) * sifting the new node down. */ void -binaryheap_replace_first(binaryheap *heap, Datum d) +binaryheap_replace_first(binaryheap *heap, void *d) { Assert(!binaryheap_empty(heap) && heap->bh_has_heap_property); - heap->bh_nodes[0] = d; + bh_set(heap, 0, d); if (heap->bh_size > 1) sift_down(heap, 0); @@ -221,26 +242,26 @@ binaryheap_replace_first(binaryheap *heap, Datum d) static void sift_up(binaryheap *heap, int node_off) { - Datum node_val = heap->bh_nodes[node_off]; + memmove(heap->bh_hole, bh_node(heap, node_off), heap->bh_elem_size); /* * Within the loop, the node_off'th array entry is a "hole" that - * notionally holds node_val, but we don't actually store node_val there - * till the end, saving some unnecessary data copying steps. + * notionally holds the swapped value, but we don't actually store the + * value there till the end, saving some unnecessary data copying steps. */ while (node_off != 0) { int cmp; int parent_off; - Datum parent_val; + void *parent_val; /* * If this node is smaller than its parent, the heap condition is * satisfied, and we're done. */ parent_off = parent_offset(node_off); - parent_val = heap->bh_nodes[parent_off]; - cmp = heap->bh_compare(node_val, + parent_val = bh_node(heap, parent_off); + cmp = heap->bh_compare(heap->bh_hole, parent_val, heap->bh_arg); if (cmp <= 0) @@ -250,11 +271,11 @@ sift_up(binaryheap *heap, int node_off) * Otherwise, swap the parent value with the hole, and go on to check * the node's new parent. */ - heap->bh_nodes[node_off] = parent_val; + bh_set(heap, node_off, parent_val); node_off = parent_off; } /* Re-fill the hole */ - heap->bh_nodes[node_off] = node_val; + bh_set(heap, node_off, heap->bh_hole); } /* @@ -264,12 +285,12 @@ sift_up(binaryheap *heap, int node_off) static void sift_down(binaryheap *heap, int node_off) { - Datum node_val = heap->bh_nodes[node_off]; + memmove(heap->bh_hole, bh_node(heap, node_off), heap->bh_elem_size); /* * Within the loop, the node_off'th array entry is a "hole" that - * notionally holds node_val, but we don't actually store node_val there - * till the end, saving some unnecessary data copying steps. + * notionally holds the swapped value, but we don't actually store the + * value there till the end, saving some unnecessary data copying steps. */ while (true) { @@ -279,21 +300,21 @@ sift_down(binaryheap *heap, int node_off) /* Is the left child larger than the parent? */ if (left_off < heap->bh_size && - heap->bh_compare(node_val, - heap->bh_nodes[left_off], + heap->bh_compare(heap->bh_hole, + bh_node(heap, left_off), heap->bh_arg) < 0) swap_off = left_off; /* Is the right child larger than the parent? */ if (right_off < heap->bh_size && - heap->bh_compare(node_val, - heap->bh_nodes[right_off], + heap->bh_compare(heap->bh_hole, + bh_node(heap, right_off), heap->bh_arg) < 0) { /* swap with the larger child */ if (!swap_off || - heap->bh_compare(heap->bh_nodes[left_off], - heap->bh_nodes[right_off], + heap->bh_compare(bh_node(heap, left_off), + bh_node(heap, right_off), heap->bh_arg) < 0) swap_off = right_off; } @@ -309,9 +330,9 @@ sift_down(binaryheap *heap, int node_off) * Otherwise, swap the hole with the child that violates the heap * property; then go on to check its children. */ - heap->bh_nodes[node_off] = heap->bh_nodes[swap_off]; + bh_set(heap, node_off, bh_node(heap, swap_off)); node_off = swap_off; } /* Re-fill the hole */ - heap->bh_nodes[node_off] = node_val; + bh_set(heap, node_off, heap->bh_hole); } diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c index 46af349564..f7ee95d8f9 100644 --- a/src/backend/postmaster/pgarch.c +++ b/src/backend/postmaster/pgarch.c @@ -145,7 +145,7 @@ static bool pgarch_readyXlog(char *xlog); static void pgarch_archiveDone(char *xlog); static void pgarch_die(int code, Datum arg); static void HandlePgArchInterrupts(void); -static int ready_file_comparator(Datum a, Datum b, void *arg); +static int ready_file_comparator(const void *a, const void *b, void *arg); static void LoadArchiveLibrary(void); static void pgarch_call_module_shutdown_cb(int code, Datum arg); @@ -250,6 +250,7 @@ PgArchiverMain(void) /* Initialize our max-heap for prioritizing files to archive. */ arch_files->arch_heap = binaryheap_allocate(NUM_FILES_PER_DIRECTORY_SCAN, + sizeof(char *), ready_file_comparator, NULL); /* Load the archive_library. */ @@ -631,22 +632,27 @@ pgarch_readyXlog(char *xlog) /* If the heap isn't full yet, quickly add it. */ arch_file = arch_files->arch_filenames[arch_files->arch_heap->bh_size]; strcpy(arch_file, basename); - binaryheap_add_unordered(arch_files->arch_heap, CStringGetDatum(arch_file)); + binaryheap_add_unordered(arch_files->arch_heap, &arch_file); /* If we just filled the heap, make it a valid one. */ if (arch_files->arch_heap->bh_size == NUM_FILES_PER_DIRECTORY_SCAN) binaryheap_build(arch_files->arch_heap); } - else if (ready_file_comparator(binaryheap_first(arch_files->arch_heap), - CStringGetDatum(basename), NULL) > 0) + else { - /* - * Remove the lowest priority file and add the current one to the - * heap. - */ - arch_file = DatumGetCString(binaryheap_remove_first(arch_files->arch_heap)); - strcpy(arch_file, basename); - binaryheap_add(arch_files->arch_heap, CStringGetDatum(arch_file)); + char *root; + + binaryheap_first(arch_files->arch_heap, &root); + if (ready_file_comparator(&root, &basename, NULL) > 0) + { + /* + * Remove the lowest priority file and add the current one to + * the heap. + */ + binaryheap_remove_first(arch_files->arch_heap, &arch_file); + strcpy(arch_file, basename); + binaryheap_add(arch_files->arch_heap, &arch_file); + } } } FreeDir(rldir); @@ -668,7 +674,7 @@ pgarch_readyXlog(char *xlog) */ arch_files->arch_files_size = arch_files->arch_heap->bh_size; for (int i = 0; i < arch_files->arch_files_size; i++) - arch_files->arch_files[i] = DatumGetCString(binaryheap_remove_first(arch_files->arch_heap)); + binaryheap_remove_first(arch_files->arch_heap, &arch_files->arch_files[i]); /* Return the highest priority file. */ arch_files->arch_files_size--; @@ -686,10 +692,10 @@ pgarch_readyXlog(char *xlog) * If "a" and "b" have equivalent values, 0 will be returned. */ static int -ready_file_comparator(Datum a, Datum b, void *arg) +ready_file_comparator(const void *a, const void *b, void *arg) { - char *a_str = DatumGetCString(a); - char *b_str = DatumGetCString(b); + const char *a_str = *((const char **) a); + const char *b_str = *((const char **) b); bool a_history = IsTLHistoryFileName(a_str); bool b_history = IsTLHistoryFileName(b_str); diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 12edc5772a..e22680da0b 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -1223,11 +1223,11 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid, * Binary heap comparison function. */ static int -ReorderBufferIterCompare(Datum a, Datum b, void *arg) +ReorderBufferIterCompare(const void *a, const void *b, void *arg) { ReorderBufferIterTXNState *state = (ReorderBufferIterTXNState *) arg; - XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn; - XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn; + XLogRecPtr pos_a = state->entries[*((const int *) a)].lsn; + XLogRecPtr pos_b = state->entries[*((const int *) b)].lsn; if (pos_a < pos_b) return 1; @@ -1297,6 +1297,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn, /* allocate heap */ state->heap = binaryheap_allocate(state->nr_txns, + sizeof(int), ReorderBufferIterCompare, state); @@ -1330,7 +1331,8 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn, state->entries[off].change = cur_change; state->entries[off].txn = txn; - binaryheap_add_unordered(state->heap, Int32GetDatum(off++)); + binaryheap_add_unordered(state->heap, &off); + off++; } /* add subtransactions if they contain changes */ @@ -1359,7 +1361,8 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn, state->entries[off].change = cur_change; state->entries[off].txn = cur_txn; - binaryheap_add_unordered(state->heap, Int32GetDatum(off++)); + binaryheap_add_unordered(state->heap, &off); + off++; } } @@ -1384,7 +1387,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state) if (state->heap->bh_size == 0) return NULL; - off = DatumGetInt32(binaryheap_first(state->heap)); + binaryheap_first(state->heap, &off); entry = &state->entries[off]; /* free memory we might have "leaked" in the previous *Next call */ @@ -1414,7 +1417,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state) state->entries[off].lsn = next_change->lsn; state->entries[off].change = next_change; - binaryheap_replace_first(state->heap, Int32GetDatum(off)); + binaryheap_replace_first(state->heap, &off); return change; } @@ -1450,14 +1453,14 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state) /* txn stays the same */ state->entries[off].lsn = next_change->lsn; state->entries[off].change = next_change; - binaryheap_replace_first(state->heap, Int32GetDatum(off)); + binaryheap_replace_first(state->heap, &off); return change; } } /* ok, no changes there anymore, remove */ - binaryheap_remove_first(state->heap); + binaryheap_remove_first(state->heap, NULL); return change; } diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 3bd82dbfca..7c78a0d625 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -501,7 +501,7 @@ static void CheckForBufferLeaks(void); static int rlocator_comparator(const void *p1, const void *p2); static inline int buffertag_comparator(const BufferTag *ba, const BufferTag *bb); static inline int ckpt_buforder_comparator(const CkptSortItem *a, const CkptSortItem *b); -static int ts_ckpt_progress_comparator(Datum a, Datum b, void *arg); +static int ts_ckpt_progress_comparator(const void *a, const void *b, void *arg); /* @@ -2640,6 +2640,7 @@ BufferSync(int flags) * processed buffer is. */ ts_heap = binaryheap_allocate(num_spaces, + sizeof(CkptTsStatus *), ts_ckpt_progress_comparator, NULL); @@ -2649,7 +2650,7 @@ BufferSync(int flags) ts_stat->progress_slice = (float8) num_to_scan / ts_stat->num_to_scan; - binaryheap_add_unordered(ts_heap, PointerGetDatum(ts_stat)); + binaryheap_add_unordered(ts_heap, &ts_stat); } binaryheap_build(ts_heap); @@ -2665,8 +2666,9 @@ BufferSync(int flags) while (!binaryheap_empty(ts_heap)) { BufferDesc *bufHdr = NULL; - CkptTsStatus *ts_stat = (CkptTsStatus *) - DatumGetPointer(binaryheap_first(ts_heap)); + CkptTsStatus *ts_stat; + + binaryheap_first(ts_heap, &ts_stat); buf_id = CkptBufferIds[ts_stat->index].buf_id; Assert(buf_id != -1); @@ -2708,12 +2710,12 @@ BufferSync(int flags) /* Have all the buffers from the tablespace been processed? */ if (ts_stat->num_scanned == ts_stat->num_to_scan) { - binaryheap_remove_first(ts_heap); + binaryheap_remove_first(ts_heap, NULL); } else { /* update heap with the new progress */ - binaryheap_replace_first(ts_heap, PointerGetDatum(ts_stat)); + binaryheap_replace_first(ts_heap, &ts_stat); } /* @@ -5416,10 +5418,10 @@ ckpt_buforder_comparator(const CkptSortItem *a, const CkptSortItem *b) * progress. */ static int -ts_ckpt_progress_comparator(Datum a, Datum b, void *arg) +ts_ckpt_progress_comparator(const void *a, const void *b, void *arg) { - CkptTsStatus *sa = (CkptTsStatus *) a; - CkptTsStatus *sb = (CkptTsStatus *) b; + const CkptTsStatus *sa = *((const CkptTsStatus **) a); + const CkptTsStatus *sb = *((const CkptTsStatus **) b); /* we want a min-heap, so return 1 for the a < b */ if (sa->progress < sb->progress) diff --git a/src/include/lib/binaryheap.h b/src/include/lib/binaryheap.h index 52f7b06b25..b8c7ef81ef 100644 --- a/src/include/lib/binaryheap.h +++ b/src/include/lib/binaryheap.h @@ -15,7 +15,7 @@ * For a max-heap, the comparator must return <0 iff a < b, 0 iff a == b, * and >0 iff a > b. For a min-heap, the conditions are reversed. */ -typedef int (*binaryheap_comparator) (Datum a, Datum b, void *arg); +typedef int (*binaryheap_comparator) (const void *a, const void *b, void *arg); /* * binaryheap @@ -25,29 +25,32 @@ typedef int (*binaryheap_comparator) (Datum a, Datum b, void *arg); * bh_has_heap_property no unordered operations since last heap build * bh_compare comparison function to define the heap property * bh_arg user data for comparison function - * bh_nodes variable-length array of "space" nodes + * bh_hole extra node used during sifting operations + * bh_nodes variable-length array of "space + 1" nodes */ typedef struct binaryheap { int bh_size; int bh_space; + size_t bh_elem_size; bool bh_has_heap_property; /* debugging cross-check */ binaryheap_comparator bh_compare; void *bh_arg; - Datum bh_nodes[FLEXIBLE_ARRAY_MEMBER]; + void *bh_hole; + char bh_nodes[FLEXIBLE_ARRAY_MEMBER]; } binaryheap; -extern binaryheap *binaryheap_allocate(int capacity, +extern binaryheap *binaryheap_allocate(int capacity, size_t elem_size, binaryheap_comparator compare, void *arg); extern void binaryheap_reset(binaryheap *heap); extern void binaryheap_free(binaryheap *heap); -extern void binaryheap_add_unordered(binaryheap *heap, Datum d); +extern void binaryheap_add_unordered(binaryheap *heap, void *d); extern void binaryheap_build(binaryheap *heap); -extern void binaryheap_add(binaryheap *heap, Datum d); -extern Datum binaryheap_first(binaryheap *heap); -extern Datum binaryheap_remove_first(binaryheap *heap); -extern void binaryheap_replace_first(binaryheap *heap, Datum d); +extern void binaryheap_add(binaryheap *heap, void *d); +extern void binaryheap_first(binaryheap *heap, void *result); +extern void binaryheap_remove_first(binaryheap *heap, void *result); +extern void binaryheap_replace_first(binaryheap *heap, void *d); #define binaryheap_empty(h) ((h)->bh_size == 0) -- 2.25.1 --DocE+STaALJfprDB Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-Make-binaryheap-available-to-frontend-code.patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 5/6] Use background worker to do logical decoding. @ 2026-01-27 10:48 Antonin Houska <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: Antonin Houska @ 2026-01-27 10:48 UTC (permalink / raw) If the backend performing REPACK (CONCURRENTLY) does both data copying and logical decoding, it has to "travel in time" back and forth and therefore it has to invalidate system caches quite a few times. (The copying and the decoding work with different catalog snapshots.) As the decoding worker has separate caches, the switching is not necessary. Without the worker, it'd also be difficult to switch between potentially long running tasks like index build and WAL decoding. (No decoding during that time at all can suspend archiving / recycling of WAL segments for some time, which in turn may result in full disk.) Another problem is that, after having acquired AccessExclusiveLock (in order to swap the files), the backend needs to both decode and apply the data changes that took place while it was waiting for the lock. With the decoding worker, the decoding runs all the time, so the backend only needs to apply the changes. This can reduce the time the exclusive lock is held for. Note that the code added in order to handle ERRORs in the background worker almost duplicates the existing code that does the same for other types of workers (See ProcessParallelMessages() and ProcessParallelApplyMessages()). Refactoring of the existing code might be useful, to reduce the duplication. --- src/backend/access/heap/heapam_handler.c | 44 - src/backend/commands/cluster.c | 1179 +++++++++++++---- src/backend/libpq/pqmq.c | 5 + src/backend/postmaster/bgworker.c | 4 + src/backend/replication/logical/logical.c | 6 +- .../pgoutput_repack/pgoutput_repack.c | 54 +- src/backend/storage/ipc/procsignal.c | 4 + src/backend/tcop/postgres.c | 4 + .../utils/activity/wait_event_names.txt | 1 + src/include/access/tableam.h | 7 +- src/include/commands/cluster.h | 71 +- src/include/storage/procsignal.h | 1 + src/tools/pgindent/typedefs.list | 3 +- 13 files changed, 981 insertions(+), 402 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 908f1ef66c6..8589b3c940e 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -33,7 +33,6 @@ #include "catalog/index.h" #include "catalog/storage.h" #include "catalog/storage_xlog.h" -#include "commands/cluster.h" #include "commands/progress.h" #include "executor/executor.h" #include "miscadmin.h" @@ -688,7 +687,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, Relation OldIndex, bool use_sort, TransactionId OldestXmin, Snapshot snapshot, - LogicalDecodingContext *decoding_ctx, TransactionId *xid_cutoff, MultiXactId *multi_cutoff, double *num_tuples, @@ -710,7 +708,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; - XLogRecPtr end_of_wal_prev = GetFlushRecPtr(NULL); /* Remember if it's a system catalog */ is_system_catalog = IsSystemRelation(OldHeap); @@ -971,31 +968,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, ct_val[1] = *num_tuples; pgstat_progress_update_multi_param(2, ct_index, ct_val); } - - /* - * Process the WAL produced by the load, as well as by other - * transactions, so that the replication slot can advance and WAL does - * not pile up. Use wal_segment_size as a threshold so that we do not - * introduce the decoding overhead too often. - * - * Of course, we must not apply the changes until the initial load has - * completed. - * - * Note that our insertions into the new table should not be decoded - * as we (intentionally) do not write the logical decoding specific - * information to WAL. - */ - if (concurrent) - { - XLogRecPtr end_of_wal; - - end_of_wal = GetFlushRecPtr(NULL); - if ((end_of_wal - end_of_wal_prev) > wal_segment_size) - { - repack_decode_concurrent_changes(decoding_ctx, end_of_wal); - end_of_wal_prev = end_of_wal; - } - } } if (indexScan != NULL) @@ -1041,22 +1013,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); - - /* - * Try to keep the amount of not-yet-decoded WAL small, like - * above. - */ - if (concurrent) - { - XLogRecPtr end_of_wal; - - end_of_wal = GetFlushRecPtr(NULL); - if ((end_of_wal - end_of_wal_prev) > wal_segment_size) - { - repack_decode_concurrent_changes(decoding_ctx, end_of_wal); - end_of_wal_prev = end_of_wal; - } - } } tuplesort_end(tuplesort); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 03ccf10b782..e988a7a7296 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -12,12 +12,13 @@ * In concurrent mode, we lock the table with only ShareUpdateExclusiveLock, * then do an initial copy as above. However, while the tuples are being * copied, concurrent transactions could modify the table. To cope with those - * changes, we rely on logical decoding to obtain them from WAL. The changes - * are accumulated in a tuplestore. Once the initial copy is complete, we - * read the changes from the tuplestore and re-apply them on the new heap. - * Then we upgrade our ShareUpdateExclusiveLock to AccessExclusiveLock and - * swap the relfilenodes. This way, the time we hold a strong lock on the - * table is much reduced, and the bloat is eliminated. + * changes, we rely on logical decoding to obtain them from WAL. A bgworker + * consumes WAL while the initial copy is ongoing (to prevent excessive WAL + * from being reserved), and accumulates the changes in a file. Once the + * initial copy is complete, we read the changes from the file and re-apply + * them on the new heap. Then we upgrade our ShareUpdateExclusiveLock to + * AccessExclusiveLock and swap the relfilenodes. This way, the time we hold + * a strong lock on the table is much reduced, and the bloat is eliminated. * * There is hardly anything left of Paul Brown's original implementation... * @@ -45,6 +46,7 @@ #include "access/xlog_internal.h" #include "access/xloginsert.h" #include "access/xlogutils.h" +#include "access/xlogwait.h" #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" @@ -61,6 +63,8 @@ #include "commands/tablecmds.h" #include "commands/vacuum.h" #include "executor/executor.h" +#include "libpq/pqformat.h" +#include "libpq/pqmq.h" #include "miscadmin.h" #include "optimizer/optimizer.h" #include "pgstat.h" @@ -71,6 +75,8 @@ #include "storage/ipc.h" #include "storage/lmgr.h" #include "storage/predicate.h" +#include "storage/procsignal.h" +#include "tcop/tcopprot.h" #include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/guc.h" @@ -117,6 +123,12 @@ typedef struct IndexInsertState /* The WAL segment being decoded. */ static XLogSegNo repack_current_segment = 0; +/* + * The first file exported by the decoding worker must contain a snapshot, the + * following ones contain the data changes. + */ +#define WORKER_FILE_SNAPSHOT 0 + /* * Information needed to apply concurrent data changes. */ @@ -136,8 +148,113 @@ typedef struct ChangeDest /* Needed to update indexes of rel_dst. */ IndexInsertState *iistate; + + /* + * Sequential number of the file containing the changes. + * + * TODO This field makes the structure name less descriptive. Should we + * rename it, e.g. to ChangeApplyInfo? + */ + int file_seq; } ChangeDest; +/* + * Layout of shared memory used for communication between backend and the + * worker that performs logical decoding of data changes + */ +typedef struct DecodingWorkerShared +{ + /* Is the decoding initialized? */ + bool initialized; + + /* + * Once the worker has reached this LSN, it should close the current + * output file and either create a new one or exit, according to the field + * 'done'. If the value is InvalidXLogRecPtr, the worker should decode all + * the WAL available and keep checking this field. It is ok if the worker + * had already decoded records whose LSN is >= lsn_upto before this field + * has been set. + */ + XLogRecPtr lsn_upto; + + /* Exit after closing the current file? */ + bool done; + + /* The output is stored here. */ + SharedFileSet sfs; + + /* Number of the last file exported by the worker. */ + int last_exported; + + /* Synchronize access to the fields above. */ + slock_t mutex; + + /* Database to connect to. */ + Oid dbid; + + /* Role to connect as. */ + Oid roleid; + + /* Decode data changes of this relation. */ + Oid relid; + + /* The backend uses this to wait for the worker. */ + ConditionVariable cv; + + /* Info to signal the backend. */ + PGPROC *backend_proc; + pid_t backend_pid; + ProcNumber backend_proc_number; + + /* Error queue. */ + shm_mq *error_mq; + + /* + * Memory the queue is located in. + * + * For considerations on the value see the comments of + * PARALLEL_ERROR_QUEUE_SIZE. + */ +#define REPACK_ERROR_QUEUE_SIZE 16384 + char error_queue[FLEXIBLE_ARRAY_MEMBER]; +} DecodingWorkerShared; + +/* + * Generate worker's output file name. If relations of the same 'relid' happen + * to be processed at the same time, they must be from different databases and + * therefore different backends must be involved. (PID is already present in + * the fileset name.) + */ +static inline void +DecodingWorkerFileName(char *fname, Oid relid, uint32 seq) +{ + snprintf(fname, MAXPGPATH, "%u-%u", relid, seq); +} + +/* + * Backend-local information to control the decoding worker. + */ +typedef struct DecodingWorker +{ + /* The worker. */ + BackgroundWorkerHandle *handle; + + /* DecodingWorkerShared is in this segment. */ + dsm_segment *seg; + + /* Handle of the error queue. */ + shm_mq_handle *error_mqh; +} DecodingWorker; + +/* Pointer to currently running decoding worker. */ +static DecodingWorker *decoding_worker = NULL; + +/* + * Is there a message sent by a repack worker that the backend needs to + * receive? + */ +volatile sig_atomic_t RepackMessagePending = false; + static bool cluster_rel_recheck(RepackCommand cmd, Relation OldHeap, Oid indexOid, Oid userid, LOCKMODE lmode, int options); @@ -145,7 +262,7 @@ static void check_repack_concurrently_requirements(Relation rel); static void rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent); static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, - Snapshot snapshot, LogicalDecodingContext *decoding_ctx, + Snapshot snapshot, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, @@ -158,12 +275,10 @@ static List *get_tables_to_repack_partitioned(RepackCommand cmd, static bool cluster_is_permitted_for_relation(RepackCommand cmd, Oid relid, Oid userid); -static void begin_concurrent_repack(Relation rel); -static void end_concurrent_repack(void); static LogicalDecodingContext *setup_logical_decoding(Oid relid); -static HeapTuple get_changed_tuple(char *change); -static void apply_concurrent_changes(RepackDecodingState *dstate, - ChangeDest *dest); +static bool decode_concurrent_changes(LogicalDecodingContext *ctx, + DecodingWorkerShared *shared); +static void apply_concurrent_changes(BufFile *file, ChangeDest *dest); static void apply_concurrent_insert(Relation rel, HeapTuple tup, IndexInsertState *iistate, TupleTableSlot *index_slot); @@ -175,9 +290,9 @@ static void apply_concurrent_delete(Relation rel, HeapTuple tup_target); static HeapTuple find_target_tuple(Relation rel, ChangeDest *dest, HeapTuple tup_key, TupleTableSlot *ident_slot); -static void process_concurrent_changes(LogicalDecodingContext *decoding_ctx, - XLogRecPtr end_of_wal, - ChangeDest *dest); +static void process_concurrent_changes(XLogRecPtr end_of_wal, + ChangeDest *dest, + bool done); static IndexInsertState *get_index_insert_state(Relation relation, Oid ident_index_id, Relation *ident_index_p); @@ -186,7 +301,6 @@ static ScanKey build_identity_key(Oid ident_idx_oid, Relation rel_src, static void free_index_insert_state(IndexInsertState *iistate); static void cleanup_logical_decoding(LogicalDecodingContext *ctx); static void rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, - LogicalDecodingContext *decoding_ctx, TransactionId frozenXid, MultiXactId cutoffMulti); static List *build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes); @@ -196,6 +310,13 @@ static Relation process_single_relation(RepackStmt *stmt, ClusterParams *params); static Oid determine_clustered_index(Relation rel, bool usingindex, const char *indexname); +static void start_decoding_worker(Oid relid); +static void stop_decoding_worker(void); +static void repack_worker_internal(dsm_segment *seg); +static void export_initial_snapshot(Snapshot snapshot, + DecodingWorkerShared *shared); +static Snapshot get_initial_snapshot(DecodingWorker *worker); +static void ProcessRepackMessage(StringInfo msg); static const char *RepackCommandAsString(RepackCommand cmd); @@ -619,20 +740,20 @@ cluster_rel(RepackCommand cmd, Relation OldHeap, Oid indexOid, /* rebuild_relation does all the dirty work */ PG_TRY(); { - /* - * For concurrent processing, make sure that our logical decoding - * ignores data changes of other tables than the one we are - * processing. - */ - if (concurrent) - begin_concurrent_repack(OldHeap); - rebuild_relation(OldHeap, index, verbose, concurrent); } PG_FINALLY(); { if (concurrent) - end_concurrent_repack(); + { + /* + * Since during normal operation the worker was already asked to + * exit, stopping it explicitly is especially important on ERROR. + * However it still seems a good practice to make sure that the + * worker never survives the REPACK command. + */ + stop_decoding_worker(); + } } PG_END_TRY(); @@ -929,7 +1050,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent bool swap_toast_by_content; TransactionId frozenXid; MultiXactId cutoffMulti; - LogicalDecodingContext *decoding_ctx = NULL; Snapshot snapshot = NULL; #if USE_ASSERT_CHECKING LOCKMODE lmode; @@ -943,19 +1063,36 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent if (concurrent) { /* - * Prepare to capture the concurrent data changes. + * The worker needs to be member of the locking group we're the leader + * of. We ought to become the leader before the worker starts. The + * worker will join the group as soon as it starts. * - * Note that this call waits for all transactions with XID already - * assigned to finish. If some of those transactions is waiting for a - * lock conflicting with ShareUpdateExclusiveLock on our table (e.g. - * it runs CREATE INDEX), we can end up in a deadlock. Not sure this - * risk is worth unlocking/locking the table (and its clustering - * index) and checking again if its still eligible for REPACK - * CONCURRENTLY. + * This is to make sure that the deadlock described below is + * detectable by deadlock.c: if the worker waits for a transaction to + * complete and we are waiting for the worker output, then effectively + * we (i.e. this backend) are waiting for that transaction. */ - decoding_ctx = setup_logical_decoding(tableOid); + BecomeLockGroupLeader(); + + /* + * Start the worker that decodes data changes applied while we're + * copying the table contents. + * + * Note that the worker has to wait for all transactions with XID + * already assigned to finish. If some of those transactions is + * waiting for a lock conflicting with ShareUpdateExclusiveLock on our + * table (e.g. it runs CREATE INDEX), we can end up in a deadlock. + * Not sure this risk is worth unlocking/locking the table (and its + * clustering index) and checking again if it's still eligible for + * REPACK CONCURRENTLY. + */ + start_decoding_worker(tableOid); + + /* + * Wait until the worker has the initial snapshot and retrieve it. + */ + snapshot = get_initial_snapshot(decoding_worker); - snapshot = SnapBuildInitialSnapshotForRepack(decoding_ctx->snapshot_builder); PushActiveSnapshot(snapshot); } @@ -980,7 +1117,7 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent NewHeap = table_open(OIDNewHeap, NoLock); /* Copy the heap data into the new table in the desired order */ - copy_table_data(NewHeap, OldHeap, index, snapshot, decoding_ctx, verbose, + copy_table_data(NewHeap, OldHeap, index, snapshot, verbose, &swap_toast_by_content, &frozenXid, &cutoffMulti); /* The historic snapshot won't be needed anymore. */ @@ -1001,14 +1138,11 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, bool concurrent if (index) index_close(index, NoLock); - rebuild_relation_finish_concurrent(NewHeap, OldHeap, decoding_ctx, - frozenXid, cutoffMulti); + rebuild_relation_finish_concurrent(NewHeap, OldHeap, frozenXid, + cutoffMulti); pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_FINAL_CLEANUP); - - /* Done with decoding. */ - cleanup_logical_decoding(decoding_ctx); } else { @@ -1179,8 +1313,7 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, Oid NewAccessMethod, */ static void copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, - Snapshot snapshot, LogicalDecodingContext *decoding_ctx, - bool verbose, bool *pSwapToastByContent, + Snapshot snapshot, bool verbose, bool *pSwapToastByContent, TransactionId *pFreezeXid, MultiXactId *pCutoffMulti) { Relation relRelation; @@ -1341,7 +1474,6 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex, */ table_relation_copy_for_cluster(OldHeap, NewHeap, OldIndex, use_sort, cutoffs.OldestXmin, snapshot, - decoding_ctx, &cutoffs.FreezeLimit, &cutoffs.MultiXactCutoff, &num_tuples, &tups_vacuumed, @@ -2371,62 +2503,10 @@ RepackCommandAsString(RepackCommand cmd) case REPACK_COMMAND_CLUSTER: return "CLUSTER"; } - return "???"; /* keep compiler quiet */ + return "???"; /* keep compiler quiet */ } -/* - * Call this function before REPACK CONCURRENTLY starts to setup logical - * decoding. It makes sure that other users of the table put enough - * information into WAL. - * - * The point is that at various places we expect that the table we're - * processing is treated like a system catalog. For example, we need to be - * able to scan it using a "historic snapshot" anytime during the processing - * (as opposed to scanning only at the start point of the decoding, as logical - * replication does during initial table synchronization), in order to apply - * concurrent UPDATE / DELETE commands. - * - * Note that TOAST table needs no attention here as it's not scanned using - * historic snapshot. - */ -static void -begin_concurrent_repack(Relation rel) -{ - Oid toastrelid; - - /* - * Avoid logical decoding of other relations by this backend. The lock we - * have guarantees that the actual locator cannot be changed concurrently: - * TRUNCATE needs AccessExclusiveLock. - */ - Assert(CheckRelationLockedByMe(rel, ShareUpdateExclusiveLock, false)); - repacked_rel_locator = rel->rd_locator; - toastrelid = rel->rd_rel->reltoastrelid; - if (OidIsValid(toastrelid)) - { - Relation toastrel; - - /* Avoid logical decoding of other TOAST relations. */ - toastrel = table_open(toastrelid, AccessShareLock); - repacked_rel_toast_locator = toastrel->rd_locator; - table_close(toastrel, AccessShareLock); - } -} - -/* - * Call this when done with REPACK CONCURRENTLY. - */ -static void -end_concurrent_repack(void) -{ - /* - * Restore normal function of (future) logical decoding for this backend. - */ - repacked_rel_locator.relNumber = InvalidOid; - repacked_rel_toast_locator.relNumber = InvalidOid; -} - /* * Is this backend performing logical decoding on behalf of REPACK * (CONCURRENTLY) ? @@ -2491,9 +2571,10 @@ static LogicalDecodingContext * setup_logical_decoding(Oid relid) { Relation rel; - TupleDesc tupdesc; + Oid toastrelid; LogicalDecodingContext *ctx; - RepackDecodingState *dstate = palloc0_object(RepackDecodingState); + NameData slotname; + RepackDecodingState *dstate; /* * REPACK CONCURRENTLY is not allowed in a transaction block, so this @@ -2501,20 +2582,21 @@ setup_logical_decoding(Oid relid) */ Assert(!TransactionIdIsValid(GetTopTransactionIdIfAny())); - /* - * A single backend should not execute multiple REPACK commands at a time, - * so use PID to make the slot unique. - */ - snprintf(NameStr(dstate->slotname), NAMEDATALEN, "repack_%d", MyProcPid); - /* * Make sure we can use logical decoding. */ CheckSlotPermissions(); CheckLogicalDecodingRequirements(); - /* RS_TEMPORARY so that the slot gets cleaned up on ERROR. */ - ReplicationSlotCreate(NameStr(dstate->slotname), true, RS_TEMPORARY, - false, false, false); + /* + * A single backend should not execute multiple REPACK commands at a time, + * so use PID to make the slot unique. + * + * RS_TEMPORARY so that the slot gets cleaned up on ERROR. + */ + snprintf(NameStr(slotname), NAMEDATALEN, "repack_%d", MyProcPid); + ReplicationSlotCreate(NameStr(slotname), true, RS_TEMPORARY, false, false, + false); + EnsureLogicalDecodingEnabled(); /* @@ -2537,104 +2619,109 @@ setup_logical_decoding(Oid relid) DecodingContextFindStartpoint(ctx); + /* + * decode_concurrent_changes() needs non-blocking callback. + */ + ctx->reader->routine.page_read = read_local_xlog_page_no_wait; + + /* + * read_local_xlog_page_no_wait() needs to be able to indicate the end of + * WAL. + */ + ctx->reader->private_data = MemoryContextAllocZero(ctx->context, + sizeof(ReadLocalXLogPageNoWaitPrivate)); + + /* Some WAL records should have been read. */ Assert(ctx->reader->EndRecPtr != InvalidXLogRecPtr); + /* + * Initialize repack_current_segment so that we can notice WAL segment + * boundaries. + */ XLByteToSeg(ctx->reader->EndRecPtr, repack_current_segment, wal_segment_size); - /* - * Setup structures to store decoded changes. - */ + dstate = palloc0_object(RepackDecodingState); dstate->relid = relid; - dstate->tstore = tuplestore_begin_heap(false, false, - maintenance_work_mem); - /* Caller should already have the table locked. */ - rel = table_open(relid, NoLock); - tupdesc = CreateTupleDescCopy(RelationGetDescr(rel)); - dstate->tupdesc = tupdesc; - table_close(rel, NoLock); + /* + * Tuple descriptor may be needed to flatten a tuple before we write it to + * a file. A copy is needed because the decoding worker invalidates system + * caches before it starts to do the actual work. + */ + rel = table_open(relid, AccessShareLock); + dstate->tupdesc = CreateTupleDescCopy(RelationGetDescr(rel)); - /* Initialize the descriptor to store the changes ... */ - dstate->tupdesc_change = CreateTemplateTupleDesc(1); + /* Avoid logical decoding of other relations. */ + repacked_rel_locator = rel->rd_locator; + toastrelid = rel->rd_rel->reltoastrelid; + if (OidIsValid(toastrelid)) + { + Relation toastrel; - TupleDescInitEntry(dstate->tupdesc_change, 1, NULL, BYTEAOID, -1, 0); - /* ... as well as the corresponding slot. */ - dstate->tsslot = MakeSingleTupleTableSlot(dstate->tupdesc_change, - &TTSOpsMinimalTuple); + /* Avoid logical decoding of other TOAST relations. */ + toastrel = table_open(toastrelid, AccessShareLock); + repacked_rel_toast_locator = toastrel->rd_locator; + table_close(toastrel, AccessShareLock); + } + table_close(rel, AccessShareLock); - dstate->resowner = ResourceOwnerCreate(CurrentResourceOwner, - "logical decoding"); + /* The file will be set as soon as we have it opened. */ + dstate->file = NULL; ctx->output_writer_private = dstate; + return ctx; } /* - * Retrieve tuple from ConcurrentChange structure. + * Decode logical changes from the WAL sequence and store them to a file. * - * The input data starts with the structure but it might not be appropriately - * aligned. - */ -static HeapTuple -get_changed_tuple(char *change) -{ - HeapTupleData tup_data; - HeapTuple result; - char *src; - - /* - * Ensure alignment before accessing the fields. (This is why we can't use - * heap_copytuple() instead of this function.) - */ - src = change + offsetof(ConcurrentChange, tup_data); - memcpy(&tup_data, src, sizeof(HeapTupleData)); - - result = (HeapTuple) palloc(HEAPTUPLESIZE + tup_data.t_len); - memcpy(result, &tup_data, sizeof(HeapTupleData)); - result->t_data = (HeapTupleHeader) ((char *) result + HEAPTUPLESIZE); - src = change + SizeOfConcurrentChange; - memcpy(result->t_data, src, result->t_len); - - return result; -} - -/* - * Decode logical changes from the WAL sequence up to end_of_wal. + * If true is returned, there is no more work for the worker. */ -void -repack_decode_concurrent_changes(LogicalDecodingContext *ctx, - XLogRecPtr end_of_wal) +static bool +decode_concurrent_changes(LogicalDecodingContext *ctx, + DecodingWorkerShared *shared) { RepackDecodingState *dstate; - ResourceOwner resowner_old; + XLogRecPtr lsn_upto; + bool done; + char fname[MAXPGPATH]; dstate = (RepackDecodingState *) ctx->output_writer_private; - resowner_old = CurrentResourceOwner; - CurrentResourceOwner = dstate->resowner; - PG_TRY(); + /* Open the output file. */ + DecodingWorkerFileName(fname, shared->relid, shared->last_exported + 1); + dstate->file = BufFileCreateFileSet(&shared->sfs.fs, fname); + + SpinLockAcquire(&shared->mutex); + lsn_upto = shared->lsn_upto; + done = shared->done; + SpinLockRelease(&shared->mutex); + + while (true) { - while (ctx->reader->EndRecPtr < end_of_wal) - { - XLogRecord *record; - XLogSegNo segno_new; - char *errm = NULL; - XLogRecPtr end_lsn; + XLogRecord *record; + XLogSegNo segno_new; + char *errm = NULL; + XLogRecPtr end_lsn; - record = XLogReadRecord(ctx->reader, &errm); - if (errm) - elog(ERROR, "%s", errm); + CHECK_FOR_INTERRUPTS(); - if (record != NULL) - LogicalDecodingProcessRecord(ctx, ctx->reader); + record = XLogReadRecord(ctx->reader, &errm); + if (record) + { + LogicalDecodingProcessRecord(ctx, ctx->reader); /* * If WAL segment boundary has been crossed, inform the decoding - * system that the catalog_xmin can advance. (We can confirm more - * often, but a filling a single WAL segment should not take much - * time.) + * system that the catalog_xmin can advance. + * + * TODO Does it make sense to confirm more often? Segment size + * seems appropriate for restart_lsn (because less than a segment + * cannot be recycled anyway), however more frequent checks might + * be beneficial for catalog_xmin. */ end_lsn = ctx->reader->EndRecPtr; XLByteToSeg(end_lsn, segno_new, wal_segment_size); @@ -2645,80 +2732,137 @@ repack_decode_concurrent_changes(LogicalDecodingContext *ctx, (uint32) (end_lsn >> 32), (uint32) end_lsn); repack_current_segment = segno_new; } + } + else + { + ReadLocalXLogPageNoWaitPrivate *priv; + + if (errm) + ereport(ERROR, (errmsg("%s", errm))); - CHECK_FOR_INTERRUPTS(); + /* + * In the decoding loop we do not want to get blocked when there + * is no more WAL available, otherwise the loop would become + * uninterruptible. + */ + priv = (ReadLocalXLogPageNoWaitPrivate *) + ctx->reader->private_data; + if (priv->end_of_wal) + /* Do not miss the end of WAL condition next time. */ + priv->end_of_wal = false; + else + ereport(ERROR, (errmsg("could not read WAL record"))); + } + + /* + * Whether we could read new record or not, keep checking if + * 'lsn_upto' was specified. + */ + if (XLogRecPtrIsInvalid(lsn_upto)) + { + SpinLockAcquire(&shared->mutex); + lsn_upto = shared->lsn_upto; + /* 'done' should be set at the same time as 'lsn_upto' */ + done = shared->done; + SpinLockRelease(&shared->mutex); + } + if (!XLogRecPtrIsInvalid(lsn_upto) && + ctx->reader->EndRecPtr >= lsn_upto) + break; + + if (record == NULL) + { + int64 timeout = 0; + WaitLSNResult res; + + /* + * Before we retry reading, wait until new WAL is flushed. + * + * There is a race condition such that the backend executing + * REPACK determines 'lsn_upto', but before it sets the shared + * variable, we reach the end of WAL. In that case we'd need to + * wait until the next WAL flush (unrelated to REPACK). Although + * that should not be a problem in a busy system, it might be + * noticeable in other cases, including regression tests (which + * are not necessarily executed in parallel). Therefore it makes + * sense to use timeout. + * + * If lsn_upto is valid, WAL records having LSN lower than that + * should already have been flushed to disk. + */ + if (XLogRecPtrIsInvalid(lsn_upto)) + timeout = 100L; + res = WaitForLSN(WAIT_LSN_TYPE_PRIMARY_FLUSH, + ctx->reader->EndRecPtr + 1, + timeout); + if (res != WAIT_LSN_RESULT_SUCCESS && + res != WAIT_LSN_RESULT_TIMEOUT) + ereport(ERROR, (errmsg("waiting for WAL failed"))); } - InvalidateSystemCaches(); - CurrentResourceOwner = resowner_old; - } - PG_CATCH(); - { - /* clear all timetravel entries */ - InvalidateSystemCaches(); - CurrentResourceOwner = resowner_old; - PG_RE_THROW(); } - PG_END_TRY(); + + /* + * Close the file so we can make it available to the backend. + */ + BufFileClose(dstate->file); + dstate->file = NULL; + SpinLockAcquire(&shared->mutex); + shared->lsn_upto = InvalidXLogRecPtr; + shared->last_exported++; + SpinLockRelease(&shared->mutex); + ConditionVariableSignal(&shared->cv); + + return done; } /* * Apply changes stored in 'file'. */ static void -apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest) +apply_concurrent_changes(BufFile *file, ChangeDest *dest) { + char kind; + uint32 t_len; Relation rel = dest->rel; TupleTableSlot *index_slot, *ident_slot; HeapTuple tup_old = NULL; - if (dstate->nchanges == 0) - return; - /* TupleTableSlot is needed to pass the tuple to ExecInsertIndexTuples(). */ - index_slot = MakeSingleTupleTableSlot(dstate->tupdesc, &TTSOpsHeapTuple); + index_slot = MakeSingleTupleTableSlot(RelationGetDescr(rel), + &TTSOpsHeapTuple); /* A slot to fetch tuples from identity index. */ ident_slot = table_slot_create(rel, NULL); - while (tuplestore_gettupleslot(dstate->tstore, true, false, - dstate->tsslot)) + while (true) { - bool shouldFree; - HeapTuple tup_change, - tup, + size_t nread; + HeapTuple tup, tup_exist; - char *change_raw, - *src; - ConcurrentChange change; - bool isnull[1]; - Datum values[1]; CHECK_FOR_INTERRUPTS(); - /* Get the change from the single-column tuple. */ - tup_change = ExecFetchSlotHeapTuple(dstate->tsslot, false, &shouldFree); - heap_deform_tuple(tup_change, dstate->tupdesc_change, values, isnull); - Assert(!isnull[0]); - - /* Make sure we access aligned data. */ - change_raw = (char *) DatumGetByteaP(values[0]); - src = (char *) VARDATA(change_raw); - memcpy(&change, src, SizeOfConcurrentChange); + nread = BufFileReadMaybeEOF(file, &kind, 1, true); + /* Are we done with the file? */ + if (nread == 0) + break; - /* - * Extract the tuple from the change. The tuple is copied here because - * it might be assigned to 'tup_old', in which case it needs to - * survive into the next iteration. - */ - tup = get_changed_tuple(src); + /* Read the tuple. */ + BufFileReadExact(file, &t_len, sizeof(t_len)); + tup = (HeapTuple) palloc(HEAPTUPLESIZE + t_len); + tup->t_data = (HeapTupleHeader) ((char *) tup + HEAPTUPLESIZE); + BufFileReadExact(file, tup->t_data, t_len); + tup->t_len = t_len; + ItemPointerSetInvalid(&tup->t_self); + tup->t_tableOid = RelationGetRelid(dest->rel); - if (change.kind == CHANGE_UPDATE_OLD) + if (kind == CHANGE_UPDATE_OLD) { Assert(tup_old == NULL); tup_old = tup; } - else if (change.kind == CHANGE_INSERT) + else if (kind == CHANGE_INSERT) { Assert(tup_old == NULL); @@ -2726,12 +2870,11 @@ apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest) pfree(tup); } - else if (change.kind == CHANGE_UPDATE_NEW || - change.kind == CHANGE_DELETE) + else if (kind == CHANGE_UPDATE_NEW || kind == CHANGE_DELETE) { HeapTuple tup_key; - if (change.kind == CHANGE_UPDATE_NEW) + if (kind == CHANGE_UPDATE_NEW) { tup_key = tup_old != NULL ? tup_old : tup; } @@ -2748,7 +2891,7 @@ apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest) if (tup_exist == NULL) elog(ERROR, "failed to find target tuple"); - if (change.kind == CHANGE_UPDATE_NEW) + if (kind == CHANGE_UPDATE_NEW) apply_concurrent_update(rel, tup, tup_exist, dest->iistate, index_slot); else @@ -2763,26 +2906,19 @@ apply_concurrent_changes(RepackDecodingState *dstate, ChangeDest *dest) pfree(tup); } else - elog(ERROR, "unrecognized kind of change: %d", change.kind); + elog(ERROR, "unrecognized kind of change: %d", kind); /* * If a change was applied now, increment CID for next writes and * update the snapshot so it sees the changes we've applied so far. */ - if (change.kind != CHANGE_UPDATE_OLD) + if (kind != CHANGE_UPDATE_OLD) { CommandCounterIncrement(); UpdateActiveSnapshotCommandId(); } - - /* TTSOpsMinimalTuple has .get_heap_tuple==NULL. */ - Assert(shouldFree); - pfree(tup_change); } - tuplestore_clear(dstate->tstore); - dstate->nchanges = 0; - /* Cleanup. */ ExecDropSingleTupleTableSlot(index_slot); ExecDropSingleTupleTableSlot(ident_slot); @@ -2957,25 +3093,59 @@ find_target_tuple(Relation rel, ChangeDest *dest, HeapTuple tup_key, } /* - * Decode and apply concurrent changes. + * Decode and apply concurrent changes, up to (and including) the record whose + * LSN is 'end_of_wal'. */ static void -process_concurrent_changes(LogicalDecodingContext *decoding_ctx, - XLogRecPtr end_of_wal, ChangeDest *dest) +process_concurrent_changes(XLogRecPtr end_of_wal, ChangeDest *dest, bool done) { - RepackDecodingState *dstate; + DecodingWorkerShared *shared; + char fname[MAXPGPATH]; + BufFile *file; pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_CATCH_UP); - dstate = (RepackDecodingState *) decoding_ctx->output_writer_private; + /* Ask the worker for the file. */ + shared = (DecodingWorkerShared *) dsm_segment_address(decoding_worker->seg); + SpinLockAcquire(&shared->mutex); + shared->lsn_upto = end_of_wal; + shared->done = done; + SpinLockRelease(&shared->mutex); + + /* + * The worker needs to finish processing of the current WAL record. Even + * if it's idle, it'll need to close the output file. Thus we're likely to + * wait, so prepare for sleep. + */ + ConditionVariablePrepareToSleep(&shared->cv); + for (;;) + { + int last_exported; - repack_decode_concurrent_changes(decoding_ctx, end_of_wal); + SpinLockAcquire(&shared->mutex); + last_exported = shared->last_exported; + SpinLockRelease(&shared->mutex); - if (dstate->nchanges == 0) - return; + /* + * Has the worker exported the file we are waiting for? + */ + if (last_exported == dest->file_seq) + break; - apply_concurrent_changes(dstate, dest); + ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT); + } + ConditionVariableCancelSleep(); + + /* Open the file. */ + DecodingWorkerFileName(fname, shared->relid, dest->file_seq); + file = BufFileOpenFileSet(&shared->sfs.fs, fname, O_RDONLY, false); + apply_concurrent_changes(file, dest); + + BufFileClose(file); + + /* Get ready for the next file. */ + dest->file_seq++; } /* @@ -3101,15 +3271,10 @@ cleanup_logical_decoding(LogicalDecodingContext *ctx) dstate = (RepackDecodingState *) ctx->output_writer_private; - ExecDropSingleTupleTableSlot(dstate->tsslot); - FreeTupleDesc(dstate->tupdesc_change); FreeTupleDesc(dstate->tupdesc); - tuplestore_end(dstate->tstore); - FreeDecodingContext(ctx); - ReplicationSlotRelease(); - ReplicationSlotDrop(NameStr(dstate->slotname), false); + ReplicationSlotDropAcquired(); pfree(dstate); } @@ -3123,7 +3288,6 @@ cleanup_logical_decoding(LogicalDecodingContext *ctx) */ static void rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, - LogicalDecodingContext *decoding_ctx, TransactionId frozenXid, MultiXactId cutoffMulti) { @@ -3204,6 +3368,7 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, &chgdst.ident_index); chgdst.ident_key = build_identity_key(ident_idx_new, OldHeap, &chgdst.ident_key_nentries); + chgdst.file_seq = WORKER_FILE_SNAPSHOT + 1; /* * During testing, wait for another backend to perform concurrent data @@ -3225,7 +3390,7 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, * hold AccessExclusiveLock. (Quite some amount of WAL could have been * written during the data copying and index creation.) */ - process_concurrent_changes(decoding_ctx, end_of_wal, &chgdst); + process_concurrent_changes(end_of_wal, &chgdst, false); /* * Acquire AccessExclusiveLock on the table, its TOAST relation (if there @@ -3306,8 +3471,11 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, XLogFlush(wal_insert_ptr); end_of_wal = GetFlushRecPtr(NULL); - /* Apply the concurrent changes again. */ - process_concurrent_changes(decoding_ctx, end_of_wal, &chgdst); + /* + * Apply the concurrent changes again. Indicate that the decoding worker + * won't be needed anymore. + */ + process_concurrent_changes(end_of_wal, &chgdst, true); /* Remember info about rel before closing OldHeap */ relpersistence = OldHeap->rd_rel->relpersistence; @@ -3417,3 +3585,510 @@ build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes) return result; } + +/* + * Try to start a background worker to perform logical decoding of data + * changes applied to relation while REPACK CONCURRENTLY is copying its + * contents to a new table. + */ +static void +start_decoding_worker(Oid relid) +{ + Size size; + dsm_segment *seg; + DecodingWorkerShared *shared; + shm_mq *mq; + shm_mq_handle *mqh; + BackgroundWorker bgw; + + /* Setup shared memory. */ + size = BUFFERALIGN(offsetof(DecodingWorkerShared, error_queue)) + + BUFFERALIGN(REPACK_ERROR_QUEUE_SIZE); + seg = dsm_create(size, 0); + shared = (DecodingWorkerShared *) dsm_segment_address(seg); + shared->lsn_upto = InvalidXLogRecPtr; + shared->done = false; + SharedFileSetInit(&shared->sfs, seg); + shared->last_exported = -1; + SpinLockInit(&shared->mutex); + shared->dbid = MyDatabaseId; + + /* + * This is the UserId set in cluster_rel(). Security context shouldn't be + * needed for decoding worker. + */ + shared->roleid = GetUserId(); + shared->relid = relid; + ConditionVariableInit(&shared->cv); + shared->backend_proc = MyProc; + shared->backend_pid = MyProcPid; + shared->backend_proc_number = MyProcNumber; + + mq = shm_mq_create((char *) BUFFERALIGN(shared->error_queue), + REPACK_ERROR_QUEUE_SIZE); + shm_mq_set_receiver(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + + memset(&bgw, 0, sizeof(bgw)); + snprintf(bgw.bgw_name, BGW_MAXLEN, + "REPACK decoding worker for relation \"%s\"", + get_rel_name(relid)); + snprintf(bgw.bgw_type, BGW_MAXLEN, "REPACK decoding worker"); + bgw.bgw_flags = BGWORKER_SHMEM_ACCESS | + BGWORKER_BACKEND_DATABASE_CONNECTION; + bgw.bgw_start_time = BgWorkerStart_RecoveryFinished; + bgw.bgw_restart_time = BGW_NEVER_RESTART; + snprintf(bgw.bgw_library_name, MAXPGPATH, "postgres"); + snprintf(bgw.bgw_function_name, BGW_MAXLEN, "RepackWorkerMain"); + bgw.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(seg)); + bgw.bgw_notify_pid = MyProcPid; + + decoding_worker = palloc0_object(DecodingWorker); + if (!RegisterDynamicBackgroundWorker(&bgw, &decoding_worker->handle)) + ereport(ERROR, + (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED), + errmsg("out of background worker slots"), + errhint("You might need to increase \"%s\".", "max_worker_processes"))); + + decoding_worker->seg = seg; + decoding_worker->error_mqh = mqh; + + /* + * The decoding setup must be done before the caller can have XID assigned + * for any reason, otherwise the worker might end up in a deadlock, + * waiting for the caller's transaction to end. Therefore wait here until + * the worker indicates that it has the logical decoding initialized. + */ + ConditionVariablePrepareToSleep(&shared->cv); + for (;;) + { + bool initialized; + + SpinLockAcquire(&shared->mutex); + initialized = shared->initialized; + SpinLockRelease(&shared->mutex); + + if (initialized) + break; + + ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT); + } + ConditionVariableCancelSleep(); +} + +/* + * Stop the decoding worker and cleanup the related resources. + * + * The worker stops on its own when it knows there is no more work to do, but + * we need to stop it explicitly at least on ERROR in the launching backend. + */ +static void +stop_decoding_worker(void) +{ + BgwHandleStatus status; + + /* Haven't reached the worker startup? */ + if (decoding_worker == NULL) + return; + + /* Could not register the worker? */ + if (decoding_worker->handle == NULL) + return; + + TerminateBackgroundWorker(decoding_worker->handle); + /* The worker should really exit before the REPACK command does. */ + HOLD_INTERRUPTS(); + status = WaitForBackgroundWorkerShutdown(decoding_worker->handle); + RESUME_INTERRUPTS(); + + if (status == BGWH_POSTMASTER_DIED) + ereport(FATAL, + (errcode(ERRCODE_ADMIN_SHUTDOWN), + errmsg("postmaster exited during REPACK command"))); + + shm_mq_detach(decoding_worker->error_mqh); + + /* + * If we could not cancel the current sleep due to ERROR, do that before + * we detach from the shared memory the condition variable is located in. + * If we did not, the bgworker ERROR handling code would try and fail + * badly. + */ + ConditionVariableCancelSleep(); + + dsm_detach(decoding_worker->seg); + pfree(decoding_worker); + decoding_worker = NULL; +} + +/* Is this process a REPACK worker? */ +static bool is_repack_worker = false; + +static pid_t backend_pid; +static ProcNumber backend_proc_number; + +/* + * See ParallelWorkerShutdown for details. + */ +static void +RepackWorkerShutdown(int code, Datum arg) +{ + SendProcSignal(backend_pid, + PROCSIG_REPACK_MESSAGE, + backend_proc_number); + + dsm_detach((dsm_segment *) DatumGetPointer(arg)); +} + +/* REPACK decoding worker entry point */ +void +RepackWorkerMain(Datum main_arg) +{ + dsm_segment *seg; + DecodingWorkerShared *shared; + shm_mq *mq; + shm_mq_handle *mqh; + + is_repack_worker = true; + + /* + * Override the default bgworker_die() with die() so we can use + * CHECK_FOR_INTERRUPTS(). + */ + pqsignal(SIGTERM, die); + BackgroundWorkerUnblockSignals(); + + seg = dsm_attach(DatumGetUInt32(main_arg)); + if (seg == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("could not map dynamic shared memory segment"))); + + shared = (DecodingWorkerShared *) dsm_segment_address(seg); + + /* Arrange to signal the leader if we exit. */ + backend_pid = shared->backend_pid; + backend_proc_number = shared->backend_proc_number; + before_shmem_exit(RepackWorkerShutdown, PointerGetDatum(seg)); + + /* + * Join locking group - see the comments around the call of + * start_decoding_worker(). + */ + if (!BecomeLockGroupMember(shared->backend_proc, backend_pid)) + /* The leader is not running anymore. */ + return; + + /* + * Setup a queue to send error messages to the backend that launched this + * worker. + */ + mq = (shm_mq *) (char *) BUFFERALIGN(shared->error_queue); + shm_mq_set_sender(mq, MyProc); + mqh = shm_mq_attach(mq, seg, NULL); + pq_redirect_to_shm_mq(seg, mqh); + pq_set_parallel_leader(shared->backend_pid, + shared->backend_proc_number); + + /* Connect to the database. */ + BackgroundWorkerInitializeConnectionByOid(shared->dbid, shared->roleid, 0); + + repack_worker_internal(seg); +} + +static void +repack_worker_internal(dsm_segment *seg) +{ + DecodingWorkerShared *shared; + LogicalDecodingContext *decoding_ctx; + SharedFileSet *sfs; + Snapshot snapshot; + + /* + * Transaction is needed to open relation, and it also provides us with a + * resource owner. + */ + StartTransactionCommand(); + + shared = (DecodingWorkerShared *) dsm_segment_address(seg); + + /* + * Not sure the spinlock is needed here - the backend should not change + * anything in the shared memory until we have serialized the snapshot. + */ + SpinLockAcquire(&shared->mutex); + Assert(XLogRecPtrIsInvalid(shared->lsn_upto)); + sfs = &shared->sfs; + SpinLockRelease(&shared->mutex); + + SharedFileSetAttach(sfs, seg); + + /* + * Prepare to capture the concurrent data changes ourselves. + */ + decoding_ctx = setup_logical_decoding(shared->relid); + + /* Announce that we're ready. */ + SpinLockAcquire(&shared->mutex); + shared->initialized = true; + SpinLockRelease(&shared->mutex); + ConditionVariableSignal(&shared->cv); + + /* Build the initial snapshot and export it. */ + snapshot = SnapBuildInitialSnapshotForRepack(decoding_ctx->snapshot_builder); + export_initial_snapshot(snapshot, shared); + + /* + * Only historic snapshots should be used now. Do not let us restrict the + * progress of xmin horizon. + */ + InvalidateCatalogSnapshot(); + + while (!decode_concurrent_changes(decoding_ctx, shared)) + ; + + /* Cleanup. */ + cleanup_logical_decoding(decoding_ctx); + CommitTransactionCommand(); +} + +/* + * Make snapshot available to the backend that launched the decoding worker. + */ +static void +export_initial_snapshot(Snapshot snapshot, DecodingWorkerShared *shared) +{ + char fname[MAXPGPATH]; + BufFile *file; + Size snap_size; + char *snap_space; + + snap_size = EstimateSnapshotSpace(snapshot); + snap_space = (char *) palloc(snap_size); + SerializeSnapshot(snapshot, snap_space); + FreeSnapshot(snapshot); + + DecodingWorkerFileName(fname, shared->relid, shared->last_exported + 1); + file = BufFileCreateFileSet(&shared->sfs.fs, fname); + /* To make restoration easier, write the snapshot size first. */ + BufFileWrite(file, &snap_size, sizeof(snap_size)); + BufFileWrite(file, snap_space, snap_size); + pfree(snap_space); + BufFileClose(file); + + /* Increase the counter to tell the backend that the file is available. */ + SpinLockAcquire(&shared->mutex); + shared->last_exported++; + SpinLockRelease(&shared->mutex); + ConditionVariableSignal(&shared->cv); +} + +/* + * Get the initial snapshot from the decoding worker. + */ +static Snapshot +get_initial_snapshot(DecodingWorker *worker) +{ + DecodingWorkerShared *shared; + char fname[MAXPGPATH]; + BufFile *file; + Size snap_size; + char *snap_space; + Snapshot snapshot; + + shared = (DecodingWorkerShared *) dsm_segment_address(worker->seg); + + /* + * The worker needs to initialize the logical decoding, which usually + * takes some time. Therefore it makes sense to prepare for the sleep + * first. + */ + ConditionVariablePrepareToSleep(&shared->cv); + for (;;) + { + int last_exported; + + SpinLockAcquire(&shared->mutex); + last_exported = shared->last_exported; + SpinLockRelease(&shared->mutex); + + /* + * Has the worker exported the file we are waiting for? + */ + if (last_exported == WORKER_FILE_SNAPSHOT) + break; + + ConditionVariableSleep(&shared->cv, WAIT_EVENT_REPACK_WORKER_EXPORT); + } + ConditionVariableCancelSleep(); + + /* Read the snapshot from a file. */ + DecodingWorkerFileName(fname, shared->relid, WORKER_FILE_SNAPSHOT); + file = BufFileOpenFileSet(&shared->sfs.fs, fname, O_RDONLY, false); + BufFileReadExact(file, &snap_size, sizeof(snap_size)); + snap_space = (char *) palloc(snap_size); + BufFileReadExact(file, snap_space, snap_size); + BufFileClose(file); + + /* Restore it. */ + snapshot = RestoreSnapshot(snap_space); + pfree(snap_space); + + return snapshot; +} + +bool +IsRepackWorker(void) +{ + return is_repack_worker; +} + +/* + * Handle receipt of an interrupt indicating a repack worker message. + * + * Note: this is called within a signal handler! All we can do is set + * a flag that will cause the next CHECK_FOR_INTERRUPTS() to invoke + * ProcessRepackMessages(). + */ +void +HandleRepackMessageInterrupt(void) +{ + InterruptPending = true; + RepackMessagePending = true; + SetLatch(MyLatch); +} + +/* + * Process any queued protocol messages received from parallel workers. + */ +void +ProcessRepackMessages(void) +{ + MemoryContext oldcontext; + + static MemoryContext hpm_context = NULL; + + /* + * Nothing to do if we haven't launched the worker yet or have already + * terminated it. + */ + if (decoding_worker == NULL) + return; + + /* + * This is invoked from ProcessInterrupts(), and since some of the + * functions it calls contain CHECK_FOR_INTERRUPTS(), there is a potential + * for recursive calls if more signals are received while this runs. It's + * unclear that recursive entry would be safe, and it doesn't seem useful + * even if it is safe, so let's block interrupts until done. + */ + HOLD_INTERRUPTS(); + + /* + * Moreover, CurrentMemoryContext might be pointing almost anywhere. We + * don't want to risk leaking data into long-lived contexts, so let's do + * our work here in a private context that we can reset on each use. + */ + if (hpm_context == NULL) /* first time through? */ + hpm_context = AllocSetContextCreate(TopMemoryContext, + "ProcessRepackMessages", + ALLOCSET_DEFAULT_SIZES); + else + MemoryContextReset(hpm_context); + + oldcontext = MemoryContextSwitchTo(hpm_context); + + /* OK to process messages. Reset the flag saying there are more to do. */ + RepackMessagePending = false; + + /* + * Read as many messages as we can from each worker, but stop when no more + * messages can be read from the worker without blocking. + */ + while (true) + { + shm_mq_result res; + Size nbytes; + void *data; + + res = shm_mq_receive(decoding_worker->error_mqh, &nbytes, + &data, true); + if (res == SHM_MQ_WOULD_BLOCK) + break; + else if (res == SHM_MQ_SUCCESS) + { + StringInfoData msg; + + initStringInfo(&msg); + appendBinaryStringInfo(&msg, data, nbytes); + ProcessRepackMessage(&msg); + pfree(msg.data); + } + else + { + /* + * The decoding worker is special in that it exits as soon as it + * has its work done. Thus the DETACHED result code is fine. + */ + Assert(res == SHM_MQ_DETACHED); + + break; + } + } + + MemoryContextSwitchTo(oldcontext); + + /* Might as well clear the context on our way out */ + MemoryContextReset(hpm_context); + + RESUME_INTERRUPTS(); +} + +/* + * Process a single protocol message received from a single parallel worker. + */ +static void +ProcessRepackMessage(StringInfo msg) +{ + char msgtype; + + msgtype = pq_getmsgbyte(msg); + + switch (msgtype) + { + case PqMsg_ErrorResponse: + case PqMsg_NoticeResponse: + { + ErrorData edata; + + /* Parse ErrorResponse or NoticeResponse. */ + pq_parse_errornotice(msg, &edata); + + /* Death of a worker isn't enough justification for suicide. */ + edata.elevel = Min(edata.elevel, ERROR); + + /* + * If desired, add a context line to show that this is a + * message propagated from a parallel worker. Otherwise, it + * can sometimes be confusing to understand what actually + * happened. + */ + if (edata.context) + edata.context = psprintf("%s\n%s", edata.context, + _("decoding worker")); + else + edata.context = pstrdup(_("decoding worker")); + + /* Rethrow error or print notice. */ + ThrowErrorData(&edata); + + break; + } + + default: + { + elog(ERROR, "unrecognized message type received from decoding worker: %c (message length %d bytes)", + msgtype, msg->len); + } + } +} diff --git a/src/backend/libpq/pqmq.c b/src/backend/libpq/pqmq.c index 6e4bbfb5aa1..42f6fa472c5 100644 --- a/src/backend/libpq/pqmq.c +++ b/src/backend/libpq/pqmq.c @@ -14,6 +14,7 @@ #include "postgres.h" #include "access/parallel.h" +#include "commands/cluster.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "libpq/pqmq.h" @@ -175,6 +176,10 @@ mq_putmessage(char msgtype, const char *s, size_t len) SendProcSignal(pq_mq_parallel_leader_pid, PROCSIG_PARALLEL_APPLY_MESSAGE, pq_mq_parallel_leader_proc_number); + else if (IsRepackWorker()) + SendProcSignal(pq_mq_parallel_leader_pid, + PROCSIG_REPACK_MESSAGE, + pq_mq_parallel_leader_proc_number); else { Assert(IsParallelWorker()); diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c index 65deabe91a7..334bb708c5b 100644 --- a/src/backend/postmaster/bgworker.c +++ b/src/backend/postmaster/bgworker.c @@ -13,6 +13,7 @@ #include "postgres.h" #include "access/parallel.h" +#include "commands/cluster.h" #include "libpq/pqsignal.h" #include "miscadmin.h" #include "pgstat.h" @@ -136,6 +137,9 @@ static const struct }, { "SequenceSyncWorkerMain", SequenceSyncWorkerMain + }, + { + "RepackWorkerMain", RepackWorkerMain } }; diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index b0ef1a12520..35a46988285 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -194,7 +194,11 @@ StartupDecodingContext(List *output_plugin_options, ctx->slot = slot; - ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, ctx); + /* + * TODO A separate patch for PG core, unless there's really a reason to + * pass ctx for private_data (May extensions expect ctx?). + */ + ctx->reader = XLogReaderAllocate(wal_segment_size, NULL, xl_routine, NULL); if (!ctx->reader) ereport(ERROR, (errcode(ERRCODE_OUT_OF_MEMORY), diff --git a/src/backend/replication/pgoutput_repack/pgoutput_repack.c b/src/backend/replication/pgoutput_repack/pgoutput_repack.c index a074eed393d..4bc47a72371 100644 --- a/src/backend/replication/pgoutput_repack/pgoutput_repack.c +++ b/src/backend/replication/pgoutput_repack/pgoutput_repack.c @@ -169,17 +169,13 @@ store_change(LogicalDecodingContext *ctx, ConcurrentChangeKind kind, HeapTuple tuple) { RepackDecodingState *dstate; - char *change_raw; - ConcurrentChange change; + char kind_byte = (char) kind; bool flattened = false; - Size size; - Datum values[1]; - bool isnull[1]; - char *dst; dstate = (RepackDecodingState *) ctx->output_writer_private; - size = VARHDRSZ + SizeOfConcurrentChange; + /* Store the change kind. */ + BufFileWrite(dstate->file, &kind_byte, 1); /* * ReorderBufferCommit() stores the TOAST chunks in its private memory @@ -196,46 +192,12 @@ store_change(LogicalDecodingContext *ctx, ConcurrentChangeKind kind, tuple = toast_flatten_tuple(tuple, dstate->tupdesc); flattened = true; } + /* Store the tuple size ... */ + BufFileWrite(dstate->file, &tuple->t_len, sizeof(tuple->t_len)); + /* ... and the tuple itself. */ + BufFileWrite(dstate->file, tuple->t_data, tuple->t_len); - size += tuple->t_len; - if (size >= MaxAllocSize) - elog(ERROR, "Change is too big."); - - /* Construct the change. */ - change_raw = (char *) palloc0(size); - SET_VARSIZE(change_raw, size); - - /* - * Since the varlena alignment might not be sufficient for the structure, - * set the fields in a local instance and remember where it should - * eventually be copied. - */ - change.kind = kind; - dst = (char *) VARDATA(change_raw); - - /* - * Copy the tuple. - * - * Note: change->tup_data.t_data must be fixed on retrieval! - */ - memcpy(&change.tup_data, tuple, sizeof(HeapTupleData)); - memcpy(dst, &change, SizeOfConcurrentChange); - dst += SizeOfConcurrentChange; - memcpy(dst, tuple->t_data, tuple->t_len); - - /* The data has been copied. */ + /* Free the flat copy if created above. */ if (flattened) pfree(tuple); - - /* Store as tuple of 1 bytea column. */ - values[0] = PointerGetDatum(change_raw); - isnull[0] = false; - tuplestore_putvalues(dstate->tstore, dstate->tupdesc_change, - values, isnull); - - /* Accounting. */ - dstate->nchanges++; - - /* Cleanup. */ - pfree(change_raw); } diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c index 8e56922dcea..6f9e7a7aab7 100644 --- a/src/backend/storage/ipc/procsignal.c +++ b/src/backend/storage/ipc/procsignal.c @@ -19,6 +19,7 @@ #include "access/parallel.h" #include "commands/async.h" +#include "commands/cluster.h" #include "miscadmin.h" #include "pgstat.h" #include "port/pg_bitutils.h" @@ -697,6 +698,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS) if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE)) HandleParallelApplyMessageInterrupt(); + if (CheckProcSignal(PROCSIG_REPACK_MESSAGE)) + HandleRepackMessageInterrupt(); + if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE)) HandleRecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE); diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..fc81ad87615 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -36,6 +36,7 @@ #include "access/xact.h" #include "catalog/pg_type.h" #include "commands/async.h" +#include "commands/cluster.h" #include "commands/event_trigger.h" #include "commands/explain_state.h" #include "commands/prepare.h" @@ -3541,6 +3542,9 @@ ProcessInterrupts(void) if (ParallelApplyMessagePending) ProcessParallelApplyMessages(); + + if (RepackMessagePending) + ProcessRepackMessages(); } /* diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 5537a2d2530..56d01ccd5af 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -154,6 +154,7 @@ RECOVERY_CONFLICT_SNAPSHOT "Waiting for recovery conflict resolution for a vacuu RECOVERY_CONFLICT_TABLESPACE "Waiting for recovery conflict resolution for dropping a tablespace." RECOVERY_END_COMMAND "Waiting for <xref linkend="guc-recovery-end-command"/> to complete." RECOVERY_PAUSE "Waiting for recovery to be resumed." +REPACK_WORKER_EXPORT "Waiting for decoding worker to export a new output file." REPLICATION_ORIGIN_DROP "Waiting for a replication origin to become inactive so it can be dropped." REPLICATION_SLOT_DROP "Waiting for a replication slot to become inactive so it can be dropped." RESTORE_COMMAND "Waiting for <xref linkend="guc-restore-command"/> to complete." diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 76aa993009a..15760363a1a 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -22,7 +22,6 @@ #include "access/xact.h" #include "commands/vacuum.h" #include "executor/tuptable.h" -#include "replication/logical.h" #include "storage/read_stream.h" #include "utils/rel.h" #include "utils/snapshot.h" @@ -631,7 +630,6 @@ typedef struct TableAmRoutine bool use_sort, TransactionId OldestXmin, Snapshot snapshot, - LogicalDecodingContext *decoding_ctx, TransactionId *xid_cutoff, MultiXactId *multi_cutoff, double *num_tuples, @@ -1651,8 +1649,6 @@ table_relation_copy_data(Relation rel, const RelFileLocator *newrlocator) * - *multi_cutoff - ditto * - snapshot - if != NULL, ignore data changes done by transactions that this * (MVCC) snapshot considers still in-progress or in the future. - * - decoding_ctx - logical decoding context, to capture concurrent data - * changes. * * Output parameters: * - *xid_cutoff - rel's new relfrozenxid value, may be invalid @@ -1666,7 +1662,6 @@ table_relation_copy_for_cluster(Relation OldTable, Relation NewTable, bool use_sort, TransactionId OldestXmin, Snapshot snapshot, - LogicalDecodingContext *decoding_ctx, TransactionId *xid_cutoff, MultiXactId *multi_cutoff, double *num_tuples, @@ -1675,7 +1670,7 @@ table_relation_copy_for_cluster(Relation OldTable, Relation NewTable, { OldTable->rd_tableam->relation_copy_for_cluster(OldTable, NewTable, OldIndex, use_sort, OldestXmin, - snapshot, decoding_ctx, + snapshot, xid_cutoff, multi_cutoff, num_tuples, tups_vacuumed, tups_recently_dead); diff --git a/src/include/commands/cluster.h b/src/include/commands/cluster.h index 6a5c476294a..1b05d5d418b 100644 --- a/src/include/commands/cluster.h +++ b/src/include/commands/cluster.h @@ -17,11 +17,13 @@ #include "nodes/parsenodes.h" #include "parser/parse_node.h" #include "replication/decode.h" +#include "postmaster/bgworker.h" #include "replication/logical.h" +#include "storage/buffile.h" #include "storage/lock.h" +#include "storage/shm_mq.h" #include "utils/relcache.h" #include "utils/resowner.h" -#include "utils/tuplestore.h" /* flag bits for ClusterParams->options */ @@ -44,6 +46,9 @@ typedef struct ClusterParams * The following definitions are used by REPACK CONCURRENTLY. */ +/* + * Stored as a single byte in the output file. + */ typedef enum { CHANGE_INSERT, @@ -52,68 +57,30 @@ typedef enum CHANGE_DELETE } ConcurrentChangeKind; -typedef struct ConcurrentChange -{ - /* See the enum above. */ - ConcurrentChangeKind kind; - - /* - * The actual tuple. - * - * The tuple data follows the ConcurrentChange structure. Before use make - * sure the tuple is correctly aligned (ConcurrentChange can be stored as - * bytea) and that tuple->t_data is fixed. - */ - HeapTupleData tup_data; -} ConcurrentChange; - -#define SizeOfConcurrentChange (offsetof(ConcurrentChange, tup_data) + \ - sizeof(HeapTupleData)) - /* * Logical decoding state. * - * Here we store the data changes that we decode from WAL while the table - * contents is being copied to a new storage. Also the necessary metadata - * needed to apply these changes to the table is stored here. + * The output plugin uses it to store the data changes that it decodes from + * WAL while the table contents is being copied to a new storage. */ typedef struct RepackDecodingState { /* The relation whose changes we're decoding. */ Oid relid; - /* Replication slot name. */ - NameData slotname; - - /* - * Decoded changes are stored here. Although we try to avoid excessive - * batches, it can happen that the changes need to be stored to disk. The - * tuplestore does this transparently. - */ - Tuplestorestate *tstore; - - /* The current number of changes in tstore. */ - double nchanges; - - /* - * Descriptor to store the ConcurrentChange structure serialized (bytea). - * We can't store the tuple directly because tuplestore only supports - * minimum tuple and we may need to transfer OID system column from the - * output plugin. Also we need to transfer the change kind, so it's better - * to put everything in the structure than to use 2 tuplestores "in - * parallel". - */ - TupleDesc tupdesc_change; - - /* Tuple descriptor needed to update indexes. */ + /* Tuple descriptor of the relation being processed. */ TupleDesc tupdesc; - /* Slot to retrieve data from tstore. */ - TupleTableSlot *tsslot; - - ResourceOwner resowner; + /* The current output file. */ + BufFile *file; } RepackDecodingState; +extern PGDLLIMPORT volatile sig_atomic_t RepackMessagePending; + +extern bool IsRepackWorker(void); +extern void HandleRepackMessageInterrupt(void); +extern void ProcessRepackMessages(void); + extern void ExecRepack(ParseState *pstate, RepackStmt *stmt, bool isTopLevel); extern void cluster_rel(RepackCommand command, Relation OldHeap, Oid indexOid, @@ -136,6 +103,6 @@ extern void finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, extern bool am_decoding_for_repack(void); extern bool change_useless_for_repack(XLogRecordBuffer *buf); -extern void repack_decode_concurrent_changes(LogicalDecodingContext *ctx, - XLogRecPtr end_of_wal); + +extern void RepackWorkerMain(Datum main_arg); #endif /* CLUSTER_H */ diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h index e52b8eb7697..3ef35ca6b80 100644 --- a/src/include/storage/procsignal.h +++ b/src/include/storage/procsignal.h @@ -36,6 +36,7 @@ typedef enum PROCSIG_BARRIER, /* global barrier interrupt */ PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */ PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */ + PROCSIG_REPACK_MESSAGE, /* Message from repack worker */ /* Recovery conflict reasons */ PROCSIG_RECOVERY_CONFLICT_FIRST, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 1aa4fb442f1..1e2e4ba80e9 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -498,7 +498,6 @@ CompressFileHandle CompressionLocation CompressorState ComputeXidHorizonsResult -ConcurrentChange ConcurrentChangeKind ConditionVariable ConditionVariableMinimallyPadded @@ -638,6 +637,8 @@ DeclareCursorStmt DecodedBkpBlock DecodedXLogRecord DecodingOutputState +DecodingWorker +DecodingWorkerShared DefElem DefElemAction DefaultACLInfo -- 2.47.3 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v32-0006-Use-multiple-snapshots-to-copy-the-data.patch ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-01 08:01 Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-01 08:01 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Thanks a lot for your feedback Mahendra. I have rebased the changes. I'll follow up on your other comments regarding cleanup and the shortened flag name. I'm currently uploading the latest version post rebase as I've made some changes to fix the test failures. Will let cfbot run those once. Regards Nitin Motiani, Google Attachments: [application/x-patch] v10-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC94C+2jM5dikDzv-KkhbhOUz3Zm-Bd62Xycuh9XCPG2x8g@mail.gmail.com/3-v10-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From 64a7d4d5ba904822ec650a9c99a5b4604e6d7db4 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v10 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 74fc651f6f4..2b18c3c8270 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 703b2677b4e..cbac8bc2520 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v10-0005-POC-Add-test-in-pg_dump.pl.patch (5.6K, ../../CAH5HC94C+2jM5dikDzv-KkhbhOUz3Zm-Bd62Xycuh9XCPG2x8g@mail.gmail.com/4-v10-0005-POC-Add-test-in-pg_dump.pl.patch) download | inline diff: From e702603beba8943efcda66c9b751d5ee56ad9084 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 1 May 2026 07:49:14 +0000 Subject: [PATCH v10 5/5] [POC] Add test in pg_dump.pl * Make a couple of code changes to fix the test. If they pass, we'll rebase. --- src/bin/pg_dump/compress_none.c | 10 ++++++---- src/bin/pg_dump/pg_backup_archiver.c | 27 +++++++++++++++++++++++++-- src/bin/pg_dump/pg_dump.c | 20 ++++++++++---------- src/bin/pg_dump/t/002_pg_dump.pl | 19 +++++++++++++++++++ 4 files changed, 60 insertions(+), 16 deletions(-) diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 8c2f95b520d..9ecdf18eec4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,14 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; if (CFH->path_is_pipe_command) - ret = pclose(fp); + ret = pclose_check(fp); else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_log_error("could not close file: %m"); + } } return ret == 0; diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4ef9dae49ed..8358bd97df5 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2463,7 +2475,18 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..e4ae64c1495 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -420,6 +420,8 @@ main(int argc, char **argv) int c; const char *filename = NULL; bool filename_is_pipe = false; + bool file_specified = false; + bool pipe_specified = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -608,11 +610,7 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } + file_specified = true; filename = pg_strdup(optarg); filename_is_pipe = false; /* it already is, setting again * here just for clarity */ @@ -809,11 +807,7 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } + pipe_specified = true; filename = pg_strdup(optarg); filename_is_pipe = true; break; @@ -825,6 +819,12 @@ main(int argc, char **argv) } } + if (file_specified && pipe_specified) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3bc8e51561d..06b2f2619e2 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -223,6 +223,25 @@ my %pgdump_runs = ( ], }, + # This test kept failing. + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe-command' => "cat > $tempdir/defaults_dir_format/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + '--statistics', + "$tempdir/defaults_dir_format", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v10-0004-Add-documentation-for-pipe-command-in-pg_dump-an.patch (7.5K, ../../CAH5HC94C+2jM5dikDzv-KkhbhOUz3Zm-Bd62Xycuh9XCPG2x8g@mail.gmail.com/5-v10-0004-Add-documentation-for-pipe-command-in-pg_dump-an.patch) download | inline diff: From 177b0ef5e3416dd598fdc88dc6e63e660015d4f6 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 4 Apr 2025 14:34:48 +0000 Subject: [PATCH v10 4/5] Add documentation for pipe-command in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..05323ddc918 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe-command</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe-command</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe-command can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe-command to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe-command="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe-command to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe-command="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe-command to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe-command="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 6ae2cdcfc10..cb585c40361 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe-command</option> can't be set + at the same time. + If neither this option nor <option>--pipe-command</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe-command</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe-command can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe-command</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe-command to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe-commnad="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe-command to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe-commnad="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe-command along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe-commnad="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v10-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.8K, ../../CAH5HC94C+2jM5dikDzv-KkhbhOUz3Zm-Bd62Xycuh9XCPG2x8g@mail.gmail.com/6-v10-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From b592449a155ea46158578ac587a82c080b5da688 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v10 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 26 +++++-- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 60 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..8c2f95b520d 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -233,7 +236,6 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) if (fd >= 0) { int dup_fd = dup(fd); - if (dup_fd < 0) return false; CFH->private_data = fdopen(dup_fd, mode); @@ -245,7 +247,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +264,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +284,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +297,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index fda912ba0a9..6466bd4bded 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index fecf6f2d1ce..4ef9dae49ed 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9c3aca6543a..9fdb67c109d 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index d6a1428c67a..74fc651f6f4 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 95f4ac110b9..703b2677b4e 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v10-0003-Add-basic-tests-for-pipe-command.patch (2.6K, ../../CAH5HC94C+2jM5dikDzv-KkhbhOUz3Zm-Bd62Xycuh9XCPG2x8g@mail.gmail.com/7-v10-0003-Add-basic-tests-for-pipe-command.patch) download | inline diff: From 1a7588b3057dca5d65cfcdad59b96ebee746bc40 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v10 3/5] Add basic tests for pipe-command * This currently only adds a few basic tests for pg_dump with --pipe-command. * These tests include the invalid usages of --pipe-command with other flags. * We are still working on adding other tests in pg_dump.pl. But we ran into some issues which might be related to setup. --- src/bin/pg_dump/t/001_basic.pl | 36 ++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 509f4f9ce7d..9f49a1fbc32 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,42 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe-command="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: hint: Only one of [--file, --pipe-command] allowed\E/, + 'pg_dump: hint: Only one of [--file, --pipe-command] allowed' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe-command="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: hint: Option --pipe-command is not supported with any compression type\E/, + 'pg_dump: hint: Option --pipe-command is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe-command="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: hint: Option --pipe-command is not supported with any compression type\E/, + 'pg_dump: hint: Option --pipe-command is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe-command="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: hint: Option --pipe-command is not supported with any compression type\E/, + 'pg_dump: hint: Option --pipe-command is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe-command="cat"', 'test'], + qr/\Qpg_dump: hint: Option --pipe-command is only supported with directory format.\E/, + 'pg_dump: hint: Option --pipe-command is only supported with directory format.' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe-command="cat"', 'test'], + qr/\Qpg_dump: hint: Option --pipe-command is only supported with directory format.\E/, + 'pg_dump: hint: Option --pipe-command is only supported with directory format.' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, -- 2.54.0.545.g6539524ca2-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-04 04:39 Nitin Motiani <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-04 04:39 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Hi Mahendra, I'm attaching the latest version of the patch. This incorporates several suggestions Mahendra made. I've also fixed the failing test and added support for Windows in the test by using perlbin instead of cat. Here is the list of changes in this version: 1. Changed pipe-command to pip. 2. Added better error handling for invalid command. 3. Added shell escaping in the command before setting it as the file path. 4. I decided to change the mode to PG_BINARY_W for the large object toc files even in the standard case. If there is a concern, we can revert to the older version. 5. Added a bunch of new tests for various scenarios and parallelism. Currently these changes are in a separate commit. After the review, I can squash the first 3 commits of pg_dump, pg_restore, and the fixes in one commit. On Sat, Mar 14, 2026 at 10:37 PM Mahendra Singh Thalor <[email protected]> wrote: > > Comment5: I think we can support this new pipe option with pg_dumpall also as we support directory mode in pg_dumpall from v19. Regarding this, I haven't looked into the details of how that would work. If the rest look good, we can consider implementing this in another patch. For the time being, I've skipped this option in pg_dumpall and the global restore. Let me know what you think. Thanks Nitin Motiani, Google Attachments: [application/x-patch] v11-0003-Fixes-and-refactors-in-pipe-command.patch (40.2K, ../../CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com/2-v11-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From 1d3dac88a58c799b2c047b52d3d66e97bbee2a8b Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v11 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Use appendShellString in file substitutions for shell escaping. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 6 +- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 61 +++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 118 +++++++++++--------------- src/bin/pg_dump/pg_dump.c | 53 +++++------- src/bin/pg_dump/pg_dumpall.c | 9 ++ src/bin/pg_dump/pg_restore.c | 69 ++++++++------- 17 files changed, 225 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..3857eff2179 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,7 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; /* * Private data to be used by the compressor. @@ -201,7 +201,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +210,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 8c2f95b520d..2dae62aadd4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,31 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: %s", reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } } return ret == 0; @@ -228,6 +247,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -236,6 +272,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) if (fd >= 0) { int dup_fd = dup(fd); + if (dup_fd < 0) return false; CFH->private_data = fdopen(dup_fd, mode); @@ -247,10 +284,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -265,12 +299,9 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -285,7 +316,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -297,7 +328,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 6466bd4bded..8efeb549d76 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4ef9dae49ed..107173f2b53 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9fdb67c109d..0384b39bd97 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 2b18c3c8270..d3b9be5317e 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -43,6 +43,7 @@ #include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" +#include "fe_utils/string_utils.h" #include "parallel.h" #include "pg_backup_utils.h" @@ -158,7 +159,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +172,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +300,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +358,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +421,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +432,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +440,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +552,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +619,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +654,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,11 +718,30 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + PQExpBuffer esc_filename = createPQExpBuffer(); + + /* + * Security: Escape the filename before substituting it into the + * command string. This prevents shell injection if the filename + * contains special characters (like spaces, quotes, or semicolons). + * While internal filenames used by directory format (like toc.dat, + * 1234.dat, blob_567.dat) are currently safe, this follows defensive + * programming practices to ensure the pipeline remains secure even if + * internal naming conventions change. + */ + appendShellString(esc_filename, relativeFilename); + + pipe = replace_percent_placeholders(dname, "pipe", "f", esc_filename->data); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); + destroyPQExpBuffer(esc_filename); } else /* replace all ocurrences of %f in dname with * relativeFilename */ @@ -809,23 +794,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..21157c568b8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,20 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +983,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1316,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index cbac8bc2520..08a642d14b7 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,20 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +626,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +637,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +658,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +677,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +686,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +723,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +784,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1179,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1343,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v11-0004-Add-tests-for-pipe.patch (14.9K, ../../CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com/3-v11-0004-Add-tests-for-pipe.patch) download | inline diff: From 1dc5546344afaa2e3507b070da603ba9f8541483 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v11 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 ++++++- src/bin/pg_dump/t/002_pg_dump.pl | 227 ++++++++++++++++++++ src/bin/pg_dump/t/004_pg_dump_parallel.pl | 30 +++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 345 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 509f4f9ce7d..878ebf33271 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3bc8e51561d..1388e6792b0 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,6 +7,7 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; @@ -46,6 +47,24 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + +# Check for external gzip program for pipe tests. +my $gzip_bin = $ENV{GZIP_PROGRAM} || 'gzip'; +my $has_gzip_bin = (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +242,140 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/defaults_dir_format/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + '--pipe' => "$perl_cat $tempdir/defaults_dir_format/%f", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/pipe_cross_dump/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + '--pipe' => ($supports_gzip && !$PostgreSQL::Test::Utils::windows_os) + ? "if [ -f $tempdir/pipe_cross_restore/%f.gz ]; then $gzip_bin -d -c $tempdir/pipe_cross_restore/%f.gz; else $perl_cat $tempdir/pipe_cross_restore/%f; fi" + : "$perl_cat $tempdir/pipe_cross_restore/%f", + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$gzip_bin | $perl_cat > $tempdir/pipe_out_dir_complex/%f.gz", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_complex/%f.gz | $gzip_bin -d", + '--statistics', + ], + }, + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_lo/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_lo/%f", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +680,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + '--pipe' => "$perl_cat > $tempdir/schema_only_pipe_dir/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + '--pipe' => "$perl_cat $tempdir/schema_only_pipe_dir/%f", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5217,6 +5386,13 @@ foreach my $run (sort keys %pgdump_runs) my $test_key = $run; my $run_db = 'postgres'; + if (defined($pgdump_runs{$run}->{skip_unless}) && + !${ $pgdump_runs{$run}->{skip_unless} }) + { + note "skipping run $run"; + next; + } + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); @@ -5334,6 +5510,57 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "$perlbin -pe \"END { exit 1 }\""; + +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$failing_perl_cat > %f", 'postgres' ], + qr/pipe command failed/, + 'pg_dump pipe command error reporting' +); + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$failing_perl_cat " . "$tempdir/pipe_cross_dump" . '/%f' ], + qr/pipe command failed/, + 'pg_restore pipe command error reporting' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: Broken pipe|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: Broken pipe|not found/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', "--pipe=$perl_cat > %f", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$perl_cat %f", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + ######################################### # Stop the database instance, which will be removed at the end of the tests. diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..efcff1272d9 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,9 +8,15 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; @@ -21,6 +27,7 @@ my $backupdir = $node->backup_dir; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); $node->safe_psql( $dbname1, @@ -87,4 +94,27 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + '--pipe' => "$perl_cat > $backupdir/dump_pipe/%f", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + '--pipe' => "$perl_cat $backupdir/dump_pipe/%f", + ], + 'parallel restore with pipe'); + done_testing(); diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index b2630ef2897..83358696e18 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v11-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.4K, ../../CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com/4-v11-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From cf5c2267905709e3781e57b90dfe7e2a30b04968 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 4 Apr 2025 14:34:48 +0000 Subject: [PATCH v11 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 6ae2cdcfc10..c1222d7d361 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe-commnad="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe-commnad="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe-commnad="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v11-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com/5-v11-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From c1a2b056af1d9b2be16b4f5ee6e2731f6ce7bb3a Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v11 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 74fc651f6f4..2b18c3c8270 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 703b2677b4e..cbac8bc2520 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v11-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.8K, ../../CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com/6-v11-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From 53f34cc049d455d9a76de28e495a0b0edd9c4677 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v11 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 26 +++++-- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 60 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..8c2f95b520d 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -233,7 +236,6 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) if (fd >= 0) { int dup_fd = dup(fd); - if (dup_fd < 0) return false; CFH->private_data = fdopen(dup_fd, mode); @@ -245,7 +247,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +264,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +284,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +297,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index fda912ba0a9..6466bd4bded 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index fecf6f2d1ce..4ef9dae49ed 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9c3aca6543a..9fdb67c109d 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index d6a1428c67a..74fc651f6f4 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 95f4ac110b9..703b2677b4e 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.545.g6539524ca2-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-07 08:35 Nitin Motiani <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-07 08:35 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers I did another round of rebase and added a couple of extra tests. Changed error messages in the tests for windows. > 3. Added shell escaping in the command before setting it as the file path. Removed this because it might have been causing Windows test failures. Since the command runs from the client, this step isn't necessary. Thanks Nitin Motiani Google Attachments: [application/x-patch] v12-0004-Add-tests-for-pipe.patch (15.7K, ../../CAH5HC95hD0tS2FKi8B6=o4g_Pf9WWbg8ZfD0Mu3m-h7ecfiK8A@mail.gmail.com/2-v12-0004-Add-tests-for-pipe.patch) download | inline diff: From da48c820c9d34cd6f39e51e3851311c5ec71368c Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v12 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 +++++- src/bin/pg_dump/t/002_pg_dump.pl | 239 ++++++++++++++++++++ src/bin/pg_dump/t/004_pg_dump_parallel.pl | 30 +++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 357 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 509f4f9ce7d..878ebf33271 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3bc8e51561d..496ffb5df05 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,6 +7,7 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; @@ -46,6 +47,24 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + +# Check for external gzip program for pipe tests. +my $gzip_bin = $ENV{GZIP_PROGRAM} || 'gzip'; +my $has_gzip_bin = (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +242,140 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/defaults_dir_format/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + '--pipe' => "$perl_cat $tempdir/defaults_dir_format/%f", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/pipe_cross_dump/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + '--pipe' => ($supports_gzip && !$PostgreSQL::Test::Utils::windows_os) + ? "if [ -f $tempdir/pipe_cross_restore/%f.gz ]; then $gzip_bin -d -c $tempdir/pipe_cross_restore/%f.gz; else $perl_cat $tempdir/pipe_cross_restore/%f; fi" + : "$perl_cat $tempdir/pipe_cross_restore/%f", + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$gzip_bin | $perl_cat > $tempdir/pipe_out_dir_complex/%f.gz", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_complex/%f.gz | $gzip_bin -d", + '--statistics', + ], + }, + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_lo/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_lo/%f", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +680,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + '--pipe' => "$perl_cat > $tempdir/schema_only_pipe_dir/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + '--pipe' => "$perl_cat $tempdir/schema_only_pipe_dir/%f", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5217,6 +5386,13 @@ foreach my $run (sort keys %pgdump_runs) my $test_key = $run; my $run_db = 'postgres'; + if (defined($pgdump_runs{$run}->{skip_unless}) && + !${ $pgdump_runs{$run}->{skip_unless} }) + { + note "skipping run $run"; + next; + } + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); @@ -5334,6 +5510,69 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "$perlbin -pe \"END { exit 1 }\""; + +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$failing_perl_cat > %f", 'postgres' ], + qr/pipe command failed/, + 'pg_dump pipe command error reporting' +); + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$failing_perl_cat " . "$tempdir/pipe_cross_dump" . '/%f' ], + qr/pipe command failed/, + 'pg_restore pipe command error reporting' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|not found|not recognized/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', "--pipe=$perl_cat > %f", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$perl_cat %f", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + +# Test that pg_dump --pipe bypasses local directory existence check. +# We use a pipe command that writes to a subdirectory that hasn't been created. +# The dump itself will fail when the pipe command tries to write to the +# non-existent directory, but the error should come from the pipe command/write +# failure, not from pg_dump's directory initialization. +my $remote_dir = "$tempdir/non_existent_remote_dir"; +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$perl_cat > $remote_dir/%f", 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|pipe command failed/, + 'pg_dump --pipe bypasses local directory existence check' +); + ######################################### # Stop the database instance, which will be removed at the end of the tests. diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..efcff1272d9 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,9 +8,15 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; @@ -21,6 +27,7 @@ my $backupdir = $node->backup_dir; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); $node->safe_psql( $dbname1, @@ -87,4 +94,27 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + '--pipe' => "$perl_cat > $backupdir/dump_pipe/%f", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + '--pipe' => "$perl_cat $backupdir/dump_pipe/%f", + ], + 'parallel restore with pipe'); + done_testing(); diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index b2630ef2897..83358696e18 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v12-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC95hD0tS2FKi8B6=o4g_Pf9WWbg8ZfD0Mu3m-h7ecfiK8A@mail.gmail.com/3-v12-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From 40740fe2739efd4b89cc29c86febb442e770a16c Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v12 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 74fc651f6f4..2b18c3c8270 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c31d262e71a..c657149d658 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v12-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.6K, ../../CAH5HC95hD0tS2FKi8B6=o4g_Pf9WWbg8ZfD0Mu3m-h7ecfiK8A@mail.gmail.com/4-v12-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From ac5d839b2f349256ab63ce18f801387cd42efe93 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v12 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 25 ++++++- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 59 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..4cf02843185 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -245,7 +248,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +265,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +285,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +298,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index fda912ba0a9..6466bd4bded 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index fecf6f2d1ce..4ef9dae49ed 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9c3aca6543a..9fdb67c109d 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index d6a1428c67a..74fc651f6f4 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 48fdcb0fae1..c31d262e71a 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v12-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.3K, ../../CAH5HC95hD0tS2FKi8B6=o4g_Pf9WWbg8ZfD0Mu3m-h7ecfiK8A@mail.gmail.com/5-v12-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From f17eb09a6aacc4434ed71ebebd7d2f02edd592dd Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 4 Apr 2025 14:34:48 +0000 Subject: [PATCH v12 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 5e77ddd556f..6db9cbc12af 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.545.g6539524ca2-goog [application/x-patch] v12-0003-Fixes-and-refactors-in-pipe-command.patch (39.5K, ../../CAH5HC95hD0tS2FKi8B6=o4g_Pf9WWbg8ZfD0Mu3m-h7ecfiK8A@mail.gmail.com/6-v12-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From 872692907f2a2021740a6747f1fc76bd6bc05d95 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v12 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Use appendShellString in file substitutions for shell escaping. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 6 +- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 60 ++++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 111 ++++++++++---------------- src/bin/pg_dump/pg_dump.c | 53 +++++------- src/bin/pg_dump/pg_dumpall.c | 9 +++ src/bin/pg_dump/pg_restore.c | 69 +++++++++------- 17 files changed, 217 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..3857eff2179 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,7 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; /* * Private data to be used by the compressor. @@ -201,7 +201,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +210,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 4cf02843185..2dae62aadd4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,31 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: %s", reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } } return ret == 0; @@ -228,6 +247,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -248,10 +284,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -266,12 +299,9 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -286,7 +316,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -298,7 +328,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 6466bd4bded..8efeb549d76 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4ef9dae49ed..107173f2b53 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9fdb67c109d..0384b39bd97 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 2b18c3c8270..c4109f0a589 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -158,7 +158,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +431,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +439,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +551,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +618,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +653,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,10 +717,23 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + /* + * Unlike commands synthesized by the backend, this is a user-provided + * template running client-side. We perform literal substitution + * rather than using appendShellString() to avoid interfering with the + * user's intentional shell quoting (e.g., for Windows vs Unix + * differences). Since this is a client-side execution, there are no + * privilege escalation concerns. + */ + pipe = replace_percent_placeholders(dname, "pipe", "f", relativeFilename); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); } else /* replace all ocurrences of %f in dname with @@ -809,23 +787,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..21157c568b8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,20 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +983,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1316,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c657149d658..35dc5b492bb 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,20 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +626,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +637,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +658,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +677,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +686,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +723,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +784,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1179,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1343,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.545.g6539524ca2-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-08 17:07 Nitin Motiani <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-08 17:07 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Hi, Made another small test change. And rebased the code. I'm attaching the latest patches. Thanks Nitin Motiani Google Attachments: [application/x-patch] v13-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.6K, ../../CAH5HC97VKPYVbfbidyTPiiOZnxYmBKWm7EpWCFYkJvyiGjLepQ@mail.gmail.com/2-v13-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From bcb5f9869d0b19aadea938cf11b10130a3de556c Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v13 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 25 ++++++- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 59 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..4cf02843185 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -245,7 +248,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +265,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +285,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +298,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index fda912ba0a9..6466bd4bded 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index fecf6f2d1ce..4ef9dae49ed 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9c3aca6543a..9fdb67c109d 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index d6a1428c67a..74fc651f6f4 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 48fdcb0fae1..c31d262e71a 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v13-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC97VKPYVbfbidyTPiiOZnxYmBKWm7EpWCFYkJvyiGjLepQ@mail.gmail.com/3-v13-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From c3b297f659c288c375b40de84eb4be3a380675b4 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v13 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 74fc651f6f4..2b18c3c8270 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c31d262e71a..c657149d658 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v13-0004-Add-tests-for-pipe.patch (16.3K, ../../CAH5HC97VKPYVbfbidyTPiiOZnxYmBKWm7EpWCFYkJvyiGjLepQ@mail.gmail.com/4-v13-0004-Add-tests-for-pipe.patch) download | inline diff: From 43918db8e304e1492298c20dde6cc13153b827c9 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v13 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 +++++- src/bin/pg_dump/t/002_pg_dump.pl | 243 ++++++++++++++++++++ src/bin/pg_dump/t/004_pg_dump_parallel.pl | 34 +++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 365 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 509f4f9ce7d..878ebf33271 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3bc8e51561d..79c601371ab 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,6 +7,7 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; @@ -46,6 +47,28 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -Mopen=IO,:raw -pe 1"; + +# Check for external gzip program for pipe tests. +my $gzip_bin = $ENV{GZIP_PROGRAM} || 'gzip'; +my $has_gzip_bin = (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +246,140 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/defaults_dir_format/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + '--pipe' => "$perl_cat $tempdir/defaults_dir_format/%f", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/pipe_cross_dump/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + '--pipe' => ($supports_gzip && !$PostgreSQL::Test::Utils::windows_os) + ? "if [ -f $tempdir/pipe_cross_restore/%f.gz ]; then $gzip_bin -d -c $tempdir/pipe_cross_restore/%f.gz; else $perl_cat $tempdir/pipe_cross_restore/%f; fi" + : "$perl_cat $tempdir/pipe_cross_restore/%f", + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$gzip_bin | $perl_cat > $tempdir/pipe_out_dir_complex/%f.gz", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_complex/%f.gz | $gzip_bin -d", + '--statistics', + ], + }, + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_lo/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_lo/%f", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +684,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + '--pipe' => "$perl_cat > $tempdir/schema_only_pipe_dir/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + '--pipe' => "$perl_cat $tempdir/schema_only_pipe_dir/%f", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5217,6 +5390,13 @@ foreach my $run (sort keys %pgdump_runs) my $test_key = $run; my $run_db = 'postgres'; + if (defined($pgdump_runs{$run}->{skip_unless}) && + !${ $pgdump_runs{$run}->{skip_unless} }) + { + note "skipping run $run"; + next; + } + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); @@ -5334,6 +5514,69 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "$perlbin -Mopen=IO,:raw -pe \"END { exit 1 }\""; + +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$failing_perl_cat > %f", 'postgres' ], + qr/pipe command failed/, + 'pg_dump pipe command error reporting' +); + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$failing_perl_cat " . "$tempdir/pipe_cross_dump" . '/%f' ], + qr/pipe command failed/, + 'pg_restore pipe command error reporting' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|not found|not recognized/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', "--pipe=$perl_cat > %f", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$perl_cat %f", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + +# Test that pg_dump --pipe bypasses local directory existence check. +# We use a pipe command that writes to a subdirectory that hasn't been created. +# The dump itself will fail when the pipe command tries to write to the +# non-existent directory, but the error should come from the pipe command/write +# failure, not from pg_dump's directory initialization. +my $remote_dir = "$tempdir/non_existent_remote_dir"; +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$perl_cat > $remote_dir/%f", 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|pipe command failed/, + 'pg_dump --pipe bypasses local directory existence check' +); + ######################################### # Stop the database instance, which will be removed at the end of the tests. diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..d7f02a6e142 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,9 +8,19 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -Mopen=IO,:raw -pe 1"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; @@ -21,6 +31,7 @@ my $backupdir = $node->backup_dir; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); $node->safe_psql( $dbname1, @@ -87,4 +98,27 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + '--pipe' => "$perl_cat > $backupdir/dump_pipe/%f", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + '--pipe' => "$perl_cat $backupdir/dump_pipe/%f", + ], + 'parallel restore with pipe'); + done_testing(); diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index b2630ef2897..83358696e18 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v13-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.3K, ../../CAH5HC97VKPYVbfbidyTPiiOZnxYmBKWm7EpWCFYkJvyiGjLepQ@mail.gmail.com/5-v13-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From 6ba7e3834c56b27aad1e109f0051b1ef4fd77323 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 4 Apr 2025 14:34:48 +0000 Subject: [PATCH v13 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 5e77ddd556f..6db9cbc12af 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v13-0003-Fixes-and-refactors-in-pipe-command.patch (39.5K, ../../CAH5HC97VKPYVbfbidyTPiiOZnxYmBKWm7EpWCFYkJvyiGjLepQ@mail.gmail.com/6-v13-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From 419cfd1fb7e3ea9cb94901f8e43d75a1b234b74e Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v13 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Use appendShellString in file substitutions for shell escaping. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 6 +- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 60 ++++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 111 ++++++++++---------------- src/bin/pg_dump/pg_dump.c | 53 +++++------- src/bin/pg_dump/pg_dumpall.c | 9 +++ src/bin/pg_dump/pg_restore.c | 69 +++++++++------- 17 files changed, 217 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..3857eff2179 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,7 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; /* * Private data to be used by the compressor. @@ -201,7 +201,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +210,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 4cf02843185..2dae62aadd4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,31 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: %s", reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } } return ret == 0; @@ -228,6 +247,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -248,10 +284,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -266,12 +299,9 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -286,7 +316,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -298,7 +328,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 6466bd4bded..8efeb549d76 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4ef9dae49ed..107173f2b53 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9fdb67c109d..0384b39bd97 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 2b18c3c8270..c4109f0a589 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -158,7 +158,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +431,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +439,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +551,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +618,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +653,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,10 +717,23 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + /* + * Unlike commands synthesized by the backend, this is a user-provided + * template running client-side. We perform literal substitution + * rather than using appendShellString() to avoid interfering with the + * user's intentional shell quoting (e.g., for Windows vs Unix + * differences). Since this is a client-side execution, there are no + * privilege escalation concerns. + */ + pipe = replace_percent_placeholders(dname, "pipe", "f", relativeFilename); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); } else /* replace all ocurrences of %f in dname with @@ -809,23 +787,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..21157c568b8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,20 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +983,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1316,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c657149d658..35dc5b492bb 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,20 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +626,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +637,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +658,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +677,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +686,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +723,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +784,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1179,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1343,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.563.g4f69b47b94-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-11 05:34 Nitin Motiani <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-11 05:34 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Attaching another version with minor changes to the test code for windows backslash paths. Thanks Nitin Motiani Google Attachments: [application/x-patch] v14-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.6K, ../../CAH5HC94w4XA87UiKhgw9KKhCfVgU_p+ayT7jJF2gVEGvUibA_A@mail.gmail.com/2-v14-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From bcb5f9869d0b19aadea938cf11b10130a3de556c Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v14 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 25 ++++++- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 59 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..4cf02843185 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -245,7 +248,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +265,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +285,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +298,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index fda912ba0a9..6466bd4bded 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index fecf6f2d1ce..4ef9dae49ed 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9c3aca6543a..9fdb67c109d 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index d6a1428c67a..74fc651f6f4 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 48fdcb0fae1..c31d262e71a 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v14-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC94w4XA87UiKhgw9KKhCfVgU_p+ayT7jJF2gVEGvUibA_A@mail.gmail.com/3-v14-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From c3b297f659c288c375b40de84eb4be3a380675b4 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v14 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 74fc651f6f4..2b18c3c8270 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c31d262e71a..c657149d658 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v14-0003-Fixes-and-refactors-in-pipe-command.patch (39.5K, ../../CAH5HC94w4XA87UiKhgw9KKhCfVgU_p+ayT7jJF2gVEGvUibA_A@mail.gmail.com/4-v14-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From dec97039b98dbbbb78fffc44315961909504a35c Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v14 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 6 +- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 60 ++++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 111 ++++++++++---------------- src/bin/pg_dump/pg_dump.c | 53 +++++------- src/bin/pg_dump/pg_dumpall.c | 9 +++ src/bin/pg_dump/pg_restore.c | 69 +++++++++------- 17 files changed, 217 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..3857eff2179 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,7 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; /* * Private data to be used by the compressor. @@ -201,7 +201,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +210,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 4cf02843185..2dae62aadd4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,31 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: %s", reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } } return ret == 0; @@ -228,6 +247,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -248,10 +284,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -266,12 +299,9 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -286,7 +316,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -298,7 +328,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 6466bd4bded..8efeb549d76 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4ef9dae49ed..107173f2b53 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 9fdb67c109d..0384b39bd97 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 2b18c3c8270..c4109f0a589 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -158,7 +158,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +431,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +439,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +551,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +618,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +653,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,10 +717,23 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + /* + * Unlike commands synthesized by the backend, this is a user-provided + * template running client-side. We perform literal substitution + * rather than using appendShellString() to avoid interfering with the + * user's intentional shell quoting (e.g., for Windows vs Unix + * differences). Since this is a client-side execution, there are no + * privilege escalation concerns. + */ + pipe = replace_percent_placeholders(dname, "pipe", "f", relativeFilename); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); } else /* replace all ocurrences of %f in dname with @@ -809,23 +787,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..21157c568b8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,20 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +983,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1316,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c657149d658..35dc5b492bb 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,20 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +626,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +637,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +658,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +677,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +686,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +723,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +784,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1179,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1343,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v14-0004-Add-tests-for-pipe.patch (16.6K, ../../CAH5HC94w4XA87UiKhgw9KKhCfVgU_p+ayT7jJF2gVEGvUibA_A@mail.gmail.com/5-v14-0004-Add-tests-for-pipe.patch) download | inline diff: From 3274d91d0ee03e1d8ba1bd81080b13df92f8d3ba Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v14 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 +++++- src/bin/pg_dump/t/002_pg_dump.pl | 245 ++++++++++++++++++++ src/bin/pg_dump/t/004_pg_dump_parallel.pl | 35 +++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 368 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 509f4f9ce7d..878ebf33271 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3bc8e51561d..316b1c2bb35 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,8 +7,10 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; +$tempdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; ############################################################### # Definition of the pg_dump runs to make. @@ -46,6 +48,28 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -Mopen=IO,:raw -pe 1"; + +# Check for external gzip program for pipe tests. +my $gzip_bin = $ENV{GZIP_PROGRAM} || 'gzip'; +my $has_gzip_bin = (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +247,141 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/defaults_dir_format/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + '--pipe' => "$perl_cat $tempdir/defaults_dir_format/%f", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$perl_cat > $tempdir/pipe_cross_dump/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + '--pipe' => ($supports_gzip && !$PostgreSQL::Test::Utils::windows_os) + ? "if [ -f \"$tempdir/pipe_cross_restore/%f.gz\" ]; then $gzip_bin -d -c \"$tempdir/pipe_cross_restore/%f.gz\"; else $perl_cat \"$tempdir/pipe_cross_restore/%f\"; fi" + : "$perl_cat \"$tempdir/pipe_cross_restore/%f\"", + + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel/%f", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_parallel_8/%f", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--pipe' => "$gzip_bin | $perl_cat > $tempdir/pipe_out_dir_complex/%f.gz", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_complex/%f.gz | $gzip_bin -d", + '--statistics', + ], + }, + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + '--pipe' => "$perl_cat > $tempdir/pipe_out_dir_lo/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + '--pipe' => "$perl_cat $tempdir/pipe_out_dir_lo/%f", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +686,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + '--pipe' => "$perl_cat > $tempdir/schema_only_pipe_dir/%f", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + '--pipe' => "$perl_cat \"$tempdir/schema_only_pipe_dir/%f\"", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5217,6 +5392,13 @@ foreach my $run (sort keys %pgdump_runs) my $test_key = $run; my $run_db = 'postgres'; + if (defined($pgdump_runs{$run}->{skip_unless}) && + !${ $pgdump_runs{$run}->{skip_unless} }) + { + note "skipping run $run"; + next; + } + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); @@ -5334,6 +5516,69 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "$perlbin -Mopen=IO,:raw -pe \"END { exit 1 }\""; + +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$failing_perl_cat > \"%f\"", 'postgres' ], + qr/pipe command failed/, + 'pg_dump pipe command error reporting' +); + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$failing_perl_cat \"$tempdir/pipe_cross_dump/%f\"" ], + qr/pipe command failed/, + 'pg_restore pipe command error reporting' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|not found|not recognized/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', "--pipe=$perl_cat > \"%f\"", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$perl_cat \"%f\"", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + +# Test that pg_dump --pipe bypasses local directory existence check. +# We use a pipe command that writes to a subdirectory that hasn't been created. +# The dump itself will fail when the pipe command tries to write to the +# non-existent directory, but the error should come from the pipe command/write +# failure, not from pg_dump's directory initialization. +my $remote_dir = "$tempdir/non_existent_remote_dir"; +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$perl_cat > \"$remote_dir/%f\"", 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|pipe command failed/, + 'pg_dump --pipe bypasses local directory existence check' +); + ######################################### # Stop the database instance, which will be removed at the end of the tests. diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..ed4259ee242 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,19 +8,31 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -Mopen=IO,:raw -pe 1"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; $node->start; my $backupdir = $node->backup_dir; +$backupdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); $node->safe_psql( $dbname1, @@ -87,4 +99,27 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + '--pipe' => "$perl_cat > $backupdir/dump_pipe/%f", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + '--pipe' => "$perl_cat \"$backupdir/dump_pipe/%f\"", + ], + 'parallel restore with pipe'); + done_testing(); diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index b2630ef2897..83358696e18 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v14-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.3K, ../../CAH5HC94w4XA87UiKhgw9KKhCfVgU_p+ayT7jJF2gVEGvUibA_A@mail.gmail.com/6-v14-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From 4ea4945704be4494c8b331896380cfb3d4788986 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 4 Apr 2025 14:34:48 +0000 Subject: [PATCH v14 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 5e77ddd556f..6db9cbc12af 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.563.g4f69b47b94-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-14 10:09 Nitin Motiani <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-14 10:09 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers I rebased again and cleaned up the test files to remove the split command in the Windows test. Attaching the latest version. Attachments: [application/x-patch] v15-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.6K, ../../CAH5HC96kAB-HU-Nz0TMgD_V=+P7Y2f0drb1gYNCO9fidqyLhCA@mail.gmail.com/2-v15-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From a22767d57ebc887cb2bf74428ce6206b7422c785 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v15 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 25 ++++++- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 59 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..4cf02843185 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -245,7 +248,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +265,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +285,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +298,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 28e7ff6fa16..549703af622 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 2fd773ad84f..4b6bb7b8a14 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 1218bf6a6a1..cdc12a54f5e 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 562868cd2ad..49a7ab91050 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 48fdcb0fae1..c31d262e71a 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v15-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.3K, ../../CAH5HC96kAB-HU-Nz0TMgD_V=+P7Y2f0drb1gYNCO9fidqyLhCA@mail.gmail.com/3-v15-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From b3c18d74f56286826739ede345136a5f530c3004 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Fri, 4 Apr 2025 14:34:48 +0000 Subject: [PATCH v15 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 5e77ddd556f..6db9cbc12af 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v15-0004-Add-tests-for-pipe.patch (17.5K, ../../CAH5HC96kAB-HU-Nz0TMgD_V=+P7Y2f0drb1gYNCO9fidqyLhCA@mail.gmail.com/4-v15-0004-Add-tests-for-pipe.patch) download | inline diff: From 0320c2d096ad54771a774589966bd1044efe9ae8 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v15 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 +++++- src/bin/pg_dump/t/002_pg_dump.pl | 266 ++++++++++++++++++++ src/bin/pg_dump/t/004_pg_dump_parallel.pl | 38 +++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 392 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 687e842cde9..92d47e4fd93 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3ee9fda50e4..400f19c360d 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,8 +7,10 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; +$tempdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; ############################################################### # Definition of the pg_dump runs to make. @@ -46,6 +48,49 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe 1"; + +# Check for external gzip program for pipe tests. +my $gzip_path = $ENV{GZIP_PROGRAM} || 'gzip'; +my $gzip_bin = "\"$gzip_path\""; +my $has_gzip_bin = + (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Pre-calculate complex pipe commands to keep the test definitions readable +# and ensure unified --pipe=... syntax for Windows stability. +my $pipe_defaults_dir = "$perl_cat > \"$tempdir/defaults_dir_format/%f\""; +my $pipe_defaults_res = "$perl_cat \"$tempdir/defaults_dir_format/%f\""; +my $pipe_cross_dump = "$perl_cat > \"$tempdir/pipe_cross_dump/%f\""; +my $pipe_cross_restore = ($supports_gzip && !$PostgreSQL::Test::Utils::windows_os) + ? "if [ -f \"$tempdir/pipe_cross_restore/%f.gz\" ]; then $gzip_bin -d -c \"$tempdir/pipe_cross_restore/%f.gz\"; else $perl_cat \"$tempdir/pipe_cross_restore/%f\"; fi" + : "$perl_cat \"$tempdir/pipe_cross_restore/%f\""; +my $pipe_parallel_out = "$perl_cat > \"$tempdir/pipe_out_dir_parallel/%f\""; +my $pipe_parallel_in = "$perl_cat \"$tempdir/pipe_out_dir_parallel/%f\""; +my $pipe_parallel_8_out = "$perl_cat > \"$tempdir/pipe_out_dir_parallel_8/%f\""; +my $pipe_parallel_8_in = "$perl_cat \"$tempdir/pipe_out_dir_parallel_8/%f\""; +my $pipe_complex_out = "$gzip_bin | $perl_cat > \"$tempdir/pipe_out_dir_complex/%f.gz\""; +my $pipe_complex_in = "$perl_cat \"$tempdir/pipe_out_dir_complex/%f.gz\" | $gzip_bin -d"; +my $pipe_lo_out = "$perl_cat > \"$tempdir/pipe_out_dir_lo/%f\""; +my $pipe_lo_in = "$perl_cat \"$tempdir/pipe_out_dir_lo/%f\""; +my $pipe_schema_out = "$perl_cat > \"$tempdir/schema_only_pipe_dir/%f\""; +my $pipe_schema_in = "$perl_cat \"$tempdir/schema_only_pipe_dir/%f\""; + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +268,139 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_defaults_dir", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + "--pipe=$pipe_defaults_res", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_cross_dump", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + "--pipe=$pipe_cross_restore", + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + "--pipe=$pipe_parallel_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + "--pipe=$pipe_parallel_in", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + "--pipe=$pipe_parallel_8_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + "--pipe=$pipe_parallel_8_in", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_complex_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + "--pipe=$pipe_complex_in", + '--statistics', + ], + }, + + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + "--pipe=$pipe_lo_out", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + "--pipe=$pipe_lo_in", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +705,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + "--pipe=$pipe_schema_out", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + "--pipe=$pipe_schema_in", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5217,6 +5411,13 @@ foreach my $run (sort keys %pgdump_runs) my $test_key = $run; my $run_db = 'postgres'; + if (defined($pgdump_runs{$run}->{skip_unless}) && + !${ $pgdump_runs{$run}->{skip_unless} }) + { + note "skipping run $run"; + next; + } + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); @@ -5334,9 +5535,74 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe \"END { exit 1 }\""; + +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$failing_perl_cat > \"%f\"", 'postgres' ], + qr/pipe command failed/, + 'pg_dump pipe command error reporting' +); + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$failing_perl_cat \"$tempdir/pipe_cross_dump/%f\"" ], + qr/pipe command failed/, + 'pg_restore pipe command error reporting' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|not found|not recognized/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', "--pipe=$perl_cat > \"%f\"", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', "--pipe=$perl_cat \"%f\"", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + +# Test that pg_dump --pipe bypasses local directory existence check. +# We use a pipe command that writes to a subdirectory that hasn't been created. +# The dump itself will fail when the pipe command tries to write to the +# non-existent directory, but the error should come from the pipe command/write +# failure, not from pg_dump's directory initialization. +my $remote_dir = "$tempdir/non_existent_remote_dir"; +$node->command_fails_like( + [ 'pg_dump', '-Fd', "--pipe=$perl_cat > \"$remote_dir/%f\"", 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|pipe command failed/, + 'pg_dump --pipe bypasses local directory existence check' +); + + ######################################### # Stop the database instance, which will be removed at the end of the tests. $node->stop('fast'); done_testing(); + diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..9c390ce63b7 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,19 +8,31 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe 1"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; $node->start; my $backupdir = $node->backup_dir; +$backupdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); $node->safe_psql( $dbname1, @@ -87,4 +99,30 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +# Pre-calculate pipe commands for readability and unified syntax. +my $pipe_dump = "$perl_cat > \"$backupdir/dump_pipe/%f\""; +my $pipe_restore = "$perl_cat \"$backupdir/dump_pipe/%f\""; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + "--pipe=$pipe_dump", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + "--pipe=$pipe_restore", + ], + 'parallel restore with pipe'); done_testing(); diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index cecf0442088..f4b1afd8ef6 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v15-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC96kAB-HU-Nz0TMgD_V=+P7Y2f0drb1gYNCO9fidqyLhCA@mail.gmail.com/5-v15-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From 2da7bfae1b2bd84faa7f6641672b16f9fb43a81d Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v15 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 49a7ab91050..15ce45fb9e9 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c31d262e71a..c657149d658 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.563.g4f69b47b94-goog [application/x-patch] v15-0003-Fixes-and-refactors-in-pipe-command.patch (39.5K, ../../CAH5HC96kAB-HU-Nz0TMgD_V=+P7Y2f0drb1gYNCO9fidqyLhCA@mail.gmail.com/6-v15-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From 5889951576301be365e3036d69320b2cfde1b7e5 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v15 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 6 +- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 60 ++++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 111 ++++++++++---------------- src/bin/pg_dump/pg_dump.c | 53 +++++------- src/bin/pg_dump/pg_dumpall.c | 9 +++ src/bin/pg_dump/pg_restore.c | 69 +++++++++------- 17 files changed, 217 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..3857eff2179 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,7 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; /* * Private data to be used by the compressor. @@ -201,7 +201,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +210,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 4cf02843185..2dae62aadd4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,31 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: %s", reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } } return ret == 0; @@ -228,6 +247,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -248,10 +284,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -266,12 +299,9 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -286,7 +316,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -298,7 +328,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 549703af622..c1148a66635 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4b6bb7b8a14..bb14a83b80b 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index cdc12a54f5e..9555d44ae29 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 15ce45fb9e9..3a6f47d5483 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -158,7 +158,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +431,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +439,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +551,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +618,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +653,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,10 +717,23 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + /* + * Unlike commands synthesized by the backend, this is a user-provided + * template running client-side. We perform literal substitution + * rather than using appendShellString() to avoid interfering with the + * user's intentional shell quoting (e.g., for Windows vs Unix + * differences). Since this is a client-side execution, there are no + * privilege escalation concerns. + */ + pipe = replace_percent_placeholders(dname, "pipe", "f", relativeFilename); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); } else /* replace all ocurrences of %f in dname with @@ -809,23 +787,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..21157c568b8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,20 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +983,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1316,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c657149d658..35dc5b492bb 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,20 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +626,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +637,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +658,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +677,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +686,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +723,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +784,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1179,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1343,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.563.g4f69b47b94-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-21 09:56 Nitin Motiani <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-05-21 09:56 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Changed how pipe commands are quoted in the Windows test. The latest versions are attached. Thanks Nitin Motiani Google Attachments: [application/x-patch] v16-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com/2-v16-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From 9c85862782a6fe018c22dc469bd5abfa596cf1f9 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v16 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 49a7ab91050..15ce45fb9e9 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c31d262e71a..c657149d658 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.669.g59709faab0-goog [application/x-patch] v16-0004-Add-tests-for-pipe.patch (20.1K, ../../CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com/3-v16-0004-Add-tests-for-pipe.patch) download | inline diff: From 76ee3b3c375c32055edff86f6348af007012a922 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v16 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 ++++- src/bin/pg_dump/t/002_pg_dump.pl | 292 +++++++++++++++++++- src/bin/pg_dump/t/004_pg_dump_parallel.pl | 43 +++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 415 insertions(+), 10 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 687e842cde9..92d47e4fd93 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3ee9fda50e4..51582b3caf7 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,8 +7,10 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; +$tempdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; ############################################################### # Definition of the pg_dump runs to make. @@ -46,6 +48,69 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe 1"; + +# Check for external gzip program for pipe tests. +my $gzip_path = $ENV{GZIP_PROGRAM} || 'gzip'; +my $gzip_bin = "\"$gzip_path\""; +my $has_gzip_bin = + (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Pre-calculate complex pipe commands to keep the test definitions readable +# and ensure unified --pipe=... syntax for Windows stability. +# On Windows, we use double-layer quoting: internal quotes for paths with +# spaces, and an outer set of escaped quotes to protect shell operators +# like | and >. On other platforms, we avoid the outer wrap to satisfy /bin/sh. +my $is_win = $PostgreSQL::Test::Utils::windows_os; + +my $raw_pipe_defaults_dir = "$perl_cat > \"$tempdir/defaults_dir_format/%f\""; +my $raw_pipe_defaults_res = "$perl_cat \"$tempdir/defaults_dir_format/%f\""; +my $raw_pipe_cross_dump = "$perl_cat > \"$tempdir/pipe_cross_dump/%f\""; +my $raw_pipe_cross_restore = ($supports_gzip && !$is_win) + ? "if [ -f \"$tempdir/pipe_cross_restore/%f.gz\" ]; then $gzip_bin -d -c \"$tempdir/pipe_cross_restore/%f.gz\"; else $perl_cat \"$tempdir/pipe_cross_restore/%f\"; fi" + : "$perl_cat \"$tempdir/pipe_cross_restore/%f\""; +my $raw_pipe_parallel_out = "$perl_cat > \"$tempdir/pipe_out_dir_parallel/%f\""; +my $raw_pipe_parallel_in = "$perl_cat \"$tempdir/pipe_out_dir_parallel/%f\""; +my $raw_pipe_parallel_8_out = "$perl_cat > \"$tempdir/pipe_out_dir_parallel_8/%f\""; +my $raw_pipe_parallel_8_in = "$perl_cat \"$tempdir/pipe_out_dir_parallel_8/%f\""; +my $raw_pipe_complex_out = "$gzip_bin | $perl_cat > \"$tempdir/pipe_out_dir_complex/%f.gz\""; +my $raw_pipe_complex_in = "$perl_cat \"$tempdir/pipe_out_dir_complex/%f.gz\" | $gzip_bin -d"; +my $raw_pipe_lo_out = "$perl_cat > \"$tempdir/pipe_out_dir_lo/%f\""; +my $raw_pipe_lo_in = "$perl_cat \"$tempdir/pipe_out_dir_lo/%f\""; +my $raw_pipe_schema_out = "$perl_cat > \"$tempdir/schema_only_pipe_dir/%f\""; +my $raw_pipe_schema_in = "$perl_cat \"$tempdir/schema_only_pipe_dir/%f\""; + +my $pipe_defaults_dir = $is_win ? "\"$raw_pipe_defaults_dir\"" : $raw_pipe_defaults_dir; +my $pipe_defaults_res = $is_win ? "\"$raw_pipe_defaults_res\"" : $raw_pipe_defaults_res; +my $pipe_cross_dump = $is_win ? "\"$raw_pipe_cross_dump\"" : $raw_pipe_cross_dump; +my $pipe_cross_restore = $is_win ? "\"$raw_pipe_cross_restore\"" : $raw_pipe_cross_restore; +my $pipe_parallel_out = $is_win ? "\"$raw_pipe_parallel_out\"" : $raw_pipe_parallel_out; +my $pipe_parallel_in = $is_win ? "\"$raw_pipe_parallel_in\"" : $raw_pipe_parallel_in; +my $pipe_parallel_8_out = $is_win ? "\"$raw_pipe_parallel_8_out\"" : $raw_pipe_parallel_8_out; +my $pipe_parallel_8_in = $is_win ? "\"$raw_pipe_parallel_8_in\"" : $raw_pipe_parallel_8_in; +my $pipe_complex_out = $is_win ? "\"$raw_pipe_complex_out\"" : $raw_pipe_complex_out; +my $pipe_complex_in = $is_win ? "\"$raw_pipe_complex_in\"" : $raw_pipe_complex_in; +my $pipe_lo_out = $is_win ? "\"$raw_pipe_lo_out\"" : $raw_pipe_lo_out; +my $pipe_lo_in = $is_win ? "\"$raw_pipe_lo_in\"" : $raw_pipe_lo_in; +my $pipe_schema_out = $is_win ? "\"$raw_pipe_schema_out\"" : $raw_pipe_schema_out; +my $pipe_schema_in = $is_win ? "\"$raw_pipe_schema_in\"" : $raw_pipe_schema_in; + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +288,139 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_defaults_dir", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + "--pipe=$pipe_defaults_res", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_cross_dump", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + "--pipe=$pipe_cross_restore", + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + "--pipe=$pipe_parallel_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + "--pipe=$pipe_parallel_in", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + "--pipe=$pipe_parallel_8_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + "--pipe=$pipe_parallel_8_in", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_complex_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + "--pipe=$pipe_complex_in", + '--statistics', + ], + }, + + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + "--pipe=$pipe_lo_out", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + "--pipe=$pipe_lo_in", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +725,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + "--pipe=$pipe_schema_out", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + "--pipe=$pipe_schema_in", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5212,25 +5426,24 @@ command_fails_like( ######################################### # Run all runs + foreach my $run (sort keys %pgdump_runs) { my $test_key = $run; - my $run_db = 'postgres'; + my $run_db = 'postgres'; $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); if ($pgdump_runs{$run}->{glob_patterns}) { - my $glob_patterns = $pgdump_runs{$run}->{glob_patterns}; - foreach my $glob_pattern (@{$glob_patterns}) + foreach my $glob_pattern (@{ $pgdump_runs{$run}->{glob_patterns} }) { - my @glob_output = glob($glob_pattern); my $ok = 0; - # certainly found some files if glob() returned multiple matches - $ok = 1 if (scalar(@glob_output) > 1); - # if just one match, we need to check if it's real - $ok = 1 if (scalar(@glob_output) == 1 && -f $glob_output[0]); + foreach my $file (glob("$glob_pattern")) + { + $ok = 1 if -e $file; + } is($ok, 1, "$run: glob check for $glob_pattern"); } } @@ -5334,6 +5547,69 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe \"END { exit 1 }\""; + +$node->command_fails_like( + [ 'pg_dump', '-Fd', $is_win ? "--pipe=\"$failing_perl_cat > \\\"%f\\\"\"" : "--pipe=$failing_perl_cat > \"%f\"", 'postgres' ], + qr/pipe command failed/, + 'pg_dump pipe command error reporting' +); + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', $is_win ? "--pipe=\"$failing_perl_cat \\\"$tempdir/pipe_cross_dump/%f\\\"\"" : "--pipe=$failing_perl_cat \"$tempdir/pipe_cross_dump/%f\"", ], + qr/pipe command failed/, + 'pg_restore pipe command error reporting' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|not found|not recognized/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', $is_win ? "--pipe=\"$perl_cat > \\\"%f\\\"\"" : "--pipe=$perl_cat > \"%f\"", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', $is_win ? "--pipe=\"$perl_cat \\\"%f\\\"\"" : "--pipe=$perl_cat \"%f\"", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + +# Test that pg_dump --pipe bypasses local directory existence check. +# We use a pipe command that writes to a subdirectory that hasn't been created. +# The dump itself will fail when the pipe command tries to write to the +# non-existent directory, but the error should come from the pipe command/write +# failure, not from pg_dump's directory initialization. +my $remote_dir = "$tempdir/non_existent_remote_dir"; +$node->command_fails_like( + [ 'pg_dump', '-Fd', $is_win ? "--pipe=\"$perl_cat > \\\"$remote_dir/%f\\\"\"" : "--pipe=$perl_cat > \"$remote_dir/%f\"", 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|pipe command failed/, + 'pg_dump --pipe bypasses local directory existence check' +); + ######################################### # Stop the database instance, which will be removed at the end of the tests. diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..63cd3ba016d 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,19 +8,31 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe 1"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; $node->start; my $backupdir = $node->backup_dir; +$backupdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); $node->safe_psql( $dbname1, @@ -87,4 +99,35 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +# Pre-calculate pipe commands for readability and unified syntax. +# Use double-layer quoting only on Windows to protect shell operators. +my $is_win = $PostgreSQL::Test::Utils::windows_os; +my $raw_pipe_dump = "$perl_cat > \"$backupdir/dump_pipe/%f\""; +my $raw_pipe_restore = "$perl_cat \"$backupdir/dump_pipe/%f\""; + +my $pipe_dump = $is_win ? "\"$raw_pipe_dump\"" : $raw_pipe_dump; +my $pipe_restore = $is_win ? "\"$raw_pipe_restore\"" : $raw_pipe_restore; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + "--pipe=$pipe_dump", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + "--pipe=$pipe_restore", + ], + 'parallel restore with pipe'); done_testing(); diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index cecf0442088..f4b1afd8ef6 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.669.g59709faab0-goog [application/x-patch] v16-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.6K, ../../CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com/4-v16-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From d17b8e90c1b17e5d9b6497c8034996370e02c3c0 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v16 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 25 ++++++- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 59 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..4cf02843185 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -245,7 +248,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +265,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +285,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +298,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 28e7ff6fa16..549703af622 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 2fd773ad84f..4b6bb7b8a14 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 1218bf6a6a1..cdc12a54f5e 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 562868cd2ad..49a7ab91050 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index d56dcc701ce..7345e6c7a4b 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 48fdcb0fae1..c31d262e71a 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.669.g59709faab0-goog [application/x-patch] v16-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.4K, ../../CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com/5-v16-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From 680027b837f2d8b6f5ae195b5fe23abb1d4f1f71 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Thu, 21 May 2026 09:52:25 +0000 Subject: [PATCH v16 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 5e77ddd556f..6db9cbc12af 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.669.g59709faab0-goog [application/x-patch] v16-0003-Fixes-and-refactors-in-pipe-command.patch (39.5K, ../../CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com/6-v16-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From 34ef34f70469972c7784395b037f55533a9396ef Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v16 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 6 +- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 60 ++++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 111 ++++++++++---------------- src/bin/pg_dump/pg_dump.c | 53 +++++------- src/bin/pg_dump/pg_dumpall.c | 9 +++ src/bin/pg_dump/pg_restore.c | 69 +++++++++------- 17 files changed, 217 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..3857eff2179 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,7 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; /* * Private data to be used by the compressor. @@ -201,7 +201,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +210,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 4cf02843185..2dae62aadd4 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,31 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: %s", reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } } return ret == 0; @@ -228,6 +247,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -248,10 +284,7 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -266,12 +299,9 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; @@ -286,7 +316,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -298,7 +328,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 549703af622..c1148a66635 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4b6bb7b8a14..bb14a83b80b 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index cdc12a54f5e..9555d44ae29 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 15ce45fb9e9..3a6f47d5483 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -158,7 +158,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +431,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +439,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +551,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +618,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +653,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,10 +717,23 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + /* + * Unlike commands synthesized by the backend, this is a user-provided + * template running client-side. We perform literal substitution + * rather than using appendShellString() to avoid interfering with the + * user's intentional shell quoting (e.g., for Windows vs Unix + * differences). Since this is a client-side execution, there are no + * privilege escalation concerns. + */ + pipe = replace_percent_placeholders(dname, "pipe", "f", relativeFilename); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); } else /* replace all ocurrences of %f in dname with @@ -809,23 +787,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 7345e6c7a4b..21157c568b8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,20 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +983,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1316,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c657149d658..35dc5b492bb 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,20 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +626,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +637,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +658,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +677,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +686,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +723,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +784,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1179,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1343,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.669.g59709faab0-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-05-22 10:34 solai v <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: solai v @ 2026-05-22 10:34 UTC (permalink / raw) To: Nitin Motiani <[email protected]>; +Cc: Hannu Krosing <[email protected]>; Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Hi all, Thank you for the updated patch. On Fri, May 22, 2026 at 1:03 PM Nitin Motiani <[email protected]> wrote: > > Changed how pipe commands are quoted in the Windows test. The latest > versions are attached. I worked on reproducing the current limitation around parallel dumps and then tested the latest v16 patch adding --pipe support for pg_dump. To begin with, I verified the existing behavior. For example: pg_dump postgres | gzip > dump.sql.gz works, but does not support parallelism, whereas: pg_dump -Fd -j 4 -f dumpdir postgres du -sh dumpdir 21M dumpdir requires intermediate disk storage. This demonstrates the current limitation where users must choose between parallelism and streaming pipelines. I then tested the patch introducing --pipe support. The feature is quite useful for modern workflows where users want to stream dump output directly to compression or upload pipelines without relying on intermediate storage. Basic functionality worked as expected. For example: pg_dump -p 55432 -Fd -j 4 --pipe="cat > dump.out" postgres, produced a ~38MB output file, and: pg_dump -p 55432 -Fd -j 4 --pipe="gzip > dump.gz" postgres produced, a compressed file (~11MB). The initial contents appeared valid: gunzip -c dump.gz | head 1 2 3 ... Also, no intermediate directory was created, confirming that the patch enables streaming without filesystem-backed staging. Error handling also behaved correctly. For example: --pipe="invalid_cmd" resulted in: pg_dump: error: pipe command failed: command not found and: --pipe="gzip | false" resulted in: pg_dump: error: pipe command failed: child process exited with exit code 1 However, I observed an important issue when using the feature with multiple parallel workers. Since the pipe command is executed per output file, using: --pipe="gzip > dump.gz", it results in multiple workers invoking independent gzip processes that all write to the same output file. This leads to corrupted or truncated output. In my testing: gunzip -c dump.gz > dump.sql failed with: gzip: dump.gz: unexpected end of file This suggests that concurrent writes to a shared output target are not coordinated and can result in invalid dumps. It would be helpful to clarify expected usage patterns here. For example: whether users are expected to generate distinct outputs per worker, or whether safeguards should be implemented to prevent multiple workers from writing to the same destination. Additionally, during failure scenarios I observed backend logs such as: FATAL: connection to client lost Broken pipe While this is expected when the pipe terminates prematurely, it may be worth considering whether error messaging or cleanup behavior can be made clearer from the user perspective. Overall, the feature is valuable and aligns well with modern backup workflows. However, behavior in multi-worker scenarios with shared pipe targets may need further clarification or safeguards to avoid data corruption. Looking forward to more feedback. Regards. Solai ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-06-09 06:34 Nitin Motiani <[email protected]> parent: solai v <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Nitin Motiani @ 2026-06-09 06:34 UTC (permalink / raw) To: solai v <[email protected]>; +Cc: Hannu Krosing <[email protected]>; Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers On Fri, May 22, 2026 at 4:04 PM solai v <[email protected]> wrote: > I then tested the patch introducing --pipe support. The feature is > quite useful for modern workflows where users want to stream dump > output directly to compression or upload pipelines without relying on > intermediate storage. Thank you for the feedback. > gzip: dump.gz: unexpected end of file > This suggests that concurrent writes to a shared output target are not > coordinated and can result in invalid dumps. It would be helpful to > clarify expected usage patterns here. For example: whether users are > expected to generate distinct outputs per worker, or whether > safeguards should be implemented to prevent multiple workers from > writing to the same destination. I added a warning for cases where the pipe command provided with parallel dump and restore doesn't contain a `%f`. We can also add it to the documentation. Let me know what you think. > scenarios I observed backend logs such as: > FATAL: connection to client lost > Broken pipe > While this is expected when the pipe terminates prematurely, it may be > worth considering whether error messaging or cleanup behavior can be > made clearer from the user perspective. I added the failed command to the error message. I'm not sure if we can do any auto-cleanup commands which succeeded. Thanks & Regards Nitin Motiani Google Attachments: [application/x-patch] v17-0002-Add-pipe-command-support-in-pg_restore.patch (9.5K, ../../CAH5HC94-gFS63cnQbSXZ+hDvSUNV51U+KeRU-XiY-ngu_KtJ5A@mail.gmail.com/2-v17-0002-Add-pipe-command-support-in-pg_restore.patch) download | inline diff: From edbbfea8a4819aa14e60031b0bbe6a38d6e759e9 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 08:05:25 +0000 Subject: [PATCH v17 2/5] Add pipe-command support in pg_restore * This is same as the pg_dump change. We add support for --pipe-command in directory archive format. This can be used to read from multiple streams and do pre-processing (decompression with a custom algorithm, filtering etc) before restore. Currently that is not possible because the pg_dump output of directory format can't just be piped. * Like pg_dump, here also either filename or --pipe-command can be set. If neither are set, the standard input is used as before. * This is only supported with compression none and archive format directory. * We reuse the inputFileSpec field for the pipe-command. And add a bool to specify if it is a pipe. * The changes made for pg_dump to handle the pipe case with popen and pclose also work here. * The logic of %f format specifier to read from the pg_dump output is the same too. Most of the code from the pg_dump commit works. We add similar logic to the function to read large objects. * The --pipe command works -l and -L option. --- src/bin/pg_dump/compress_io.c | 30 +++++++++------ src/bin/pg_dump/pg_backup_directory.c | 16 +++++++- src/bin/pg_dump/pg_restore.c | 53 ++++++++++++++++++++------- 3 files changed, 72 insertions(+), 27 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index bc521dd274b..88488186b34 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -260,22 +260,28 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, fname = pg_strdup(path); - if (hasSuffix(fname, ".gz")) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (hasSuffix(fname, ".lz4")) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (hasSuffix(fname, ".zst")) - compression_spec.algorithm = PG_COMPRESSION_ZSTD; - else + /* + * If the path is a pipe command, the compression algorithm is none. + */ + if (!path_is_pipe_command) { - if (stat(path, &st) == 0) - compression_spec.algorithm = PG_COMPRESSION_NONE; - else if (check_compressed_file(path, &fname, "gz")) + if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; - else if (check_compressed_file(path, &fname, "lz4")) + else if (hasSuffix(fname, ".lz4")) compression_spec.algorithm = PG_COMPRESSION_LZ4; - else if (check_compressed_file(path, &fname, "zst")) + else if (hasSuffix(fname, ".zst")) compression_spec.algorithm = PG_COMPRESSION_ZSTD; + else + { + if (stat(path, &st) == 0) + compression_spec.algorithm = PG_COMPRESSION_NONE; + else if (check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = PG_COMPRESSION_GZIP; + else if (check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = PG_COMPRESSION_LZ4; + else if (check_compressed_file(path, &fname, "zst")) + compression_spec.algorithm = PG_COMPRESSION_ZSTD; + } } CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 49a7ab91050..15ce45fb9e9 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -439,7 +439,21 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) tocfname, line); StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + + /* + * XXX : Create a helper function for blob files naming common to + * _LoadLOs an _StartLO. + */ + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); + strcpy(path, pipe); + pfree(pipe); + } + else + { + snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); + } _PrintFileData(AH, path); EndRestoreLO(AH, oid); } diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c31d262e71a..c657149d658 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts); + int numWorkers, bool append_data, bool filespec_is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -93,6 +93,7 @@ main(int argc, char **argv) int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; + bool filespec_is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -173,6 +174,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, + {"pipe-command", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -356,6 +358,11 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; + case 8: /* pipe-command */ + inputFileSpec = pg_strdup(optarg); + filespec_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -363,11 +370,29 @@ main(int argc, char **argv) } } - /* Get file name from command line */ + /* + * Get file name from command line. Note that filename argument and + * pipe-command can't both be set. + */ if (optind < argc) + { + if (filespec_is_pipe) + { + pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); + exit_nicely(1); + } inputFileSpec = argv[optind++]; - else + } + + /* + * Even if the file argument is not provided, if the pipe-command is + * specified, we need to use that as the file arg and not fallback to + * stdio. + */ + else if (!filespec_is_pipe) + { inputFileSpec = NULL; + } /* Complain if any arguments remain */ if (optind < argc) @@ -594,7 +619,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts); + n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -606,7 +631,7 @@ main(int argc, char **argv) { /* Now restore all the databases from map.dat */ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers); + opts, numWorkers, filespec_is_pipe); } /* Free db pattern list. */ @@ -626,7 +651,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false); + n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -645,7 +670,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) { Archive *AH; int nerror = 0; @@ -654,7 +679,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -691,12 +716,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data) + int numWorkers, bool append_data, bool filespec_is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, false); + AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -1145,7 +1170,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers) + int numWorkers, bool filespec_is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1309,7 +1334,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); n_errors_total += n_errors; -- 2.54.0.1064.gd145956f57-goog [application/x-patch] v17-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch (7.3K, ../../CAH5HC94-gFS63cnQbSXZ+hDvSUNV51U+KeRU-XiY-ngu_KtJ5A@mail.gmail.com/3-v17-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch) download | inline diff: From 13b1f2511a5e8107abff78f6dc8f288557990b8f Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Mon, 8 Jun 2026 11:33:43 +0000 Subject: [PATCH v17 5/5] Add documentation for pipe in pg_dump and pg_restore * Add the descriptions of the new flags and constraints regarding which mode and other flags they can't be used with. * Explain the purpose of the flags. * Add a few examples of the usage of the flags. --- doc/src/sgml/ref/pg_dump.sgml | 56 ++++++++++++++++++++++++++ doc/src/sgml/ref/pg_restore.sgml | 68 +++++++++++++++++++++++++++++++- 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index ae1bc14d2f2..6458b032d25 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -297,6 +297,7 @@ PostgreSQL documentation specifies the target directory instead of a file. In this case the directory is created by <command>pg_dump</command> unless the directory exists and is empty. + This option and <option>--pipe</option> can't be used together. </para> </listitem> </varlistentry> @@ -1224,6 +1225,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to write to multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes the pg_dump output to this + process. + This option is not valid if <option>--file</option> + is also specified. + </para> + <para> + The pipe can be used to perform operations like compress + using a custom algorithm, filter, or write the output to a cloud + storage etc. The user would need a way to pipe the final output of + each stream to a file. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>--file</option>. + See <xref linkend="pg-dump-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--quote-all-identifiers</option></term> <listitem> @@ -1803,6 +1830,35 @@ CREATE DATABASE foo WITH TEMPLATE template0; </screen> </para> + <para> + To use pipe to dump a database into a directory-format archive + (the directory <literal>dumpdir</literal> needs to exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to dump a database into a directory-format archive + in parallel with 5 worker jobs (the directory <literal>dumpdir</literal> needs to exist + before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb -j 5 --pipe="cat > dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pipe to compress and dump a database into a + directory-format archive (the directory <literal>dumpdir</literal> needs to + exist before running the command). + +<screen> +<prompt>$</prompt> <userinput>pg_dump -Fd mydb --pipe="gzip > dumpdir/%f.gz"</userinput> +</screen> + </para> + <para> To reload an archive file into a (freshly created) database named <literal>newdb</literal>: diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 5e77ddd556f..6db9cbc12af 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -118,7 +118,10 @@ PostgreSQL documentation <para> Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. - If not specified, the standard input is used. + This option and <option>--pipe</option> can't be set + at the same time. + If neither this option nor <option>--pipe</option> is specified, + the standard input is used. </para> </listitem> </varlistentry> @@ -919,6 +922,32 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--pipe</option></term> + <listitem> + <para> + This option is only supported with the directory output + format. It can be used to read from multiple streams which + otherwise would not be possible with the directory mode. + For each stream, it starts a process which runs the + specified command and pipes its output to the pg_restore process. + This option is not valid if <option>filename</option> is also specified. + </para> + <para> + The pipe can be used to perform operations like + decompress using a custom algorithm, filter, or read from + a cloud storage. When reading from the pg_dump output, + the user would need a way to read the correct file in each + stream. To handle that, the pipe command supports a format + specifier %f. And all the instances of %f in the command string + will be replaced with the corresponding file name which + would have been used in the directory mode with <option>filename</option>. + This is same as the <option>--pipe</option> of pg-dump. + See <xref linkend="app-pgrestore-examples"/> below. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--section=<replaceable class="parameter">sectionname</replaceable></option></term> <listitem> @@ -1364,6 +1393,43 @@ CREATE DATABASE foo WITH TEMPLATE template0; <prompt>$</prompt> <userinput>pg_restore -L db.list db.dump</userinput> </screen></para> + <para> + To use pg_restore with pipe to recreate from a dump in + directory-archive format. The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f"</userinput> +</screen> + </para> + + <para> + To use pg_restore with pipe to first decompress and then + recreate from a dump in directory-archive format. The database + should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in <literal>dumpdir</literal>. And all files are + <literal>gzip</literal> compressed. + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f.gz | gunzip"</userinput> +</screen> + </para> + + <para> + To use pipe along with <option>-L</option> to recreate only + selectd items from a dump in the directory-archive format. + The database should not exist beforehand. + Assume in this example that the dump in directory-archive format is + stored in dumpdir. + The <literal>db.list</literal> file is the same as one used in the previous example with <option>-L</option> + +<screen> +<prompt>$</prompt> <userinput>pg_restore -C -Fd -d postgres --pipe="cat dumpdir/%f" -L db.list</userinput> +</screen> + </para> + </refsect1> <refsect1> -- 2.54.0.1064.gd145956f57-goog [application/x-patch] v17-0003-Fixes-and-refactors-in-pipe-command.patch (40.4K, ../../CAH5HC94-gFS63cnQbSXZ+hDvSUNV51U+KeRU-XiY-ngu_KtJ5A@mail.gmail.com/4-v17-0003-Fixes-and-refactors-in-pipe-command.patch) download | inline diff: From 8f50e50e2c264ceee4f3b190c331dd908d63b010 Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sun, 3 May 2026 12:37:46 +0000 Subject: [PATCH v17 3/5] Fixes and refactors in pipe command Fix pclose bug with fdopen case for stdout by ensuring fclose is called. Add better error handling to pclose and show a clearer error message using wait_result_to_str() Changed pipe-command flag to pipe as recommended in review. Change the mode from 'ab' to 'w' for large object TOC. Refactor and document the code. --- src/bin/pg_dump/compress_gzip.c | 6 +- src/bin/pg_dump/compress_gzip.h | 2 +- src/bin/pg_dump/compress_io.c | 25 +++--- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 8 +- src/bin/pg_dump/compress_lz4.h | 2 +- src/bin/pg_dump/compress_none.c | 76 ++++++++++++++---- src/bin/pg_dump/compress_none.h | 2 +- src/bin/pg_dump/compress_zstd.c | 8 +- src/bin/pg_dump/compress_zstd.h | 2 +- src/bin/pg_dump/pg_backup.h | 4 +- src/bin/pg_dump/pg_backup_archiver.c | 30 ++++++- src/bin/pg_dump/pg_backup_archiver.h | 2 +- src/bin/pg_dump/pg_backup_directory.c | 111 ++++++++++---------------- src/bin/pg_dump/pg_dump.c | 56 ++++++------- src/bin/pg_dump/pg_dumpall.c | 9 +++ src/bin/pg_dump/pg_restore.c | 72 ++++++++++------- 17 files changed, 244 insertions(+), 182 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 0ce15847d9a..6a02f9b3907 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -430,9 +430,9 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("cPipe command not supported for Gzip"); CFH->open_func = Gzip_open; @@ -460,7 +460,7 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index f77c5c86c56..952c9223836 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -20,6 +20,6 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 88488186b34..b4d84ef17d1 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -192,28 +192,27 @@ free_keep_errno(void *p) */ CompressFileHandle * InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); /* - * Always set to non-compressed when path_is_pipe_command assuming that - * external compressor as part of pipe is more efficient. Can review in - * the future. + * Always set to non-compressed when is_pipe assuming that external + * compressor as part of pipe is more efficient. Can review in the future. */ - if (path_is_pipe_command) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + if (is_pipe) + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleNone(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleGzip(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleLZ4(CFH, compression_spec, is_pipe); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); + InitCompressFileHandleZstd(CFH, compression_spec, is_pipe); return CFH; } @@ -247,7 +246,7 @@ check_compressed_file(const char *path, char **fname, char *ext) */ CompressFileHandle * InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command) + bool is_pipe) { CompressFileHandle *CFH = NULL; struct stat st; @@ -263,7 +262,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, /* * If the path is a pipe command, the compression algorithm is none. */ - if (!path_is_pipe_command) + if (!is_pipe) { if (hasSuffix(fname, ".gz")) compression_spec.algorithm = PG_COMPRESSION_GZIP; @@ -284,7 +283,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode, } } - CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); + CFH = InitCompressFileHandle(compression_spec, is_pipe); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index bd0fc2634dc..463e3047654 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -189,7 +189,12 @@ struct CompressFileHandle /* * Compression specification for this file handle. */ - bool path_is_pipe_command; + bool is_pipe; + + /* + * The command path or template (only used if is_pipe is true). + */ + char *command; /* * Private data to be used by the compressor. @@ -201,7 +206,7 @@ struct CompressFileHandle * Initialize a compress file handle with the requested compression. */ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); /* * Initialize a compress file stream. Infer the compression algorithm @@ -210,6 +215,6 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, const char *mode, - bool path_is_pipe_command); + bool is_pipe); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 2bc4c37c5db..79595556715 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -767,11 +767,11 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { LZ4State *state; - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for LZ4"); CFH->open_func = LZ4Stream_open; @@ -789,7 +789,7 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = state; } @@ -804,7 +804,7 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 490141ee8a1..2c235cf3a50 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -20,6 +20,6 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 4cf02843185..6698825df2f 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -14,6 +14,7 @@ #include "postgres_fe.h" #include <unistd.h> +#include "port.h" #include "compress_none.h" #include "pg_backup_utils.h" @@ -210,13 +211,40 @@ close_none(CompressFileHandle *CFH) if (fp) { - errno = 0; - if (CFH->path_is_pipe_command) + if (CFH->is_pipe) + { ret = pclose(fp); + if (ret != 0) + { + /* + * For pipe commands, pclose() returns the exit status of the + * child process. If the shell command itself fails (e.g. + * "command not found"), pclose() will return a non-zero exit + * status, but errno will likely remain 0 (Success). We use + * wait_result_to_str to decode the status and pg_fatal to + * prevent the caller from logging a generic and misleading + * "could not close file: Success" message. We also include + * the failing command string to help the user debug. + */ + char *reason = wait_result_to_str(ret); + + pg_fatal("pipe command failed: \"%s\": %s", + CFH->command ? CFH->command : "unknown", + reason); + } + } else + { ret = fclose(fp); - if (ret != 0) - pg_log_error("could not close file: %m"); + if (ret != 0) + pg_fatal("could not close file: %m"); + } + } + + if (CFH->command) + { + pg_free(CFH->command); + CFH->command = NULL; } return ret == 0; @@ -228,6 +256,23 @@ eof_none(CompressFileHandle *CFH) return feof((FILE *) CFH->private_data) != 0; } +static FILE * +open_handle_none(const char *path, const char *mode, bool is_pipe) +{ + if (is_pipe) + { + /* + * If the path is a pipe, we use popen(). Note that we do not track + * the child PID for cleanup during fatal errors. We intentionally + * rely on standard POSIX semantics: if pg_dump crashes, the OS will + * close our end of the pipe, sending EOF to the child process, which + * will then cleanly exit on its own. + */ + return popen(path, mode); + } + return fopen(path, mode); +} + static bool open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) { @@ -248,13 +293,13 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; + + if (CFH->is_pipe) + CFH->command = pg_strdup(path); } return true; @@ -266,16 +311,16 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) Assert(CFH->private_data == NULL); pg_log_debug("Opening %s, pipe is %s", - path, CFH->path_is_pipe_command ? "true" : "false"); + path, CFH->is_pipe ? "true" : "false"); - if (CFH->path_is_pipe_command) - CFH->private_data = popen(path, mode); - else - CFH->private_data = fopen(path, mode); + CFH->private_data = open_handle_none(path, mode, CFH->is_pipe); if (CFH->private_data == NULL) return false; + if (CFH->is_pipe) + CFH->command = pg_strdup(path); + return true; } @@ -286,7 +331,7 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -298,7 +343,8 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; + CFH->command = NULL; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index d898a2d411c..57943ceff7f 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -20,6 +20,6 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index e4830d35ec0..57c4ad16500 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -28,7 +28,7 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -576,9 +576,9 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command) + bool is_pipe) { - if (path_is_pipe_command) + if (is_pipe) pg_fatal("Pipe command not supported for Zstd"); CFH->open_func = Zstd_open; @@ -592,7 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; - CFH->path_is_pipe_command = path_is_pipe_command; + CFH->is_pipe = is_pipe; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1f23e7266bf..8b06657bc80 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -21,6 +21,6 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, - bool path_is_pipe_command); + bool is_pipe); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 549703af622..c1148a66635 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,7 +316,7 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool is_pipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, @@ -324,7 +324,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, DataDirSyncMethod sync_method, - bool FileSpecIsPipe); + bool is_pipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4b6bb7b8a14..bb14a83b80b 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -1744,7 +1744,19 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout) or fopen() (for a regular file), using pclose() + * on it is a bug that causes failures on BSD-based systems (like FreeBSD + * or macOS). + */ + CFH = InitCompressFileHandle(compression_spec, false); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2442,7 +2454,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; - AH->fSpecIsPipe = FileSpecIsPipe; + AH->is_pipe = FileSpecIsPipe; AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ @@ -2463,7 +2475,19 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); + + /* + * The output handle (usually stdout) should never be a pipe command + * managed by our popen logic, even if the archive itself is a pipe. Our + * pipe command implementation for directory mode is a template for the + * data files, not for this primary output stream. + * + * Furthermore, marking this as a pipe command would cause it to be closed + * with pclose() instead of fclose(). Since this handle is opened via + * fdopen() (for stdout), using pclose() on it is a bug that causes + * failures on BSD-based systems (like FreeBSD or macOS). + */ + CFH = InitCompressFileHandle(out_compress_spec, false); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index cdc12a54f5e..9555d44ae29 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,7 +301,7 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ - bool fSpecIsPipe; /* fSpec is a pipe command template requiring + bool is_pipe; /* fSpec is a pipe command template requiring * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 15ce45fb9e9..3a6f47d5483 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -158,7 +158,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - if (!AH->fSpecIsPipe) /* no checks for pipe */ + if (!AH->is_pipe) /* no checks for pipe */ { /* we accept an empty existing directory */ create_or_open_dir(ctx->directory); @@ -171,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->is_pipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -299,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -357,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->is_pipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -420,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->is_pipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -431,7 +431,6 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; - char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -440,20 +439,8 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) StartRestoreLO(AH, oid, AH->public.ropt->dropSchema); - /* - * XXX : Create a helper function for blob files naming common to - * _LoadLOs an _StartLO. - */ - if (AH->fSpecIsPipe) - { - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", lofname); - strcpy(path, pipe); - pfree(pipe); - } - else - { - snprintf(path, MAXPGPATH, "%s/%s", ctx->directory, lofname); - } + setFilePath(AH, path, lofname); + _PrintFileData(AH, path); EndRestoreLO(AH, oid); } @@ -564,7 +551,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + tocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -631,39 +618,27 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->is_pipe); /* - * XXX: We can probably simplify this code by using the mode 'w' for all - * cases. The current implementation is due to historical reason that the - * mode for the LOs TOC file has been "ab" from the start. That is - * something we can't do for pipe-command as popen only supports read and - * write. So here a different mode is used for pipes. + * We use 'w' (PG_BINARY_W) mode for the LOs TOC file in all cases. + * Historically, the mode for this file was "ab". However, append mode is + * entirely redundant due to how large objects are partitioned. * - * But in future we can evaluate using 'w' for everything.there is one - * ToCEntry There is only one ToCEntry per blob group. And it is written - * by @WriteDataChunksForToCEntry. This function calls _StartLOs once - * before the dumper function and and _EndLOs once after the dumper. And - * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only - * opened once and closed after all the entries are written. Therefore the - * mode can be made 'w' for all the cases. We tested changing the mode to - * PG_BINARY_W and the tests passed. But in case there are some missing - * scenarios, we have not made that change here. Instead for now only - * doing it for the pipe command. + * pg_dump splits large objects into chunks of up to 1000 blobs per + * archive entry. Each chunk receives a completely unique dumpId, and the + * TOC file is named using that ID (e.g., blobs_123.toc). Furthermore, + * WriteDataChunksForTocEntry ensures a strict sequential lifecycle for + * each entry: it calls _StartLOs (opens the file), then the dumper + * function (writes the chunk), and finally _EndLOs (closes the file). * - * Another alternative is to keep the 'ab' mode for regular files and use - * 'w' mode for pipe files but now also cache the pipe handle to keep it - * open till all the LOs in the dump group are done. This is not needed - * because of the same reason listed above that a file handle is only - * opened once. In short there are 3 solutions : 1. Change the mode for - * everything (preferred) 2. Change it only for pipe-command (current) 3. - * Change it for pipe-command and then cache those handles and close them - * in the end (not needed). + * Because a blobs_NNN.toc file is guaranteed to be unique and is only + * opened exactly once, written to sequentially, and then closed forever, + * there is no scenario where "ab" is required. This change to "w" is + * necessary because popen() for pipe-commands only supports "r" and "w". */ - if (AH->fSpecIsPipe) - mode = PG_BINARY_W; - else - mode = "ab"; + mode = PG_BINARY_W; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -678,22 +653,12 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; - char *pipe; char blob_name[MAXPGPATH]; - if (AH->fSpecIsPipe) - { - snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); - pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); - strcpy(fname, pipe); - pfree(pipe); - } - else - { - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); - } + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + setFilePath(AH, fname, blob_name); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->is_pipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -752,10 +717,23 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) dname = ctx->directory; - if (AH->fSpecIsPipe) + if (AH->is_pipe) { - pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + /* + * Unlike commands synthesized by the backend, this is a user-provided + * template running client-side. We perform literal substitution + * rather than using appendShellString() to avoid interfering with the + * user's intentional shell quoting (e.g., for Windows vs Unix + * differences). Since this is a client-side execution, there are no + * privilege escalation concerns. + */ + pipe = replace_percent_placeholders(dname, "pipe", "f", relativeFilename); + + if (strlen(pipe) >= MAXPGPATH) + pg_fatal("pipe command too long: \"%s\"", pipe); + strcpy(buf, pipe); + pfree(pipe); } else /* replace all ocurrences of %f in dname with @@ -809,23 +787,18 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); - pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { - if (AH->fSpecIsPipe) - pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) { - pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); - pg_log_error("filename: %s", fname); } if (stat(fname, &st) == 0) diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index b65f869208a..65a9ce222c8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,7 +419,8 @@ main(int argc, char **argv) { int c; const char *filename = NULL; - bool filename_is_pipe = false; + char *pipe_command = NULL; + bool is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -536,7 +537,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, - {"pipe-command", required_argument, NULL, 26}, + {"pipe", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -608,14 +609,9 @@ main(int argc, char **argv) break; case 'f': - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } filename = pg_strdup(optarg); - filename_is_pipe = false; /* it already is, setting again - * here just for clarity */ + is_pipe = false; /* it already is, setting again here just + * for clarity */ break; case 'F': @@ -809,13 +805,8 @@ main(int argc, char **argv) break; case 26: /* pipe command */ - if (filename != NULL) - { - pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); - exit_nicely(1); - } - filename = pg_strdup(optarg); - filename_is_pipe = true; + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -825,6 +816,10 @@ main(int argc, char **argv) } } + if (filename && pipe_command) + pg_fatal("options %s and %s cannot be used together", + "-f/--file", "--pipe"); + /* * Non-option argument specifies database name as long as it wasn't * already specified with -d / --dbname @@ -926,26 +921,23 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); - if (filename_is_pipe && archiveFormat != archDirectory) - { - pg_log_error_hint("Option --pipe-command is only supported with directory format."); - exit_nicely(1); - } + if (is_pipe && archiveFormat != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); - if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) - { - pg_log_error_hint("Option --pipe-command is not supported with any compression type."); - exit_nicely(1); - } + if (is_pipe && strcmp(compression_algorithm_str, "none") != 0) + pg_fatal("option --pipe is not supported with any compression type"); + + if (is_pipe && numWorkers > 1 && strstr(pipe_command, "%f") == NULL) + pg_log_warning("parallel jobs with --pipe usually require the \"%%f\" placeholder to avoid data corruption from multiple workers writing to the same file"); /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. If directory format is being used with pipe-command, - * no compression is done. + * done by default. If directory format is being used with pipe, no + * compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !filename_is_pipe && !user_compression_defined) + !is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -994,8 +986,8 @@ main(int argc, char **argv) pg_fatal("parallel backup only supported by the directory format"); /* Open the output file */ - fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); + fout = CreateArchive(is_pipe ? pipe_command : filename, archiveFormat, compression_spec, + dosync, archiveMode, setupDumpWorker, sync_method, is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); @@ -1327,6 +1319,8 @@ help(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -f, --file=FILENAME output file or directory name\n")); + printf(_(" --pipe=COMMAND execute command for each output file and\n" + " write data to it via pipe\n")); printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n" " plain text (default))\n")); printf(_(" -j, --jobs=NUM use this many parallel jobs to dump\n")); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 2d551365180..bf69a44fa23 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -298,6 +298,15 @@ main(int argc, char *argv[]) case 'F': format_name = pg_strdup(optarg); break; + + /* + * Note: support for --pipe is currently skipped for + * pg_dumpall due to the complexity of avoiding path + * collisions between multiple databases and coordinating + * nested directory structures. This could be considered as a + * future enhancement. + */ + case 'g': globals_only = true; break; diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index c657149d658..1c6d7b2f72b 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -60,11 +60,11 @@ static void usage(const char *progname); static void read_restore_filters(const char *filename, RestoreOptions *opts); static bool file_exists_in_directory(const char *dir, const char *filename); static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe); -static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe); + int numWorkers, bool append_data, bool is_pipe); +static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe); static int restore_all_databases(const char *inputFileSpec, - SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool filespec_is_pipe); + SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers, bool is_pipe); static int get_dbnames_list_to_restore(PGconn *conn, SimplePtrList *dbname_oid_list, SimpleStringList db_exclude_patterns); @@ -87,13 +87,14 @@ main(int argc, char **argv) RestoreOptions *opts; int c; int numWorkers = 1; - char *inputFileSpec; + char *inputFileSpec = NULL; + char *pipe_command = NULL; bool data_only = false; bool schema_only = false; int n_errors = 0; bool globals_only = false; SimpleStringList db_exclude_patterns = {NULL, NULL}; - bool filespec_is_pipe = false; + bool is_pipe = false; static int disable_triggers = 0; static int enable_row_security = 0; static int if_exists = 0; @@ -174,7 +175,7 @@ main(int argc, char **argv) {"filter", required_argument, NULL, 4}, {"restrict-key", required_argument, NULL, 6}, {"exclude-database", required_argument, NULL, 7}, - {"pipe-command", required_argument, NULL, 8}, + {"pipe", required_argument, NULL, 8}, {NULL, 0, NULL, 0} }; @@ -358,9 +359,9 @@ main(int argc, char **argv) simple_string_list_append(&db_exclude_patterns, optarg); break; - case 8: /* pipe-command */ - inputFileSpec = pg_strdup(optarg); - filespec_is_pipe = true; + case 8: /* pipe */ + pipe_command = pg_strdup(optarg); + is_pipe = true; break; default: @@ -371,25 +372,21 @@ main(int argc, char **argv) } /* - * Get file name from command line. Note that filename argument and - * pipe-command can't both be set. + * Get file name from command line. Note that filename argument and pipe + * can't both be set. */ if (optind < argc) { - if (filespec_is_pipe) - { - pg_log_error_hint("Only one of [filespec, --pipe-command] allowed"); - exit_nicely(1); - } + if (is_pipe) + pg_fatal("cannot specify both an input file and --pipe"); inputFileSpec = argv[optind++]; } /* - * Even if the file argument is not provided, if the pipe-command is - * specified, we need to use that as the file arg and not fallback to - * stdio. + * Even if the file argument is not provided, if the pipe is specified, we + * need to use that as the file arg and not fallback to stdio. */ - else if (!filespec_is_pipe) + else if (!is_pipe) { inputFileSpec = NULL; } @@ -539,10 +536,23 @@ main(int argc, char **argv) pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"", opts->formatName); } + else + opts->format = archUnknown; + + if (is_pipe && opts->format != archDirectory) + pg_fatal("option --pipe is only supported with directory format"); + + if (is_pipe && numWorkers > 1 && strstr(pipe_command, "%f") == NULL) + pg_log_warning("parallel jobs with --pipe usually require the \"%%f\" placeholder to avoid data corruption from multiple workers reading from the same file"); /* * If toc.glo file is present, then restore all the databases from * map.dat, but skip restoring those matching --exclude-database patterns. + * + * Note: support for --pipe is currently skipped for cluster archives + * (archives containing toc.glo) due to the added complexity of handling + * nested directory paths and multiple databases. This could be considered + * as a future enhancement. */ if (inputFileSpec != NULL && (file_exists_in_directory(inputFileSpec, "toc.glo"))) @@ -619,7 +629,7 @@ main(int argc, char **argv) snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec); if (!no_globals) - n_errors = restore_global_objects(global_path, tmpopts, filespec_is_pipe); + n_errors = restore_global_objects(global_path, tmpopts, is_pipe); else pg_log_info("skipping restore of global objects because %s was specified", "--no-globals"); @@ -630,8 +640,8 @@ main(int argc, char **argv) else { /* Now restore all the databases from map.dat */ - n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns, - opts, numWorkers, filespec_is_pipe); + n_errors = n_errors + restore_all_databases(is_pipe ? pipe_command : inputFileSpec, db_exclude_patterns, + opts, numWorkers, is_pipe); } /* Free db pattern list. */ @@ -651,7 +661,7 @@ main(int argc, char **argv) "-g/--globals-only"); /* Process if toc.glo file does not exist. */ - n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, filespec_is_pipe); + n_errors = restore_one_database(is_pipe ? pipe_command : inputFileSpec, opts, numWorkers, false, is_pipe); } /* Done, print a summary of ignored errors during restore. */ @@ -670,7 +680,7 @@ main(int argc, char **argv) * This restore all global objects. */ static int -restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool filespec_is_pipe) +restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool is_pipe) { Archive *AH; int nerror = 0; @@ -679,7 +689,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -716,12 +726,12 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts, bool fil */ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts, - int numWorkers, bool append_data, bool filespec_is_pipe) + int numWorkers, bool append_data, bool is_pipe) { Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format, filespec_is_pipe); + AH = OpenArchive(inputFileSpec, opts->format, is_pipe); SetArchiveOptions(AH, NULL, opts); @@ -777,6 +787,8 @@ usage(const char *progname) printf(_("\nGeneral options:\n")); printf(_(" -d, --dbname=NAME connect to database name\n")); printf(_(" -f, --file=FILENAME output file name (- for stdout)\n")); + printf(_(" --pipe=COMMAND execute command for each input file and\n" + " read data from it via pipe\n")); printf(_(" -F, --format=c|d|t backup file format (should be automatic)\n")); printf(_(" -l, --list print summarized TOC of the archive\n")); printf(_(" -v, --verbose verbose mode\n")); @@ -1170,7 +1182,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, static int restore_all_databases(const char *inputFileSpec, SimpleStringList db_exclude_patterns, RestoreOptions *opts, - int numWorkers, bool filespec_is_pipe) + int numWorkers, bool is_pipe) { SimplePtrList dbname_oid_list = {NULL, NULL}; int num_db_restore = 0; @@ -1334,7 +1346,7 @@ restore_all_databases(const char *inputFileSpec, } /* Restore the single database. */ - n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, filespec_is_pipe); + n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true, is_pipe); n_errors_total += n_errors; -- 2.54.0.1064.gd145956f57-goog [application/x-patch] v17-0004-Add-tests-for-pipe.patch (22.1K, ../../CAH5HC94-gFS63cnQbSXZ+hDvSUNV51U+KeRU-XiY-ngu_KtJ5A@mail.gmail.com/5-v17-0004-Add-tests-for-pipe.patch) download | inline diff: From b4773e4cf28dfa62605981134f62c2b72cddbabc Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Sat, 15 Feb 2025 04:29:17 +0000 Subject: [PATCH v17 4/5] Add tests for pipe * These tests include the invalid usages of --pipe-command with other flags. * Also test pg_dump and pg_restore with pipe command along with various other flags. --- src/bin/pg_dump/t/001_basic.pl | 72 ++++- src/bin/pg_dump/t/002_pg_dump.pl | 296 +++++++++++++++++++- src/bin/pg_dump/t/004_pg_dump_parallel.pl | 88 ++++++ src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 18 ++ 4 files changed, 464 insertions(+), 10 deletions(-) diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl index 687e842cde9..92d47e4fd93 100644 --- a/src/bin/pg_dump/t/001_basic.pl +++ b/src/bin/pg_dump/t/001_basic.pl @@ -74,6 +74,48 @@ command_fails_like( 'pg_dump: options --statistics-only and --no-statistics cannot be used together' ); +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-f', 'testdir', 'test'], + qr/\Qpg_dump: error: options -f\/--file and --pipe cannot be used together\E/, + 'pg_dump: options -f/--file and --pipe cannot be used together' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', 'gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=lz4', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '--compress=gzip', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fd', '--pipe="cat"', '-Z', '1', 'test'], + qr/\Qpg_dump: error: option --pipe is not supported with any compression type\E/, + 'pg_dump: option --pipe is not supported with any compression type' +); + +command_fails_like( + [ 'pg_dump', '-Fc', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_dump', '--format=tar', '--pipe="cat"', 'test'], + qr/\Qpg_dump: error: option --pipe is only supported with directory format\E/, + 'pg_dump: option --pipe is only supported with directory format' +); + command_fails_like( [ 'pg_dump', '-j2', '--include-foreign-data=xxx' ], qr/\Qpg_dump: error: option --include-foreign-data is not supported with parallel backup\E/, @@ -94,12 +136,38 @@ command_fails_like( command_fails_like( [ 'pg_restore', '-d', 'xxx', '-f', 'xxx' ], qr/\Qpg_restore: error: options -d\/--dbname and -f\/--file cannot be used together\E/, - 'pg_restore: options -d/--dbname and -f/--file cannot be used together'); + 'pg_restore: options -d/--dbname and -f/--file cannot be used together' +); + +command_fails_like( + [ 'pg_restore', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fd', '-f', '-', '--pipe="cat"', 'dumpdir' ], + qr/\Qpg_restore: error: cannot specify both an input file and --pipe\E/, + 'pg_restore: cannot specify both an input file and --pipe' +); + +command_fails_like( + [ 'pg_restore', '-Fc', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); + +command_fails_like( + [ 'pg_restore', '--format=tar', '-f', '-', '--pipe="cat"' ], + qr/\Qpg_restore: error: option --pipe is only supported with directory format\E/, + 'pg_restore: option --pipe is only supported with directory format' +); command_fails_like( [ 'pg_dump', '-c', '-a' ], qr/\Qpg_dump: error: options -c\/--clean and -a\/--data-only cannot be used together\E/, - 'pg_dump: options -c/--clean and -a/--data-only cannot be used together'); + 'pg_dump: options -c/--clean and -a/--data-only cannot be used together' +); command_fails_like( [ 'pg_dumpall', '-c', '-a' ], diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 3ee9fda50e4..2e37dcb4482 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -7,8 +7,10 @@ use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use File::Spec; my $tempdir = PostgreSQL::Test::Utils::tempdir; +$tempdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; ############################################################### # Definition of the pg_dump runs to make. @@ -46,6 +48,68 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir; my $supports_icu = ($ENV{with_icu} eq 'yes'); my $supports_gzip = check_pg_config("#define HAVE_LIBZ 1"); +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe 1"; + +# Check for external gzip program for pipe tests. +my $gzip_path = $ENV{GZIP_PROGRAM} || 'gzip'; +my $gzip_bin = "\"$gzip_path\""; +my $has_gzip_bin = + (system("$gzip_bin --version >" . File::Spec->devnull() . " 2>&1") == 0); + +# Pre-calculate complex pipe commands to keep the test definitions readable +# and ensure unified --pipe=... syntax for Windows stability. +# On Windows, we use space-wrapped quoting (e.g., " ... ") to protect +# internal quotes and shell operators from destructive cmd.exe /c stripping. +my $is_win = $PostgreSQL::Test::Utils::windows_os; + +my $raw_pipe_defaults_dir = "$perl_cat > \"$tempdir/defaults_dir_format/%f\""; +my $raw_pipe_defaults_res = "$perl_cat \"$tempdir/defaults_dir_format/%f\""; +my $raw_pipe_cross_dump = "$perl_cat > \"$tempdir/pipe_cross_dump/%f\""; +my $raw_pipe_cross_restore = ($supports_gzip && !$is_win) + ? "if [ -f \"$tempdir/pipe_cross_restore/%f.gz\" ]; then $gzip_bin -d -c \"$tempdir/pipe_cross_restore/%f.gz\"; else $perl_cat \"$tempdir/pipe_cross_restore/%f\"; fi" + : "$perl_cat \"$tempdir/pipe_cross_restore/%f\""; +my $raw_pipe_parallel_out = "$perl_cat > \"$tempdir/pipe_out_dir_parallel/%f\""; +my $raw_pipe_parallel_in = "$perl_cat \"$tempdir/pipe_out_dir_parallel/%f\""; +my $raw_pipe_parallel_8_out = "$perl_cat > \"$tempdir/pipe_out_dir_parallel_8/%f\""; +my $raw_pipe_parallel_8_in = "$perl_cat \"$tempdir/pipe_out_dir_parallel_8/%f\""; +my $raw_pipe_complex_out = "$gzip_bin | $perl_cat > \"$tempdir/pipe_out_dir_complex/%f.gz\""; +my $raw_pipe_complex_in = "$perl_cat \"$tempdir/pipe_out_dir_complex/%f.gz\" | $gzip_bin -d"; +my $raw_pipe_lo_out = "$perl_cat > \"$tempdir/pipe_out_dir_lo/%f\""; +my $raw_pipe_lo_in = "$perl_cat \"$tempdir/pipe_out_dir_lo/%f\""; +my $raw_pipe_schema_out = "$perl_cat > \"$tempdir/schema_only_pipe_dir/%f\""; +my $raw_pipe_schema_in = "$perl_cat \"$tempdir/schema_only_pipe_dir/%f\""; + +my $pipe_defaults_dir = $is_win ? "\" $raw_pipe_defaults_dir \"" : $raw_pipe_defaults_dir; +my $pipe_defaults_res = $is_win ? "\" $raw_pipe_defaults_res \"" : $raw_pipe_defaults_res; +my $pipe_cross_dump = $is_win ? "\" $raw_pipe_cross_dump \"" : $raw_pipe_cross_dump; +my $pipe_cross_restore = $is_win ? "\" $raw_pipe_cross_restore \"" : $raw_pipe_cross_restore; +my $pipe_parallel_out = $is_win ? "\" $raw_pipe_parallel_out \"" : $raw_pipe_parallel_out; +my $pipe_parallel_in = $is_win ? "\" $raw_pipe_parallel_in \"" : $raw_pipe_parallel_in; +my $pipe_parallel_8_out = $is_win ? "\" $raw_pipe_parallel_8_out \"" : $raw_pipe_parallel_8_out; +my $pipe_parallel_8_in = $is_win ? "\" $raw_pipe_parallel_8_in \"" : $raw_pipe_parallel_8_in; +my $pipe_complex_out = $is_win ? "\" $raw_pipe_complex_out \"" : $raw_pipe_complex_out; +my $pipe_complex_in = $is_win ? "\" $raw_pipe_complex_in \"" : $raw_pipe_complex_in; +my $pipe_lo_out = $is_win ? "\" $raw_pipe_lo_out \"" : $raw_pipe_lo_out; +my $pipe_lo_in = $is_win ? "\" $raw_pipe_lo_in \"" : $raw_pipe_lo_in; +my $pipe_schema_out = $is_win ? "\" $raw_pipe_schema_out \"" : $raw_pipe_schema_out; +my $pipe_schema_in = $is_win ? "\" $raw_pipe_schema_in \"" : $raw_pipe_schema_in; + +# Create output directories for pipe tests +mkdir "$tempdir/pipe_out_dir_parallel"; +mkdir "$tempdir/pipe_out_dir_parallel_8"; +mkdir "$tempdir/pipe_out_dir_complex"; +mkdir "$tempdir/pipe_out_dir_lo"; +mkdir "$tempdir/pipe_cross_dump"; +mkdir "$tempdir/pipe_cross_restore"; +mkdir "$tempdir/schema_only_pipe_dir"; + my %pgdump_runs = ( binary_upgrade => { dump_cmd => [ @@ -223,6 +287,139 @@ my %pgdump_runs = ( ], }, + defaults_dir_format_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_defaults_dir", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe.sql", + "--pipe=$pipe_defaults_res", + '--statistics', + ], + }, + + defaults_dir_format_pipe_dump_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_cross_dump", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_dump_only.sql", + '--statistics', + "$tempdir/pipe_cross_dump", + ], + }, + + defaults_dir_format_pipe_restore_only => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--file' => "$tempdir/pipe_cross_restore", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_dir_format_pipe_restore_only.sql", + "--pipe=$pipe_cross_restore", + '--statistics', + ], + }, + + defaults_parallel_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 2, + "--pipe=$pipe_parallel_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_pipe.sql", + "--pipe=$pipe_parallel_in", + '--statistics', + ], + }, + + defaults_parallel_8_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--jobs' => 8, + "--pipe=$pipe_parallel_8_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_parallel_8_pipe.sql", + "--pipe=$pipe_parallel_8_in", + '--statistics', + ], + }, + + defaults_complex_pipe => { + test_key => 'defaults', + skip_unless => \$has_gzip_bin, + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + "--pipe=$pipe_complex_out", + '--statistics', + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_complex_pipe.sql", + "--pipe=$pipe_complex_in", + '--statistics', + ], + }, + + defaults_lo_pipe => { + test_key => 'defaults', + dump_cmd => [ + 'pg_dump', + '--format' => 'directory', + '--statistics', + "--pipe=$pipe_lo_out", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/defaults_lo_pipe.sql", + '--statistics', + "--pipe=$pipe_lo_in", + ], + glob_patterns => [ + "$tempdir/pipe_out_dir_lo/toc.dat", + "$tempdir/pipe_out_dir_lo/blobs_*.toc", + ], + }, + # Do not use --no-sync to give test coverage for data sync. defaults_parallel => { test_key => 'defaults', @@ -527,6 +724,22 @@ my %pgdump_runs = ( 'postgres', ], }, + schema_only_pipe => { + test_key => 'schema_only', + dump_cmd => [ + 'pg_dump', '--no-sync', + '--format' => 'directory', + '--schema-only', + "--pipe=$pipe_schema_out", + 'postgres', + ], + restore_cmd => [ + 'pg_restore', + '--format' => 'directory', + '--file' => "$tempdir/schema_only_pipe.sql", + "--pipe=$pipe_schema_in", + ], + }, section_pre_data => { dump_cmd => [ 'pg_dump', '--no-sync', @@ -5212,25 +5425,24 @@ command_fails_like( ######################################### # Run all runs + foreach my $run (sort keys %pgdump_runs) { my $test_key = $run; - my $run_db = 'postgres'; + my $run_db = 'postgres'; $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, "$run: pg_dump runs"); if ($pgdump_runs{$run}->{glob_patterns}) { - my $glob_patterns = $pgdump_runs{$run}->{glob_patterns}; - foreach my $glob_pattern (@{$glob_patterns}) + foreach my $glob_pattern (@{ $pgdump_runs{$run}->{glob_patterns} }) { - my @glob_output = glob($glob_pattern); my $ok = 0; - # certainly found some files if glob() returned multiple matches - $ok = 1 if (scalar(@glob_output) > 1); - # if just one match, we need to check if it's real - $ok = 1 if (scalar(@glob_output) == 1 && -f $glob_output[0]); + foreach my $file (glob("$glob_pattern")) + { + $ok = 1 if -e $file; + } is($ok, 1, "$run: glob check for $glob_pattern"); } } @@ -5334,6 +5546,74 @@ foreach my $run (sort keys %pgdump_runs) } } +######################################### +# Test error reporting for a failing pipe command. +# We use a perl one-liner that exits with 1 after processing input. +# This ensures we test the error handling in pclose() at the end of the dump, +# verifying that the child's exit status is correctly captured and reported. +my $failing_perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe \"END { exit 1 }\""; + +# Verify that pg_dump's error message includes the exact failing command, +# including the specific "-pe "END { exit 1 }"" payload. +$node->command_fails_like( + [ 'pg_dump', '-Fd', $is_win ? "--pipe=\"$failing_perl_cat > \\\"%f\\\"\"" : "--pipe=$failing_perl_cat > \"%f\"", 'postgres' ], + qr/pipe command failed: ".*perl.*-Mopen=IO,:raw.*-pe.*END \{ exit 1 \}.*": /, + 'pg_dump pipe command error reporting includes full command string' +); + + +# Verify that pg_restore's error message also includes the exact failing command +# and its arguments, including the target file path. +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', $is_win ? "--pipe=\" $failing_perl_cat \\\"$tempdir/pipe_cross_dump/%f\\\" \"" : "--pipe=$failing_perl_cat \"$tempdir/pipe_cross_dump/%f\"", ], + qr/pipe command failed: ".*perl.*-Mopen=IO,:raw.*-pe.*END \{ exit 1 \}.*pipe_cross_dump.*": /, + 'pg_restore pipe command error reporting includes full command string' +); + +# Targeted Edge Case Tests +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=/nonexistent/binary', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|Permission denied/, + 'pg_dump early pipe command execution failure' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '--pipe=no_such_command_at_all', 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|not found|not recognized/, + 'pg_dump command not found error reporting' +); + +$node->command_fails_like( + [ 'pg_dump', '-Fd', '-f', '-', $is_win ? "--pipe=\" $perl_cat > \\\"%f\\\" \"" : "--pipe=$perl_cat > \"%f\"", 'postgres' ], + qr/options -f\/--file and --pipe cannot be used together/, + 'pg_dump options -f/--file and --pipe conflict check' +); + +# Test that pg_restore rejects a positional argument when --pipe is used. +# We create a dummy cluster archive (containing toc.glo) to verify that +# even in cluster mode, the mutual exclusivity holds. +mkdir "$tempdir/dummy_cluster_archive"; +open my $fh, '>', "$tempdir/dummy_cluster_archive/toc.glo"; +close $fh; + +$node->command_fails_like( + [ 'pg_restore', '-Fd', '-l', $is_win ? "--pipe=\" $perl_cat \\\"%f\\\" \"" : "--pipe=$perl_cat \"%f\"", "$tempdir/dummy_cluster_archive" ], + qr/cannot specify both an input file and --pipe/, + 'pg_restore --pipe rejects positional argument even for cluster archive' +); + +# Test that pg_dump --pipe bypasses local directory existence check. +# We use a pipe command that writes to a subdirectory that hasn't been created. +# The dump itself will fail when the pipe command tries to write to the +# non-existent directory, but the error should come from the pipe command/write +# failure, not from pg_dump's directory initialization. +my $remote_dir = "$tempdir/non_existent_remote_dir"; +$node->command_fails_like( + [ 'pg_dump', '-Fd', $is_win ? "--pipe=\" $perl_cat > \\\"$remote_dir/%f\\\" \"" : "--pipe=$perl_cat > \"$remote_dir/%f\"", 'postgres' ], + qr/could not write to file: (?:Broken pipe|The pipe has been ended)|pipe command failed/, + 'pg_dump --pipe bypasses local directory existence check' +); + ######################################### # Stop the database instance, which will be removed at the end of the tests. diff --git a/src/bin/pg_dump/t/004_pg_dump_parallel.pl b/src/bin/pg_dump/t/004_pg_dump_parallel.pl index 738f34b1c1b..57edf5697c0 100644 --- a/src/bin/pg_dump/t/004_pg_dump_parallel.pl +++ b/src/bin/pg_dump/t/004_pg_dump_parallel.pl @@ -8,19 +8,33 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +# On Windows, perl opens file handles in text mode by default, which corrupts +# binary archive data by translating newlines and interpreting EOF characters. +# We use -Mopen=IO,:raw to force raw binary mode. We use -pe 1 instead of +# -pe '' to avoid shell quoting issues with empty strings on Windows cmd.exe. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "\"$perlbin\" -Mopen=IO,:raw -pe 1"; + my $dbname1 = 'regression_src'; my $dbname2 = 'regression_dest1'; my $dbname3 = 'regression_dest2'; +my $dbname4 = 'regression_dest3'; +my $dbname5 = 'regression_dest_warning'; my $node = PostgreSQL::Test::Cluster->new('main'); $node->init; $node->start; my $backupdir = $node->backup_dir; +$backupdir =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; $node->run_log([ 'createdb', $dbname1 ]); $node->run_log([ 'createdb', $dbname2 ]); $node->run_log([ 'createdb', $dbname3 ]); +$node->run_log([ 'createdb', $dbname4 ]); +$node->run_log([ 'createdb', $dbname5 ]); $node->safe_psql( $dbname1, @@ -87,4 +101,78 @@ $node->command_ok( ], 'parallel restore as inserts'); +mkdir "$backupdir/dump_pipe"; + +# Pre-calculate pipe commands for readability and unified syntax. +# Use space-wrapped quoting only on Windows to protect shell operators. +my $is_win = $PostgreSQL::Test::Utils::windows_os; +my $raw_pipe_dump = "$perl_cat > \"$backupdir/dump_pipe/%f\""; +my $raw_pipe_restore = "$perl_cat \"$backupdir/dump_pipe/%f\""; + +my $pipe_dump = $is_win ? "\" $raw_pipe_dump \"" : $raw_pipe_dump; +my $pipe_restore = $is_win ? "\" $raw_pipe_restore \"" : $raw_pipe_restore; + +$node->command_ok( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + "--pipe=$pipe_dump", + $node->connstr($dbname1), + ], + 'parallel dump with pipe'); + +$node->command_ok( + [ + 'pg_restore', '--verbose', + '--dbname' => $node->connstr($dbname4), + '--format' => 'directory', + '--jobs' => 3, + "--pipe=$pipe_restore", + ], + 'parallel restore with pipe'); + +# Test warning when parallel jobs are used without %f in pipe command. +# Use a simple command that doesn't use %f. +my $pipe_no_f = $is_win ? "\" $perl_cat > \\\"$backupdir/no_f.out\\\" \"" : "$perl_cat > \"$backupdir/no_f.out\""; +$node->command_checks_all( + [ + 'pg_dump', + '--format' => 'directory', + '--no-sync', + '--jobs' => 2, + "--pipe=$pipe_no_f", + $node->connstr($dbname1), + ], + 0, + [], + [qr/parallel jobs with --pipe usually require the "%f" placeholder/], + 'parallel dump without %f issues warning'); + +# Test warning in pg_restore when parallel jobs are used without %f. +# We restore into a clean database ($dbname5) using --schema-only to prevent +# pg_restore from attempting to read table data files. We configure the pipe +# command to cat the TOC file directly (lacking %f), which allows pg_restore +# to successfully read and restore the metadata and exit with 0, while still +# emitting the warning on stderr. +my $pipe_restore_no_f_success = $is_win + ? "\" $perl_cat \\\"$backupdir/dump_pipe/toc.dat\\\" \"" + : "$perl_cat \"$backupdir/dump_pipe/toc.dat\""; + +$node->command_checks_all( + [ + 'pg_restore', + '--format' => 'directory', + '--jobs' => 2, + '--schema-only', + "--pipe=$pipe_restore_no_f_success", + '--dbname' => $node->connstr($dbname5), + ], + 0, + [], + [qr/parallel jobs with --pipe usually require the "%f" placeholder/], + 'parallel restore without %f issues warning and succeeds'); + done_testing(); + diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl index cecf0442088..f4b1afd8ef6 100644 --- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl +++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl @@ -8,6 +8,11 @@ use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +# Use perl one-liner as a portable 'cat' replacement for Windows compatibility. +my $perlbin = $^X; +$perlbin =~ s!\\!/!g if $PostgreSQL::Test::Utils::windows_os; +my $perl_cat = "$perlbin -pe ''"; + my $tempdir = PostgreSQL::Test::Utils::tempdir; my $inputfile; @@ -98,6 +103,19 @@ command_ok( ], "filter file without patterns"); +mkdir "$backupdir/dump_pipe_filter"; + +command_ok( + [ + 'pg_dump', + '--port' => $port, + '--format' => 'directory', + '--pipe' => "$perl_cat > $backupdir/dump_pipe_filter/%f", + '--filter' => "$tempdir/inputfile.txt", + 'postgres' + ], + "filter file without patterns with pipe"); + my $dump = slurp_file($plainfile); like($dump, qr/^CREATE TABLE public\.table_one/m, "table one dumped"); -- 2.54.0.1064.gd145956f57-goog [application/x-patch] v17-0001-Add-pipe-command-support-for-directory-mode-of-p.patch (31.6K, ../../CAH5HC94-gFS63cnQbSXZ+hDvSUNV51U+KeRU-XiY-ngu_KtJ5A@mail.gmail.com/6-v17-0001-Add-pipe-command-support-for-directory-mode-of-p.patch) download | inline diff: From 3c9eff08513054c0cf0db278f142c30fd999667a Mon Sep 17 00:00:00 2001 From: Nitin Motiani <[email protected]> Date: Tue, 11 Feb 2025 08:31:02 +0000 Subject: [PATCH v17 1/5] Add pipe-command support for directory mode of pg_dump * We add a new flag --pipe-command which can be used in directory mode. This allows us to support multiple streams and we can do post processing like compression, filtering etc. This is currently not possible with directory-archive format. * Currently this flag is only supported with compression none and archive format directory. * This flag can't be used with the flag --file. Only one of the two flags can be used at a time. * We reuse the filename field for the --pipe-command also. And add a bool to specify that the field will be used as a pipe command. * Most of the code remains as it is. The core change is that in case of --pipe-command, instead of fopen we do popen. * The user would need a way to store the post-processing output in files. For that we support the same format as the directory mode currently does with the flag --file. We allow the user to add a format specifier %f to the --pipe-command. And for each stream, the format specifier is replaced with the corresponding file name. This file name is the same as it would have been if the flag --file had been used. * To enable the above, there are a few places in the code where we change the file name creation logic. Currently the file name is appended to the directory name which is provided with --file flag. In case of --pipe-command, we instead replace %f with the file name. This change is made for the common use case and separately for blob files. * There is an open question on what mode to use in case of large objects TOC file. Currently the code uses "ab" but that won't work for popen. We have proposed a few options in the comments regarding this. For the time being we are using mode PG_BINARY_W for the pipe use case. --- src/bin/pg_dump/compress_gzip.c | 9 ++- src/bin/pg_dump/compress_gzip.h | 3 +- src/bin/pg_dump/compress_io.c | 26 +++++-- src/bin/pg_dump/compress_io.h | 11 ++- src/bin/pg_dump/compress_lz4.c | 11 ++- src/bin/pg_dump/compress_lz4.h | 3 +- src/bin/pg_dump/compress_none.c | 25 ++++++- src/bin/pg_dump/compress_none.h | 3 +- src/bin/pg_dump/compress_zstd.c | 10 ++- src/bin/pg_dump/compress_zstd.h | 3 +- src/bin/pg_dump/pg_backup.h | 5 +- src/bin/pg_dump/pg_backup_archiver.c | 22 +++--- src/bin/pg_dump/pg_backup_archiver.h | 2 + src/bin/pg_dump/pg_backup_directory.c | 103 +++++++++++++++++++++----- src/bin/pg_dump/pg_dump.c | 37 ++++++++- src/bin/pg_dump/pg_dumpall.c | 2 +- src/bin/pg_dump/pg_restore.c | 6 +- 17 files changed, 222 insertions(+), 59 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 60c553ba25a..0ce15847d9a 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -429,8 +429,12 @@ Gzip_open_write(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("cPipe command not supported for Gzip"); + CFH->open_func = Gzip_open; CFH->open_write_func = Gzip_open_write; CFH->read_func = Gzip_read; @@ -455,7 +459,8 @@ InitCompressorGzip(CompressorState *cs, void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "gzip"); } diff --git a/src/bin/pg_dump/compress_gzip.h b/src/bin/pg_dump/compress_gzip.h index af1a2a3445e..f77c5c86c56 100644 --- a/src/bin/pg_dump/compress_gzip.h +++ b/src/bin/pg_dump/compress_gzip.h @@ -19,6 +19,7 @@ extern void InitCompressorGzip(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleGzip(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_GZIP_H_ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 52652b0d979..bc521dd274b 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -191,20 +191,29 @@ free_keep_errno(void *p) * Initialize a compress file handle for the specified compression algorithm. */ CompressFileHandle * -InitCompressFileHandle(const pg_compress_specification compression_spec) +InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CompressFileHandle *CFH; CFH = pg_malloc0_object(CompressFileHandle); - if (compression_spec.algorithm == PG_COMPRESSION_NONE) - InitCompressFileHandleNone(CFH, compression_spec); + /* + * Always set to non-compressed when path_is_pipe_command assuming that + * external compressor as part of pipe is more efficient. Can review in + * the future. + */ + if (path_is_pipe_command) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); + + else if (compression_spec.algorithm == PG_COMPRESSION_NONE) + InitCompressFileHandleNone(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_GZIP) - InitCompressFileHandleGzip(CFH, compression_spec); + InitCompressFileHandleGzip(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_LZ4) - InitCompressFileHandleLZ4(CFH, compression_spec); + InitCompressFileHandleLZ4(CFH, compression_spec, path_is_pipe_command); else if (compression_spec.algorithm == PG_COMPRESSION_ZSTD) - InitCompressFileHandleZstd(CFH, compression_spec); + InitCompressFileHandleZstd(CFH, compression_spec, path_is_pipe_command); return CFH; } @@ -237,7 +246,8 @@ check_compressed_file(const char *path, char **fname, char *ext) * On failure, return NULL with an error code in errno. */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, + bool path_is_pipe_command) { CompressFileHandle *CFH = NULL; struct stat st; @@ -268,7 +278,7 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) compression_spec.algorithm = PG_COMPRESSION_ZSTD; } - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, path_is_pipe_command); errno = 0; if (!CFH->open_func(fname, -1, mode, CFH)) { diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index ed7b14f0963..bd0fc2634dc 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -186,6 +186,11 @@ struct CompressFileHandle */ pg_compress_specification compression_spec; + /* + * Compression specification for this file handle. + */ + bool path_is_pipe_command; + /* * Private data to be used by the compressor. */ @@ -195,7 +200,8 @@ struct CompressFileHandle /* * Initialize a compress file handle with the requested compression. */ -extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec); +extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specification compression_spec, + bool path_is_pipe_command); /* * Initialize a compress file stream. Infer the compression algorithm @@ -203,6 +209,7 @@ extern CompressFileHandle *InitCompressFileHandle(const pg_compress_specificatio * suffixes in 'path'. */ extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + bool path_is_pipe_command); extern bool EndCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index 0a7872116e7..2bc4c37c5db 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -766,10 +766,14 @@ LZ4Stream_open_write(const char *path, const char *mode, CompressFileHandle *CFH */ void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { LZ4State *state; + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for LZ4"); + CFH->open_func = LZ4Stream_open; CFH->open_write_func = LZ4Stream_open_write; CFH->read_func = LZ4Stream_read; @@ -785,6 +789,8 @@ InitCompressFileHandleLZ4(CompressFileHandle *CFH, if (CFH->compression_spec.level >= 0) state->prefs.compressionLevel = CFH->compression_spec.level; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = state; } #else /* USE_LZ4 */ @@ -797,7 +803,8 @@ InitCompressorLZ4(CompressorState *cs, void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 7360a469fc0..490141ee8a1 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -19,6 +19,7 @@ extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleLZ4(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/compress_none.c b/src/bin/pg_dump/compress_none.c index 743e2ce94b5..4cf02843185 100644 --- a/src/bin/pg_dump/compress_none.c +++ b/src/bin/pg_dump/compress_none.c @@ -211,7 +211,10 @@ close_none(CompressFileHandle *CFH) if (fp) { errno = 0; - ret = fclose(fp); + if (CFH->path_is_pipe_command) + ret = pclose(fp); + else + ret = fclose(fp); if (ret != 0) pg_log_error("could not close file: %m"); } @@ -245,7 +248,11 @@ open_none(const char *path, int fd, const char *mode, CompressFileHandle *CFH) } else { - CFH->private_data = fopen(path, mode); + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; } @@ -258,7 +265,14 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) { Assert(CFH->private_data == NULL); - CFH->private_data = fopen(path, mode); + pg_log_debug("Opening %s, pipe is %s", + path, CFH->path_is_pipe_command ? "true" : "false"); + + if (CFH->path_is_pipe_command) + CFH->private_data = popen(path, mode); + else + CFH->private_data = fopen(path, mode); + if (CFH->private_data == NULL) return false; @@ -271,7 +285,8 @@ open_write_none(const char *path, const char *mode, CompressFileHandle *CFH) void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { CFH->open_func = open_none; CFH->open_write_func = open_write_none; @@ -283,5 +298,7 @@ InitCompressFileHandleNone(CompressFileHandle *CFH, CFH->eof_func = eof_none; CFH->get_error_func = get_error_none; + CFH->path_is_pipe_command = path_is_pipe_command; + CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_none.h b/src/bin/pg_dump/compress_none.h index 5134f012ee9..d898a2d411c 100644 --- a/src/bin/pg_dump/compress_none.h +++ b/src/bin/pg_dump/compress_none.h @@ -19,6 +19,7 @@ extern void InitCompressorNone(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleNone(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* _COMPRESS_NONE_H_ */ diff --git a/src/bin/pg_dump/compress_zstd.c b/src/bin/pg_dump/compress_zstd.c index 68f1d815917..e4830d35ec0 100644 --- a/src/bin/pg_dump/compress_zstd.c +++ b/src/bin/pg_dump/compress_zstd.c @@ -27,7 +27,8 @@ InitCompressorZstd(CompressorState *cs, const pg_compress_specification compress } void -InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressFileHandleZstd(CompressFileHandle *CFH, const pg_compress_specification compression_spec, + bool path_is_pipe_command) { pg_fatal("this build does not support compression with %s", "ZSTD"); } @@ -574,8 +575,12 @@ Zstd_get_error(CompressFileHandle *CFH) void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec) + const pg_compress_specification compression_spec, + bool path_is_pipe_command) { + if (path_is_pipe_command) + pg_fatal("Pipe command not supported for Zstd"); + CFH->open_func = Zstd_open; CFH->open_write_func = Zstd_open_write; CFH->read_func = Zstd_read; @@ -587,6 +592,7 @@ InitCompressFileHandleZstd(CompressFileHandle *CFH, CFH->get_error_func = Zstd_get_error; CFH->compression_spec = compression_spec; + CFH->path_is_pipe_command = path_is_pipe_command; CFH->private_data = NULL; } diff --git a/src/bin/pg_dump/compress_zstd.h b/src/bin/pg_dump/compress_zstd.h index 1222d7107d9..1f23e7266bf 100644 --- a/src/bin/pg_dump/compress_zstd.h +++ b/src/bin/pg_dump/compress_zstd.h @@ -20,6 +20,7 @@ extern void InitCompressorZstd(CompressorState *cs, const pg_compress_specification compression_spec); extern void InitCompressFileHandleZstd(CompressFileHandle *CFH, - const pg_compress_specification compression_spec); + const pg_compress_specification compression_spec, + bool path_is_pipe_command); #endif /* COMPRESS_ZSTD_H */ diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 28e7ff6fa16..549703af622 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -316,14 +316,15 @@ extern void ProcessArchiveRestoreOptions(Archive *AHX); extern void RestoreArchive(Archive *AHX, bool append_data); /* Open an existing archive */ -extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt); +extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe); /* Create a new archive */ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, + bool FileSpecIsPipe); /* The --list option */ extern void PrintTOCSummary(Archive *AHX); diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 2fd773ad84f..4b6bb7b8a14 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -57,7 +57,7 @@ static ArchiveHandle *_allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupWorkerPtr, - DataDirSyncMethod sync_method); + DataDirSyncMethod sync_method, bool FileSpecIsPipe); static void _getObjectDescription(PQExpBuffer buf, const TocEntry *te); static void _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx); static void _doSetFixedOutputState(ArchiveHandle *AH); @@ -233,11 +233,12 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, SetupWorkerPtrType setupDumpWorker, - DataDirSyncMethod sync_method) + DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH = _allocAH(FileSpec, fmt, compression_spec, - dosync, mode, setupDumpWorker, sync_method); + dosync, mode, setupDumpWorker, sync_method, FileSpecIsPipe); return (Archive *) AH; } @@ -245,7 +246,7 @@ CreateArchive(const char *FileSpec, const ArchiveFormat fmt, /* Open an existing archive */ /* Public */ Archive * -OpenArchive(const char *FileSpec, const ArchiveFormat fmt) +OpenArchive(const char *FileSpec, const ArchiveFormat fmt, bool FileSpecIsPipe) { ArchiveHandle *AH; pg_compress_specification compression_spec = {0}; @@ -253,7 +254,7 @@ OpenArchive(const char *FileSpec, const ArchiveFormat fmt) compression_spec.algorithm = PG_COMPRESSION_NONE; AH = _allocAH(FileSpec, fmt, compression_spec, true, archModeRead, setupRestoreWorker, - DATA_DIR_SYNC_METHOD_FSYNC); + DATA_DIR_SYNC_METHOD_FSYNC, FileSpecIsPipe); return (Archive *) AH; } @@ -1743,7 +1744,7 @@ SetOutput(ArchiveHandle *AH, const char *filename, else mode = PG_BINARY_W; - CFH = InitCompressFileHandle(compression_spec); + CFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!CFH->open_func(filename, fn, mode, CFH)) { @@ -2399,7 +2400,8 @@ static ArchiveHandle * _allocAH(const char *FileSpec, const ArchiveFormat fmt, const pg_compress_specification compression_spec, bool dosync, ArchiveMode mode, - SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method) + SetupWorkerPtrType setupWorkerPtr, DataDirSyncMethod sync_method, + bool FileSpecIsPipe) { ArchiveHandle *AH; CompressFileHandle *CFH; @@ -2440,6 +2442,8 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, else AH->fSpec = NULL; + AH->fSpecIsPipe = FileSpecIsPipe; + AH->currUser = NULL; /* unknown */ AH->currSchema = NULL; /* ditto */ AH->currTablespace = NULL; /* ditto */ @@ -2452,14 +2456,14 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt, AH->mode = mode; AH->compression_spec = compression_spec; - AH->dosync = dosync; + AH->dosync = FileSpecIsPipe ? false : dosync; AH->sync_method = sync_method; memset(&(AH->sqlparse), 0, sizeof(AH->sqlparse)); /* Open stdout with no compression for AH output handle */ out_compress_spec.algorithm = PG_COMPRESSION_NONE; - CFH = InitCompressFileHandle(out_compress_spec); + CFH = InitCompressFileHandle(out_compress_spec, AH->fSpecIsPipe); if (!CFH->open_func(NULL, fileno(stdout), PG_BINARY_A, CFH)) pg_fatal("could not open stdout for appending: %m"); AH->OF = CFH; diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index 1218bf6a6a1..cdc12a54f5e 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -301,6 +301,8 @@ struct _archiveHandle int loCount; /* # of LOs restored */ char *fSpec; /* Archive File Spec */ + bool fSpecIsPipe; /* fSpec is a pipe command template requiring + * replacing %f with file name */ FILE *FH; /* General purpose file handle */ void *OF; /* Output file */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 562868cd2ad..49a7ab91050 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -39,7 +39,8 @@ #include <dirent.h> #include <sys/stat.h> -#include "common/file_utils.h" +/* #include "common/file_utils.h" */ +#include "common/percentrepl.h" #include "compress_io.h" #include "dumputils.h" #include "parallel.h" @@ -157,8 +158,11 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) if (AH->mode == archModeWrite) { - /* we accept an empty existing directory */ - create_or_open_dir(ctx->directory); + if (!AH->fSpecIsPipe) /* no checks for pipe */ + { + /* we accept an empty existing directory */ + create_or_open_dir(ctx->directory); + } } else { /* Read Mode */ @@ -167,7 +171,7 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, AH->fSpecIsPipe); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -295,7 +299,7 @@ _StartData(ArchiveHandle *AH, TocEntry *te) setFilePath(AH, fname, tctx->filename); - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); @@ -353,7 +357,7 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, AH->fSpecIsPipe); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -416,7 +420,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) else setFilePath(AH, tocfname, tctx->filename); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, AH->fSpecIsPipe); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -427,6 +431,7 @@ _LoadLOs(ArchiveHandle *AH, TocEntry *te) { char lofname[MAXPGPATH + 1]; char path[MAXPGPATH]; + char *pipe; /* Can't overflow because line and lofname are the same length */ if (sscanf(line, "%u %" CppAsString2(MAXPGPATH) "s\n", &oid, lofname) != 2) @@ -545,7 +550,7 @@ _CloseArchive(ArchiveHandle *AH) /* The TOC is always created uncompressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - tocFH = InitCompressFileHandle(compression_spec); + tocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); if (!tocFH->open_write_func(fname, PG_BINARY_W, tocFH)) pg_fatal("could not open output file \"%s\": %m", fname); ctx->dataFH = tocFH; @@ -606,13 +611,46 @@ _StartLOs(ArchiveHandle *AH, TocEntry *te) lclTocEntry *tctx = (lclTocEntry *) te->formatData; pg_compress_specification compression_spec = {0}; char fname[MAXPGPATH]; + const char *mode; setFilePath(AH, fname, tctx->filename); /* The LO TOC file is never compressed */ compression_spec.algorithm = PG_COMPRESSION_NONE; - ctx->LOsTocFH = InitCompressFileHandle(compression_spec); - if (!ctx->LOsTocFH->open_write_func(fname, "ab", ctx->LOsTocFH)) + ctx->LOsTocFH = InitCompressFileHandle(compression_spec, AH->fSpecIsPipe); + + /* + * XXX: We can probably simplify this code by using the mode 'w' for all + * cases. The current implementation is due to historical reason that the + * mode for the LOs TOC file has been "ab" from the start. That is + * something we can't do for pipe-command as popen only supports read and + * write. So here a different mode is used for pipes. + * + * But in future we can evaluate using 'w' for everything.there is one + * ToCEntry There is only one ToCEntry per blob group. And it is written + * by @WriteDataChunksForToCEntry. This function calls _StartLOs once + * before the dumper function and and _EndLOs once after the dumper. And + * the dumper dumps all the LOs in the group. So a blob_NNN.toc is only + * opened once and closed after all the entries are written. Therefore the + * mode can be made 'w' for all the cases. We tested changing the mode to + * PG_BINARY_W and the tests passed. But in case there are some missing + * scenarios, we have not made that change here. Instead for now only + * doing it for the pipe command. + * + * Another alternative is to keep the 'ab' mode for regular files and use + * 'w' mode for pipe files but now also cache the pipe handle to keep it + * open till all the LOs in the dump group are done. This is not needed + * because of the same reason listed above that a file handle is only + * opened once. In short there are 3 solutions : 1. Change the mode for + * everything (preferred) 2. Change it only for pipe-command (current) 3. + * Change it for pipe-command and then cache those handles and close them + * in the end (not needed). + */ + if (AH->fSpecIsPipe) + mode = PG_BINARY_W; + else + mode = "ab"; + if (!ctx->LOsTocFH->open_write_func(fname, mode, ctx->LOsTocFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -626,10 +664,22 @@ _StartLO(ArchiveHandle *AH, TocEntry *te, Oid oid) { lclContext *ctx = (lclContext *) AH->formatData; char fname[MAXPGPATH]; + char *pipe; + char blob_name[MAXPGPATH]; - snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + if (AH->fSpecIsPipe) + { + snprintf(blob_name, MAXPGPATH, "blob_%u.dat", oid); + pipe = replace_percent_placeholders(ctx->directory, "pipe-command", "f", blob_name); + strcpy(fname, pipe); + pfree(pipe); + } + else + { + snprintf(fname, MAXPGPATH, "%s/blob_%u.dat", ctx->directory, oid); + } - ctx->dataFH = InitCompressFileHandle(AH->compression_spec); + ctx->dataFH = InitCompressFileHandle(AH->compression_spec, AH->fSpecIsPipe); if (!ctx->dataFH->open_write_func(fname, PG_BINARY_W, ctx->dataFH)) pg_fatal("could not open output file \"%s\": %m", fname); } @@ -683,15 +733,27 @@ setFilePath(ArchiveHandle *AH, char *buf, const char *relativeFilename) { lclContext *ctx = (lclContext *) AH->formatData; char *dname; + char *pipe; dname = ctx->directory; - if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) - pg_fatal("file name too long: \"%s\"", dname); - strcpy(buf, dname); - strcat(buf, "/"); - strcat(buf, relativeFilename); + if (AH->fSpecIsPipe) + { + pipe = replace_percent_placeholders(dname, "pipe-command", "f", relativeFilename); + strcpy(buf, pipe); + pfree(pipe); + } + else /* replace all ocurrences of %f in dname with + * relativeFilename */ + { + if (strlen(dname) + 1 + strlen(relativeFilename) + 1 > MAXPGPATH) + pg_fatal("file name too long: \"%s\"", dname); + + strcpy(buf, dname); + strcat(buf, "/"); + strcat(buf, relativeFilename); + } } /* @@ -733,17 +795,24 @@ _PrepParallelRestore(ArchiveHandle *AH) * only need an approximate indicator of that. */ setFilePath(AH, fname, tctx->filename); + pg_log_error("filename: %s", fname); if (stat(fname, &st) == 0) te->dataLength = st.st_size; else if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE) { + if (AH->fSpecIsPipe) + pg_log_error("pipe and compressed"); if (AH->compression_spec.algorithm == PG_COMPRESSION_GZIP) strlcat(fname, ".gz", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_LZ4) strlcat(fname, ".lz4", sizeof(fname)); else if (AH->compression_spec.algorithm == PG_COMPRESSION_ZSTD) + { + pg_log_error("filename: %s", fname); strlcat(fname, ".zst", sizeof(fname)); + pg_log_error("filename: %s", fname); + } if (stat(fname, &st) == 0) te->dataLength = st.st_size; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index a0f7f8e2168..b65f869208a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -419,6 +419,7 @@ main(int argc, char **argv) { int c; const char *filename = NULL; + bool filename_is_pipe = false; const char *format = "p"; TableInfo *tblinfo; int numTables; @@ -535,6 +536,7 @@ main(int argc, char **argv) {"exclude-extension", required_argument, NULL, 17}, {"sequence-data", no_argument, &dopt.sequence_data, 1}, {"restrict-key", required_argument, NULL, 25}, + {"pipe-command", required_argument, NULL, 26}, {NULL, 0, NULL, 0} }; @@ -606,7 +608,14 @@ main(int argc, char **argv) break; case 'f': + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } filename = pg_strdup(optarg); + filename_is_pipe = false; /* it already is, setting again + * here just for clarity */ break; case 'F': @@ -799,6 +808,16 @@ main(int argc, char **argv) dopt.restrict_key = pg_strdup(optarg); break; + case 26: /* pipe command */ + if (filename != NULL) + { + pg_log_error_hint("Only one of [--file, --pipe-command] allowed"); + exit_nicely(1); + } + filename = pg_strdup(optarg); + filename_is_pipe = true; + break; + default: /* getopt_long already emitted a complaint */ pg_log_error_hint("Try \"%s --help\" for more information.", progname); @@ -907,14 +926,26 @@ main(int argc, char **argv) else if (dopt.restrict_key) pg_fatal("option %s can only be used with %s", "--restrict-key", "--format=plain"); + if (filename_is_pipe && archiveFormat != archDirectory) + { + pg_log_error_hint("Option --pipe-command is only supported with directory format."); + exit_nicely(1); + } + + if (filename_is_pipe && strcmp(compression_algorithm_str, "none") != 0) + { + pg_log_error_hint("Option --pipe-command is not supported with any compression type."); + exit_nicely(1); + } /* * Custom and directory formats are compressed by default with gzip when * available, not the others. If gzip is not available, no compression is - * done by default. + * done by default. If directory format is being used with pipe-command, + * no compression is done. */ if ((archiveFormat == archCustom || archiveFormat == archDirectory) && - !user_compression_defined) + !filename_is_pipe && !user_compression_defined) { #ifdef HAVE_LIBZ compression_algorithm_str = "gzip"; @@ -964,7 +995,7 @@ main(int argc, char **argv) /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compression_spec, - dosync, archiveMode, setupDumpWorker, sync_method); + dosync, archiveMode, setupDumpWorker, sync_method, filename_is_pipe); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index c1f43113c53..2d551365180 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -678,7 +678,7 @@ main(int argc, char *argv[]) /* Open the output file */ fout = CreateArchive(global_path, archCustom, compression_spec, - dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC); + dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC, false); /* Make dump options accessible right away */ SetArchiveOptions(fout, &dopt, NULL); diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 48fdcb0fae1..c31d262e71a 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -1,5 +1,5 @@ /*------------------------------------------------------------------------- - * +* * pg_restore.c * pg_restore is an utility extracting postgres database definitions * from a backup archive created by pg_dump/pg_dumpall using the archiver @@ -654,7 +654,7 @@ restore_global_objects(const char *inputFileSpec, RestoreOptions *opts) opts->format = archCustom; opts->txn_size = 0; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); @@ -696,7 +696,7 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts, Archive *AH; int n_errors; - AH = OpenArchive(inputFileSpec, opts->format); + AH = OpenArchive(inputFileSpec, opts->format, false); SetArchiveOptions(AH, NULL, opts); -- 2.54.0.1064.gd145956f57-goog ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Adding pg_dump flag for parallel export to pipes @ 2026-06-12 05:20 solai v <[email protected]> parent: Nitin Motiani <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: solai v @ 2026-06-12 05:20 UTC (permalink / raw) To: Nitin Motiani <[email protected]>; +Cc: Hannu Krosing <[email protected]>; Mahendra Singh Thalor <[email protected]>; Dilip Kumar <[email protected]>; Thomas Munro <[email protected]>; pgsql-hackers Hi all, On Tue, Jun 9, 2026 at 12:04 PM Nitin Motiani <[email protected]> wrote: > > On Fri, May 22, 2026 at 4:04 PM solai v <[email protected]> wrote: > > I then tested the patch introducing --pipe support. The feature is > > quite useful for modern workflows where users want to stream dump > > output directly to compression or upload pipelines without relying on > > intermediate storage. > > Thank you for the feedback. > > > gzip: dump.gz: unexpected end of file > > This suggests that concurrent writes to a shared output target are not > > coordinated and can result in invalid dumps. It would be helpful to > > clarify expected usage patterns here. For example: whether users are > > expected to generate distinct outputs per worker, or whether > > safeguards should be implemented to prevent multiple workers from > > writing to the same destination. > > I added a warning for cases where the pipe command provided with > parallel dump and restore doesn't contain a `%f`. We can also add it > to the documentation. Let me know what you think. > > > scenarios I observed backend logs such as: > > FATAL: connection to client lost > > Broken pipe > > While this is expected when the pipe terminates prematurely, it may be > > worth considering whether error messaging or cleanup behavior can be > > made clearer from the user perspective. > > I added the failed command to the error message. I'm not sure if we > can do any auto-cleanup commands which succeeded. > Thank you for the updated patches. I reviewed and tested the latest v17 patch series in my current cluster. I tested the new --pipe functionality with both serial and parallel directory dumps and correctly generated: pipe_dump/ toc.dat 3475.dat 3476.dat The same behavior was verified with parallel jobs (-j 4). I also tested compression through an external command: pg_dump -p 55432 -Fd -j 4 \ --pipe='gzip > pipe_dump/%f.gz' \ postgres which successfully generated: pipe_dump/ toc.dat.gz 3475.dat.gz 3476.dat.gz without creating an intermediate dump directory. I verified error handling (--pipe='invalid_cmd') as well and correctly reported the error as "pg_dump: error: pipe command failed: "invalid_cmd": command not found". Similarly, --pipe='gzip | false' correctly propagated the child process failure. I also performed an end-to-end validation of the new functionality by restoring the generated archive using pg_restore --pipe. The restore completed successfully, and row counts for the test tables matched the original database confirming that dump generation, compression, restore, and data integrity all worked correctly. One particularly useful improvement I noticed is the handling of parallel jobs when the %f placeholder is omitted, producing the warning: "pg_dump: warning: parallel jobs with --pipe usually require the "%f" placeholder to avoid data corruption from multiple workers writing to the same file". This is an usability improvement, as it explicitly warns users against a common misuse that could otherwise lead to corrupted output when multiple workers write to the same destination file. Overall, the patch series worked well and the new --pipe support behaves correctly in my testing, also the newly added warning for missing %f significantly improves the user experience for parallel jobs. The patch looks good to me. Regards, Solai ^ permalink raw reply [nested|flat] 13+ messages in thread
end of thread, other threads:[~2026-06-12 05:20 UTC | newest] Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-07-05 14:24 [PATCH 03/17] Teach postmaster to accept encryption key Antonin Houska <[email protected]> 2023-09-04 22:04 [PATCH v7 1/5] Allow binaryheap to use any type. Nathan Bossart <[email protected]> 2026-01-27 10:48 [PATCH 5/6] Use background worker to do logical decoding. Antonin Houska <[email protected]> 2026-05-01 08:01 Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-04 04:39 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-07 08:35 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-08 17:07 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-11 05:34 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-14 10:09 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-21 09:56 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-05-22 10:34 ` Re: Adding pg_dump flag for parallel export to pipes solai v <[email protected]> 2026-06-09 06:34 ` Re: Adding pg_dump flag for parallel export to pipes Nitin Motiani <[email protected]> 2026-06-12 05:20 ` Re: Adding pg_dump flag for parallel export to pipes solai v <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox