Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hjPPU-0002G7-UG for pgsql-hackers@arkaria.postgresql.org; Fri, 05 Jul 2019 14:42:02 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1hjPPT-0005Nw-2o for pgsql-hackers@arkaria.postgresql.org; Fri, 05 Jul 2019 14:41:59 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hjPPR-0005F0-Jl for pgsql-hackers@lists.postgresql.org; Fri, 05 Jul 2019 14:41:58 +0000 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hjPPA-0005xC-LB for pgsql-hackers@postgresql.org; Fri, 05 Jul 2019 14:41:56 +0000 Received: by mail-wm1-x341.google.com with SMTP id w9so7663188wmd.1 for ; Fri, 05 Jul 2019 07:41:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec-at.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:in-reply-to:references:comments:mime-version :date:message-id; bh=NjMYVbIGd88G5DbK5ldqZuALAooBv+fY3X43sBbc4G8=; b=OEhb1J1VZrlIBlFizUfOn4iCxVwr+dSKw4wgcHl+yCcujyO8nJZrPMg9LwkBRB4bNx vhyFSFAgff+FIK3BXk4sKWCMmy930JPPZdlA2zNN4V97kreVX3DES5eMybGbzBWs3vaU pcFe2pTBRpWgpZwd/F0ZWW1NWBduPgcPK05UWmuaKJR1njOzPrhgVNTtio3n8NDGrr7Q +inz9ngSZLtZA4G+MdgjBBXykv8GgtQYPMLoAc5OBnjpt7cb3kBiIcExCl/T5up2Q+k7 RKjA/Qe0S5oMjI8wGRlTO839PQtICkKGk9XBu6tt50HNTId7faPMCQLrUQTInQJkMxEY OLpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :comments:mime-version:date:message-id; bh=NjMYVbIGd88G5DbK5ldqZuALAooBv+fY3X43sBbc4G8=; b=gvWlY7VdvF3NpRYOxY8S9VwoFbeZtFaAJfHVFHGDy97PhlM9VAgfahv4CL/te3KPRy G5F9uFwzYH8jmmBqC31wZiNB2LxZvoSUojmlREcokn+djGP6eHZZqp1aIMi9QO9JrNtg dVM4fJSexiEJvf/Vcguoy/H2Xko9bz5sXQlRau+/w9DCXV4rw7eKZFctSUQv/luM/WhY PnOcBORs8e5bDm/YF7PB+XENG+nI0vvAEtIgBH3yRvp4TCc5ccr1VLwa/QasjF7joR6x ewHj1ThbQGpm5WkH5Ap7mfZuBPflt4GvmQhDkY1GkAbDCcVGO3FILY33MHGMER7lPhhy eurg== X-Gm-Message-State: APjAAAV2CMTh19ahX6OfY45S5z2oAm0TiKte5LWzyt24zPAuX1q61MCc sZQFL2PMgfW/KapVWZCFjorRFA== X-Google-Smtp-Source: APXvYqxn/7i0lLck7Hv3JTnBuCafNRhzBXxUHcTwATEdmjc/jOCwCh9dcwDqRTTapUQwl79Dop7nHg== X-Received: by 2002:a1c:3883:: with SMTP id f125mr3671773wma.18.1562337697851; Fri, 05 Jul 2019 07:41:37 -0700 (PDT) Received: from localhost ([77.87.240.5]) by smtp.gmail.com with ESMTPSA id d10sm10716728wro.18.2019.07.05.07.41.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Jul 2019 07:41:35 -0700 (PDT) From: Antonin Houska To: Robert Haas cc: Toshi Harada , "pgsql-hackers@postgresql.org" Subject: Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3 In-reply-to: <32714.1559057215@localhost> References: <5907.1534140361@localhost> <201808150211.w7F2ApMj023097@ccmail04.silk.ntt-tx.co.jp> <19039.1536933944@localhost> <14646.1552570130@localhost> <11812.1554386232@localhost> <7567.1556117300@localhost> <32714.1559057215@localhost> Comments: In-reply-to Antonin Houska message dated "Tue, 28 May 2019 17:26:55 +0200." X-Mailer: MH-E 8.6+git; nmh 1.7+dev; GNU Emacs 26.0.91 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Date: Fri, 05 Jul 2019 16:41:34 +0200 Message-ID: <7082.1562337694@localhost> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --=-=-= Content-Type: text/plain Antonin Houska wrote: > Thanks of another round of review. > > Robert Haas wrote: > > > On Wed, Apr 24, 2019 at 10:48 AM Antonin Houska wrote: > > > Attached is the next version. It tries to address various problems pointed out > > > upthread, including documentation. This is the next version. I've reworked the way key is passed to postmaster (v04-0003-...) so it's easier to call the encryption key command interactively, adjusted pg_upgrade so that it preserves database, tablespace and relfilenode OIDs (v04-0014-...), reorganized the code a bit and split the code into more diffs. -- Antonin Houska Web: https://www.cybertec-postgresql.com --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0001-Make-postgres-aware-of-the-encryption.patch From a6cdae26675a74d102dfe38a3a8f77430ea5480d Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 01/17] Make postgres aware of the encryption This patch makes postgres aware of the encryption (i.e. sets up encryption key and initializes the encryption library) but does not actually encrypt any data. If initdb is passed the "encryption key command" (the -K option), it executes it and reads the encryption key from the command's stdout: initdb -D data -K "echo 71916b8b93063c280e4e0ee646395f450bda4f1bac85c57bc3a6afd168f3ab47" -K option (without any argument) is then passed to bootstrap process just so it knows that the new cluster should be encrypted. The first thing the bootstrap process does in such a case is that it reads the encryption key from stdin. The point is that a flag is eventually sets a control file that indidicates that encryption is enabled for the cluster. A sample string is also encrypted and stored in the control file. It'll be used on each startup to verify that user passed the correct key. pg_controldata utility shows two new fields now: "Data encryption" and "Data encryption fingerprint". Once a standalone backend it started, it reads the control file and recognizes the cluster is encrypted. Therefore it expects the encryption key to be the first user's input: postgres --single -D data postgres key>71916b8b93063c280e4e0ee646395f450bda4f1bac85c57bc3a6afd168f3ab47 Once the correct key is passed, the session can continue as if there was no encryption. The only difference user can see is that "data_encryption" variable is set: PostgreSQL stand-alone backend 13devel backend> SHOW data_encryption; 1: data_encryption (typeid = 25, len = -1, typmod = -1, byval = f) ---- 1: data_encryption = "on" (typeid = 25, len = -1, typmod = -1, byval = f) ---- --- src/backend/access/transam/xlog.c | 72 ++++++++ src/backend/bootstrap/bootstrap.c | 48 ++++- src/backend/storage/file/Makefile | 2 +- src/backend/storage/file/encryption.c | 313 ++++++++++++++++++++++++++++++++ src/backend/tcop/postgres.c | 30 +++ src/backend/utils/misc/guc.c | 12 ++ src/bin/initdb/.gitignore | 1 + src/bin/initdb/Makefile | 7 +- src/bin/initdb/initdb.c | 74 +++++++- src/bin/pg_controldata/pg_controldata.c | 9 + src/common/string.c | 39 ++++ src/fe_utils/Makefile | 3 +- src/fe_utils/encryption.c | 90 +++++++++ src/include/catalog/pg_control.h | 11 ++ src/include/common/string.h | 1 + src/include/fe_utils/encryption.h | 18 ++ src/include/pg_config_manual.h | 7 + src/include/storage/encryption.h | 99 ++++++++++ 18 files changed, 829 insertions(+), 7 deletions(-) create mode 100644 src/backend/storage/file/encryption.c create mode 100644 src/fe_utils/encryption.c create mode 100644 src/include/fe_utils/encryption.h create mode 100644 src/include/storage/encryption.h diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13e0d2366f..6b58e0ac76 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -55,6 +55,7 @@ #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" +#include "storage/encryption.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/large_object.h" @@ -77,6 +78,7 @@ #include "pg_trace.h" extern uint32 bootstrap_data_checksum_version; +extern char *bootstrap_encryption_sample; /* Unsupported old recovery command file names (relative to $PGDATA) */ #define RECOVERY_COMMAND_FILE "recovery.conf" @@ -4774,6 +4776,58 @@ ReadControlFile(void) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("\"max_wal_size\" must be at least twice \"wal_segment_size\""))); + /* + * Initialize encryption, but not if the current backend has already done + * that. + */ + if (ControlFile->data_cipher > PG_CIPHER_NONE && !data_encrypted) + { + /* + * Set data_encryption for caller to know that he needs to retrieve + * the key and initialize the encryption library. + */ + SetConfigOption("data_encryption", "true", PGC_INTERNAL, + PGC_S_OVERRIDE); + + /* + * Save the verification string. We'll perform the actual verification + * as soon as the encryption setup is done. + */ + memcpy(encryption_verification, + ControlFile->encryption_verification, + ENCRYPTION_SAMPLE_SIZE); + + /* + * full_page_writes must be set because torn page write of an + * encrypted page implies that decryption of the page will produce + * garbage. This damage can affect even those parts of the page that + * haven't been modified by any access method. And since no access + * method modified those parts, there might be no XLOG records to + * repair them during crash recovery. So full page image is the only + * way to fix such a page. + * + * Do not enforce this setting in binary upgrade mode, since + * pg_upgrade sets it to off for performance reasons. (If the upgrade + * crashes, the new cluster must be created from scratch anyway.) + * + * XXX It would be nice to have guc.c check so that we don't have to + * copy and paste the error message. However it's unclear how to + * ensure that either ERROR is raised or nothing happens at all. It + * seems that set_config_option() can change + * config_generic.reset_source if the check succeeded, but that's too + * invasive. + */ + if (!fullPageWrites && !IsBinaryUpgrade) + ereport(FATAL, + (errmsg("invalid value for parameter \"full_page_writes\": %d", + fullPageWrites), + errdetail("Cannot disable parameter when the cluster is encrypted."))); + } + + /* + * This calculation relies on data_encryption (in particular the header + * sizes do), so we could not do it earlier. + */ UsableBytesInSegment = (wal_segment_size / XLOG_BLCKSZ * UsableBytesInPage) - (SizeOfXLogLongPHD - SizeOfXLogShortPHD); @@ -5255,6 +5309,24 @@ BootStrapXLOG(void) ControlFile->track_commit_timestamp = track_commit_timestamp; ControlFile->data_checksum_version = bootstrap_data_checksum_version; + if (data_encrypted) + { + char sample[ENCRYPTION_SAMPLE_SIZE]; + + ControlFile->data_cipher = PG_CIPHER_AES_BLOCK_CBC_256_STREAM_CTR_256; + + sample_encryption(sample); + + memcpy(ControlFile->encryption_verification, sample, + ENCRYPTION_SAMPLE_SIZE); + } + else + { + ControlFile->data_cipher = PG_CIPHER_NONE; + memset(ControlFile->encryption_verification, 0, + ENCRYPTION_SAMPLE_SIZE); + } + /* some additional ControlFile fields are set in WriteControlFile() */ WriteControlFile(); diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 43627ab8f4..406fff51a9 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -40,6 +40,7 @@ #include "storage/bufmgr.h" #include "storage/bufpage.h" #include "storage/condition_variable.h" +#include "storage/encryption.h" #include "storage/ipc.h" #include "storage/proc.h" #include "tcop/tcopprot.h" @@ -58,6 +59,7 @@ uint32 bootstrap_data_checksum_version = 0; /* No checksum */ static void CheckerModeMain(void); static void BootstrapModeMain(void); +static int bootstrap_getc(void); static void bootstrap_signals(void); static void ShutdownAuxiliaryProcess(int code, Datum arg); static Form_pg_attribute AllocateAttribute(void); @@ -226,7 +228,7 @@ AuxiliaryProcessMain(int argc, char *argv[]) /* If no -x argument, we are a CheckerProcess */ MyAuxProcType = CheckerProcess; - while ((flag = getopt(argc, argv, "B:c:d:D:Fkr:x:X:-:")) != -1) + while ((flag = getopt(argc, argv, "B:c:d:D:FkKr:x:X:-:")) != -1) { switch (flag) { @@ -252,6 +254,22 @@ AuxiliaryProcessMain(int argc, char *argv[]) case 'F': SetConfigOption("fsync", "false", PGC_POSTMASTER, PGC_S_ARGV); break; +#ifdef USE_ENCRYPTION + case 'K': + /* + * When auxiliary process (bootstrap) starts, the control file + * does not exist yet, so command line option needs to be used + * to indicate that the encryption is enabled. + * + * Postmaster should not pass this option. Instead, it just + * sets data_encrypted according to the control file and child + * processes inherit that. + */ + Assert(!IsUnderPostmaster); + data_encrypted = true; + + break; +#endif /* USE_ENCRYPTION */ case 'k': bootstrap_data_checksum_version = PG_DATA_CHECKSUM_VERSION; break; @@ -373,6 +391,24 @@ AuxiliaryProcessMain(int argc, char *argv[]) if (!IsUnderPostmaster) InitializeMaxBackends(); + /* + * If data_encryption is set because of command line option, do the setup + * now. (If set by postmaster, postmaster should have performed the + * setup.) + * + * This should only be useful for the bootstrap process. Anyone else + * detects the encryption via ReadControlFile(). + */ + if (data_encrypted && MyAuxProcType == BootstrapProcess) + { + Assert(!IsUnderPostmaster); + + /* Read the key from stdin. */ + read_encryption_key(bootstrap_getc); + + setup_encryption(); + } + BaseInit(); /* @@ -551,6 +587,16 @@ BootstrapModeMain(void) */ /* + * Read a single character from stdin. This is a callback for + * read_encryption_key(). + */ +static int +bootstrap_getc(void) +{ + return getc(stdin); +} + +/* * Set up signal handling for a bootstrap process */ static void diff --git a/src/backend/storage/file/Makefile b/src/backend/storage/file/Makefile index ca6a0e4f7d..9f277755fc 100644 --- a/src/backend/storage/file/Makefile +++ b/src/backend/storage/file/Makefile @@ -12,6 +12,6 @@ subdir = src/backend/storage/file top_builddir = ../../../.. include $(top_builddir)/src/Makefile.global -OBJS = fd.o buffile.o copydir.o reinit.o sharedfileset.o +OBJS = fd.o buffile.o copydir.o reinit.o sharedfileset.o encryption.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/storage/file/encryption.c b/src/backend/storage/file/encryption.c new file mode 100644 index 0000000000..629138fae7 --- /dev/null +++ b/src/backend/storage/file/encryption.c @@ -0,0 +1,313 @@ +/*------------------------------------------------------------------------- + * + * encryption.c + * This code handles encryption and decryption of data. + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * See src/backend/storage/file/README.encryption for explanation of the + * design. + * + * IDENTIFICATION + * src/backend/storage/file/encryption.c + * + * NOTES + * This file is compiled as both front-end and backend code, so the + * FRONTEND macro must be used to distinguish the case if we need to + * report error or if server-defined variable / function seems useful. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include + +#include "access/xlog.h" +#include "access/xlogdefs.h" +#include "common/fe_memutils.h" +#include "common/sha2.h" +#include "common/string.h" +#include "catalog/pg_control.h" +#include "storage/bufpage.h" +#include "storage/encryption.h" + +#ifndef FRONTEND +#include "port.h" +#include "storage/shmem.h" +#include "storage/fd.h" +#include "utils/memutils.h" +#endif /* FRONTEND */ + +#ifdef USE_ENCRYPTION +#include +#include +#include + +EVP_CIPHER_CTX *ctx_encrypt, + *ctx_decrypt, + *ctx_encrypt_stream, + *ctx_decrypt_stream; +#endif /* USE_ENCRYPTION */ + +unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; + +bool data_encrypted = false; + +char encryption_verification[ENCRYPTION_SAMPLE_SIZE]; + +bool encryption_setup_done = false; + +#ifdef USE_ENCRYPTION +static void init_encryption_context(EVP_CIPHER_CTX **ctx_p, bool stream); +static void evp_error(void); +#endif /* USE_ENCRYPTION */ + +#ifndef FRONTEND +/* + * Read encryption key in hexadecimal form from stdin and store it in + * encryption_key variable. + */ +void +read_encryption_key(read_encryption_key_cb read_char) +{ + char *buf; + int read_len, i, c; + + buf = (char *) palloc(ENCRYPTION_KEY_CHARS); + + read_len = 0; + while ((c = (*read_char)()) != EOF && c != '\n') + { + if (read_len >= ENCRYPTION_KEY_CHARS) + ereport(FATAL, (errmsg("Encryption key is too long"))); + + buf[read_len++] = c; + } + + if (read_len < ENCRYPTION_KEY_CHARS) + ereport(FATAL, (errmsg("Encryption key is too short"))); + + /* Turn the hexadecimal representation into an array of bytes. */ + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + { + if (sscanf(buf + 2 * i, "%2hhx", encryption_key + i) == 0) + { + ereport(FATAL, + (errmsg("Invalid character in encryption key at position %d", + 2 * i))); + } + } + + pfree(buf); +} +#endif /* FRONTEND */ + +/* + * Initialize encryption subsystem for use. Must be called before any + * encryptable data is read from or written to data directory. + */ +void +setup_encryption(void) +{ +#ifdef USE_ENCRYPTION + /* + * Setup OpenSSL. + * + * None of these functions should return a value or raise error. + */ + ERR_load_crypto_strings(); + OpenSSL_add_all_algorithms(); + + /* + * TODO Find out if this needs to be called for OpenSSL < 1.1.0. + */ + /* OPENSSL_config(NULL); */ + + init_encryption_context(&ctx_encrypt, false); + init_encryption_context(&ctx_decrypt, false); + + init_encryption_context(&ctx_encrypt_stream, true); + init_encryption_context(&ctx_decrypt_stream, true); + + encryption_setup_done = true; +#else /* !USE_ENCRYPTION */ +#ifndef FRONTEND + /* + * If no encryption implementation is linked and caller requests + * encryption, we should error out here and thus cause the calling process + * to fail (preferably postmaster, so the child processes don't make the + * same mistake). + */ + ereport(FATAL, (errmsg(ENCRYPTION_NOT_SUPPORTED_MSG))); +#else + /* Front-end shouldn't actually get here, but be careful. */ + fprintf(stderr, "%s\n", ENCRYPTION_NOT_SUPPORTED_MSG); + exit(EXIT_FAILURE); +#endif /* FRONTEND */ +#endif /* USE_ENCRYPTION */ +} + +/* + * Encrypts a fixed value into *buf to verify that encryption key is correct. + * Caller provided buf needs to be able to hold at least ENCRYPTION_SAMPLE_SIZE + * bytes. + */ +void +sample_encryption(char *buf) +{ + char tweak[TWEAK_SIZE]; + int i; + + for (i = 0; i < TWEAK_SIZE; i++) + tweak[i] = i; + + encrypt_block("postgresqlcrypt", buf, ENCRYPTION_SAMPLE_SIZE, tweak, + false); +} + +/* + * Encrypts one block of data with a specified tweak value. May only be called + * when encryption_enabled is true. + * + * Input and output buffer may point to the same location. + * + * "size" must be a (non-zero) multiple of ENCRYPTION_BLOCK. + * + * "tweak" value must be TWEAK_SIZE bytes long. + * + * If "stream" is set, stream cipher is used instead of block one. + * + * All-zero blocks are not encrypted to correctly handle relation extension, + * and also to simplify handling of holes created by seek past EOF and + * consequent write (see buffile.c). + */ +void +encrypt_block(const char *input, char *output, Size size, char *tweak, + bool stream) +{ +#ifdef USE_ENCRYPTION + int out_size; + EVP_CIPHER_CTX *ctx; + + Assert(data_encrypted); + + /* + * Block cipher should only be used if the size is whole multiple of + * encryption block size. + */ + Assert((size >= ENCRYPTION_BLOCK && size % ENCRYPTION_BLOCK == 0) || + stream); + + /* + * Empty page is not worth encryption. Do not waste cycles checking for + * stream cipher as this is currently used only for XLOG pages, and empty + * XLOG page should not be written to disk. + */ + if (!stream && IsAllZero(input, size)) + { + memset(output, 0, size); + return; + } + + ctx = !stream ? ctx_encrypt : ctx_encrypt_stream; + + /* The remaining initialization. */ + if (EVP_EncryptInit_ex(ctx, NULL, NULL, encryption_key, + (unsigned char *) tweak) != 1) + evp_error(); + + /* Do the actual encryption. */ + if (EVP_EncryptUpdate(ctx, (unsigned char *) output, + &out_size, (unsigned char *) input, size) != 1) + evp_error(); + + Assert(out_size == size); +#else + /* data_encrypted should not be set */ + Assert(false); +#endif /* USE_ENCRYPTION */ +} + + +#ifdef USE_ENCRYPTION +/* + * Initialize the OpenSSL context for passed cipher. + * + * On server side this happens during postmaster startup, so other processes + * inherit the initialized context via fork(). There's no reason to this again + * and again in encrypt_block() / decrypt_block(), also because we cannot + * handle out-of-memory conditions encountered by OpenSSL in another way than + * ereport(FATAL). The OOM is much less likely to happen during postmaster + * startup, and even if it happens, troubleshooting should be easier than if + * it happened during normal operation. + * + * XXX Do we need to call EVP_CIPHER_CTX_cleanup() (via on_proc_exit callback + * for server processes and other way for front-ends)? Not sure it's + * necessary, as the initialization does not involve any shared resources + * (e.g. files). + */ +static void +init_encryption_context(EVP_CIPHER_CTX **ctx_p, bool stream) +{ + EVP_CIPHER_CTX *ctx; + const EVP_CIPHER *cipher; +#ifdef USE_ASSERT_CHECKING + int block_size; +#endif /* USE_ASSERT_CHECKING */ + + cipher = !stream ? EVP_aes_256_cbc() : EVP_aes_256_ctr(); + + if ((*ctx_p = EVP_CIPHER_CTX_new()) == NULL) + evp_error(); + ctx = *ctx_p; + if (EVP_EncryptInit_ex(ctx, cipher, NULL, NULL, NULL) != 1) + evp_error(); + + /* + * No padding is needed. For a block cipher, the input block size should + * already be a multiple of ENCRYPTION_BLOCK. For stream cipher, we don't + * need padding anyway. This might save some cycles at the OpenSSL end. + * XXX Is it setting worth when we don't call EVP_DecryptFinal_ex() + * anyway? + */ + EVP_CIPHER_CTX_set_padding(ctx, 0); + + Assert(EVP_CIPHER_CTX_iv_length(ctx) == TWEAK_SIZE); + Assert(EVP_CIPHER_CTX_key_length(ctx) == ENCRYPTION_KEY_LENGTH); + block_size = EVP_CIPHER_CTX_block_size(ctx); +#ifdef USE_ASSERT_CHECKING + if (!stream) + Assert(block_size == ENCRYPTION_BLOCK); + else + Assert(block_size == 1); +#endif /* USE_ASSERT_CHECKING */ +} + +#endif /* USE_ENCRYPTION */ + +#ifdef USE_ENCRYPTION +/* + * Error callback for openssl. + */ +static void +evp_error(void) +{ + ERR_print_errors_fp(stderr); +#ifndef FRONTEND + + /* + * FATAL is the appropriate level because backend can hardly fix anything + * if encryption / decryption has failed. + * + * XXX Do we yet need EVP_CIPHER_CTX_cleanup() here? + */ + elog(FATAL, "OpenSSL encountered error during encryption or decryption."); +#else + fprintf(stderr, + "OpenSSL encountered error during encryption or decryption."); + exit(EXIT_FAILURE); +#endif /* FRONTEND */ +} +#endif /* USE_ENCRYPTION */ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 44a59e1d4f..38dad9311f 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -63,6 +63,7 @@ #include "replication/walsender.h" #include "rewrite/rewriteHandler.h" #include "storage/bufmgr.h" +#include "storage/encryption.h" #include "storage/ipc.h" #include "storage/proc.h" #include "storage/procsignal.h" @@ -3868,6 +3869,35 @@ PostgresMain(int argc, char *argv[], PG_SETMASK(&UnBlockSig); /* + * Standalone backend operating on an encrypted cluster needs encryption + * key. + */ + if (!IsUnderPostmaster && data_encrypted) + { + char sample[ENCRYPTION_SAMPLE_SIZE]; + + /* Display a prompt for user to enter the encryption key. */ + printf("key> "); + fflush(stdout); + + /* + * Read the key from stdin. Pass interactive_getc() as the callback so + * that the reading is interruptible. + */ + read_encryption_key(interactive_getc); + + setup_encryption(); + + /* Verify the key. */ + sample_encryption(sample); + if (memcmp(encryption_verification, sample, ENCRYPTION_SAMPLE_SIZE)) + ereport(FATAL, + (errmsg("invalid encryption key"), + errdetail("The passed encryption key does not match" + " database encryption key."))); + } + + /* * General initialization. * * NOTE: if you are tempted to add code in this vicinity, consider putting diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 631f16f5fe..2b7fc2fb0f 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -71,6 +71,7 @@ #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" +#include "storage/encryption.h" #include "storage/dsm_impl.h" #include "storage/standby.h" #include "storage/fd.h" @@ -1836,6 +1837,17 @@ static struct config_bool ConfigureNamesBool[] = }, { + {"data_encryption", PGC_INTERNAL, PRESET_OPTIONS, + gettext_noop("Shows whether data encryption is turned on for this cluster."), + NULL, + GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE + }, + &data_encrypted, + false, + NULL, NULL, NULL + }, + + { {"syslog_sequence_numbers", PGC_SIGHUP, LOGGING_WHERE, gettext_noop("Add sequence number to syslog messages to avoid duplicate suppression."), NULL diff --git a/src/bin/initdb/.gitignore b/src/bin/initdb/.gitignore index 71a899ffb8..0f33b5c67d 100644 --- a/src/bin/initdb/.gitignore +++ b/src/bin/initdb/.gitignore @@ -1,5 +1,6 @@ /encnames.c /localtime.c +/encryption.c /initdb diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile index 7c404430a9..b72de55d62 100644 --- a/src/bin/initdb/Makefile +++ b/src/bin/initdb/Makefile @@ -26,7 +26,7 @@ ifneq (,$(with_system_tzdata)) override CPPFLAGS += '-DSYSTEMTZDIR="$(with_system_tzdata)"' endif -OBJS= initdb.o findtimezone.o localtime.o encnames.o $(WIN32RES) +OBJS= initdb.o findtimezone.o localtime.o encnames.o encryption.o $(WIN32RES) all: initdb @@ -45,6 +45,9 @@ encnames.c: % : $(top_srcdir)/src/backend/utils/mb/% localtime.c: % : $(top_srcdir)/src/timezone/% rm -f $@ && $(LN_S) $< . +encryption.c: % : $(top_srcdir)/src/backend/storage/file/% + rm -f $@ && $(LN_S) $< . + install: all installdirs $(INSTALL_PROGRAM) initdb$(X) '$(DESTDIR)$(bindir)/initdb$(X)' @@ -55,7 +58,7 @@ uninstall: rm -f '$(DESTDIR)$(bindir)/initdb$(X)' clean distclean maintainer-clean: - rm -f initdb$(X) $(OBJS) encnames.c localtime.c + rm -f initdb$(X) $(OBJS) encnames.c localtime.c encryption.c rm -rf tmp_check # ensure that changes in datadir propagate into object file diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index f1acbdfcf2..f2a582e975 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -68,11 +68,13 @@ #include "common/logging.h" #include "common/restricted_token.h" #include "common/username.h" +#include "fe_utils/encryption.h" #include "fe_utils/string_utils.h" #include "getaddrinfo.h" #include "getopt_long.h" #include "mb/pg_wchar.h" #include "miscadmin.h" +#include "storage/encryption.h" /* Ideally this would be in a .h file, but it hardly seems worth the trouble */ @@ -171,6 +173,8 @@ static bool output_failed = false; static int output_errno = 0; static char *pgdata_native; +unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; + /* defaults */ static int n_connections = 10; static int n_buffers = 50; @@ -239,6 +243,7 @@ static char **filter_lines_with_token(char **lines, const char *token); static char **readfile(const char *path); static void writefile(char *path, char **lines); static FILE *popen_check(const char *command, const char *mode); +static void send_encryption_key(FILE *f); static char *get_id(void); static int get_encoding_id(const char *encoding_name); static void set_input(char **dest, const char *filename); @@ -584,6 +589,22 @@ popen_check(const char *command, const char *mode) } /* + * Send encryption key in hexadecimal format to the file stream passed. + * + * The backend processes could actually receive binary data but that would + * make startup of postgres in single-user mode less convenient. + */ +static void +send_encryption_key(FILE *f) +{ + int i; + + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + fprintf(f, "%.2x", encryption_key[i]); + fputc('\n', f); +} + +/* * clean up any files we created on failure * if we created the data directory remove it too */ @@ -1379,6 +1400,7 @@ bootstrap_template1(void) char **bki_lines; char headerline[MAXPGPATH]; char buf[64]; + static char *encr_opt_str = NULL; printf(_("running bootstrap script ... ")); fflush(stdout); @@ -1446,17 +1468,37 @@ bootstrap_template1(void) /* Also ensure backend isn't confused by this environment var: */ unsetenv("PGCLIENTENCODING"); + /* Prepare the -K option for the backend. */ + if (encryption_key_command) + { + size_t len; + + len = 3; + encr_opt_str = (char *) pg_malloc(len); + snprintf(encr_opt_str, len, "-K"); + } + else + { + encr_opt_str = (char *) pg_malloc(1); + encr_opt_str[0] = '\0'; + } + snprintf(cmd, sizeof(cmd), - "\"%s\" --boot -x1 -X %u %s %s %s", + "\"%s\" --boot -x1 -X %u %s %s %s %s", backend_exec, wal_segment_size_mb * (1024 * 1024), data_checksums ? "-k" : "", + encr_opt_str, boot_options, debug ? "-d 5" : ""); PG_CMD_OPEN; + /* If the cluster is encrypted, first send the encryption key. */ + if (encryption_key_command) + send_encryption_key(cmdfd); + for (line = bki_lines; *line != NULL; line++) { PG_CMD_PUTS(*line); @@ -2378,6 +2420,10 @@ usage(const char *progname) printf(_("\nLess commonly used options:\n")); printf(_(" -d, --debug generate lots of debugging output\n")); printf(_(" -k, --data-checksums use data page checksums\n")); +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command\n" + " command that returns encryption key\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -L DIRECTORY where to find the input files\n")); printf(_(" -n, --no-clean do not clean up after errors\n")); printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n")); @@ -2957,6 +3003,13 @@ initialize_data_directory(void) /* Top level PG_VERSION is checked by bootstrapper, so make it first */ write_version_file(NULL); + /* + * If the cluster will be encrypted, run the command to generate the + * encryption key. + */ + if (encryption_key_command) + run_encryption_key_command(encryption_key); + /* Select suitable configuration settings */ set_null_conf(); test_config_settings(); @@ -2986,6 +3039,10 @@ initialize_data_directory(void) PG_CMD_OPEN; + /* If the cluster is encrypted, first send the encryption key. */ + if (encryption_key_command) + send_encryption_key(cmdfd); + setup_auth(cmdfd); setup_depend(cmdfd); @@ -3054,6 +3111,9 @@ main(int argc, char *argv[]) {"waldir", required_argument, NULL, 'X'}, {"wal-segsize", required_argument, NULL, 12}, {"data-checksums", no_argument, NULL, 'k'}, +#ifdef USE_ENCRYPTION + {"encryption-key-command", required_argument, NULL, 'K'}, +#endif /* USE_ENCRYPTION */ {"allow-group-access", no_argument, NULL, 'g'}, {NULL, 0, NULL, 0} }; @@ -3096,7 +3156,7 @@ main(int argc, char *argv[]) /* process command-line options */ - while ((c = getopt_long(argc, argv, "dD:E:kL:nNU:WA:sST:X:g", long_options, &option_index)) != -1) + while ((c = getopt_long(argc, argv, "dD:E:kK:L:nNU:WA:sST:X:g", long_options, &option_index)) != -1) { switch (c) { @@ -3148,6 +3208,11 @@ main(int argc, char *argv[]) case 'k': data_checksums = true; break; +#ifdef USE_ENCRYPTION + case 'K': + encryption_key_command = pg_strdup(optarg); + break; +#endif /* USE_ENCRYPTION */ case 'L': share_path = pg_strdup(optarg); break; @@ -3318,6 +3383,11 @@ main(int argc, char *argv[]) if (pwprompt || pwfilename) get_su_pwd(); + if (encryption_key_command) + printf(_("Data encryption is enabled.\n")); + else + printf(_("Data encryption is disabled.\n")); + printf("\n"); initialize_data_directory(); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 390ea0a939..3eef925f97 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -336,5 +336,14 @@ main(int argc, char *argv[]) ControlFile->data_checksum_version); printf(_("Mock authentication nonce: %s\n"), mock_auth_nonce_str); + printf(_("Data encryption: %s\n"), + ControlFile->data_cipher > PG_CIPHER_NONE ? _("on") : _("off")); + if (ControlFile->data_cipher > PG_CIPHER_NONE) + printf(_("Data encryption fingerprint: %08X%08X%08X%08X\n"), + htonl(((uint32 *) ControlFile->encryption_verification)[0]), + htonl(((uint32 *) ControlFile->encryption_verification)[1]), + htonl(((uint32 *) ControlFile->encryption_verification)[2]), + htonl(((uint32 *) ControlFile->encryption_verification)[3]) + ); return 0; } diff --git a/src/common/string.c b/src/common/string.c index b01a56ceaa..5ba890d37d 100644 --- a/src/common/string.c +++ b/src/common/string.c @@ -42,6 +42,45 @@ pg_str_endswith(const char *str, const char *end) return strcmp(str, end) == 0; } +/* + * Helper function to check if a page is completely empty. + * + * TODO Invent name that is more consistent with that of the other function(s) + * in this module. + */ +bool +IsAllZero(const char *input, Size size) +{ + const char *pos = input; + const char *aligned_start = (char *) MAXALIGN64(input); + const char *end = input + size; + + /* Check 1 byte at a time until pos is 8 byte aligned */ + while (pos < aligned_start) + if (*pos++ != 0) + return false; + + /* + * Run 8 parallel 8 byte checks in one iteration. On 2016 hardware + * slightly faster than 4 parallel checks. + */ + while (pos + 8 * sizeof(uint64) <= end) + { + uint64 *p = (uint64 *) pos; + + if ((p[0] | p[1] | p[2] | p[3] | p[4] | p[5] | p[6] | p[7]) != 0) + return false; + pos += 8 * sizeof(uint64); + } + + /* Handle unaligned tail. */ + while (pos < end) + if (*pos++ != 0) + return false; + + return true; +} + /* * strtoint --- just like strtol, but returns int not long diff --git a/src/fe_utils/Makefile b/src/fe_utils/Makefile index 7d73800323..a508ec150e 100644 --- a/src/fe_utils/Makefile +++ b/src/fe_utils/Makefile @@ -19,7 +19,8 @@ include $(top_builddir)/src/Makefile.global override CPPFLAGS := -DFRONTEND -I$(libpq_srcdir) $(CPPFLAGS) -OBJS = mbprint.o print.o psqlscan.o simple_list.o string_utils.o conditional.o +OBJS = mbprint.o print.o psqlscan.o simple_list.o string_utils.o conditional.o \ + encryption.o all: libpgfeutils.a diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c new file mode 100644 index 0000000000..dab1435014 --- /dev/null +++ b/src/fe_utils/encryption.c @@ -0,0 +1,90 @@ +/*------------------------------------------------------------------------- + * + * encryption.c + * Front-end code to handle keys for full cluster encryption. The actual + * encryption is not performed here. + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/fe_utils/encryption.c + * + *------------------------------------------------------------------------- + */ + +#include + +#include "postgres_fe.h" + +#include "common/fe_memutils.h" +#include "common/file_perm.h" +#include "common/logging.h" +#include "fe_utils/encryption.h" +#include "storage/encryption.h" +#include "libpq-fe.h" +#include "libpq-int.h" +#include "libpq/pqcomm.h" + +char *encryption_key_command = NULL; + +/* + * Run the command that is supposed to generate encryption key and store it + * where encryption_key points to. + */ +void +run_encryption_key_command(unsigned char *encryption_key) +{ + FILE *fp; + char *buf; + int read_len, i, c; + + Assert(encryption_key_command != NULL && + strlen(encryption_key_command) > 0); + + fp = popen(encryption_key_command, "r"); + if (fp == NULL) + { + pg_log_fatal("Failed to execute \"%s\"", encryption_key_command); + exit(EXIT_FAILURE); + } + + buf = (char *) palloc(ENCRYPTION_KEY_CHARS); + + /* + * Read the key. This is very similar to backend's read_encryption_key() + * but there seems to be no straightforward way to call the function from + * here. + */ + read_len = 0; + while ((c = fgetc(fp)) != EOF && c != '\n') + { + if (read_len >= ENCRYPTION_KEY_CHARS) + { + pg_log_fatal("Encryption key is too long"); + exit(EXIT_FAILURE); + } + + buf[read_len++] = c; + } + + if (read_len < ENCRYPTION_KEY_CHARS) + { + pg_log_fatal("Encryption key is too short"); + exit(EXIT_FAILURE); + } + + /* Turn the hexadecimal representation into an array of bytes. */ + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + { + if (sscanf(buf + 2 * i, "%2hhx", encryption_key + i) == 0) + { + pg_log_fatal("Invalid character in encryption key at position %d", + 2 * i); + exit(EXIT_FAILURE); + } + } + + pfree(buf); + pclose(fp); +} diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index ff98d9e91a..c1b53e1ebf 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -19,6 +19,7 @@ #include "access/xlogdefs.h" #include "pgtime.h" /* for pg_time_t */ #include "port/pg_crc32c.h" +#include "storage/encryption.h" /* Version identifier for this pg_control format */ @@ -228,6 +229,16 @@ typedef struct ControlFileData */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; + /* + * Cipher used to encrypt data. Zero if unencrypted. + * + * The data type is actually CipherKind, but we don't want to include + * encryption.h just because of this field. + */ + uint8 data_cipher; + /* Sample value for encryption key verification */ + uint8 encryption_verification[ENCRYPTION_SAMPLE_SIZE]; + /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; diff --git a/src/include/common/string.h b/src/include/common/string.h index 77f31337ca..6128d81b8c 100644 --- a/src/include/common/string.h +++ b/src/include/common/string.h @@ -14,5 +14,6 @@ extern bool pg_str_endswith(const char *str, const char *end); extern int strtoint(const char *pg_restrict str, char **pg_restrict endptr, int base); extern void pg_clean_ascii(char *str); +extern bool IsAllZero(const char *input, Size size); #endif /* COMMON_STRING_H */ diff --git a/src/include/fe_utils/encryption.h b/src/include/fe_utils/encryption.h new file mode 100644 index 0000000000..69e3b2f756 --- /dev/null +++ b/src/include/fe_utils/encryption.h @@ -0,0 +1,18 @@ +/*------------------------------------------------------------------------- + * + * encryption.h + * Client code to support full cluster encryption. + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/include/fe_utils/encryption.h + * + *------------------------------------------------------------------------- + */ + +/* Executable to retrieve the encryption key. */ +extern char *encryption_key_command; + +extern void run_encryption_key_command(unsigned char *encryption_key); diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h index 743401cb96..b716fb4d13 100644 --- a/src/include/pg_config_manual.h +++ b/src/include/pg_config_manual.h @@ -172,6 +172,13 @@ #endif /* + * OpenSSL is currently the only implementation of encryption we use. + */ +#ifdef USE_OPENSSL +#define USE_ENCRYPTION +#endif + +/* * This is the default directory in which AF_UNIX socket files are * placed. Caution: changing this risks breaking your existing client * applications, which are likely to continue to look in the old diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h new file mode 100644 index 0000000000..8479b76a34 --- /dev/null +++ b/src/include/storage/encryption.h @@ -0,0 +1,99 @@ +/*------------------------------------------------------------------------- + * + * encryption.h + * Full database encryption support + * + * + * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/storage/encryption.h + * + *------------------------------------------------------------------------- + */ +#ifndef ENCRYPTION_H +#define ENCRYPTION_H + +#include "access/xlogdefs.h" +#include "miscadmin.h" +#include "storage/block.h" +#include "storage/relfilenode.h" +#include "port/pg_crc32c.h" + +/* + * Common error message issued when particular code path cannot be executed + * due to absence of the OpenSSL library. + */ +#define ENCRYPTION_NOT_SUPPORTED_MSG \ + "compile postgres with --with-openssl to use encryption." + +/* + * Full database encryption key. + * + * The key of EVP_aes_256_cbc() cipher is 256 bits long. + */ +#define ENCRYPTION_KEY_LENGTH 32 +/* Key length in characters (two characters per hexadecimal digit) */ +#define ENCRYPTION_KEY_CHARS (ENCRYPTION_KEY_LENGTH * 2) + +/* + * Cipher used to encrypt data. + * + * Due to very specific requirements, the ciphers are not likely to change, + * but we should be somewhat flexible. + * + * XXX If we have more than one cipher someday, have pg_controldata report the + * cipher kind (in textual form) instead of merely saying "on". + */ +typedef enum CipherKind +{ + /* The cluster is not encrypted. */ + PG_CIPHER_NONE = 0, + + /* + * AES (Rijndael) in CBC mode of operation as block cipher, and in CTR + * mode as stream cipher. Key length is always 256 bits. + */ + PG_CIPHER_AES_BLOCK_CBC_256_STREAM_CTR_256 +} CipherKind; + +/* Key to encrypt / decrypt data. */ +extern unsigned char encryption_key[]; + +/* + * The encrypted data is a series of blocks of size + * ENCRYPTION_BLOCK. Currently we use the EVP_aes_256_xts implementation. Make + * sure the following constants match if adopting another algorithm. + */ +#define ENCRYPTION_BLOCK 16 + +#define TWEAK_SIZE 16 + +/* Is the cluster encrypted? */ +extern PGDLLIMPORT bool data_encrypted; + +/* + * Number of bytes reserved to store encryption sample in ControlFileData. + */ +#define ENCRYPTION_SAMPLE_SIZE 16 + +#ifndef FRONTEND +/* Copy of the same field of ControlFileData. */ +extern char encryption_verification[]; +#endif /* FRONTEND */ + +/* Do we have encryption_key and the encryption library initialized? */ +extern bool encryption_setup_done; + +#ifndef FRONTEND +typedef int (*read_encryption_key_cb) (void); +extern void read_encryption_key(read_encryption_key_cb read_char); +#endif /* FRONTEND */ + +extern void setup_encryption(void); +extern void sample_encryption(char *buf); +extern void encrypt_block(const char *input, char *output, Size size, + char *tweak, bool stream); +extern void encryption_error(bool fatal, char *message); + +#endif /* ENCRYPTION_H */ -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0002-Prepare-ServerLoop-for-the-next-patch.patch From 9246acbc3fd7ed00de831687c8e504f4d5c46f9e Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 02/17] Prepare ServerLoop for the next patch. This is only refactoring patch that tries to make the following one easier to read. It simply moves some code from ServerLoop() to ServerLoopCheckTimeouts(). --- src/backend/postmaster/postmaster.c | 110 ++++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 50 deletions(-) diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 688ad439ed..e8e3f8a4d3 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -403,6 +403,7 @@ static void BackendInitialize(Port *port); static void BackendRun(Port *port) pg_attribute_noreturn(); static void ExitPostmaster(int status) pg_attribute_noreturn(); static int ServerLoop(void); +static void ServerLoopCheckTimeouts(void); static int BackendStartup(Port *port); static int ProcessStartupPacket(Port *port, bool secure_done); static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options); @@ -1611,6 +1612,8 @@ DetermineSleepTime(struct timeval *timeout) } } +static time_t last_lockfile_recheck_time, last_touch_time; + /* * Main idle loop of postmaster * @@ -1621,8 +1624,6 @@ ServerLoop(void) { fd_set readmask; int nSockets; - time_t last_lockfile_recheck_time, - last_touch_time; last_lockfile_recheck_time = last_touch_time = time(NULL); @@ -1632,7 +1633,6 @@ ServerLoop(void) { fd_set rmask; int selres; - time_t now; /* * Wait for a connection request to arrive. @@ -1797,58 +1797,68 @@ ServerLoop(void) * us sleep at most that long; except for SIGKILL timeout which has * special-case logic there. */ - now = time(NULL); + ServerLoopCheckTimeouts(); + } +} - /* - * If we already sent SIGQUIT to children and they are slow to shut - * down, it's time to send them SIGKILL. This doesn't happen - * normally, but under certain conditions backends can get stuck while - * shutting down. This is a last measure to get them unwedged. - * - * Note we also do this during recovery from a process crash. - */ - if ((Shutdown >= ImmediateShutdown || (FatalError && !SendStop)) && - AbortStartTime != 0 && - (now - AbortStartTime) >= SIGKILL_CHILDREN_AFTER_SECS) - { - /* We were gentle with them before. Not anymore */ - TerminateChildren(SIGKILL); - /* reset flag so we don't SIGKILL again */ - AbortStartTime = 0; - } +/* + * Subroutine of ServerLoop() that checks if various timeouts elapsed, and if + * so, takes the appropriate action. + */ +static void +ServerLoopCheckTimeouts(void) +{ + time_t now = time(NULL); - /* - * Once a minute, verify that postmaster.pid hasn't been removed or - * overwritten. If it has, we force a shutdown. This avoids having - * postmasters and child processes hanging around after their database - * is gone, and maybe causing problems if a new database cluster is - * created in the same place. It also provides some protection - * against a DBA foolishly removing postmaster.pid and manually - * starting a new postmaster. Data corruption is likely to ensue from - * that anyway, but we can minimize the damage by aborting ASAP. - */ - if (now - last_lockfile_recheck_time >= 1 * SECS_PER_MINUTE) - { - if (!RecheckDataDirLockFile()) - { - ereport(LOG, - (errmsg("performing immediate shutdown because data directory lock file is invalid"))); - kill(MyProcPid, SIGQUIT); - } - last_lockfile_recheck_time = now; - } + /* + * If we already sent SIGQUIT to children and they are slow to shut down, + * it's time to send them SIGKILL. This doesn't happen normally, but + * under certain conditions backends can get stuck while shutting down. + * This is a last measure to get them unwedged. + * + * Note we also do this during recovery from a process crash. + */ + if ((Shutdown >= ImmediateShutdown || (FatalError && !SendStop)) && + AbortStartTime != 0 && + (now - AbortStartTime) >= SIGKILL_CHILDREN_AFTER_SECS) + { + /* We were gentle with them before. Not anymore */ + TerminateChildren(SIGKILL); + /* reset flag so we don't SIGKILL again */ + AbortStartTime = 0; + } - /* - * Touch Unix socket and lock files every 58 minutes, to ensure that - * they are not removed by overzealous /tmp-cleaning tasks. We assume - * no one runs cleaners with cutoff times of less than an hour ... - */ - if (now - last_touch_time >= 58 * SECS_PER_MINUTE) + /* + * Once a minute, verify that postmaster.pid hasn't been removed or + * overwritten. If it has, we force a shutdown. This avoids having + * postmasters and child processes hanging around after their database is + * gone, and maybe causing problems if a new database cluster is created + * in the same place. It also provides some protection against a DBA + * foolishly removing postmaster.pid and manually starting a new + * postmaster. Data corruption is likely to ensue from that anyway, but + * we can minimize the damage by aborting ASAP. + */ + if (now - last_lockfile_recheck_time >= 1 * SECS_PER_MINUTE) + { + if (!RecheckDataDirLockFile()) { - TouchSocketFiles(); - TouchSocketLockFiles(); - last_touch_time = now; + ereport(LOG, + (errmsg("performing immediate shutdown because data directory lock file is invalid"))); + kill(MyProcPid, SIGQUIT); } + last_lockfile_recheck_time = now; + } + + /* + * Touch Unix socket and lock files every 58 minutes, to ensure that they + * are not removed by overzealous /tmp-cleaning tasks. We assume no one + * runs cleaners with cutoff times of less than an hour ... + */ + if (now - last_touch_time >= 58 * SECS_PER_MINUTE) + { + TouchSocketFiles(); + TouchSocketLockFiles(); + last_touch_time = now; } } -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0003-Teach-postmaster-to-accept-encryption-key.patch From af9da6990b09276e65054d0d341f758f928e0fc6 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 03/17] Teach postmaster to accept encryption key This patch teaches postmaster to accept encryption key (as a special kind of startup packet) before recovery starts. In fact the key is received by a backend that postmaster launches if a client tries to initiate connection while the postmaster is in PM_INIT state. At this stage, only the key is received, whereas regular connection attempt ends up with FATAL: the database system is starting up If postmaster receives no key for some time, it raises this error and exits: FATAL: Encryption key not received. With this patch, user can start postmaster as usual: postgres -D data and use pg_keytool utility in another session to send the key: echo 71916b8b93063c280e4e0ee646395f450bda4f1bac85c57bc3a6afd168f3ab47 | pg_keytool -s --- src/backend/postmaster/postmaster.c | 211 +++++++++++++++++++++++++++++++++- src/backend/storage/file/encryption.c | 34 ++++++ src/backend/storage/ipc/ipci.c | 2 + src/bin/Makefile | 1 + src/bin/pg_keytool/.gitignore | 1 + src/bin/pg_keytool/Makefile | 31 +++++ src/bin/pg_keytool/pg_keytool.c | 199 ++++++++++++++++++++++++++++++++ src/fe_utils/encryption.c | 101 ++++++++++++++++ src/include/fe_utils/encryption.h | 2 + src/include/libpq/pqcomm.h | 24 ++++ src/include/storage/encryption.h | 26 +++++ 11 files changed, 628 insertions(+), 4 deletions(-) create mode 100644 src/bin/pg_keytool/.gitignore create mode 100644 src/bin/pg_keytool/Makefile create mode 100644 src/bin/pg_keytool/pg_keytool.c diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index e8e3f8a4d3..47f7bc93f7 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -117,6 +117,7 @@ #include "postmaster/syslogger.h" #include "replication/logicallauncher.h" #include "replication/walsender.h" +#include "storage/encryption.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/pg_shmem.h" @@ -402,12 +403,13 @@ static void PostmasterStateMachine(void); static void BackendInitialize(Port *port); static void BackendRun(Port *port) pg_attribute_noreturn(); static void ExitPostmaster(int status) pg_attribute_noreturn(); -static int ServerLoop(void); +static int ServerLoop(bool receive_encryption_key); static void ServerLoopCheckTimeouts(void); static int BackendStartup(Port *port); static int ProcessStartupPacket(Port *port, bool secure_done); static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options); static void processCancelRequest(Port *port, void *pkt); +static void processEncryptionKey(void *pkt); static int initMasks(fd_set *rmask); static void report_fork_failure_to_client(Port *port, int errnum); static CAC_state canAcceptConnections(void); @@ -1365,6 +1367,41 @@ PostmasterMain(int argc, char *argv[]) AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STARTING); /* + * If encryption key is needed and we don't have it yet, call ServerLoop() + * in a restricted mode which allows a client application to send us the + * key. Regular client connections are not allowed yet. + */ + if (data_encrypted && !encryption_setup_done) + { + char sample[ENCRYPTION_SAMPLE_SIZE]; + + status = ServerLoop(true); + + /* No other return code should be seen here. */ + Assert(status == STATUS_OK); + + /* ServerLoop() shouldn't have exited otherwise. */ + Assert(encryption_key_shmem->initialized); + + /* + * Take a local copy of the key so that we don't have to receive it + * again if restarting after a crash. + */ + memcpy(encryption_key, encryption_key_shmem, ENCRYPTION_KEY_LENGTH); + + /* Finalize the setup. */ + setup_encryption(); + + /* Verify the key. */ + sample_encryption(sample); + if (memcmp(encryption_verification, sample, ENCRYPTION_SAMPLE_SIZE)) + ereport(FATAL, + (errmsg("invalid encryption key"), + errdetail("The passed encryption key does not match" + " database encryption key."))); + } + + /* * We're ready to rock and roll... */ StartupPID = StartupDataBase(); @@ -1375,7 +1412,7 @@ PostmasterMain(int argc, char *argv[]) /* Some workers may be scheduled to start now */ maybe_start_bgworkers(); - status = ServerLoop(); + status = ServerLoop(false); /* * ServerLoop probably shouldn't ever return, but if it does, close down. @@ -1615,18 +1652,37 @@ DetermineSleepTime(struct timeval *timeout) static time_t last_lockfile_recheck_time, last_touch_time; /* + * The maximum amount of time postmaster can wait for encryption key if the + * cluster is encrypted. It should be shorter than the time pg_ctl waits for + * postgres to start (60 seconds) and should not be too long so that failure + * to specify encryption key command is recognized soon. + */ +#define MAX_WAIT_FOR_KEY_SECS 20 + +/* When should waiting for encryption key end. */ +static TimestampTz wait_for_keys_until = 0; + +/* * Main idle loop of postmaster * + * If receive_encryption_key is true, only launch backend(s) to receive + * cluster encryption key and stop as soon as the key is in shared memory. + * * NB: Needs to be called with signals blocked */ static int -ServerLoop(void) +ServerLoop(bool receive_encryption_key) { fd_set readmask; int nSockets; last_lockfile_recheck_time = last_touch_time = time(NULL); + /* Compute wait_for_keys_until if needed and not done yet. */ + if (receive_encryption_key && wait_for_keys_until == 0) + wait_for_keys_until = PgStartTime + + MAX_WAIT_FOR_KEY_SECS * USECS_PER_SEC; + nSockets = initMasks(&readmask); for (;;) @@ -1663,6 +1719,27 @@ ServerLoop(void) /* Needs to run with blocked signals! */ DetermineSleepTime(&timeout); + /* + * If waiting for the encryption key, make sure + * MAX_WAIT_FOR_KEY_SECS is not exceeded. + */ + if (receive_encryption_key) + { + TimestampTz timeout_tz, timeout_end;; + + timeout_tz = timeout.tv_sec * USECS_PER_SEC + + timeout.tv_usec; + timeout_end = GetCurrentTimestamp() + timeout_tz; + + if (timeout_end > wait_for_keys_until) + { + timeout_tz -= (timeout_end - wait_for_keys_until); + + timeout.tv_sec = timeout_tz / USECS_PER_SEC; + timeout.tv_usec = timeout_tz % USECS_PER_SEC; + } + } + PG_SETMASK(&UnBlockSig); selres = select(nSockets, &rmask, NULL, NULL, &timeout); @@ -1714,6 +1791,33 @@ ServerLoop(void) } } + /* + * If only waiting for the encryption key, it's too early to launch + * the other processes. + */ + if (receive_encryption_key) + { + /* The regular checks should take place though. */ + ServerLoopCheckTimeouts(); + + /* Done if the key has been delivered. */ + if (encryption_key_shmem->initialized) + return STATUS_OK; + + /* + * Do not wait for the key forever. One problem we solve here is + * that pg_ctl (or custom script that starts postgres) does not + * have to check whether the cluster is encrypted. If DBA forgets + * to pass the encryption key command, he'll simply see the + * startup to fail and the appropriate message to appear in the + * server log. + */ + if (GetCurrentTimestamp() >= wait_for_keys_until) + ereport(FATAL, (errmsg("Encryption key not received."))); + + continue; + } + /* If we have lost the log collector, try to start a new one */ if (SysLoggerPID == 0 && Logging_collector) SysLoggerPID = SysLogger_Start(); @@ -1990,6 +2094,13 @@ ProcessStartupPacket(Port *port, bool secure_done) return STATUS_ERROR; } + if (proto == ENCRYPTION_KEY_MSG_CODE && data_encrypted) + { + processEncryptionKey(buf); + /* Not really an error, but we don't want to proceed further */ + return STATUS_ERROR; + } + if (proto == NEGOTIATE_SSL_CODE && !secure_done) { char SSLok; @@ -2394,6 +2505,77 @@ processCancelRequest(Port *port, void *pkt) } /* + * Process a message from backend that contains the encryption key. + * + * Currently there's no guarantee that only a single backend tries to deliver + * the key, but it's not worth trying to synchronize the access. If multiple + * callers try to process the same message, nothing should get broken. If some + * caller is delivering a wrong key (e.g. due to misconfiguration of another + * cluster), it does not help if it waits until processing of the correct key + * is finished. + * + * If the key gets messed up, the worst case is that the server fails to start + * up. (No data corruption is expected.) + * + * Client should not expect any response to this request: failure on client + * side indicates either broken connection, wrong key, bug in + * send_key_to_postmaster() or repeated call of the client - nothing to be + * handled easily by client code. Most of these failures need attention of the + * DBA. + */ +void +processEncryptionKey(void *pkt) +{ + EncryptionKeyMsg *msg = (EncryptionKeyMsg *) pkt; + + /* Backend to receive the key should not be launched otherwise. */ + Assert(data_encrypted); + + /* + * The checks below would fire in this case too, but this error message is + * clearer for the case like this. + */ + if (pmState != PM_INIT) + { + ereport(COMMERROR, + (errmsg("encryption key can only be setup during startup"), + errhint("Check if the cluster is encrypted."))); + return; + } + + /* + * Someone else already initialized the key. This is not fatal but seems + * to indicate misconfiguration which DBA should be aware of. Check also + * encryption_setup_done because shared memory could have been reset. As + * postmaster has a local copy of the key, the new key should not be used, + * but it'd be inconsistent if we didn't complain in this special case. + */ + if (encryption_key_shmem->initialized || encryption_setup_done) + { + ereport(COMMERROR, + (errmsg("received encryption key more than once"))); + return; + } + + if (msg->version != 1) + { + ereport(COMMERROR, + (errmsg("unexpected version of encryption key message %d", + msg->version))); + return; + } + + memcpy(encryption_key_shmem->data, msg->data, ENCRYPTION_KEY_LENGTH); + /* + * XXX Is memory barrier needed here to make sure that the copying is + * done? + */ + encryption_key_shmem->initialized = true; + + ereport(DEBUG1, (errmsg_internal("encryption key received"))); +} + +/* * canAcceptConnections --- check to see if database state allows connections. */ static CAC_state @@ -2418,7 +2600,8 @@ canAcceptConnections(void) else if (Shutdown > NoShutdown) return CAC_SHUTDOWN; /* shutdown is pending */ else if (!FatalError && - (pmState == PM_STARTUP || + ((data_encrypted && pmState == PM_INIT) || + pmState == PM_STARTUP || pmState == PM_RECOVERY)) return CAC_STARTUP; /* normal startup */ else if (!FatalError && @@ -2761,6 +2944,16 @@ pmdie(SIGNAL_ARGS) pmState = (pmState == PM_RUN) ? PM_WAIT_BACKUP : PM_WAIT_READONLY; } + else if (pmState == PM_INIT && data_encrypted && + !encryption_setup_done) + { + /* + * Only backends to receive encryption key may be active now, + * and these should not be involved in any transactions. + */ + SignalSomeChildren(SIGTERM, BACKEND_TYPE_NORMAL); + pmState = PM_WAIT_BACKENDS; + } /* * Now wait for online backup mode to end and backends to exit. If @@ -2828,6 +3021,16 @@ pmdie(SIGNAL_ARGS) signal_child(WalWriterPID, SIGTERM); pmState = PM_WAIT_BACKENDS; } + else if (pmState == PM_INIT && data_encrypted && + !encryption_setup_done) + { + /* + * Only backends to receive encryption key may be active now, + * and these should not be involved in any transactions. + */ + SignalSomeChildren(SIGTERM, BACKEND_TYPE_NORMAL); + pmState = PM_WAIT_BACKENDS; + } /* * Now wait for backends to exit. If there are none, diff --git a/src/backend/storage/file/encryption.c b/src/backend/storage/file/encryption.c index 629138fae7..1eda334762 100644 --- a/src/backend/storage/file/encryption.c +++ b/src/backend/storage/file/encryption.c @@ -53,6 +53,10 @@ EVP_CIPHER_CTX *ctx_encrypt, unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; +#ifndef FRONTEND +ShmemEncryptionKey *encryption_key_shmem = NULL; +#endif /* FRONTEND */ + bool data_encrypted = false; char encryption_verification[ENCRYPTION_SAMPLE_SIZE]; @@ -66,6 +70,36 @@ static void evp_error(void); #ifndef FRONTEND /* + * Report space needed for our shared memory area + */ +Size +EncryptionShmemSize(void) +{ + return sizeof(ShmemEncryptionKey); +} + +/* + * Initialize our shared memory area + */ +void +EncryptionShmemInit(void) +{ + bool found; + + encryption_key_shmem = ShmemInitStruct("Cluster Encryption Key", + EncryptionShmemSize(), + &found); + if (!IsUnderPostmaster) + { + Assert(!found); + + encryption_key_shmem->initialized = false; + } + else + Assert(found); +} + +/* * Read encryption key in hexadecimal form from stdin and store it in * encryption_key variable. */ diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index d7d733530f..771bf30c63 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -147,6 +147,7 @@ CreateSharedMemoryAndSemaphores(int port) size = add_size(size, BTreeShmemSize()); size = add_size(size, SyncScanShmemSize()); size = add_size(size, AsyncShmemSize()); + size = add_size(size, EncryptionShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -263,6 +264,7 @@ CreateSharedMemoryAndSemaphores(int port) BTreeShmemInit(); SyncScanShmemInit(); AsyncShmemInit(); + EncryptionShmemInit(); #ifdef EXEC_BACKEND diff --git a/src/bin/Makefile b/src/bin/Makefile index 903e58121f..b05a8bbb4e 100644 --- a/src/bin/Makefile +++ b/src/bin/Makefile @@ -22,6 +22,7 @@ SUBDIRS = \ pg_controldata \ pg_ctl \ pg_dump \ + pg_keytool \ pg_resetwal \ pg_rewind \ pg_test_fsync \ diff --git a/src/bin/pg_keytool/.gitignore b/src/bin/pg_keytool/.gitignore new file mode 100644 index 0000000000..249876a6ed --- /dev/null +++ b/src/bin/pg_keytool/.gitignore @@ -0,0 +1 @@ +/pg_keytool diff --git a/src/bin/pg_keytool/Makefile b/src/bin/pg_keytool/Makefile new file mode 100644 index 0000000000..6da0631f32 --- /dev/null +++ b/src/bin/pg_keytool/Makefile @@ -0,0 +1,31 @@ +# src/bin/pg_keysetup/Makefile + +PGFILEDESC = "pg_keytool - handle cluster encryption key" +PGAPPICON=win32 + +subdir = src/bin/pg_keytool +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = pg_keytool.o $(RMGRDESCOBJS) $(WIN32RES) + +override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS) +override CPPFLAGS := -DFRONTEND $(CPPFLAGS) +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) + +all: pg_keytool + +pg_keytool: $(OBJS) | submake-libpgport + $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) + +install: all installdirs + $(INSTALL_PROGRAM) pg_keytool$(X) '$(DESTDIR)$(bindir)/pg_keytool$(X)' + +installdirs: + $(MKDIR_P) '$(DESTDIR)$(bindir)' + +uninstall: + rm -f '$(DESTDIR)$(bindir)/pg_keytool$(X)' + +clean distclean maintainer-clean: + rm -f pg_keytool$(X) $(OBJS) encryption.c diff --git a/src/bin/pg_keytool/pg_keytool.c b/src/bin/pg_keytool/pg_keytool.c new file mode 100644 index 0000000000..322625da41 --- /dev/null +++ b/src/bin/pg_keytool/pg_keytool.c @@ -0,0 +1,199 @@ +/*------------------------------------------------------------------------- + * + * pg_keytool.c - Handle cluster encryption key. + * + * Copyright (c) 2013-2019, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/bin/pg_keytool/pg_keytool.c + *------------------------------------------------------------------------- + */ +/* + * TODO Adopt the new frontend logging API, after some things are clarified: + * https://www.postgresql.org/message-id/1939.1560773970%40localhost + */ +#define FRONTEND 1 +#include "postgres.h" + +#include +#include + +#include "common/fe_memutils.h" +#include "common/logging.h" +#include "fe_utils/encryption.h" +#include "libpq-fe.h" +#include "libpq-int.h" +#include "libpq/pqcomm.h" +#include "port/pg_crc32c.h" +#include "storage/encryption.h" +#include "getopt_long.h" + +#ifdef USE_ENCRYPTION +static const char *progname; + +unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; + +static void +usage(const char *progname) +{ + const char *env; + + printf(_("%s is a tool to handle cluster encryption key.\n\n"), + progname); + printf(_("Usage:\n")); + printf(_(" %s [OPTION]...\n"), progname); + printf(_("\nOptions:\n")); + /* Display default host */ + env = getenv("PGHOST"); + printf(_(" -h, --host=HOSTNAME database server host or socket directory (default: \"%s\")\n"), + env ? env : _("local socket")); + /* Display default port */ + env = getenv("PGPORT"); + printf(_(" -p, --port=PORT database server port (default: \"%s\")\n"), + env ? env : DEF_PGPORT_STR); + printf(_(" -s, send output to database server\n")); + printf(_(" -?, --help show this help, then exit\n\n")); + printf(_("Key is read from stdin and sent either to stdout or to PostgreSQL server being started\n")); +} +#endif /* USE_ENCRYPTION */ + +int +main(int argc, char **argv) +{ +/* + * If no encryption library is linked, let the utility fail immediately. It'd + * be weird if we reported incorrect usage just to say later that no useful + * work can be done anyway. + */ +#ifdef USE_ENCRYPTION + int c; + char *host = NULL; + char *port_str = NULL; + bool to_server = false; + int i, n; + int optindex; + char key_chars[ENCRYPTION_KEY_CHARS]; + + static struct option long_options[] = + { + {"pgdata", required_argument, NULL, 'D'}, + {"host", required_argument, NULL, 'h'}, + {"port", required_argument, NULL, 'p'}, + {NULL, 0, NULL, 0} + }; + + pg_logging_init(argv[0]); + progname = get_progname(argv[0]); + + if (argc > 1) + { + if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0) + { + usage(progname); + exit(0); + } + if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0) + { + puts("pg_keytool (PostgreSQL) " PG_VERSION); + exit(0); + } + } + + while ((c = getopt_long(argc, argv, "h:p:s", + long_options, &optindex)) != -1) + { + switch (c) + { + case 'h': + host = pg_strdup(optarg); + break; + + case 'p': + port_str = pg_strdup(optarg); + break; + + case 's': + to_server = true; + break; + + case '?': + /* Actual help option given */ + if (strcmp(argv[optind - 1], "-?") == 0) + { + usage(progname); + exit(EXIT_SUCCESS); + } + + default: + pg_log_error("Try \"%s --help\" for more information.", progname); + exit(1); + } + } + + /* Complain if any arguments remain */ + if (optind < argc) + { + fprintf(stderr, _("%s: too many command-line arguments (first is \"%s\")\n"), + progname, argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + if ((host || port_str) && !to_server) + { + pg_log_error("host and port can only be passed along with the -s option"); + exit(1); + } + + /* Read the key. */ + n = 0; + /* Key length in characters (two characters per hexadecimal digit) */ + while ((c = getchar()) != EOF && c != '\n') + { + if (n >= ENCRYPTION_KEY_CHARS) + { + pg_log_error("The key is too long"); + exit(EXIT_FAILURE); + } + + key_chars[n++] = c; + } + + if (n < ENCRYPTION_KEY_CHARS) + { + pg_log_error("The key is too short"); + exit(EXIT_FAILURE); + } + + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + { + if (sscanf(key_chars + 2 * i, "%2hhx", encryption_key + i) == 0) + { + pg_log_error("Invalid character in encryption key at position %d", + 2 * i); + exit(EXIT_FAILURE); + } + } + + /* + * Send the encryption key either to stdout or to server. + */ + if (!to_server) + { + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + printf("%.2x", encryption_key[i]); + printf("\n"); + } + else + { + if (!send_key_to_postmaster(host, port_str, encryption_key)) + pg_log_error("could not send encryption key to server"); + } + +#else + pg_log_fatal(ENCRYPTION_NOT_SUPPORTED_MSG); + exit(EXIT_FAILURE); +#endif /* USE_ENCRYPTION */ + return 0; +} diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c index dab1435014..ca37c9f373 100644 --- a/src/fe_utils/encryption.c +++ b/src/fe_utils/encryption.c @@ -88,3 +88,104 @@ run_encryption_key_command(unsigned char *encryption_key) pfree(buf); pclose(fp); } + +/* + * Send the contents of encryption_key in the form of special startup packet + * to a server that is being started. + * + * Returns true if we could send the message and false if not, however even + * success does not guarantee that server started up - caller should + * eventually test server connection himself. + * + * TODO Windows stuff? + */ +bool +send_key_to_postmaster(const char *host, const char *port, + const unsigned char *encryption_Key) +{ + const char **keywords = pg_malloc0(3 * sizeof(*keywords)); + const char **values = pg_malloc0(3 * sizeof(*values)); + int i; + PGconn *conn = NULL; + int sock = -1; + EncryptionKeyMsg message; + int msg_size, packet_size; + char *packet = NULL; + bool res = true; + +/* How many seconds we can wait for the postmaster to receive the key. */ +#define SEND_ENCRYPT_KEY_TIMEOUT 60 + + if (host) + { + keywords[0] = "host"; + values[0] = host; + } + keywords[1] = "port"; + values[1] = port; + + /* Compose the message. */ + message.encryptionKeyCode = pg_hton32(ENCRYPTION_KEY_MSG_CODE); + message.version = 1; + memcpy(message.data, encryption_key, ENCRYPTION_KEY_LENGTH); + msg_size = offsetof(EncryptionKeyMsg, data) + ENCRYPTION_KEY_LENGTH; + + packet_size = msg_size + 4; + packet = (char *) palloc(packet_size); + *((int32 *) packet) = pg_hton32(packet_size); + memcpy(packet + 4, &message, msg_size); + + /* + * Although we don't expect the server to accept regular libpq messages, + * we try to get at least a valid socket. + */ + for (i = 0; i < SEND_ENCRYPT_KEY_TIMEOUT + 1; i++) + { + if (i > 0) + /* Sleep for 1 second. */ + pg_usleep(1000000L); + + if (conn) + { + PQfinish(conn); + conn = NULL; + } + + conn = PQconnectStartParams(keywords, values, false); + if (conn == NULL) + continue; + + sock = PQsocket(conn); + /* Cannot send the key if there's no valid socket. */ + if (sock == -1) + continue; + + /* Non-blocking write would only make this simple case tricky. */ + if (!pg_set_block(sock)) + continue; + + retry: + /* + * Send the packet. Here we need to use low level API because the server + * does is not fully up so libpq cannot be used properly. + */ + if (send(sock, (char *) packet, packet_size, 0) != packet_size) + { + if (SOCK_ERRNO == EINTR) + /* Interrupted system call - we'll just try again */ + goto retry; + continue; + } + else + /* Success */ + break; + } + + pg_free(keywords); + pg_free(values); + if (conn) + PQfinish(conn); + pfree(packet); + + return res; +} diff --git a/src/include/fe_utils/encryption.h b/src/include/fe_utils/encryption.h index 69e3b2f756..7307557ed6 100644 --- a/src/include/fe_utils/encryption.h +++ b/src/include/fe_utils/encryption.h @@ -16,3 +16,5 @@ extern char *encryption_key_command; extern void run_encryption_key_command(unsigned char *encryption_key); +extern bool send_key_to_postmaster(const char *host, const char *port, + const unsigned char *encryption_Key); diff --git a/src/include/libpq/pqcomm.h b/src/include/libpq/pqcomm.h index baf6a4b6c0..09822801c7 100644 --- a/src/include/libpq/pqcomm.h +++ b/src/include/libpq/pqcomm.h @@ -23,6 +23,8 @@ #endif #include +#include "storage/encryption.h" + #ifdef HAVE_STRUCT_SOCKADDR_STORAGE #ifndef HAVE_STRUCT_SOCKADDR_STORAGE_SS_FAMILY @@ -205,4 +207,26 @@ typedef struct CancelRequestPacket #define NEGOTIATE_SSL_CODE PG_PROTOCOL(1234,5679) #define NEGOTIATE_GSS_CODE PG_PROTOCOL(1234,5680) +/* Client can also send cluster encryption key to the postmaster. */ +#define ENCRYPTION_KEY_MSG_CODE PG_PROTOCOL(1234,5681) + +/* + * Message containing the encryption key, received by postmaster during + * startup. + * + * TODO Consider adding extension identifier and key_length field so that + * postmaster can also receove extension-specific keys. Extension that needs a + * key would call a function from _PG_init() that tells postmaster where to + * store the key and how long the key is. (Should also postgres in standalone + * mode accept keys for extensions?) + */ +typedef struct EncryptionKeyMsg +{ + /* Note this integer field is stored in network byte order! */ + MsgType encryptionKeyCode; /* code to identify a key message */ + + unsigned char version; /* message format version. */ + char data[ENCRYPTION_KEY_LENGTH]; /* the key itself. */ +} EncryptionKeyMsg; + #endif /* PQCOMM_H */ diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h index 8479b76a34..4f7b96d4f3 100644 --- a/src/include/storage/encryption.h +++ b/src/include/storage/encryption.h @@ -60,6 +60,29 @@ typedef enum CipherKind /* Key to encrypt / decrypt data. */ extern unsigned char encryption_key[]; +#ifndef FRONTEND +/* + * Space for the encryption key in shared memory. Backend that receives the + * key during startup stores it here so postmaster can eventually take a local + * copy. + * + * We cannot protect the "initialized" field with a spinlock because the data + * will be accessed by postmaster, but it should not be necessary for bool + * type. Furthermore, synchronization is not really critical here, see + * processEncryptionKey(). + */ +typedef struct ShmemEncryptionKey +{ + char data[ENCRYPTION_KEY_LENGTH]; /* the key */ + bool initialized; /* received the key? */ +} ShmemEncryptionKey; + +/* + * Encryption key in the shared memory. + */ +extern ShmemEncryptionKey *encryption_key_shmem; +#endif /* FRONTEND */ + /* * The encrypted data is a series of blocks of size * ENCRYPTION_BLOCK. Currently we use the EVP_aes_256_xts implementation. Make @@ -86,6 +109,9 @@ extern char encryption_verification[]; extern bool encryption_setup_done; #ifndef FRONTEND +extern Size EncryptionShmemSize(void); +extern void EncryptionShmemInit(void); + typedef int (*read_encryption_key_cb) (void); extern void read_encryption_key(read_encryption_key_cb read_char); #endif /* FRONTEND */ -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0004-Minor-refactoring-of-pg_ctl.c.patch From bdc832086b1d98aec69ab6d3b6c3ae6c09560d05 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 04/17] Minor refactoring of pg_ctl.c. Added get_config_variable() function to pg_ctl.c because the next patch of the series will need to retrieve some more configuration variables from postgresql.conf. --- src/bin/pg_ctl/pg_ctl.c | 69 ++++++++++++++++++++++++++++++++--------------- src/fe_utils/encryption.c | 4 +++ 2 files changed, 52 insertions(+), 21 deletions(-) diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c index a10bc8d545..71741ffc81 100644 --- a/src/bin/pg_ctl/pg_ctl.c +++ b/src/bin/pg_ctl/pg_ctl.c @@ -133,6 +133,7 @@ static void do_logrotate(void); static void do_kill(pgpid_t pid); static void print_msg(const char *msg); static void adjust_data_dir(void); +static char *get_config_variable(const char *var_name, size_t res_size); #ifdef WIN32 #if (_MSC_VER >= 1800) @@ -2172,28 +2173,46 @@ set_starttype(char *starttypeopt) static void adjust_data_dir(void) { - char cmd[MAXPGPATH], - filename[MAXPGPATH], - *my_exec_path; - FILE *fd; + char conf_path[MAXPGPATH]; + char *filename; + FILE *fd; /* do nothing if we're working without knowledge of data dir */ if (pg_config == NULL) return; - /* If there is no postgresql.conf, it can't be a config-only dir */ - snprintf(filename, sizeof(filename), "%s/postgresql.conf", pg_config); - if ((fd = fopen(filename, "r")) == NULL) + /* If PG_VERSION exists, it can't be a config-only dir */ + snprintf(conf_path, sizeof(conf_path), "%s/PG_VERSION", pg_config); + if ((fd = fopen(conf_path, "r")) != NULL) return; - fclose(fd); - /* If PG_VERSION exists, it can't be a config-only dir */ - snprintf(filename, sizeof(filename), "%s/PG_VERSION", pg_config); - if ((fd = fopen(filename, "r")) != NULL) + filename = get_config_variable("data_directory", MAXPGPATH); + + if (filename) { - fclose(fd); - return; + free(pg_data); + pg_data = filename; + canonicalize_path(pg_data); } +} + +/* + * Retrieve value of configuration variable from configuration file. + */ +static char * +get_config_variable(const char *var_name, size_t res_size) +{ + char cmd[MAXPGPATH], + filename[MAXPGPATH], + *my_exec_path; + char *result; + FILE *fd; + + /* If there is no postgresql.conf, the data dir is not useful. */ + snprintf(filename, sizeof(filename), "%s/postgresql.conf", pg_config); + if ((fd = fopen(filename, "r")) == NULL) + return NULL; + fclose(fd); /* Must be a configuration directory, so find the data directory */ @@ -2204,27 +2223,35 @@ adjust_data_dir(void) my_exec_path = pg_strdup(exec_path); /* it's important for -C to be the first option, see main.c */ - snprintf(cmd, MAXPGPATH, "\"%s\" -C data_directory %s%s", + snprintf(cmd, MAXPGPATH, "\"%s\" -C %s %s%s", my_exec_path, + var_name, pgdata_opt ? pgdata_opt : "", post_opts ? post_opts : ""); + result = pg_malloc(res_size); fd = popen(cmd, "r"); - if (fd == NULL || fgets(filename, sizeof(filename), fd) == NULL) + if (fd == NULL || fgets(result, res_size, fd) == NULL) { - write_stderr(_("%s: could not determine the data directory using command \"%s\"\n"), progname, cmd); + write_stderr(_("%s: could not determine the value of \"%s\" \"%s\"\n"), + progname, var_name, cmd); + free(result); exit(1); } pclose(fd); free(my_exec_path); /* Remove trailing newline */ - if (strchr(filename, '\n') != NULL) - *strchr(filename, '\n') = '\0'; + if (strchr(result, '\n') != NULL) + *strchr(result, '\n') = '\0'; - free(pg_data); - pg_data = pg_strdup(filename); - canonicalize_path(pg_data); + if (strlen(result) == 0) + { + pg_free(result); + result = NULL; + } + + return result; } diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c index ca37c9f373..e888d4ae34 100644 --- a/src/fe_utils/encryption.c +++ b/src/fe_utils/encryption.c @@ -31,6 +31,10 @@ char *encryption_key_command = NULL; /* * Run the command that is supposed to generate encryption key and store it * where encryption_key points to. + * + * TODO Replace pattern (e.g. %D) in the command with data directory so that + * DBA knows for which cluster he enters the password. That should also make + * the use of pg_keytool in the command easier. */ void run_encryption_key_command(unsigned char *encryption_key) -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0005-Teach-pg_ctl-to-send-the-encryption-key-to-postmaste.patch From aa5e08b1f545dd65bffe1ff473e6b890e4831eda Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 05/17] Teach pg_ctl to send the encryption key to postmaster. With this patch the server can be started this way: pg_ctl -D data -K "echo 71916b8b93063c280e4e0ee646395f450bda4f1bac85c57bc3a6afd168f3ab47" start --- src/bin/pg_ctl/Makefile | 2 +- src/bin/pg_ctl/pg_ctl.c | 117 ++++++++++++++++++++++++++++++++++++++++++++-- src/fe_utils/encryption.c | 4 -- 3 files changed, 115 insertions(+), 8 deletions(-) diff --git a/src/bin/pg_ctl/Makefile b/src/bin/pg_ctl/Makefile index 83cbf97ed8..b9f755d93a 100644 --- a/src/bin/pg_ctl/Makefile +++ b/src/bin/pg_ctl/Makefile @@ -20,9 +20,9 @@ include $(top_builddir)/src/Makefile.global # but let's not pull that in on platforms where we don't need it. ifeq ($(PORTNAME), win32) override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS) -LDFLAGS_INTERNAL += $(libpq_pgport) SUBMAKE_LIBPQ := submake-libpq endif +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) OBJS= pg_ctl.o $(WIN32RES) diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c index 71741ffc81..7e414ac048 100644 --- a/src/bin/pg_ctl/pg_ctl.c +++ b/src/bin/pg_ctl/pg_ctl.c @@ -27,6 +27,7 @@ #include "common/controldata_utils.h" #include "common/file_perm.h" #include "common/logging.h" +#include "fe_utils/encryption.h" #include "getopt_long.h" #include "utils/pidfile.h" @@ -106,6 +107,12 @@ static char logrotate_file[MAXPGPATH]; static volatile pgpid_t postmasterPID = -1; +/* + * Define encryption_key locally rather that linking to storage/encryption.c + * only because of this one variable. + */ +unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; + #ifdef WIN32 static DWORD pgctl_start_type = SERVICE_AUTO_START; static SERVICE_STATUS status; @@ -134,6 +141,8 @@ static void do_kill(pgpid_t pid); static void print_msg(const char *msg); static void adjust_data_dir(void); static char *get_config_variable(const char *var_name, size_t res_size); +static char *get_first_csv_item(char *csv_list); +static void get_postmaster_address(char **host_p, char **port_str_p); #ifdef WIN32 #if (_MSC_VER >= 1800) @@ -825,6 +834,8 @@ do_start(void) { pgpid_t old_pid = 0; pgpid_t pm_pid; + char *host = NULL; + char *port_str = NULL; if (ctl_command != RESTART_COMMAND) { @@ -864,8 +875,36 @@ do_start(void) } #endif + if (encryption_key_command) + { + /* + * If encryption key is needed, retrieve it before trying to start + * postmaster. + */ + run_encryption_key_command(encryption_key); + + /* + * Where should the key be sent? + * + * Retrieve the info now because the time the retrieval will take + * should not overlap with waiting for the startup completion below. + */ + get_postmaster_address(&host, &port_str); + } + pm_pid = start_postmaster(); + if (encryption_key_command) + { + /* Send the key to the postmaster */ + if (!send_key_to_postmaster(host, port_str, encryption_key)) + { + write_stderr(_("%s: could not send encryption key to postmaster\n"), + progname); + exit(1); + } + } + if (do_wait) { /* @@ -2025,10 +2064,10 @@ do_help(void) printf(_("%s is a utility to initialize, start, stop, or control a PostgreSQL server.\n\n"), progname); printf(_("Usage:\n")); printf(_(" %s init[db] [-D DATADIR] [-s] [-o OPTIONS]\n"), progname); - printf(_(" %s start [-D DATADIR] [-l FILENAME] [-W] [-t SECS] [-s]\n" + printf(_(" %s start [-D DATADIR] [-l FILENAME] [-W] [-t SECS] [-K KEY-CMD] [-s]\n" " [-o OPTIONS] [-p PATH] [-c]\n"), progname); printf(_(" %s stop [-D DATADIR] [-m SHUTDOWN-MODE] [-W] [-t SECS] [-s]\n"), progname); - printf(_(" %s restart [-D DATADIR] [-m SHUTDOWN-MODE] [-W] [-t SECS] [-s]\n" + printf(_(" %s restart [-D DATADIR] [-m SHUTDOWN-MODE] [-W] [-t SECS] [-K KEY-CMD] [-s]\n" " [-o OPTIONS] [-c]\n"), progname); printf(_(" %s reload [-D DATADIR] [-s]\n"), progname); printf(_(" %s status [-D DATADIR]\n"), progname); @@ -2046,6 +2085,10 @@ do_help(void) #ifdef WIN32 printf(_(" -e SOURCE event source for logging when running as a service\n")); #endif +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command\n" + " command that returns encryption key\n\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -s, --silent only print errors, no informational messages\n")); printf(_(" -t, --timeout=SECS seconds to wait when using -w option\n")); printf(_(" -V, --version output version information, then exit\n")); @@ -2254,6 +2297,66 @@ get_config_variable(const char *var_name, size_t res_size) return result; } +/* + * Get the first item of comma-separated list or NULL if there's no valid + * item. + */ +static char * +get_first_csv_item(char *csv_list) +{ + char *start, *end, *result; + + start = csv_list; + /* First, skip the leading space. */ + while (isspace(*start)) + start++; + if (*start == '\0') + return NULL; + + end = start; + while (*end != '\0' && *end != ',' && !isspace(*end)) + end++; + if (end == start) + return NULL; + + result = pg_strdup(start); + + /* Trim the string if needed. */ + if (*end != '\0') + result[end - start] = '\0'; + + return result; +} + +/* + * Retrieve host name and port to which the encryption key should be + * sent. host can remain NULL, which will make libpq assume "localhost". + */ +static void +get_postmaster_address(char **host_p, char **port_str_p) +{ + char *socket_dirs; + + /* First, try to connect via the unix socket. */ + socket_dirs = get_config_variable("unix_socket_directories", + MAXPGPATH); + if (socket_dirs) + { + /* If there are multiple sockets, use the first one. */ + *host_p = get_first_csv_item(socket_dirs); + } + else + { + /* Let libpq assume "localhost".*/ + *host_p = NULL; + } + + /* + * The maximum length of the port number is 5 characters for the port + * number (65535 at maximum) + '\n' + terminating '\0'. + */ + *port_str_p = get_config_variable("port", 7); +} static DBState get_control_dbstate(void) @@ -2286,6 +2389,7 @@ main(int argc, char **argv) {"options", required_argument, NULL, 'o'}, {"silent", no_argument, NULL, 's'}, {"timeout", required_argument, NULL, 't'}, + {"encryption-key-command", required_argument, NULL, 'K'}, {"core-files", no_argument, NULL, 'c'}, {"wait", no_argument, NULL, 'w'}, {"no-wait", no_argument, NULL, 'W'}, @@ -2356,7 +2460,7 @@ main(int argc, char **argv) /* process command-line options */ while (optind < argc) { - while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:sS:t:U:wW", + while ((c = getopt_long(argc, argv, "cD:e:K:l:m:N:o:p:P:sS:t:U:wW", long_options, &option_index)) != -1) { switch (c) @@ -2382,6 +2486,9 @@ main(int argc, char **argv) case 'e': event_source = pg_strdup(optarg); break; + case 'K': + encryption_key_command = pg_strdup(optarg); + break; case 'l': log_file = pg_strdup(optarg); break; @@ -2558,6 +2665,10 @@ main(int argc, char **argv) umask(pg_mode_mask); } + if (encryption_key_command && ctl_command != + START_COMMAND && ctl_command != RESTART_COMMAND) + write_stderr(_("%s: ignoring the -K option, it's only useful for start or restart commands\n"), progname); + switch (ctl_command) { case INIT_COMMAND: diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c index e888d4ae34..ca37c9f373 100644 --- a/src/fe_utils/encryption.c +++ b/src/fe_utils/encryption.c @@ -31,10 +31,6 @@ char *encryption_key_command = NULL; /* * Run the command that is supposed to generate encryption key and store it * where encryption_key points to. - * - * TODO Replace pattern (e.g. %D) in the command with data directory so that - * DBA knows for which cluster he enters the password. That should also make - * the use of pg_keytool in the command easier. */ void run_encryption_key_command(unsigned char *encryption_key) -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0006-Allow-user-to-use-password-instead-of-encryption-key.patch From 0e853e3074afd029c665d14cab750cacdba149b9 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 06/17] Allow user to use password instead of encryption key. pg_keytool reads the password from stdin, uses "key derivation function (KDF) parameters" stored in the data directory to derive the key and writes the key to standard output. The cluster can be created this way initdb -D data -K "echo securepwd | pg_keytool -D data -w" and started either this way pg_ctl -D data -K "echo securepwd | pg_keytool -D data -w" start or this way postgres -D data and (in another session) echo securepwd | pg_keytool -D data -ws Both initdb and pg_ctl can substitute the data directory for %D pattern in the key encryption command. Thus user uses pg_keytool to derive the password, he can in fact say initdb -D data -K "echo securepwd | pg_keytool -D %D -w" and pg_ctl -D data -K "echo securepwd | pg_keytool -D %D -w" start respectively. --- src/bin/initdb/initdb.c | 26 +++- src/bin/pg_ctl/pg_ctl.c | 2 +- src/bin/pg_keytool/pg_keytool.c | 87 +++++++++--- src/fe_utils/encryption.c | 273 +++++++++++++++++++++++++++++++++++++- src/include/fe_utils/encryption.h | 8 +- src/include/storage/encryption.h | 8 ++ 6 files changed, 379 insertions(+), 25 deletions(-) diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index f2a582e975..bfa8f5fad4 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -3004,11 +3004,31 @@ initialize_data_directory(void) write_version_file(NULL); /* - * If the cluster will be encrypted, run the command to generate the - * encryption key. + * If the cluster will be encrypted, write the KDF file so that encryption + * key can be derived from password. */ if (encryption_key_command) - run_encryption_key_command(encryption_key); + { + /* + * XXX Since execution of encryption_key_command produce the key (as + * opposed to password), we don't know if the command received the key + * itself or a password. If DBA provided initdb with a key, he will + * never use password in the future (there was no KDF so far so the + * key could not be derived from password, and the password can hardly + * be derived from the key), so the KDF file may be useless. We don't + * have enough information to recognize this special case, so just + * initialize and write the KDF unconditionally. + */ + init_kdf(); + write_kdf_file(pg_data); + + /* + * The key command is allowed to use pg_keytool, which in turn needs + * the KDF parameters. The KDF parameters are now available so we can + * run the command. + */ + run_encryption_key_command(encryption_key, pg_data); + } /* Select suitable configuration settings */ set_null_conf(); diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c index 7e414ac048..e8e4eee162 100644 --- a/src/bin/pg_ctl/pg_ctl.c +++ b/src/bin/pg_ctl/pg_ctl.c @@ -881,7 +881,7 @@ do_start(void) * If encryption key is needed, retrieve it before trying to start * postmaster. */ - run_encryption_key_command(encryption_key); + run_encryption_key_command(encryption_key, pg_data); /* * Where should the key be sent? diff --git a/src/bin/pg_keytool/pg_keytool.c b/src/bin/pg_keytool/pg_keytool.c index 322625da41..c5151ad57c 100644 --- a/src/bin/pg_keytool/pg_keytool.c +++ b/src/bin/pg_keytool/pg_keytool.c @@ -43,6 +43,7 @@ usage(const char *progname) printf(_("Usage:\n")); printf(_(" %s [OPTION]...\n"), progname); printf(_("\nOptions:\n")); + printf(_(" -D, --pgdata=DATADIR data directory\n")); /* Display default host */ env = getenv("PGHOST"); printf(_(" -h, --host=HOSTNAME database server host or socket directory (default: \"%s\")\n"), @@ -52,8 +53,9 @@ usage(const char *progname) printf(_(" -p, --port=PORT database server port (default: \"%s\")\n"), env ? env : DEF_PGPORT_STR); printf(_(" -s, send output to database server\n")); + printf(_(" -w expect password on input, not a key\n")); printf(_(" -?, --help show this help, then exit\n\n")); - printf(_("Key is read from stdin and sent either to stdout or to PostgreSQL server being started\n")); + printf(_("Password or key is read from stdin. Key is sent to PostgreSQL server being started\n")); } #endif /* USE_ENCRYPTION */ @@ -69,9 +71,12 @@ main(int argc, char **argv) int c; char *host = NULL; char *port_str = NULL; + char *DataDir = NULL; bool to_server = false; + bool expect_password = false; int i, n; int optindex; + char password[ENCRYPTION_PWD_MAX_LENGTH]; char key_chars[ENCRYPTION_KEY_CHARS]; static struct option long_options[] = @@ -99,11 +104,15 @@ main(int argc, char **argv) } } - while ((c = getopt_long(argc, argv, "h:p:s", + while ((c = getopt_long(argc, argv, "h:D:p:sw", long_options, &optindex)) != -1) { switch (c) { + case 'D': + DataDir = optarg; + break; + case 'h': host = pg_strdup(optarg); break; @@ -116,6 +125,10 @@ main(int argc, char **argv) to_server = true; break; + case 'w': + expect_password = true; + break; + case '?': /* Actual help option given */ if (strcmp(argv[optind - 1], "-?") == 0) @@ -146,34 +159,78 @@ main(int argc, char **argv) exit(1); } - /* Read the key. */ + /* + * The KDF file is needed to derive the key from password, and this file + * is located in the data directory. + */ + if (expect_password && DataDir == NULL) + { + pg_log_error("%s: no data directory specified", progname); + pg_log_error("Try \"%s --help\" for more information.", progname); + exit(EXIT_FAILURE); + } + + /* + * Read the credentials (key or password). + */ n = 0; /* Key length in characters (two characters per hexadecimal digit) */ while ((c = getchar()) != EOF && c != '\n') { - if (n >= ENCRYPTION_KEY_CHARS) + if (!expect_password) { - pg_log_error("The key is too long"); - exit(EXIT_FAILURE); + if (n >= ENCRYPTION_KEY_CHARS) + { + pg_log_error("The key is too long"); + exit(EXIT_FAILURE); + } + + key_chars[n++] = c; } + else + { + if (n >= ENCRYPTION_PWD_MAX_LENGTH) + { + pg_log_error("The password is too long"); + exit(EXIT_FAILURE); + } - key_chars[n++] = c; + password[n++] = c; + } } - if (n < ENCRYPTION_KEY_CHARS) + /* If password was received, turn it into encryption key. */ + if (!expect_password) { - pg_log_error("The key is too short"); - exit(EXIT_FAILURE); - } + if (n < ENCRYPTION_KEY_CHARS) + { + pg_log_error("The key is too short"); + exit(EXIT_FAILURE); + } - for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + { + if (sscanf(key_chars + 2 * i, "%2hhx", encryption_key + i) == 0) + { + pg_log_error("Invalid character in encryption key at position %d", + 2 * i); + exit(EXIT_FAILURE); + } + } + } + else { - if (sscanf(key_chars + 2 * i, "%2hhx", encryption_key + i) == 0) + if (n < ENCRYPTION_PWD_MIN_LENGTH) { - pg_log_error("Invalid character in encryption key at position %d", - 2 * i); + pg_log_error("The password is too short"); exit(EXIT_FAILURE); } + + /* Read the KDF parameters. */ + read_kdf_file(DataDir); + + /* Run the KDF. */ + derive_key_from_password(encryption_key, password, n); } /* diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c index ca37c9f373..134b3bde9b 100644 --- a/src/fe_utils/encryption.c +++ b/src/fe_utils/encryption.c @@ -21,31 +21,294 @@ #include "common/file_perm.h" #include "common/logging.h" #include "fe_utils/encryption.h" -#include "storage/encryption.h" #include "libpq-fe.h" #include "libpq-int.h" #include "libpq/pqcomm.h" char *encryption_key_command = NULL; +#define KDF_PARAMS_FILE "global/kdf_params" +#define KDF_PARAMS_FILE_SIZE 512 + +/* + * Key derivation function. + */ +typedef enum KDFKind +{ + KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA = 0 +} KFDKind; + +typedef struct KDFParamsPBKDF2 +{ + unsigned long int niter; + unsigned char salt[ENCRYPTION_KDF_SALT_LEN]; +} KDFParamsPBKDF2; + +/* + * Parameters of the key derivation function. + * + * The parameters are generated by initdb and stored into a file, which is + * then read during PG startup. This is similar to storing various settings in + * pg_control. However an existing KDF file is read only, so it does not have + * to be stored in shared memory. + */ +typedef struct KDFParamsData +{ + KFDKind function; + + /* + * Function-specific parameters. + */ + union + { + KDFParamsPBKDF2 pbkdf2; + } data; + + /* CRC of all above ... MUST BE LAST! */ + pg_crc32c crc; +} KDFParamsData; + +extern KDFParamsData *KDFParams; + +/* + * Pointer to the KDF parameters. + */ +KDFParamsData *KDFParams = NULL; + +/* Initialize KDF file. */ +void +init_kdf(void) +{ + KDFParamsPBKDF2 *params; + struct timeval tv; + uint64 salt; + + /* + * The initialization should not be repeated. + */ + Assert(KDFParams == NULL); + + KDFParams = palloc0(KDF_PARAMS_FILE_SIZE); + KDFParams->function = KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA; + params = &KDFParams->data.pbkdf2; + + /* + * Currently we derive the salt in the same way as system identifier, + * however these two values are not supposed to match. XXX Is it worth the + * effort if initdb derives the system identifier, passes it to this + * function and also sends it to the bootstrap process? Not sure. + */ + gettimeofday(&tv, NULL); + salt = ((uint64) tv.tv_sec) << 32; + salt |= ((uint64) tv.tv_usec) << 12; + salt |= getpid() & 0xFFF; + + memcpy(params->salt, &salt, sizeof(uint64)); + params->niter = ENCRYPTION_KDF_NITER; +} + +/* + * Write KDFParamsData to file. + */ +void +write_kdf_file(char *dir) +{ + char path[MAXPGPATH]; + int fd; + + Assert(KDFParams != NULL); + + /* Account for both file separator and terminating NULL character. */ + if ((strlen(dir) + 1 + strlen(KDF_PARAMS_FILE) + 1) > MAXPGPATH) + { + pg_log_fatal("KDF directory is too long"); + exit(EXIT_FAILURE); + } + + snprintf(path, MAXPGPATH, "%s/%s", dir, KDF_PARAMS_FILE); + + /* Contents are protected with a CRC */ + INIT_CRC32C(KDFParams->crc); + COMP_CRC32C(KDFParams->crc, + (char *) KDFParams, + offsetof(KDFParamsData, crc)); + FIN_CRC32C(KDFParams->crc); + + fd = open(path, O_WRONLY | O_CREAT | PG_BINARY, + pg_file_create_mode); + if (fd < 0) + { + pg_log_fatal("could not create key derivation file \"%s\": %m", path); + exit(EXIT_FAILURE); + } + + if (write(fd, KDFParams, KDF_PARAMS_FILE_SIZE) != KDF_PARAMS_FILE_SIZE) + { + /* if write didn't set errno, assume problem is no disk space */ + if (errno == 0) + errno = ENOSPC; + pg_log_fatal("could not write to key derivation file \"%s\": %m", + path); + exit(EXIT_FAILURE); + } + + if (close(fd)) + { + pg_log_fatal("could not close key setup file: %m"); + exit(EXIT_FAILURE); + } +} + +/* + * Read KDFParamsData from file and store it in local memory. + * + * If dir is NULL, assume we're in the data directory. + * + * postmaster should call the function early enough for any other process to + * inherit valid pointer to the data. + */ +void +read_kdf_file(char *dir) +{ + pg_crc32c crc; + char path[MAXPGPATH]; + int fd; + + /* Account for both file separator and terminating NULL character. */ + if ((strlen(dir) + 1 + strlen(KDF_PARAMS_FILE) + 1) > MAXPGPATH) + { + pg_log_fatal("KDF directory is too long"); + exit(EXIT_FAILURE); + } + + snprintf(path, MAXPGPATH, "%s/%s", dir, KDF_PARAMS_FILE); + + KDFParams = palloc0(KDF_PARAMS_FILE_SIZE); + fd = open(path, O_RDONLY | PG_BINARY, S_IRUSR); + + if (fd < 0) + { + pg_log_fatal("could not open key setup file \"%s\": %m", path); + exit(EXIT_FAILURE); + } + + if (read(fd, KDFParams, sizeof(KDFParamsData)) != sizeof(KDFParamsData)) + { + pg_log_fatal("could not read from key setup file \"%s\": %m", path); + exit(EXIT_FAILURE); + } + + close(fd); + + /* Now check the CRC. */ + INIT_CRC32C(crc); + COMP_CRC32C(crc, + (char *) KDFParams, + offsetof(KDFParamsData, crc)); + FIN_CRC32C(crc); + + if (!EQ_CRC32C(crc, KDFParams->crc)) + { + pg_log_fatal("incorrect checksum in key setup file \"%s\"", path); + exit(EXIT_FAILURE); + } + + + if (KDFParams->function != KDF_OPENSSL_PKCS5_PBKDF2_HMAC_SHA) + { + pg_log_fatal("unsupported KDF function"); + exit(EXIT_FAILURE); + } +} + +/* + * Run the key derivation function and initialize encryption_key variable. + */ +void +derive_key_from_password(unsigned char *encryption_key, const char *password, + int len) +{ + KDFParamsPBKDF2 *params; + int rc; + + params = &KDFParams->data.pbkdf2; + rc = PKCS5_PBKDF2_HMAC(password, + len, + params->salt, + ENCRYPTION_KDF_SALT_LEN, + params->niter, + EVP_sha1(), + ENCRYPTION_KEY_LENGTH, + encryption_key); + + if (rc != 1) + { + pg_log_fatal("failed to derive key from password"); + exit(EXIT_FAILURE); + } +} + /* * Run the command that is supposed to generate encryption key and store it - * where encryption_key points to. + * where encryption_key points to. If valid string is passed for data_dir, + * it's used to replace '%D' pattern in the command. */ void -run_encryption_key_command(unsigned char *encryption_key) +run_encryption_key_command(unsigned char *encryption_key, char *data_dir) { FILE *fp; + char cmd[MAXPGPATH]; + char *sp, *dp, *endp; char *buf; int read_len, i, c; Assert(encryption_key_command != NULL && strlen(encryption_key_command) > 0); - fp = popen(encryption_key_command, "r"); + /* + * Replace %D pattern in the command with the actual data directory path. + */ + dp = cmd; + endp = cmd + MAXPGPATH - 1; + *endp = '\0'; + for (sp = encryption_key_command; *sp; sp++) + { + if (*sp == '%') + { + if (sp[1] == 'D') + { + if (data_dir == NULL) + { + pg_log_fatal("data directory is not known, %%D pattern cannot be replaced"); + exit(EXIT_FAILURE); + } + + sp++; + strlcpy(dp, data_dir, endp - dp); + make_native_path(dp); + dp += strlen(dp); + } + else if (dp < endp) + *dp++ = *sp; + else + break; + } + else + { + if (dp < endp) + *dp++ = *sp; + else + break; + } + } + *dp = '\0'; + + pg_log_debug("executing encryption key command \"%s\"", cmd); + + fp = popen(cmd, "r"); if (fp == NULL) { - pg_log_fatal("Failed to execute \"%s\"", encryption_key_command); + pg_log_fatal("Failed to execute \"%s\"", cmd); exit(EXIT_FAILURE); } diff --git a/src/include/fe_utils/encryption.h b/src/include/fe_utils/encryption.h index 7307557ed6..da302fe494 100644 --- a/src/include/fe_utils/encryption.h +++ b/src/include/fe_utils/encryption.h @@ -15,6 +15,12 @@ /* Executable to retrieve the encryption key. */ extern char *encryption_key_command; -extern void run_encryption_key_command(unsigned char *encryption_key); +extern void init_kdf(void); +extern void write_kdf_file(char *dir); +extern void read_kdf_file(char *dir); +extern void derive_key_from_password(unsigned char *encryption_key, + const char *password, int len); +extern void run_encryption_key_command(unsigned char *encryption_key, + char *data_dir); extern bool send_key_to_postmaster(const char *host, const char *port, const unsigned char *encryption_Key); diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h index 4f7b96d4f3..72bcae0972 100644 --- a/src/include/storage/encryption.h +++ b/src/include/storage/encryption.h @@ -57,6 +57,14 @@ typedef enum CipherKind PG_CIPHER_AES_BLOCK_CBC_256_STREAM_CTR_256 } CipherKind; +/* + * TODO Tune these values. + */ +#define ENCRYPTION_PWD_MIN_LENGTH 8 +#define ENCRYPTION_PWD_MAX_LENGTH 16 +#define ENCRYPTION_KDF_NITER 1048576 +#define ENCRYPTION_KDF_SALT_LEN sizeof(uint64) + /* Key to encrypt / decrypt data. */ extern unsigned char encryption_key[]; -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0007-Refactoring-patch.-This-only-moves-some-code-to-XLog.patch From 2ec11d17adb6f19216a50998728d6a39c38f53a9 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 07/17] Refactoring patch. This only moves some code to XLogWritePages() function because it needs to be used in a little bit different way for encrypted XLOG. Done in a separate diff so that the following diff is a bit easier to read. --- src/backend/access/transam/xlog.c | 67 ++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 26 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 6b58e0ac76..cce7eca641 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -875,6 +875,7 @@ static void LocalSetXLogInsertAllowed(void); static void CreateEndOfRecoveryRecord(void); static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags); static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo); +static Size XLogWritePages(char *from, int npages, uint32 startoffset); static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void); static void AdvanceXLInsertBuffer(XLogRecPtr upto, bool opportunistic); @@ -2485,35 +2486,10 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) finishing_seg) { char *from; - Size nbytes; - Size nleft; - int written; /* OK to write the page(s) */ from = XLogCtl->pages + startidx * (Size) XLOG_BLCKSZ; - nbytes = npages * (Size) XLOG_BLCKSZ; - nleft = nbytes; - do - { - errno = 0; - pgstat_report_wait_start(WAIT_EVENT_WAL_WRITE); - written = pg_pwrite(openLogFile, from, nleft, startoffset); - pgstat_report_wait_end(); - if (written <= 0) - { - if (errno == EINTR) - continue; - ereport(PANIC, - (errcode_for_file_access(), - errmsg("could not write to log file %s " - "at offset %u, length %zu: %m", - XLogFileNameP(ThisTimeLineID, openLogSegNo), - startoffset, nleft))); - } - nleft -= written; - from += written; - startoffset += written; - } while (nleft > 0); + startoffset += XLogWritePages(from, npages, startoffset); npages = 0; @@ -2630,6 +2606,45 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) } /* + * Write page(s) to the XLOG file. + * + * Returns the number of bytes written. + */ +static Size +XLogWritePages(char *from, int npages, uint32 startoffset) +{ + Size nbytes, + nleft; + Size written; + + nbytes = npages * (Size) XLOG_BLCKSZ; + nleft = nbytes; + do + { + errno = 0; + pgstat_report_wait_start(WAIT_EVENT_WAL_WRITE); + written = pg_pwrite(openLogFile, from, nleft, startoffset); + pgstat_report_wait_end(); + if (written <= 0) + { + if (errno == EINTR) + continue; + ereport(PANIC, + (errcode_for_file_access(), + errmsg("could not write to log file %s " + "at offset %u, length %zu: %m", + XLogFileNameP(ThisTimeLineID, openLogSegNo), + startoffset, nbytes))); + } + nleft -= written; + from += written; + startoffset += written; + } while (nleft > 0); + + return written; +} + +/* * Record the LSN for an asynchronous transaction commit/abort * and nudge the WALWriter if there is work for it to do. * (This should not be called for synchronous commits.) -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0008-Enable-XLOG-encryption.patch From 7947b5452c58a4743586180bbde489c35887b253 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 08/17] Enable XLOG encryption. --- src/backend/access/transam/xlog.c | 90 +++++++++++++++++- src/backend/access/transam/xlogutils.c | 58 +++++++++++- src/backend/replication/walsender.c | 164 ++++++++++++++++++++++++--------- src/backend/storage/file/encryption.c | 80 ++++++++++++++++ src/backend/utils/misc/guc.c | 16 +++- src/include/access/xlog.h | 16 +++- src/include/storage/encryption.h | 17 +++- 7 files changed, 392 insertions(+), 49 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index cce7eca641..2f026f274e 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -2489,7 +2489,61 @@ XLogWrite(XLogwrtRqst WriteRqst, bool flexible) /* OK to write the page(s) */ from = XLogCtl->pages + startidx * (Size) XLOG_BLCKSZ; - startoffset += XLogWritePages(from, npages, startoffset); + if (data_encrypted) + { + int i, + nencrypted; + char *to; + uint32 encr_offset; + + /* + * Encrypt and write multiple pages at a time, in order to + * reduce the number of syscalls. + */ + nencrypted = 0; + to = encrypt_buf_xlog; + encr_offset = startoffset; + for (i = 1; i <= npages; i++) + { + char tweak[TWEAK_SIZE]; + Size nbytes; + + XLogEncryptionTweak(tweak, ThisTimeLineID, openLogSegNo, encr_offset); + + /* + * We should not encrypt the unused space, in order to + * avoid "reused key attack". + */ + if (i == npages && ispartialpage) + nbytes = WriteRqst.Write % XLOG_BLCKSZ; + else + nbytes = XLOG_BLCKSZ; + + encrypt_block(from, to, nbytes, tweak, true); + nencrypted++; + from += XLOG_BLCKSZ; + to += XLOG_BLCKSZ; + encr_offset += XLOG_BLCKSZ; + + /* + * Write the encrypted data if the encryption buffer is + * full or if the last page has been encrypted. + */ + if (nencrypted >= XLOG_ENCRYPT_BUF_PAGES || i >= npages) + { + startoffset += XLogWritePages(encrypt_buf_xlog, + nencrypted, + startoffset); + + /* Prepare for the next round of page encryptions. */ + nencrypted = 0; + to = encrypt_buf_xlog; + encr_offset = startoffset; + } + } + } + else + startoffset += XLogWritePages(from, npages, startoffset); npages = 0; @@ -3479,6 +3533,24 @@ XLogFileCopy(XLogSegNo destsegno, TimeLineID srcTLI, XLogSegNo srcsegno, } pgstat_report_wait_end(); } + + /* + * Since timeline is being changed and since encryption tweak contains + * the timeline, we need to decrypt the buffer and encrypt it with the + * new tweak. Do not encrypt the unused space, in order to avoid + * "reused key attack". + */ + if (data_encrypted && nread > 0) + { + char tweak[TWEAK_SIZE]; + + XLogEncryptionTweak(tweak, srcTLI, srcsegno, nbytes); + decrypt_block(buffer.data, buffer.data, nread, tweak, true); + + XLogEncryptionTweak(tweak, ThisTimeLineID, destsegno, nbytes); + encrypt_block(buffer.data, buffer.data, nread, tweak, true); + } + errno = 0; pgstat_report_wait_start(WAIT_EVENT_WAL_COPY_WRITE); if ((int) write(fd, buffer.data, sizeof(buffer)) != (int) sizeof(buffer)) @@ -5273,6 +5345,14 @@ BootStrapXLOG(void) use_existent = false; openLogFile = XLogFileInit(1, &use_existent, false); + if (data_encrypted) + { + char tweak[TWEAK_SIZE]; + + XLogEncryptionTweak(tweak, ThisTimeLineID, 1, 0); + encrypt_block((char *) page, (char *) page, XLOG_BLCKSZ, tweak, true); + } + /* Write the first page with the initial record */ errno = 0; pgstat_report_wait_start(WAIT_EVENT_WAL_BOOTSTRAP_WRITE); @@ -11724,6 +11804,14 @@ retry: Assert(targetPageOff == readOff); Assert(reqLen <= readLen); + if (data_encrypted) + { + char tweak[TWEAK_SIZE]; + + XLogEncryptionTweak(tweak, curFileTLI, readSegNo, readOff); + decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak, true); + } + *readTLI = curFileTLI; /* diff --git a/src/backend/access/transam/xlogutils.c b/src/backend/access/transam/xlogutils.c index 10a663bae6..25139e7a86 100644 --- a/src/backend/access/transam/xlogutils.c +++ b/src/backend/access/transam/xlogutils.c @@ -25,6 +25,7 @@ #include "access/xlogutils.h" #include "miscadmin.h" #include "pgstat.h" +#include "storage/encryption.h" #include "storage/smgr.h" #include "utils/guc.h" #include "utils/hsearch.h" @@ -656,9 +657,11 @@ static void XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr, Size count) { - char *p; XLogRecPtr recptr; Size nbytes; + char *decrypt_p; + uint32 decryptOff; + char *p; /* state maintained across calls */ static int sendFile = -1; @@ -668,17 +671,46 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr, Assert(segsize == wal_segment_size); + /* + * TODO Currently the function is only called with startptr at page + * boundary. walsender.c:XLogRead() is more generic but it's a bit + * different, so decryption also needs to be adopted in a little different + * way (i.e. the decryption cannot be merged here mechanically). Manual + * merge is not worth the effort right now because a separate patch to + * consolidate XLogRead() functions is in the queue + * (https://commitfest.postgresql.org/23/2098/). Let's implement the + * decryption in a generic way when that patch is committed. + */ + Assert(startptr % XLOG_BLCKSZ == 0); + p = buf; recptr = startptr; nbytes = count; + decrypt_p = p; + decryptOff = XLogSegmentOffset(recptr, segsize); + while (nbytes > 0) { uint32 startoff; int segbytes; int readbytes; - startoff = XLogSegmentOffset(recptr, segsize); + if (recptr == startptr) + startoff = decryptOff; + else + { + startoff = XLogSegmentOffset(recptr, segsize); + if (startoff == 0) + { + /* + * If segment boundary was reached, decryptOff should have + * caught up, so we can (and should) sync it with startoff. + */ + Assert(decryptOff == segsize); + decryptOff = startoff; + } + } /* Do we need to switch to a different xlog segment? */ if (sendFile < 0 || !XLByteInSeg(recptr, sendSegNo, segsize) || @@ -758,7 +790,29 @@ XLogRead(char *buf, int segsize, TimeLineID tli, XLogRecPtr startptr, sendOff += readbytes; nbytes -= readbytes; p += readbytes; + + /* Decrypt completed blocks */ + if (data_encrypted) + { + while (decrypt_p + XLOG_BLCKSZ <= p) + { + char tweak[TWEAK_SIZE]; + + XLogEncryptionTweak(tweak, tli, sendSegNo, decryptOff); + decrypt_block(decrypt_p, decrypt_p, XLOG_BLCKSZ, tweak, + true); + + decrypt_p += XLOG_BLCKSZ; + decryptOff += XLOG_BLCKSZ; + } + } } + + /* + * TODO Currently the function is only called with count==XLOG_BLCKSZ. See + * the comment on consolidation of XLogRead() above. + */ + Assert(decrypt_p == p); } /* diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 92fa86fc9d..ae9c1418d1 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -77,6 +77,7 @@ #include "replication/walsender.h" #include "replication/walsender_private.h" #include "storage/condition_variable.h" +#include "storage/encryption.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/pmsignal.h" @@ -256,7 +257,9 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time); static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now); static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch); -static void XLogRead(char *buf, XLogRecPtr startptr, Size count); +static int XLogReadBuffer(char *buf, int nbytes, int startoff); +static void XLogRead(char *buf, XLogRecPtr startptr, Size count, + bool decrypt); /* Initialize walsender process before entering the main command loop */ @@ -787,7 +790,7 @@ logical_read_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr, int req count = flushptr - targetPagePtr; /* part of the page available */ /* now actually read the data, we know it's there */ - XLogRead(cur_page, targetPagePtr, XLOG_BLCKSZ); + XLogRead(cur_page, targetPagePtr, XLOG_BLCKSZ, true); return count; } @@ -2352,19 +2355,53 @@ WalSndKill(int code, Datum arg) SpinLockRelease(&walsnd->mutex); } +static int +XLogReadBuffer(char *buf, int nbytes, int startoff) +{ + int readbytes; + + /* Need to seek in the file? */ + if (sendOff != startoff) + { + if (lseek(sendFile, (off_t) startoff, SEEK_SET) < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not seek in log segment %s to offset %u: %m", + XLogFileNameP(curFileTimeLine, sendSegNo), + startoff))); + sendOff = startoff; + } + + pgstat_report_wait_start(WAIT_EVENT_WAL_READ); + readbytes = read(sendFile, buf, nbytes); + pgstat_report_wait_end(); + if (readbytes <= 0) + { + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read from log segment %s, offset %u, length %lu: %m", + XLogFileNameP(curFileTimeLine, sendSegNo), + sendOff, (unsigned long) nbytes))); + } + + return readbytes; +} + /* - * Read 'count' bytes from WAL into 'buf', starting at location 'startptr' + * Read 'count' bytes from WAL into 'buf', starting at location + * 'startptr'. Decrypt the data if it's encrypted and if caller wants it + * decrypted. * - * XXX probably this should be improved to suck data directly from the - * WAL buffers when possible. + * XXX probably this should be improved to fetch data directly from the WAL + * buffers when possible. * * Will open, and keep open, one WAL segment stored in the global file - * descriptor sendFile. This means if XLogRead is used once, there will - * always be one descriptor left open until the process ends, but never - * more than one. + * descriptor sendFile. This means if XLogRead is used once, there will always + * be one descriptor left open until the process ends, but never more than + * one. */ static void -XLogRead(char *buf, XLogRecPtr startptr, Size count) +XLogRead(char *buf, XLogRecPtr startptr, Size count, bool decrypt) { char *p; XLogRecPtr recptr; @@ -2454,42 +2491,87 @@ retry: sendOff = 0; } - /* Need to seek in the file? */ - if (sendOff != startoff) + /* Caller should not request decryption of unencrypted data. */ + Assert(!(decrypt && !data_encrypted)); + + if (data_encrypted && decrypt) { - if (lseek(sendFile, (off_t) startoff, SEEK_SET) < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not seek in log segment %s to offset %u: %m", - XLogFileNameP(curFileTimeLine, sendSegNo), - startoff))); - sendOff = startoff; - } + int pageoff = startoff % XLOG_BLCKSZ; + uint32 pagebase = startoff - pageoff; + int bufbytes, + bufend, + i; + char tweak[TWEAK_SIZE]; - /* How many bytes are within this segment? */ - if (nbytes > (wal_segment_size - startoff)) - segbytes = wal_segment_size - startoff; - else - segbytes = nbytes; + /* + * Only accept as much data as what can fit into the buffer. + */ + if (nbytes > (ENCRYPT_BUF_XLOG_SIZE - pageoff)) + bufbytes = ENCRYPT_BUF_XLOG_SIZE - pageoff; + else + bufbytes = nbytes; + bufend = pageoff + bufbytes; - pgstat_report_wait_start(WAIT_EVENT_WAL_READ); - readbytes = read(sendFile, p, segbytes); - pgstat_report_wait_end(); - if (readbytes < 0) - { - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not read from log segment %s, offset %u, length %zu: %m", - XLogFileNameP(curFileTimeLine, sendSegNo), - sendOff, (Size) segbytes))); + /* + * Read the data, including the leading part of the page which + * caller is not interested in. The tweak we passed to + * encrypt_block() for encryption was for the beginning of the + * block, so it'd be hard to start decryption anywhere else. + */ + readbytes = 0; + while (readbytes < bufend) + readbytes += XLogReadBuffer(encrypt_buf_xlog + readbytes, + bufend - readbytes, + pagebase + readbytes); + + /* + * Decrypt the data one page at a time (the tweak is only valid + * for particular page). + */ + for (i = 0; i < readbytes; i += XLOG_BLCKSZ) + { + Size nencrypt; + + XLogEncryptionTweak(tweak, + curFileTimeLine, + sendSegNo, + pagebase + i); + + /* + * If the last page is not complete, only decrypt the used + * part. + */ + if ((bufend - i) < XLOG_BLCKSZ) + nencrypt = bufend - i; + else + nencrypt = XLOG_BLCKSZ; + + decrypt_block(encrypt_buf_xlog + i, + encrypt_buf_xlog + i, + nencrypt, + tweak, + true); + } + + /* + * Caller does not care that we possibly had to read pageoff bytes + * in addition (because we cannot decrypt trailing part of the + * page alone). This overhead must not affect the accounting. + */ + readbytes = bufbytes; + + /* Copy the data to the output buffer. */ + memcpy(p, encrypt_buf_xlog + pageoff, bufbytes); } - else if (readbytes == 0) + else { - ereport(ERROR, - (errcode(ERRCODE_DATA_CORRUPTED), - errmsg("could not read from log segment %s, offset %u: read %d of %zu", - XLogFileNameP(curFileTimeLine, sendSegNo), - sendOff, readbytes, (Size) segbytes))); + /* How many bytes are within this segment? */ + if (nbytes > (wal_segment_size - startoff)) + segbytes = wal_segment_size - startoff; + else + segbytes = nbytes; + + readbytes = XLogReadBuffer(p, segbytes, startoff); } /* Update state for read */ @@ -2768,7 +2850,7 @@ XLogSendPhysical(void) * calls. */ enlargeStringInfo(&output_message, nbytes); - XLogRead(&output_message.data[output_message.len], startptr, nbytes); + XLogRead(&output_message.data[output_message.len], startptr, nbytes, false); output_message.len += nbytes; output_message.data[output_message.len] = '\0'; diff --git a/src/backend/storage/file/encryption.c b/src/backend/storage/file/encryption.c index 1eda334762..971b6928e4 100644 --- a/src/backend/storage/file/encryption.c +++ b/src/backend/storage/file/encryption.c @@ -63,6 +63,8 @@ char encryption_verification[ENCRYPTION_SAMPLE_SIZE]; bool encryption_setup_done = false; +char *encrypt_buf_xlog = NULL; + #ifdef USE_ENCRYPTION static void init_encryption_context(EVP_CIPHER_CTX **ctx_p, bool stream); static void evp_error(void); @@ -165,6 +167,21 @@ setup_encryption(void) init_encryption_context(&ctx_encrypt_stream, true); init_encryption_context(&ctx_decrypt_stream, true); + /* + * We need multiple pages here, so allocate the memory dynamically instead + * of using PGAlignedBlock. That also ensures it'll be MAXALIGNed, which + * is useful because the buffer will be used for I/O. + * + * Use TopMemoryContext because on server side this code is run by + * postmaster and postmaster context gets freed after fork(). + */ +#ifndef FRONTEND + encrypt_buf_xlog = (char *) MemoryContextAlloc(TopMemoryContext, + ENCRYPT_BUF_XLOG_SIZE); +#else + encrypt_buf_xlog = (char *) palloc(ENCRYPT_BUF_XLOG_SIZE); +#endif + encryption_setup_done = true; #else /* !USE_ENCRYPTION */ #ifndef FRONTEND @@ -264,6 +281,50 @@ encrypt_block(const char *input, char *output, Size size, char *tweak, #endif /* USE_ENCRYPTION */ } +/* + * Decrypts one block of data with a specified tweak value. May only be called + * when encryption_enabled is true. + * + * Input and output buffer may point to the same location. + * + * For detailed comments see encrypt_block(). + */ +void +decrypt_block(const char *input, char *output, Size size, char *tweak, + bool stream) +{ +#ifdef USE_ENCRYPTION + int out_size; + EVP_CIPHER_CTX *ctx; + + Assert(data_encrypted); + Assert((size >= ENCRYPTION_BLOCK && size % ENCRYPTION_BLOCK == 0) || + stream); + + if (!stream && IsAllZero(input, size)) + { + memset(output, 0, size); + return; + } + + ctx = !stream ? ctx_decrypt : ctx_decrypt_stream; + + /* The remaining initialization. */ + if (EVP_DecryptInit_ex(ctx, NULL, NULL, encryption_key, + (unsigned char *) tweak) != 1) + evp_error(); + + /* Do the actual encryption. */ + if (EVP_DecryptUpdate(ctx, (unsigned char *) output, + &out_size, (unsigned char *) input, size) != 1) + evp_error(); + + Assert(out_size == size); +#else + /* data_encrypted should not be set */ + Assert(false); +#endif /* USE_ENCRYPTION */ +} #ifdef USE_ENCRYPTION /* @@ -345,3 +406,22 @@ evp_error(void) #endif /* FRONTEND */ } #endif /* USE_ENCRYPTION */ + +/* + * Xlog is encrypted page at a time. Each xlog page gets a unique tweak via + * timeline, segment and offset. + * + * The function is located here rather than some of the xlog*.c modules so + * that front-end applications can easily use it too. + */ +void +XLogEncryptionTweak(char *tweak, TimeLineID timeline, XLogSegNo segment, + uint32 offset) +{ + memset(tweak, 0, TWEAK_SIZE); + memcpy(tweak, &timeline, sizeof(timeline)); + tweak += sizeof(timeline); + memcpy(tweak, &segment, sizeof(XLogSegNo)); + tweak += sizeof(XLogSegNo); + memcpy(tweak, &offset, sizeof(offset)); +} diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 2b7fc2fb0f..440120635e 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -173,6 +173,7 @@ static void assign_session_replication_role(int newval, void *extra); static bool check_temp_buffers(int *newval, void **extra, GucSource source); static bool check_bonjour(bool *newval, void **extra, GucSource source); static bool check_ssl(bool *newval, void **extra, GucSource source); +static bool check_full_page_writes(bool *newval, void **extra, GucSource source); static bool check_stage_log_stats(bool *newval, void **extra, GucSource source); static bool check_log_stats(bool *newval, void **extra, GucSource source); static bool check_canonical_path(char **newval, void **extra, GucSource source); @@ -1173,7 +1174,7 @@ static struct config_bool ConfigureNamesBool[] = }, &fullPageWrites, true, - NULL, NULL, NULL + check_full_page_writes, NULL, NULL }, { @@ -11105,6 +11106,19 @@ check_ssl(bool *newval, void **extra, GucSource source) } static bool +check_full_page_writes(bool *newval, void **extra, GucSource source) +{ + if (!(*newval) && data_encrypted) + { + GUC_check_errdetail("Cannot disable parameter when the cluster is encrypted."); + + return false; + } + + return true; +} + +static bool check_stage_log_stats(bool *newval, void **extra, GucSource source) { if (*newval && log_statement_stats) diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index d519252aad..2ad441623d 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -19,6 +19,7 @@ #include "lib/stringinfo.h" #include "nodes/pg_list.h" #include "storage/fd.h" +#include "storage/encryption.h" /* Sync methods */ @@ -186,10 +187,19 @@ extern PGDLLIMPORT int wal_level; * Normally, we don't WAL-log hint bit updates, but if checksums are enabled, * we have to protect them against torn page writes. When you only set * individual bits on a page, it's still consistent no matter what combination - * of the bits make it to disk, but the checksum wouldn't match. Also WAL-log - * them if forced by wal_log_hints=on. + * of the bits make it to disk, but the checksum wouldn't match. + * + * Regardless checksums, if encryption is enabled, hint bit change followed by + * a torn page write can result in such a situation that decryption produces + * page whose contents (following the hint bit) is garbage. This is because + * the block cipher we use propagates changes in the lower addresses of plain + * data to higher addresses of the cipher data. We need full-page image also + * to recover from this state. + * + * Also WAL-log the hint bits if forced by wal_log_hints=on. */ -#define XLogHintBitIsNeeded() (DataChecksumsEnabled() || wal_log_hints) +#define XLogHintBitIsNeeded() (DataChecksumsEnabled() || data_encrypted || \ + wal_log_hints) /* Do we need to WAL-log information required only for Hot Standby and logical replication? */ #define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA) diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h index 72bcae0972..c6ad9b43bd 100644 --- a/src/include/storage/encryption.h +++ b/src/include/storage/encryption.h @@ -116,6 +116,16 @@ extern char encryption_verification[]; /* Do we have encryption_key and the encryption library initialized? */ extern bool encryption_setup_done; +/* + * XLOG encryption/decryption buffer. This buffer spans multiple pages, in + * order to reduce the number of syscalls when doing I/O. + * + * XXX Fine tune the buffer size. + */ +extern char *encrypt_buf_xlog; +#define XLOG_ENCRYPT_BUF_PAGES 8 +#define ENCRYPT_BUF_XLOG_SIZE (XLOG_ENCRYPT_BUF_PAGES * XLOG_BLCKSZ) + #ifndef FRONTEND extern Size EncryptionShmemSize(void); extern void EncryptionShmemInit(void); @@ -127,7 +137,12 @@ extern void read_encryption_key(read_encryption_key_cb read_char); extern void setup_encryption(void); extern void sample_encryption(char *buf); extern void encrypt_block(const char *input, char *output, Size size, - char *tweak, bool stream); + char *tweak, bool stream); +extern void decrypt_block(const char *input, char *output, Size size, + char *tweak, bool stream); extern void encryption_error(bool fatal, char *message); +extern void XLogEncryptionTweak(char *tweak, TimeLineID timeline, + XLogSegNo segment, uint32 offset); + #endif /* ENCRYPTION_H */ -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0009-Enable-encryption-of-relations.patch From 5868d43af039b86a9b0cc6303646b7782a20f77b Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 09/17] Enable encryption of relations. --- src/backend/access/heap/rewriteheap.c | 16 ++++++ src/backend/access/nbtree/nbtsort.c | 2 + src/backend/commands/dbcommands.c | 17 +++++- src/backend/replication/basebackup.c | 2 +- src/backend/storage/buffer/bufmgr.c | 15 +++++ src/backend/storage/buffer/localbuf.c | 8 +++ src/backend/storage/file/copydir.c | 99 ++++++++++++++++++++++++++------ src/backend/storage/file/encryption.c | 104 ++++++++++++++++++++++++++++++++++ src/backend/storage/file/reinit.c | 53 ++++++++++++----- src/backend/storage/page/bufpage.c | 26 +-------- src/backend/storage/smgr/md.c | 41 +++++++++++++- src/include/storage/copydir.h | 6 +- src/include/storage/encryption.h | 22 ++++++- src/include/storage/reinit.h | 3 +- 14 files changed, 350 insertions(+), 64 deletions(-) diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c index 369694fa2e..37aba1eabe 100644 --- a/src/backend/access/heap/rewriteheap.c +++ b/src/backend/access/heap/rewriteheap.c @@ -330,18 +330,27 @@ end_heap_rewrite(RewriteState state) /* Write the last page, if any */ if (state->rs_buffer_valid) { + bool lsn_is_fake = false; + if (state->rs_use_wal) log_newpage(&state->rs_new_rel->rd_node, MAIN_FORKNUM, state->rs_blockno, state->rs_buffer, true); + else if (data_encrypted) + lsn_is_fake = EnforceLSNUpdateForEncryption((char *) + state->rs_buffer); + RelationOpenSmgr(state->rs_new_rel); PageSetChecksumInplace(state->rs_buffer, state->rs_blockno); smgrextend(state->rs_new_rel->rd_smgr, MAIN_FORKNUM, state->rs_blockno, (char *) state->rs_buffer, true); + + if (lsn_is_fake) + RestoreInvalidLSN((char *) state->rs_buffer); } /* @@ -692,6 +701,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup) if (len + saveFreeSpace > pageFreeSpace) { + bool lsn_is_fake = false; + /* Doesn't fit, so write out the existing page */ /* XLOG stuff */ @@ -701,6 +712,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup) state->rs_blockno, page, true); + else if (data_encrypted) + lsn_is_fake = EnforceLSNUpdateForEncryption((char *) page); /* * Now write the page. We say isTemp = true even if it's not a @@ -715,6 +728,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup) smgrextend(state->rs_new_rel->rd_smgr, MAIN_FORKNUM, state->rs_blockno, (char *) page, true); + if (lsn_is_fake) + RestoreInvalidLSN((char *) page); + state->rs_blockno++; state->rs_buffer_valid = false; } diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index d0b9013caf..e451d6054f 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -660,6 +660,8 @@ _bt_blwritepage(BTWriteState *wstate, Page page, BlockNumber blkno) /* We use the heap NEWPAGE record type for this */ log_newpage(&wstate->index->rd_node, MAIN_FORKNUM, blkno, page, true); } + else if (data_encrypted) + EnforceLSNUpdateForEncryption((char *) page); /* * If we have to write pages nonsequentially, fill in the space with diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 863f89f19d..7ea783d096 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -609,6 +609,8 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) Oid dsttablespace; char *srcpath; char *dstpath; + RelFileNode src_node = {srctablespace, src_dboid, InvalidOid}; + RelFileNode dst_node; struct stat st; /* No need to copy global tablespace */ @@ -630,6 +632,10 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) else dsttablespace = srctablespace; + dst_node.spcNode = dsttablespace; + dst_node.dbNode = dboid; + dst_node.relNode= InvalidOid; + dstpath = GetDatabasePath(dboid, dsttablespace); /* @@ -637,7 +643,7 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) * * We don't need to copy subdirectories */ - copydir(srcpath, dstpath, false); + copydir(srcpath, dstpath, &src_node, &dst_node); /* Record the filesystem change in XLOG */ { @@ -1272,10 +1278,13 @@ movedb(const char *dbname, const char *tblspcname) PG_ENSURE_ERROR_CLEANUP(movedb_failure_callback, PointerGetDatum(&fparms)); { + RelFileNode src_node = {src_tblspcoid, db_id, InvalidOid}; + RelFileNode dst_node = {dst_tblspcoid, db_id, InvalidOid}; + /* * Copy files from the old tablespace to the new one */ - copydir(src_dbpath, dst_dbpath, false); + copydir(src_dbpath, dst_dbpath, &src_node, &dst_node); /* * Record the filesystem change in XLOG @@ -2109,6 +2118,8 @@ dbase_redo(XLogReaderState *record) char *src_path; char *dst_path; struct stat st; + RelFileNode src_node = {xlrec->src_tablespace_id, xlrec->src_db_id, InvalidOid}; + RelFileNode dst_node = {xlrec->tablespace_id, xlrec->db_id, InvalidOid}; src_path = GetDatabasePath(xlrec->src_db_id, xlrec->src_tablespace_id); dst_path = GetDatabasePath(xlrec->db_id, xlrec->tablespace_id); @@ -2138,7 +2149,7 @@ dbase_redo(XLogReaderState *record) * * We don't need to copy subdirectories */ - copydir(src_path, dst_path, false); + copydir(src_path, dst_path, &src_node, &dst_node); } else if (info == XLOG_DBASE_DROP) { diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 7a1b38466b..678fdfa5a2 100644 --- a/src/backend/replication/basebackup.c +++ b/src/backend/replication/basebackup.c @@ -1104,7 +1104,7 @@ sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces, /* Exclude all forks for unlogged tables except the init fork */ if (isDbDir && parse_filename_for_nontemp_relation(de->d_name, &relOidChars, - &relForkNum)) + &relForkNum, NULL)) { /* Never exclude init forks */ if (relForkNum != INIT_FORKNUM) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 7332e6b590..a36fb7bea7 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -2678,6 +2678,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln) Block bufBlock; char *bufToWrite; uint32 buf_state; + bool lsn_is_fake = false; /* * Acquire the buffer's io_in_progress lock. If StartBufferIO returns @@ -2742,6 +2743,9 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln) */ bufBlock = BufHdrGetBlock(buf); + if (data_encrypted && !(buf_state & BM_PERMANENT)) + lsn_is_fake = EnforceLSNUpdateForEncryption(bufBlock); + /* * Update page checksum if desired. Since we have only shared lock on the * buffer, other processes might be updating hint bits in it, so we must @@ -2771,6 +2775,9 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln) pgBufferUsage.shared_blks_written++; + if (lsn_is_fake) + RestoreInvalidLSN(bufBlock); + /* * Mark the buffer as clean (unless BM_JUST_DIRTIED has become set) and * end the io_in_progress state. @@ -3209,6 +3216,7 @@ FlushRelationBuffers(Relation rel) { ErrorContextCallback errcallback; Page localpage; + bool lsn_is_fake = false; localpage = (char *) LocalBufHdrGetBlock(bufHdr); @@ -3218,6 +3226,10 @@ FlushRelationBuffers(Relation rel) errcallback.previous = error_context_stack; error_context_stack = &errcallback; + if (data_encrypted) + lsn_is_fake = EnforceLSNUpdateForEncryption((char *) + localpage); + PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); smgrwrite(rel->rd_smgr, @@ -3226,6 +3238,9 @@ FlushRelationBuffers(Relation rel) localpage, false); + if (lsn_is_fake) + RestoreInvalidLSN((char *) localpage); + buf_state &= ~(BM_DIRTY | BM_JUST_DIRTIED); pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 391b6d6e16..cd36ec78a7 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -20,6 +20,7 @@ #include "executor/instrument.h" #include "storage/buf_internals.h" #include "storage/bufmgr.h" +#include "storage/encryption.h" #include "utils/guc.h" #include "utils/memutils.h" #include "utils/resowner_private.h" @@ -207,10 +208,14 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, { SMgrRelation oreln; Page localpage = (char *) LocalBufHdrGetBlock(bufHdr); + bool lsn_is_fake = false; /* Find smgr relation for buffer */ oreln = smgropen(bufHdr->tag.rnode, MyBackendId); + if (data_encrypted) + lsn_is_fake = EnforceLSNUpdateForEncryption((char *) localpage); + PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); /* And write... */ @@ -220,6 +225,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, localpage, false); + if (lsn_is_fake) + RestoreInvalidLSN(localpage); + /* Mark not-dirty now in case we error out below */ buf_state &= ~BM_DIRTY; pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c index 30f6200a86..b98e9c10d8 100644 --- a/src/backend/storage/file/copydir.c +++ b/src/backend/storage/file/copydir.c @@ -23,24 +23,29 @@ #include #include "storage/copydir.h" +#include "storage/encryption.h" #include "storage/fd.h" +#include "storage/reinit.h" #include "miscadmin.h" #include "pgstat.h" /* * copydir: copy a directory * - * If recurse is false, subdirectories are ignored. Anything that's not - * a directory or a regular file is ignored. + * RelFileNode values must specify tablespace and database oids for source + * and target to support re-encryption if necessary. relNode value in provided + * structs will be clobbered. */ void -copydir(char *fromdir, char *todir, bool recurse) +copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode) { DIR *xldir; struct dirent *xlde; char fromfile[MAXPGPATH * 2]; char tofile[MAXPGPATH * 2]; + Assert(!data_encrypted || (fromNode != NULL && toNode != NULL)); + if (MakePGDirectory(todir) != 0) ereport(ERROR, (errcode_for_file_access(), @@ -67,14 +72,32 @@ copydir(char *fromdir, char *todir, bool recurse) (errcode_for_file_access(), errmsg("could not stat file \"%s\": %m", fromfile))); - if (S_ISDIR(fst.st_mode)) + if (S_ISREG(fst.st_mode)) { - /* recurse to handle subdirectories */ - if (recurse) - copydir(fromfile, tofile, true); + int oidchars; + ForkNumber forkNum; + int segment; + + /* + * For encrypted databases we need to reencrypt files with new + * tweaks. + */ + if (data_encrypted && + parse_filename_for_nontemp_relation(xlde->d_name, + &oidchars, &forkNum, &segment)) + { + char oidbuf[OIDCHARS + 1]; + + memcpy(oidbuf, xlde->d_name, oidchars); + oidbuf[oidchars] = '\0'; + + /* We scribble over the provided RelFileNodes here */ + fromNode->relNode = toNode->relNode = atol(oidbuf); + copy_file(fromfile, tofile, fromNode, toNode, forkNum, forkNum, segment); + } + else + copy_file(fromfile, tofile, NULL, NULL, 0, 0, 0); } - else if (S_ISREG(fst.st_mode)) - copy_file(fromfile, tofile); } FreeDir(xldir); @@ -121,17 +144,22 @@ copydir(char *fromdir, char *todir, bool recurse) } /* - * copy one file + * copy one file. If decryption and reencryption may be needed specify + * relfilenodes for source and target. */ void -copy_file(char *fromfile, char *tofile) +copy_file(char *fromfile, char *tofile, RelFileNode *fromNode, + RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum, + int segment) { char *buffer; int srcfd; int dstfd; int nbytes; + int bytesread; off_t offset; off_t flush_offset; + BlockNumber blockNum = segment * RELSEG_SIZE; /* Size of copy buffer (read and write requests) */ #define COPY_BUF_SIZE (8 * BLCKSZ) @@ -186,15 +214,50 @@ copy_file(char *fromfile, char *tofile) flush_offset = offset; } - pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ); - nbytes = read(srcfd, buffer, COPY_BUF_SIZE); - pgstat_report_wait_end(); - if (nbytes < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", fromfile))); + /* + * Try to read as much as we fit in the buffer so we can deal with + * complete blocks if we need to reencrypt. + */ + nbytes = 0; + while (nbytes < COPY_BUF_SIZE) + { + pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ); + bytesread = read(srcfd, buffer + nbytes, COPY_BUF_SIZE - nbytes); + pgstat_report_wait_end(); + if (bytesread < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", fromfile))); + nbytes += bytesread; + if (bytesread == 0) + break; + } if (nbytes == 0) break; + + /* + * If the database is encrypted we need to decrypt the data here and + * reencrypt it to adjust the tweak values of blocks. + */ + if (data_encrypted) + { + if (fromNode != NULL) + { + Assert(toNode != NULL); + + /* + * There's no reason not to have whole number of pages read + * Computation of the number of blocks below relies on this + * fact. + */ + Assert(nbytes % BLCKSZ == 0); + + blockNum = ReencryptBlock(buffer, nbytes / BLCKSZ, + fromNode, toNode, fromForkNum, + toForkNum, blockNum); + } + } + errno = 0; pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_WRITE); if ((int) write(dstfd, buffer, nbytes) != nbytes) diff --git a/src/backend/storage/file/encryption.c b/src/backend/storage/file/encryption.c index 971b6928e4..e79f967fbb 100644 --- a/src/backend/storage/file/encryption.c +++ b/src/backend/storage/file/encryption.c @@ -63,6 +63,7 @@ char encryption_verification[ENCRYPTION_SAMPLE_SIZE]; bool encryption_setup_done = false; +PGAlignedBlock encrypt_buf; char *encrypt_buf_xlog = NULL; #ifdef USE_ENCRYPTION @@ -425,3 +426,106 @@ XLogEncryptionTweak(char *tweak, TimeLineID timeline, XLogSegNo segment, tweak += sizeof(XLogSegNo); memcpy(tweak, &offset, sizeof(offset)); } + +/* + * Copying relations between tablespaces/databases means that the tweak values + * of each block will change. This function transcodes a series of blocks with + * new tweak values. Returns the new block number for convenience. + */ +BlockNumber +ReencryptBlock(char *buffer, int blocks, + RelFileNode *srcNode, RelFileNode *dstNode, + ForkNumber srcForkNum, ForkNumber dstForkNum, + BlockNumber blockNum) +{ + char *cur; + char srcTweak[TWEAK_SIZE]; + char dstTweak[TWEAK_SIZE]; + + for (cur = buffer; cur < buffer + blocks * BLCKSZ; cur += BLCKSZ) + { + mdtweak(srcTweak, srcNode, srcForkNum, blockNum); + mdtweak(dstTweak, dstNode, dstForkNum, blockNum); + decrypt_block(cur, cur, BLCKSZ, srcTweak, false); + encrypt_block(cur, cur, BLCKSZ, dstTweak, false); + blockNum++; + } + return blockNum; +} + +/* + * md files are encrypted block at a time. Tweak will alias higher numbered + * forks for huge tables. + */ +void +mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, BlockNumber blocknum) +{ + uint32 fork_and_block = (forknum << 24) ^ blocknum; + + memcpy(tweak, relnode, sizeof(RelFileNode)); + memcpy(tweak + sizeof(RelFileNode), &fork_and_block, 4); +} + +#ifndef FRONTEND +/* + * When page is encrypted using a cipher in the cipher-block chaining (CBC) + * mode, the fact that the page starts with LSN makes it harder for adversary + * to see which part of the plain (unencrypted) page changed: the LSN changes + * even if only the plain data at the end of the page changed, and, due to the + * chaining, the encrypted page becomes completely different. Although no LSN + * is needed for unlogged tables, we still want to keep the advantage + * described here. So we set pd_lsn to "fake LSN" before each write. + * + * Note that caller needs to restore InvalidXLogRecPtr after the write so that + * it can be recognized later that the page needs to be treated specially. + * + * LW_SHARED on the buffer contents is sufficient because pd_lsn is not + * expected to be used for other purposes. + * + * Returns true iff the LSN was updated. + */ +bool +EnforceLSNUpdateForEncryption(char *buf_contents) +{ + PageHeader hdr = (PageHeader) buf_contents; + XLogRecPtr recptr; + + /* Failure indicates incorrect user of the function. */ + Assert(data_encrypted); + + /* + * All-zero page shouldn't be encrypted, so don't care about the fake LSN. + */ + if (PageIsNew(hdr)) + return false; + + recptr = PageXLogRecPtrGet(hdr->pd_lsn); + + /* + * Valid LSN indicates that the relation is either logged or it's a page + * of unlogged GIST index. In either case we must not touch the LSN. + */ + if (!XLogRecPtrIsInvalid(recptr)) + return false; + + PageXLogRecPtrSet(hdr->pd_lsn, GetFakeLSNForUnloggedRel()); + + return true; +} + +/* + * If EnforceLSNUpdateForEncryption() returned true and if the function can be + * passed that buffer again, use this function to restore the + * InvalidXLogRecPtr value. + */ +void +RestoreInvalidLSN(char *buf_contents) +{ + PageHeader hdr = (PageHeader) buf_contents; + + /* Failure indicates incorrect user of the function. */ + Assert(data_encrypted); + + PageXLogRecPtrSet(hdr->pd_lsn, (XLogRecPtr) InvalidXLogRecPtr); +} +#endif /* FRONTEND */ diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c index ccdf80d614..6af20cfe64 100644 --- a/src/backend/storage/file/reinit.c +++ b/src/backend/storage/file/reinit.c @@ -16,6 +16,7 @@ #include +#include "catalog/pg_tablespace.h" #include "common/relpath.h" #include "storage/copydir.h" #include "storage/fd.h" @@ -24,9 +25,9 @@ #include "utils/memutils.h" static void ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, - int op); + int op, Oid spcOid); static void ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, - int op); + int op, Oid spcOid, Oid dbOid); typedef struct { @@ -68,7 +69,7 @@ ResetUnloggedRelations(int op) /* * First process unlogged files in pg_default ($PGDATA/base) */ - ResetUnloggedRelationsInTablespaceDir("base", op); + ResetUnloggedRelationsInTablespaceDir("base", op, DEFAULTTABLESPACE_OID); /* * Cycle through directories for all non-default tablespaces. @@ -77,13 +78,16 @@ ResetUnloggedRelations(int op) while ((spc_de = ReadDir(spc_dir, "pg_tblspc")) != NULL) { + Oid spcOid; + if (strcmp(spc_de->d_name, ".") == 0 || strcmp(spc_de->d_name, "..") == 0) continue; snprintf(temp_path, sizeof(temp_path), "pg_tblspc/%s/%s", spc_de->d_name, TABLESPACE_VERSION_DIRECTORY); - ResetUnloggedRelationsInTablespaceDir(temp_path, op); + spcOid = atoi(spc_de->d_name); + ResetUnloggedRelationsInTablespaceDir(temp_path, op, spcOid); } FreeDir(spc_dir); @@ -99,7 +103,8 @@ ResetUnloggedRelations(int op) * Process one tablespace directory for ResetUnloggedRelations */ static void -ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op) +ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op, + Oid spcOid) { DIR *ts_dir; struct dirent *de; @@ -126,6 +131,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op) while ((de = ReadDir(ts_dir, tsdirname)) != NULL) { + Oid dbOid; + /* * We're only interested in the per-database directories, which have * numeric names. Note that this code will also (properly) ignore "." @@ -134,9 +141,10 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op) if (strspn(de->d_name, "0123456789") != strlen(de->d_name)) continue; + dbOid = atoi(de->d_name); snprintf(dbspace_path, sizeof(dbspace_path), "%s/%s", tsdirname, de->d_name); - ResetUnloggedRelationsInDbspaceDir(dbspace_path, op); + ResetUnloggedRelationsInDbspaceDir(dbspace_path, op, spcOid, dbOid); } FreeDir(ts_dir); @@ -146,7 +154,8 @@ ResetUnloggedRelationsInTablespaceDir(const char *tsdirname, int op) * Process one per-dbspace directory for ResetUnloggedRelations */ static void -ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) +ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op, + Oid spcOid, Oid dbOid) { DIR *dbspace_dir; struct dirent *de; @@ -187,7 +196,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) /* Skip anything that doesn't look like a relation data file. */ if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars, - &forkNum)) + &forkNum, NULL)) continue; /* Also skip it unless this is the init fork. */ @@ -229,7 +238,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) /* Skip anything that doesn't look like a relation data file. */ if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars, - &forkNum)) + &forkNum, NULL)) continue; /* We never remove the init fork. */ @@ -279,13 +288,14 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) { ForkNumber forkNum; int oidchars; + int segment; char oidbuf[OIDCHARS + 1]; char srcpath[MAXPGPATH * 2]; char dstpath[MAXPGPATH]; /* Skip anything that doesn't look like a relation data file. */ if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars, - &forkNum)) + &forkNum, &segment)) continue; /* Also skip it unless this is the init fork. */ @@ -305,7 +315,13 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) /* OK, we're ready to perform the actual copy. */ elog(DEBUG2, "copying %s to %s", srcpath, dstpath); - copy_file(srcpath, dstpath); + { + RelFileNode srcNode = {spcOid, dbOid, atol(oidbuf)}; + RelFileNode dstNode = srcNode; + + copy_file(srcpath, dstpath, &srcNode, &dstNode, + INIT_FORKNUM, MAIN_FORKNUM, segment); + } } FreeDir(dbspace_dir); @@ -327,7 +343,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) /* Skip anything that doesn't look like a relation data file. */ if (!parse_filename_for_nontemp_relation(de->d_name, &oidchars, - &forkNum)) + &forkNum, NULL)) continue; /* Also skip it unless this is the init fork. */ @@ -372,9 +388,10 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) */ bool parse_filename_for_nontemp_relation(const char *name, int *oidchars, - ForkNumber *fork) + ForkNumber *fork, int *segment) { int pos; + int segstart = 0; /* Look for a non-empty string of digits (that isn't too long). */ for (pos = 0; isdigit((unsigned char) name[pos]); ++pos) @@ -401,6 +418,7 @@ parse_filename_for_nontemp_relation(const char *name, int *oidchars, { int segchar; + segstart = pos + 1; for (segchar = 1; isdigit((unsigned char) name[pos + segchar]); ++segchar) ; if (segchar <= 1) @@ -411,5 +429,14 @@ parse_filename_for_nontemp_relation(const char *name, int *oidchars, /* Now we should be at the end. */ if (name[pos] != '\0') return false; + + if (segment != NULL) + { + if (segstart == 0) + *segment = 0; + else + *segment = atoi(name + segstart); + } + return true; } diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c index 6b49810e37..b6c358a642 100644 --- a/src/backend/storage/page/bufpage.c +++ b/src/backend/storage/page/bufpage.c @@ -17,6 +17,7 @@ #include "access/htup_details.h" #include "access/itup.h" #include "access/xlog.h" +#include "common/string.h" #include "pgstat.h" #include "storage/checksum.h" #include "utils/memdebug.h" @@ -82,11 +83,8 @@ bool PageIsVerified(Page page, BlockNumber blkno) { PageHeader p = (PageHeader) page; - size_t *pagebytes; - int i; bool checksum_failure = false; bool header_sane = false; - bool all_zeroes = false; uint16 checksum = 0; /* @@ -119,26 +117,8 @@ PageIsVerified(Page page, BlockNumber blkno) return true; } - /* - * Check all-zeroes case. Luckily BLCKSZ is guaranteed to always be a - * multiple of size_t - and it's much faster to compare memory using the - * native word size. - */ - StaticAssertStmt(BLCKSZ == (BLCKSZ / sizeof(size_t)) * sizeof(size_t), - "BLCKSZ has to be a multiple of sizeof(size_t)"); - - all_zeroes = true; - pagebytes = (size_t *) page; - for (i = 0; i < (BLCKSZ / sizeof(size_t)); i++) - { - if (pagebytes[i] != 0) - { - all_zeroes = false; - break; - } - } - - if (all_zeroes) + /* Check all-zeroes case */ + if (IsAllZero((char *) page, BLCKSZ)) return true; /* diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c index 64acc3fa43..1b986eac01 100644 --- a/src/backend/storage/smgr/md.c +++ b/src/backend/storage/smgr/md.c @@ -32,6 +32,7 @@ #include "postmaster/bgwriter.h" #include "storage/fd.h" #include "storage/bufmgr.h" +#include "storage/encryption.h" #include "storage/md.h" #include "storage/relfilenode.h" #include "storage/smgr.h" @@ -86,6 +87,7 @@ typedef struct _MdfdVec static MemoryContext MdCxt; /* context for all MdfdVec objects */ +static char *md_encryption_tweak; /* Populate a file tag describing an md.c segment file. */ #define INIT_MD_FILETAG(a,xx_rnode,xx_forknum,xx_segno) \ @@ -139,6 +141,8 @@ static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno, static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg); +static void mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer); +static void mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer); /* * mdinit() -- Initialize private state for magnetic disk storage manager. @@ -149,6 +153,8 @@ mdinit(void) MdCxt = AllocSetContextCreate(TopMemoryContext, "MdSmgr", ALLOCSET_DEFAULT_SIZES); + + md_encryption_tweak = MemoryContextAllocZero(MdCxt, TWEAK_SIZE); } /* @@ -401,6 +407,12 @@ mdextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE); + if (data_encrypted) + { + mdencrypt(reln, forknum, blocknum, buffer); + buffer = encrypt_buf.data; + } + if ((nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, seekpos, WAIT_EVENT_DATA_FILE_EXTEND)) != BLCKSZ) { if (nbytes < 0) @@ -587,6 +599,7 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, off_t seekpos; int nbytes; MdfdVec *v; + char *buffer_read = buffer; TRACE_POSTGRESQL_SMGR_MD_READ_START(forknum, blocknum, reln->smgr_rnode.node.spcNode, @@ -601,7 +614,10 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE); - nbytes = FileRead(v->mdfd_vfd, buffer, BLCKSZ, seekpos, WAIT_EVENT_DATA_FILE_READ); + if (data_encrypted) + buffer_read = encrypt_buf.data; + + nbytes = FileRead(v->mdfd_vfd, buffer_read, BLCKSZ, seekpos, WAIT_EVENT_DATA_FILE_READ); TRACE_POSTGRESQL_SMGR_MD_READ_DONE(forknum, blocknum, reln->smgr_rnode.node.spcNode, @@ -636,6 +652,8 @@ mdread(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, blocknum, FilePathName(v->mdfd_vfd), nbytes, BLCKSZ))); } + else if (data_encrypted) + mddecrypt(reln, forknum, blocknum, buffer); } /* @@ -671,6 +689,11 @@ mdwrite(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, Assert(seekpos < (off_t) BLCKSZ * RELSEG_SIZE); + if (data_encrypted) + { + mdencrypt(reln, forknum, blocknum, buffer); + buffer = encrypt_buf.data; + } nbytes = FileWrite(v->mdfd_vfd, buffer, BLCKSZ, seekpos, WAIT_EVENT_DATA_FILE_WRITE); TRACE_POSTGRESQL_SMGR_MD_WRITE_DONE(forknum, blocknum, @@ -1248,6 +1271,22 @@ _mdnblocks(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg) return (BlockNumber) (len / BLCKSZ); } + +static void +mdencrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *buffer) +{ + mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum); + encrypt_block(buffer, encrypt_buf.data, BLCKSZ, md_encryption_tweak, + false); +} + +static void +mddecrypt(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum, char *dest) +{ + mdtweak(md_encryption_tweak, &(reln->smgr_rnode.node), forknum, blocknum); + decrypt_block(encrypt_buf.data, dest, BLCKSZ, md_encryption_tweak, false); +} + /* * Sync a file to disk, given a file tag. Write the path into an output * buffer so the caller can use it in error messages. diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h index 525cc6203e..21c18bb6c3 100644 --- a/src/include/storage/copydir.h +++ b/src/include/storage/copydir.h @@ -13,7 +13,9 @@ #ifndef COPYDIR_H #define COPYDIR_H -extern void copydir(char *fromdir, char *todir, bool recurse); -extern void copy_file(char *fromfile, char *tofile); +#include "storage/relfilenode.h" + +extern void copydir(char *fromdir, char *todir, RelFileNode *fromNode, RelFileNode *toNode); +extern void copy_file(char *fromfile, char *tofile, RelFileNode *fromNode, RelFileNode *toNode, ForkNumber fromForkNum, ForkNumber toForkNum, int segment); #endif /* COPYDIR_H */ diff --git a/src/include/storage/encryption.h b/src/include/storage/encryption.h index c6ad9b43bd..eab8b50e98 100644 --- a/src/include/storage/encryption.h +++ b/src/include/storage/encryption.h @@ -117,8 +117,15 @@ extern char encryption_verification[]; extern bool encryption_setup_done; /* - * XLOG encryption/decryption buffer. This buffer spans multiple pages, in - * order to reduce the number of syscalls when doing I/O. + * In some cases we need a separate copy of the data because encryption + * in-place (typically in the shared buffers) would make the data unusable for + * backends. + */ +extern PGAlignedBlock encrypt_buf; + +/* + * The same for XLOG. This buffer spans multiple pages, in order to reduce the + * number of syscalls when doing I/O. * * XXX Fine tune the buffer size. */ @@ -144,5 +151,16 @@ extern void encryption_error(bool fatal, char *message); extern void XLogEncryptionTweak(char *tweak, TimeLineID timeline, XLogSegNo segment, uint32 offset); +extern BlockNumber ReencryptBlock(char *buffer, int blocks, + RelFileNode *srcNode, RelFileNode *dstNode, + ForkNumber srcForkNum, ForkNumber dstForkNum, + BlockNumber blockNum); +extern void mdtweak(char *tweak, RelFileNode *relnode, ForkNumber forknum, + BlockNumber blocknum); + +#ifndef FRONTEND +extern bool EnforceLSNUpdateForEncryption(char *buf_contents); +extern void RestoreInvalidLSN(char *buf_contents); +#endif /* FRONTEND */ #endif /* ENCRYPTION_H */ diff --git a/src/include/storage/reinit.h b/src/include/storage/reinit.h index 63d0b56d48..0e028cac4d 100644 --- a/src/include/storage/reinit.h +++ b/src/include/storage/reinit.h @@ -20,7 +20,8 @@ extern void ResetUnloggedRelations(int op); extern bool parse_filename_for_nontemp_relation(const char *name, - int *oidchars, ForkNumber *fork); + int *oidchars, ForkNumber *fork, + int *segment); #define UNLOGGED_RELATION_CLEANUP 0x0001 #define UNLOGGED_RELATION_INIT 0x0002 -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0010-Introduce-transient-buffered-file.patch From b280f32f789a919d87978ea945de9521da837b9a Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 10/17] Introduce "transient buffered file". Currently we need it only to encrypt transaction data which gets spilled to disk during logical decoding. This way we hide the encryption specific details from reorderbuffer.c, as well as any other (future) user of transient files that need to be encrypted. The next patch of the series implements the actual encryption of those files. --- src/backend/postmaster/pgstat.c | 6 - src/backend/replication/logical/reorderbuffer.c | 457 ++++++++---------- src/backend/storage/file/buffile.c | 595 ++++++++++++++++++------ src/include/pgstat.h | 2 - src/include/replication/reorderbuffer.h | 4 - src/include/storage/buffile.h | 13 +- 6 files changed, 652 insertions(+), 425 deletions(-) diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index b4f2b28b51..54786899fd 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -4004,12 +4004,6 @@ pgstat_get_wait_io(WaitEventIO w) case WAIT_EVENT_RELATION_MAP_WRITE: event_name = "RelationMapWrite"; break; - case WAIT_EVENT_REORDER_BUFFER_READ: - event_name = "ReorderBufferRead"; - break; - case WAIT_EVENT_REORDER_BUFFER_WRITE: - event_name = "ReorderBufferWrite"; - break; case WAIT_EVENT_REORDER_LOGICAL_MAPPING_READ: event_name = "ReorderLogicalMappingRead"; break; diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index e7c32f2a13..844b67d44b 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -71,6 +71,7 @@ #include "replication/slot.h" #include "replication/snapbuild.h" /* just for SnapBuildSnapDecRefcount */ #include "storage/bufmgr.h" +#include "storage/buffile.h" #include "storage/fd.h" #include "storage/sinval.h" #include "utils/builtins.h" @@ -109,7 +110,7 @@ typedef struct ReorderBufferIterTXNEntry XLogRecPtr lsn; ReorderBufferChange *change; ReorderBufferTXN *txn; - int fd; + TransientBufFile *file; XLogSegNo segno; } ReorderBufferIterTXNEntry; @@ -192,11 +193,17 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn); static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn); static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, - int fd, ReorderBufferChange *change); + TransientBufFile *file, ReorderBufferChange *change); +static void ReorderBufferWriteData(TransientBufFile *file, void *ptr, size_t size, + ReorderBufferTXN *txn); static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn, - int *fd, XLogSegNo *segno); + TransientBufFile **file, XLogSegNo *segno); static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, - char *change); + TransientBufFile **file); +static ReorderBufferTupleBuf *ReorderBufferRestoreTuple(ReorderBuffer *rb, + TransientBufFile *file); +static void ReorderBufferReadData(TransientBufFile *file, void *ptr, size_t size, + bool *no_data_p); static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn); static void ReorderBufferCleanupSerializedTXNs(const char *slotname); static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot, @@ -267,9 +274,6 @@ ReorderBufferAllocate(void) buffer->by_txn_last_xid = InvalidTransactionId; buffer->by_txn_last_txn = NULL; - buffer->outbuf = NULL; - buffer->outbufsize = 0; - buffer->current_restart_decoding_lsn = InvalidXLogRecPtr; dlist_init(&buffer->toplevel_by_lsn); @@ -988,7 +992,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn) for (off = 0; off < state->nr_txns; off++) { - state->entries[off].fd = -1; + state->entries[off].file = NULL; state->entries[off].segno = 0; } @@ -1013,7 +1017,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn) { /* serialize remaining changes */ ReorderBufferSerializeTXN(rb, txn); - ReorderBufferRestoreChanges(rb, txn, &state->entries[off].fd, + ReorderBufferRestoreChanges(rb, txn, &state->entries[off].file, &state->entries[off].segno); } @@ -1043,7 +1047,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn) /* serialize remaining changes */ ReorderBufferSerializeTXN(rb, cur_txn); ReorderBufferRestoreChanges(rb, cur_txn, - &state->entries[off].fd, + &state->entries[off].file, &state->entries[off].segno); } cur_change = dlist_head_element(ReorderBufferChange, node, @@ -1124,7 +1128,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state) dlist_delete(&change->node); dlist_push_tail(&state->old_change, &change->node); - if (ReorderBufferRestoreChanges(rb, entry->txn, &entry->fd, + if (ReorderBufferRestoreChanges(rb, entry->txn, &entry->file, &state->entries[off].segno)) { /* successfully restored changes from disk */ @@ -1163,8 +1167,8 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb, for (off = 0; off < state->nr_txns; off++) { - if (state->entries[off].fd != -1) - CloseTransientFile(state->entries[off].fd); + if (state->entries[off].file) + BufFileCloseTransient(state->entries[off].file); } /* free memory we might have "leaked" in the last *Next call */ @@ -2212,24 +2216,6 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid) */ /* - * Ensure the IO buffer is >= sz. - */ -static void -ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz) -{ - if (!rb->outbufsize) - { - rb->outbuf = MemoryContextAlloc(rb->context, sz); - rb->outbufsize = sz; - } - else if (rb->outbufsize < sz) - { - rb->outbuf = repalloc(rb->outbuf, sz); - rb->outbufsize = sz; - } -} - -/* * Check whether the transaction tx should spill its data to disk. */ static void @@ -2254,7 +2240,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) { dlist_iter subtxn_i; dlist_mutable_iter change_i; - int fd = -1; + TransientBufFile *file = NULL; XLogSegNo curOpenSegNo = 0; Size spilled = 0; @@ -2281,13 +2267,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) * store in segment in which it belongs by start lsn, don't split over * multiple segments tho */ - if (fd == -1 || + if (file == NULL || !XLByteInSeg(change->lsn, curOpenSegNo, wal_segment_size)) { char path[MAXPGPATH]; - if (fd != -1) - CloseTransientFile(fd); + if (file) + BufFileCloseTransient(file); XLByteToSeg(change->lsn, curOpenSegNo, wal_segment_size); @@ -2299,16 +2285,11 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) curOpenSegNo); /* open segment, create it if necessary */ - fd = OpenTransientFile(path, - O_CREAT | O_WRONLY | O_APPEND | PG_BINARY); - - if (fd < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not open file \"%s\": %m", path))); + file = BufFileOpenTransient(path, + O_CREAT | O_WRONLY | O_APPEND | PG_BINARY); } - ReorderBufferSerializeChange(rb, txn, fd, change); + ReorderBufferSerializeChange(rb, txn, file, change); dlist_delete(&change->node); ReorderBufferReturnChange(rb, change); @@ -2320,8 +2301,8 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) txn->nentries_mem = 0; txn->serialized = true; - if (fd != -1) - CloseTransientFile(fd); + if (file) + BufFileCloseTransient(file); } /* @@ -2329,15 +2310,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) */ static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, - int fd, ReorderBufferChange *change) + TransientBufFile *file, ReorderBufferChange *change) { - ReorderBufferDiskChange *ondisk; + ReorderBufferDiskChange hdr; Size sz = sizeof(ReorderBufferDiskChange); - ReorderBufferSerializeReserve(rb, sz); - - ondisk = (ReorderBufferDiskChange *) rb->outbuf; - memcpy(&ondisk->change, change, sizeof(ReorderBufferChange)); + memcpy((char *) &hdr + offsetof(ReorderBufferDiskChange, change), + change, sizeof(ReorderBufferChange)); switch (change->action) { @@ -2347,7 +2326,6 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, case REORDER_BUFFER_CHANGE_DELETE: case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT: { - char *data; ReorderBufferTupleBuf *oldtup, *newtup; Size oldlen = 0; @@ -2370,66 +2348,55 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, sz += newlen; } - /* make sure we have enough space */ - ReorderBufferSerializeReserve(rb, sz); - - data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange); - /* might have been reallocated above */ - ondisk = (ReorderBufferDiskChange *) rb->outbuf; + hdr.size = sz; + ReorderBufferWriteData(file, &hdr, sizeof(ReorderBufferDiskChange), + txn); if (oldlen) { - memcpy(data, &oldtup->tuple, sizeof(HeapTupleData)); - data += sizeof(HeapTupleData); - - memcpy(data, oldtup->tuple.t_data, oldlen); - data += oldlen; + ReorderBufferWriteData(file, &oldtup->tuple, + sizeof(HeapTupleData), txn); + ReorderBufferWriteData(file, oldtup->tuple.t_data, oldlen, + txn); } if (newlen) { - memcpy(data, &newtup->tuple, sizeof(HeapTupleData)); - data += sizeof(HeapTupleData); - - memcpy(data, newtup->tuple.t_data, newlen); - data += newlen; + ReorderBufferWriteData(file, &newtup->tuple, + sizeof(HeapTupleData), txn); + ReorderBufferWriteData(file, newtup->tuple.t_data, newlen, + txn); } break; } case REORDER_BUFFER_CHANGE_MESSAGE: { - char *data; Size prefix_size = strlen(change->data.msg.prefix) + 1; sz += prefix_size + change->data.msg.message_size + sizeof(Size) + sizeof(Size); - ReorderBufferSerializeReserve(rb, sz); - data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange); - - /* might have been reallocated above */ - ondisk = (ReorderBufferDiskChange *) rb->outbuf; + hdr.size = sz; + ReorderBufferWriteData(file, &hdr, + sizeof(ReorderBufferDiskChange), + txn); /* write the prefix including the size */ - memcpy(data, &prefix_size, sizeof(Size)); - data += sizeof(Size); - memcpy(data, change->data.msg.prefix, - prefix_size); - data += prefix_size; + ReorderBufferWriteData(file, &prefix_size, sizeof(Size), txn); + ReorderBufferWriteData(file, change->data.msg.prefix, + prefix_size, txn); /* write the message including the size */ - memcpy(data, &change->data.msg.message_size, sizeof(Size)); - data += sizeof(Size); - memcpy(data, change->data.msg.message, - change->data.msg.message_size); - data += change->data.msg.message_size; + ReorderBufferWriteData(file, &change->data.msg.message_size, + sizeof(Size), txn); + ReorderBufferWriteData(file, change->data.msg.message, + change->data.msg.message_size, txn); break; } case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT: { Snapshot snap; - char *data; snap = change->data.snapshot; @@ -2438,49 +2405,37 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, sizeof(TransactionId) * snap->subxcnt ; - /* make sure we have enough space */ - ReorderBufferSerializeReserve(rb, sz); - data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange); - /* might have been reallocated above */ - ondisk = (ReorderBufferDiskChange *) rb->outbuf; + hdr.size = sz; + ReorderBufferWriteData(file, &hdr, + sizeof(ReorderBufferDiskChange), txn); - memcpy(data, snap, sizeof(SnapshotData)); - data += sizeof(SnapshotData); + ReorderBufferWriteData(file, snap, sizeof(SnapshotData), txn); if (snap->xcnt) - { - memcpy(data, snap->xip, - sizeof(TransactionId) * snap->xcnt); - data += sizeof(TransactionId) * snap->xcnt; - } + ReorderBufferWriteData(file, snap->xip, + sizeof(TransactionId) * snap->xcnt, + txn); if (snap->subxcnt) - { - memcpy(data, snap->subxip, - sizeof(TransactionId) * snap->subxcnt); - data += sizeof(TransactionId) * snap->subxcnt; - } + ReorderBufferWriteData(file, snap->subxip, + sizeof(TransactionId) * snap->subxcnt, + txn); break; } case REORDER_BUFFER_CHANGE_TRUNCATE: { Size size; - char *data; /* account for the OIDs of truncated relations */ size = sizeof(Oid) * change->data.truncate.nrelids; sz += size; - /* make sure we have enough space */ - ReorderBufferSerializeReserve(rb, sz); - - data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange); - /* might have been reallocated above */ - ondisk = (ReorderBufferDiskChange *) rb->outbuf; - - memcpy(data, change->data.truncate.relids, size); - data += size; + hdr.size = sz; + ReorderBufferWriteData(file, &hdr, sizeof(ReorderBufferDiskChange), + txn); + ReorderBufferWriteData(file, change->data.truncate.relids, size, + txn); break; } case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM: @@ -2489,27 +2444,21 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, /* ReorderBufferChange contains everything important */ break; } +} - ondisk->size = sz; - - errno = 0; - pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_WRITE); - if (write(fd, rb->outbuf, ondisk->size) != ondisk->size) - { - int save_errno = errno; - - CloseTransientFile(fd); - - /* if write didn't set errno, assume problem is no disk space */ - errno = save_errno ? save_errno : ENOSPC; +/* + * Wrapper for BufFileWriteTransient() that raises ERROR if the whole chunk + * was not written. XXX Should this be a macro? + */ +static void +ReorderBufferWriteData(TransientBufFile *file, void *ptr, size_t size, + ReorderBufferTXN *txn) +{ + if (BufFileWriteTransient(file, ptr, size) != size) ereport(ERROR, (errcode_for_file_access(), errmsg("could not write to data file for XID %u: %m", txn->xid))); - } - pgstat_report_wait_end(); - - Assert(ondisk->change.action == change->action); } /* @@ -2517,7 +2466,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, */ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn, - int *fd, XLogSegNo *segno) + TransientBufFile **file, XLogSegNo *segno) { Size restored = 0; XLogSegNo last_segno; @@ -2542,10 +2491,7 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn, while (restored < max_changes_in_memory && *segno <= last_segno) { - int readBytes; - ReorderBufferDiskChange *ondisk; - - if (*fd == -1) + if (*file == NULL) { char path[MAXPGPATH]; @@ -2562,77 +2508,24 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn, ReorderBufferSerializedPath(path, MyReplicationSlot, txn->xid, *segno); - *fd = OpenTransientFile(path, O_RDONLY | PG_BINARY); - if (*fd < 0 && errno == ENOENT) + *file = BufFileOpenTransient(path, O_RDONLY | PG_BINARY); + if (*file == NULL) { - *fd = -1; + Assert(errno == ENOENT); (*segno)++; continue; } - else if (*fd < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not open file \"%s\": %m", - path))); } - /* - * Read the statically sized part of a change which has information - * about the total size. If we couldn't read a record, we're at the - * end of this file. - */ - ReorderBufferSerializeReserve(rb, sizeof(ReorderBufferDiskChange)); - pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_READ); - readBytes = read(*fd, rb->outbuf, sizeof(ReorderBufferDiskChange)); - pgstat_report_wait_end(); - - /* eof */ - if (readBytes == 0) + ReorderBufferRestoreChange(rb, txn, file); + if (*file) + restored++; + else { - CloseTransientFile(*fd); - *fd = -1; + /* No data could be restored. */ (*segno)++; continue; } - else if (readBytes < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not read from reorderbuffer spill file: %m"))); - else if (readBytes != sizeof(ReorderBufferDiskChange)) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes", - readBytes, - (uint32) sizeof(ReorderBufferDiskChange)))); - - ondisk = (ReorderBufferDiskChange *) rb->outbuf; - - ReorderBufferSerializeReserve(rb, - sizeof(ReorderBufferDiskChange) + ondisk->size); - ondisk = (ReorderBufferDiskChange *) rb->outbuf; - - pgstat_report_wait_start(WAIT_EVENT_REORDER_BUFFER_READ); - readBytes = read(*fd, rb->outbuf + sizeof(ReorderBufferDiskChange), - ondisk->size - sizeof(ReorderBufferDiskChange)); - pgstat_report_wait_end(); - - if (readBytes < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not read from reorderbuffer spill file: %m"))); - else if (readBytes != ondisk->size - sizeof(ReorderBufferDiskChange)) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes", - readBytes, - (uint32) (ondisk->size - sizeof(ReorderBufferDiskChange))))); - - /* - * ok, read a full change from disk, now restore it into proper - * in-memory format - */ - ReorderBufferRestoreChange(rb, txn, rb->outbuf); - restored++; } return restored; @@ -2642,25 +2535,36 @@ ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn, * Convert change from its on-disk format to in-memory format and queue it onto * the TXN's ->changes list. * - * Note: although "data" is declared char*, at entry it points to a - * maxalign'd buffer, making it safe in most of this function to assume - * that the pointed-to data is suitably aligned for direct access. + * If no data was found in the file, close it and set *file to NULL. */ static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, - char *data) + TransientBufFile **file) { - ReorderBufferDiskChange *ondisk; + ReorderBufferDiskChange ondisk; + bool no_data; ReorderBufferChange *change; - ondisk = (ReorderBufferDiskChange *) data; + /* + * Read the statically sized part of a change which has information about + * the total size. If we couldn't read a record, we're at the end of this + * file. + */ + ReorderBufferReadData(*file, &ondisk, sizeof(ReorderBufferDiskChange), + &no_data); + + /* eof */ + if (no_data) + { + BufFileCloseTransient(*file); + *file = NULL; + return; + } change = ReorderBufferGetChange(rb); /* copy static part */ - memcpy(change, &ondisk->change, sizeof(ReorderBufferChange)); - - data += sizeof(ReorderBufferDiskChange); + memcpy(change, &ondisk.change, sizeof(ReorderBufferChange)); /* restore individual stuff */ switch (change->action) @@ -2671,50 +2575,10 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, case REORDER_BUFFER_CHANGE_DELETE: case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT: if (change->data.tp.oldtuple) - { - uint32 tuplelen = ((HeapTuple) data)->t_len; - - change->data.tp.oldtuple = - ReorderBufferGetTupleBuf(rb, tuplelen - SizeofHeapTupleHeader); - - /* restore ->tuple */ - memcpy(&change->data.tp.oldtuple->tuple, data, - sizeof(HeapTupleData)); - data += sizeof(HeapTupleData); - - /* reset t_data pointer into the new tuplebuf */ - change->data.tp.oldtuple->tuple.t_data = - ReorderBufferTupleBufData(change->data.tp.oldtuple); - - /* restore tuple data itself */ - memcpy(change->data.tp.oldtuple->tuple.t_data, data, tuplelen); - data += tuplelen; - } + change->data.tp.oldtuple = ReorderBufferRestoreTuple(rb, *file); if (change->data.tp.newtuple) - { - /* here, data might not be suitably aligned! */ - uint32 tuplelen; - - memcpy(&tuplelen, data + offsetof(HeapTupleData, t_len), - sizeof(uint32)); - - change->data.tp.newtuple = - ReorderBufferGetTupleBuf(rb, tuplelen - SizeofHeapTupleHeader); - - /* restore ->tuple */ - memcpy(&change->data.tp.newtuple->tuple, data, - sizeof(HeapTupleData)); - data += sizeof(HeapTupleData); - - /* reset t_data pointer into the new tuplebuf */ - change->data.tp.newtuple->tuple.t_data = - ReorderBufferTupleBufData(change->data.tp.newtuple); - - /* restore tuple data itself */ - memcpy(change->data.tp.newtuple->tuple.t_data, data, tuplelen); - data += tuplelen; - } + change->data.tp.newtuple = ReorderBufferRestoreTuple(rb, *file); break; case REORDER_BUFFER_CHANGE_MESSAGE: @@ -2722,44 +2586,44 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, Size prefix_size; /* read prefix */ - memcpy(&prefix_size, data, sizeof(Size)); - data += sizeof(Size); + ReorderBufferReadData(*file, &prefix_size, sizeof(Size), NULL); change->data.msg.prefix = MemoryContextAlloc(rb->context, prefix_size); - memcpy(change->data.msg.prefix, data, prefix_size); + ReorderBufferReadData(*file, change->data.msg.prefix, + prefix_size, NULL); Assert(change->data.msg.prefix[prefix_size - 1] == '\0'); - data += prefix_size; /* read the message */ - memcpy(&change->data.msg.message_size, data, sizeof(Size)); - data += sizeof(Size); + ReorderBufferReadData(*file, &change->data.msg.message_size, + sizeof(Size), NULL); change->data.msg.message = MemoryContextAlloc(rb->context, change->data.msg.message_size); - memcpy(change->data.msg.message, data, - change->data.msg.message_size); - data += change->data.msg.message_size; + ReorderBufferReadData(*file, change->data.msg.message, + change->data.msg.message_size, NULL); break; } case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT: { - Snapshot oldsnap; + SnapshotData oldsnap; Snapshot newsnap; Size size; - oldsnap = (Snapshot) data; + ReorderBufferReadData(*file, &oldsnap, sizeof(SnapshotData), NULL); size = sizeof(SnapshotData) + - sizeof(TransactionId) * oldsnap->xcnt + - sizeof(TransactionId) * (oldsnap->subxcnt + 0); + sizeof(TransactionId) * oldsnap.xcnt + + sizeof(TransactionId) * (oldsnap.subxcnt + 0); change->data.snapshot = MemoryContextAllocZero(rb->context, size); newsnap = change->data.snapshot; - memcpy(newsnap, data, size); + memcpy(newsnap, &oldsnap, sizeof(SnapshotData)); newsnap->xip = (TransactionId *) (((char *) newsnap) + sizeof(SnapshotData)); + ReorderBufferReadData(*file, newsnap->xip, + size - sizeof(SnapshotData), NULL); newsnap->subxip = newsnap->xip + newsnap->xcnt; newsnap->copied = true; break; @@ -2771,7 +2635,9 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, relids = ReorderBufferGetRelids(rb, change->data.truncate.nrelids); - memcpy(relids, data, change->data.truncate.nrelids * sizeof(Oid)); + ReorderBufferReadData(*file, relids, + change->data.truncate.nrelids * sizeof(Oid), + NULL); change->data.truncate.relids = relids; break; @@ -2787,6 +2653,77 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, } /* + * Convert heap tuple from its on-disk format to in-memory format. + */ +static ReorderBufferTupleBuf * +ReorderBufferRestoreTuple(ReorderBuffer *rb, TransientBufFile *file) +{ + HeapTupleData tupdata; + uint32 tuplelen; + ReorderBufferTupleBuf *result; + + ReorderBufferReadData(file, &tupdata, sizeof(HeapTupleData), NULL); + tuplelen = tupdata.t_len; + + result = ReorderBufferGetTupleBuf(rb, tuplelen - SizeofHeapTupleHeader); + + /* restore ->tuple */ + memcpy(&result->tuple, &tupdata, sizeof(HeapTupleData)); + + /* reset t_data pointer into the new tuplebuf */ + result->tuple.t_data = ReorderBufferTupleBufData(result); + + /* restore tuple data itself */ + ReorderBufferReadData(file, result->tuple.t_data, tuplelen, NULL); + + return result; +} + +/* + * Wrapper for BufFileReadTransient() that raises ERROR if the expected amount + * of bytes was not read. + * + * If valid pointer is passed for no_data_p, set *no_data_p to indicate + * whether zero bytes was read. If NULL is passed, do not tolerate missing + * data. + */ +static void +ReorderBufferReadData(TransientBufFile *file, void *ptr, size_t size, + bool *no_data_p) +{ + int readBytes; + + /* + * Caller should not request zero bytes. This assumption simplifies + * setting of *no_data_p below. + */ + Assert(size > 0); + + if ((readBytes = BufFileReadTransient(file, ptr, size)) != size) + { + if (no_data_p) + *no_data_p = readBytes == 0; + + /* + * It is o.k. to receive exactly zero bytes if caller passed valid + * no_data_p. + */ + if (no_data_p && *no_data_p) + return; + + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read from reorderbuffer spill file: read %d instead of %u bytes", + readBytes, (uint32) size))); + } + else if (no_data_p) + { + /* Given that size is non-zero, readBytes must be non-zero too. */ + *no_data_p = false; + } +} + +/* * Remove all on-disk stored for the passed in transaction. */ static void diff --git a/src/backend/storage/file/buffile.c b/src/backend/storage/file/buffile.c index b773a76049..b31324627a 100644 --- a/src/backend/storage/file/buffile.c +++ b/src/backend/storage/file/buffile.c @@ -41,6 +41,8 @@ #include "postgres.h" +#include + #include "commands/tablespace.h" #include "executor/instrument.h" #include "miscadmin.h" @@ -59,19 +61,44 @@ #define BUFFILE_SEG_SIZE (MAX_PHYSICAL_FILESIZE / BLCKSZ) /* + * Fields that both BufFile and TransientBufFile structures need. It must be + * the first field of those structures. + */ +typedef struct BufFileCommon +{ + bool dirty; /* does buffer need to be written? */ + int pos; /* next read/write position in buffer */ + int nbytes; /* total # of valid bytes in buffer */ + + /* + * "current pos" is position of start of buffer within the logical file. + * Position as seen by user of BufFile is (curFile, curOffset + pos). + */ + int curFile; /* file index (0..n) part of current pos, + * always zero for TransientBufFile */ + off_t curOffset; /* offset part of current pos */ + + bool readOnly; /* has the file been set to read only? */ + + bool append; /* should new data be appended to the end? */ + + PGAlignedBlock buffer; +} BufFileCommon; + +/* * This data structure represents a buffered file that consists of one or * more physical files (each accessed through a virtual file descriptor * managed by fd.c). */ struct BufFile { + BufFileCommon common; /* Common fields, see above. */ + int numFiles; /* number of physical files in set */ /* all files except the last have length exactly MAX_PHYSICAL_FILESIZE */ File *files; /* palloc'd array with numFiles entries */ bool isInterXact; /* keep open over transactions? */ - bool dirty; /* does buffer need to be written? */ - bool readOnly; /* has the file been set to read only? */ SharedFileSet *fileset; /* space for segment files if shared */ const char *name; /* name of this BufFile if shared */ @@ -82,16 +109,20 @@ struct BufFile * because after creation we only repalloc our arrays larger.) */ ResourceOwner resowner; +}; - /* - * "current pos" is position of start of buffer within the logical file. - * Position as seen by user of BufFile is (curFile, curOffset + pos). - */ - int curFile; /* file index (0..n) part of current pos */ - off_t curOffset; /* offset part of current pos */ - int pos; /* next read/write position in buffer */ - int nbytes; /* total # of valid bytes in buffer */ - PGAlignedBlock buffer; +/* + * Buffered variant of a transient file. Unlike BufFile this is simpler in + * several ways: 1) it's not split into segments, 2) there's no need of seek, + * 3) there's no need to combine read and write access. + */ +struct TransientBufFile +{ + BufFileCommon common; /* Common fields, see above. */ + + /* The underlying file. */ + char *path; + int fd; }; static BufFile *makeBufFileCommon(int nfiles); @@ -102,22 +133,32 @@ static void BufFileDumpBuffer(BufFile *file); static int BufFileFlush(BufFile *file); static File MakeNewSharedSegment(BufFile *file, int segment); +static void BufFileLoadBufferTransient(TransientBufFile *file); +static void BufFileDumpBufferTransient(TransientBufFile *file); + +static size_t BufFileReadCommon(BufFileCommon *file, void *ptr, size_t size, + bool is_transient); +static size_t BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, + bool is_transient); + /* * Create BufFile and perform the common initialization. */ static BufFile * makeBufFileCommon(int nfiles) { - BufFile *file = (BufFile *) palloc(sizeof(BufFile)); + BufFile *file = (BufFile *) palloc0(sizeof(BufFile)); + BufFileCommon *fcommon = &file->common; + + fcommon->dirty = false; + fcommon->curFile = 0; + fcommon->curOffset = 0L; + fcommon->pos = 0; + fcommon->nbytes = 0; file->numFiles = nfiles; file->isInterXact = false; - file->dirty = false; file->resowner = CurrentResourceOwner; - file->curFile = 0; - file->curOffset = 0L; - file->pos = 0; - file->nbytes = 0; return file; } @@ -133,7 +174,7 @@ makeBufFile(File firstfile) file->files = (File *) palloc(sizeof(File)); file->files[0] = firstfile; - file->readOnly = false; + file->common.readOnly = false; file->fileset = NULL; file->name = NULL; @@ -264,7 +305,7 @@ BufFileCreateShared(SharedFileSet *fileset, const char *name) file->name = pstrdup(name); file->files = (File *) palloc(sizeof(File)); file->files[0] = MakeNewSharedSegment(file, 0); - file->readOnly = false; + file->common.readOnly = false; return file; } @@ -321,7 +362,7 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name) file = makeBufFileCommon(nfiles); file->files = files; - file->readOnly = true; /* Can't write to files opened this way */ + file->common.readOnly = true; /* Can't write to files opened this way */ file->fileset = fileset; file->name = pstrdup(name); @@ -376,10 +417,10 @@ BufFileExportShared(BufFile *file) Assert(file->fileset != NULL); /* It's probably a bug if someone calls this twice. */ - Assert(!file->readOnly); + Assert(!file->common.readOnly); BufFileFlush(file); - file->readOnly = true; + file->common.readOnly = true; } /* @@ -406,7 +447,7 @@ BufFileClose(BufFile *file) * BufFileLoadBuffer * * Load some data into buffer, if possible, starting from curOffset. - * At call, must have dirty = false, pos and nbytes = 0. + * At call, must have dirty = false, nbytes = 0. * On exit, nbytes is number of bytes loaded. */ static void @@ -417,27 +458,27 @@ BufFileLoadBuffer(BufFile *file) /* * Advance to next component file if necessary and possible. */ - if (file->curOffset >= MAX_PHYSICAL_FILESIZE && - file->curFile + 1 < file->numFiles) + if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE && + file->common.curFile + 1 < file->numFiles) { - file->curFile++; - file->curOffset = 0L; + file->common.curFile++; + file->common.curOffset = 0L; } /* * Read whatever we can get, up to a full bufferload. */ - thisfile = file->files[file->curFile]; - file->nbytes = FileRead(thisfile, - file->buffer.data, - sizeof(file->buffer), - file->curOffset, - WAIT_EVENT_BUFFILE_READ); - if (file->nbytes < 0) - file->nbytes = 0; + thisfile = file->files[file->common.curFile]; + file->common.nbytes = FileRead(thisfile, + file->common.buffer.data, + sizeof(file->common.buffer), + file->common.curOffset, + WAIT_EVENT_BUFFILE_READ); + if (file->common.nbytes < 0) + file->common.nbytes = 0; /* we choose not to advance curOffset here */ - if (file->nbytes > 0) + if (file->common.nbytes > 0) pgBufferUsage.temp_blks_read++; } @@ -459,44 +500,44 @@ BufFileDumpBuffer(BufFile *file) * Unlike BufFileLoadBuffer, we must dump the whole buffer even if it * crosses a component-file boundary; so we need a loop. */ - while (wpos < file->nbytes) + while (wpos < file->common.nbytes) { off_t availbytes; /* * Advance to next component file if necessary and possible. */ - if (file->curOffset >= MAX_PHYSICAL_FILESIZE) + if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE) { - while (file->curFile + 1 >= file->numFiles) + while (file->common.curFile + 1 >= file->numFiles) extendBufFile(file); - file->curFile++; - file->curOffset = 0L; + file->common.curFile++; + file->common.curOffset = 0L; } /* * Determine how much we need to write into this file. */ - bytestowrite = file->nbytes - wpos; - availbytes = MAX_PHYSICAL_FILESIZE - file->curOffset; + bytestowrite = file->common.nbytes - wpos; + availbytes = MAX_PHYSICAL_FILESIZE - file->common.curOffset; if ((off_t) bytestowrite > availbytes) bytestowrite = (int) availbytes; - thisfile = file->files[file->curFile]; + thisfile = file->files[file->common.curFile]; bytestowrite = FileWrite(thisfile, - file->buffer.data + wpos, + file->common.buffer.data + wpos, bytestowrite, - file->curOffset, + file->common.curOffset, WAIT_EVENT_BUFFILE_WRITE); if (bytestowrite <= 0) return; /* failed to write */ - file->curOffset += bytestowrite; + file->common.curOffset += bytestowrite; wpos += bytestowrite; pgBufferUsage.temp_blks_written++; } - file->dirty = false; + file->common.dirty = false; /* * At this point, curOffset has been advanced to the end of the buffer, @@ -504,19 +545,19 @@ BufFileDumpBuffer(BufFile *file) * logical file position, ie, original value + pos, in case that is less * (as could happen due to a small backwards seek in a dirty buffer!) */ - file->curOffset -= (file->nbytes - file->pos); - if (file->curOffset < 0) /* handle possible segment crossing */ + file->common.curOffset -= (file->common.nbytes - file->common.pos); + if (file->common.curOffset < 0) /* handle possible segment crossing */ { - file->curFile--; - Assert(file->curFile >= 0); - file->curOffset += MAX_PHYSICAL_FILESIZE; + file->common.curFile--; + Assert(file->common.curFile >= 0); + file->common.curOffset += MAX_PHYSICAL_FILESIZE; } /* * Now we can set the buffer empty without changing the logical position */ - file->pos = 0; - file->nbytes = 0; + file->common.pos = 0; + file->common.nbytes = 0; } /* @@ -527,43 +568,7 @@ BufFileDumpBuffer(BufFile *file) size_t BufFileRead(BufFile *file, void *ptr, size_t size) { - size_t nread = 0; - size_t nthistime; - - if (file->dirty) - { - if (BufFileFlush(file) != 0) - return 0; /* could not flush... */ - Assert(!file->dirty); - } - - while (size > 0) - { - if (file->pos >= file->nbytes) - { - /* Try to load more data into buffer. */ - file->curOffset += file->pos; - file->pos = 0; - file->nbytes = 0; - BufFileLoadBuffer(file); - if (file->nbytes <= 0) - break; /* no more data available */ - } - - nthistime = file->nbytes - file->pos; - if (nthistime > size) - nthistime = size; - Assert(nthistime > 0); - - memcpy(ptr, file->buffer.data + file->pos, nthistime); - - file->pos += nthistime; - ptr = (void *) ((char *) ptr + nthistime); - size -= nthistime; - nread += nthistime; - } - - return nread; + return BufFileReadCommon(&file->common, ptr, size, false); } /* @@ -574,48 +579,7 @@ BufFileRead(BufFile *file, void *ptr, size_t size) size_t BufFileWrite(BufFile *file, void *ptr, size_t size) { - size_t nwritten = 0; - size_t nthistime; - - Assert(!file->readOnly); - - while (size > 0) - { - if (file->pos >= BLCKSZ) - { - /* Buffer full, dump it out */ - if (file->dirty) - { - BufFileDumpBuffer(file); - if (file->dirty) - break; /* I/O error */ - } - else - { - /* Hmm, went directly from reading to writing? */ - file->curOffset += file->pos; - file->pos = 0; - file->nbytes = 0; - } - } - - nthistime = BLCKSZ - file->pos; - if (nthistime > size) - nthistime = size; - Assert(nthistime > 0); - - memcpy(file->buffer.data + file->pos, ptr, nthistime); - - file->dirty = true; - file->pos += nthistime; - if (file->nbytes < file->pos) - file->nbytes = file->pos; - ptr = (void *) ((char *) ptr + nthistime); - size -= nthistime; - nwritten += nthistime; - } - - return nwritten; + return BufFileWriteCommon(&file->common, ptr, size, false); } /* @@ -626,10 +590,10 @@ BufFileWrite(BufFile *file, void *ptr, size_t size) static int BufFileFlush(BufFile *file) { - if (file->dirty) + if (file->common.dirty) { BufFileDumpBuffer(file); - if (file->dirty) + if (file->common.dirty) return EOF; } @@ -667,8 +631,8 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) * fileno. Note that large offsets (> 1 gig) risk overflow in this * add, unless we have 64-bit off_t. */ - newFile = file->curFile; - newOffset = (file->curOffset + file->pos) + offset; + newFile = file->common.curFile; + newOffset = (file->common.curOffset + file->common.pos) + offset; break; #ifdef NOT_USED case SEEK_END: @@ -685,9 +649,9 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) return EOF; newOffset += MAX_PHYSICAL_FILESIZE; } - if (newFile == file->curFile && - newOffset >= file->curOffset && - newOffset <= file->curOffset + file->nbytes) + if (newFile == file->common.curFile && + newOffset >= file->common.curOffset && + newOffset <= file->common.curOffset + file->common.nbytes) { /* * Seek is to a point within existing buffer; we can just adjust @@ -695,7 +659,7 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) * whether reading or writing, but buffer remains dirty if we were * writing. */ - file->pos = (int) (newOffset - file->curOffset); + file->common.pos = (int) (newOffset - file->common.curOffset); return 0; } /* Otherwise, must reposition buffer, so flush any dirty data */ @@ -723,18 +687,18 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) if (newFile >= file->numFiles) return EOF; /* Seek is OK! */ - file->curFile = newFile; - file->curOffset = newOffset; - file->pos = 0; - file->nbytes = 0; + file->common.curFile = newFile; + file->common.curOffset = newOffset; + file->common.pos = 0; + file->common.nbytes = 0; return 0; } void BufFileTell(BufFile *file, int *fileno, off_t *offset) { - *fileno = file->curFile; - *offset = file->curOffset + file->pos; + *fileno = file->common.curFile; + *offset = file->common.curOffset + file->common.pos; } /* @@ -768,8 +732,8 @@ BufFileTellBlock(BufFile *file) { long blknum; - blknum = (file->curOffset + file->pos) / BLCKSZ; - blknum += file->curFile * BUFFILE_SEG_SIZE; + blknum = (file->common.curOffset + file->common.pos) / BLCKSZ; + blknum += file->common.curFile * BUFFILE_SEG_SIZE; return blknum; } @@ -828,8 +792,8 @@ BufFileAppend(BufFile *target, BufFile *source) int i; Assert(target->fileset != NULL); - Assert(source->readOnly); - Assert(!source->dirty); + Assert(source->common.readOnly); + Assert(!source->common.dirty); Assert(source->fileset != NULL); if (target->resowner != source->resowner) @@ -843,3 +807,330 @@ BufFileAppend(BufFile *target, BufFile *source) return startBlock; } + +/* + * Open TransientBufFile at given path or create one if it does not + * exist. User will be allowed either to write to the file or to read from it, + * according to fileFlags, but not both. + */ +TransientBufFile * +BufFileOpenTransient(const char *path, int fileFlags) +{ + bool readOnly; + bool append = false; + TransientBufFile *file; + BufFileCommon *fcommon; + int fd; + off_t size; + + /* Either read or write mode, but not both. */ + Assert((fileFlags & O_RDWR) == 0); + + /* Check whether user wants read or write access. */ + readOnly = (fileFlags & O_WRONLY) == 0; + + /* + * Append mode for read access is not useful, so don't bother implementing + * it. + */ + Assert(!(readOnly && append)); + + errno = 0; + fd = OpenTransientFile(path, fileFlags); + if (fd < 0) + { + /* + * If caller wants to read from file and the file is not there, he + * should be able to handle the condition on his own. + * + * XXX Shouldn't we always let caller evaluate errno? + */ + if (errno == ENOENT && (fileFlags & O_RDONLY)) + return NULL; + + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\": %m", path))); + } + + file = (TransientBufFile *) palloc(sizeof(TransientBufFile)); + fcommon = &file->common; + fcommon->dirty = false; + fcommon->pos = 0; + fcommon->nbytes = 0; + fcommon->readOnly = readOnly; + fcommon->append = append; + fcommon->curFile = 0; + + file->path = pstrdup(path); + file->fd = fd; + + errno = 0; + size = lseek(file->fd, 0, SEEK_END); + if (errno > 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not initialize TransientBufFile for file \"%s\": %m", + path))); + + if (fcommon->append) + { + /* Position the buffer at the end of the file. */ + fcommon->curOffset = size; + } + else + fcommon->curOffset = 0L; + + return file; +} + +/* + * Close a TransientBufFile. + */ +void +BufFileCloseTransient(TransientBufFile *file) +{ + /* Flush any unwritten data. */ + if (!file->common.readOnly && + file->common.dirty && file->common.nbytes > 0) + { + BufFileDumpBufferTransient(file); + + /* + * Caller of BufFileWriteTransient() recognizes the failure to flush + * buffer by the returned value, however this function has no return + * code. + */ + if (file->common.dirty) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not flush file \"%s\": %m", file->path))); + } + + if (CloseTransientFile(file->fd)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", file->path))); + + pfree(file->path); + pfree(file); +} + +/* + * Load some data into buffer, if possible, starting from file->offset. At + * call, must have dirty = false, pos and nbytes = 0. On exit, nbytes is + * number of bytes loaded. + */ +static void +BufFileLoadBufferTransient(TransientBufFile *file) +{ + Assert(file->common.readOnly); + Assert(!file->common.dirty); + Assert(file->common.pos == 0 && file->common.nbytes == 0); + +retry: + + /* + * Read whatever we can get, up to a full bufferload. + */ + errno = 0; + pgstat_report_wait_start(WAIT_EVENT_BUFFILE_READ); + file->common.nbytes = pg_pread(file->fd, + file->common.buffer.data, + sizeof(file->common.buffer), + file->common.curOffset); + pgstat_report_wait_end(); + + if (file->common.nbytes < 0) + { + /* TODO The W32 specific code, see FileWrite. */ + + /* OK to retry if interrupted */ + if (errno == EINTR) + goto retry; + + return; + } + /* we choose not to advance offset here */ +} + +/* + * Write contents of a transient file buffer to disk. + */ +static void +BufFileDumpBufferTransient(TransientBufFile *file) +{ + int nwritten; + + /* This function should only be needed during write access ... */ + Assert(!file->common.readOnly); + + /* ... and if there's some work to do. */ + Assert(file->common.dirty); + Assert(file->common.nbytes > 0); + +retry: + errno = 0; + pgstat_report_wait_start(WAIT_EVENT_BUFFILE_WRITE); + nwritten = pg_pwrite(file->fd, + file->common.buffer.data, + file->common.nbytes, + file->common.curOffset); + pgstat_report_wait_end(); + + /* if write didn't set errno, assume problem is no disk space */ + if (nwritten != file->common.nbytes && errno == 0) + errno = ENOSPC; + + if (nwritten < 0) + { + /* TODO The W32 specific code, see FileWrite. */ + + /* OK to retry if interrupted */ + if (errno == EINTR) + goto retry; + + return; /* failed to write */ + } + + file->common.dirty = false; + + file->common.pos = 0; + file->common.nbytes = 0; +} + +/* + * Like BufFileRead() except it receives pointer to TransientBufFile. + */ +size_t +BufFileReadTransient(TransientBufFile *file, void *ptr, size_t size) +{ + return BufFileReadCommon(&file->common, ptr, size, true); +} + +/* + * Like BufFileWrite() except it receives pointer to TransientBufFile. + */ +size_t +BufFileWriteTransient(TransientBufFile *file, void *ptr, size_t size) +{ + return BufFileWriteCommon(&file->common, ptr, size, true); +} + +/* + * BufFileWriteCommon + * + * Functionality needed by both BufFileRead() and BufFileReadTransient(). + */ +static size_t +BufFileReadCommon(BufFileCommon *file, void *ptr, size_t size, + bool is_transient) +{ + size_t nread = 0; + size_t nthistime; + + if (file->dirty) + { + /* + * Transient file currently does not allow both read and write access, + * so this function should not see dirty buffer. + */ + Assert(!is_transient); + + if (BufFileFlush((BufFile *) file) != 0) + return 0; /* could not flush... */ + Assert(!file->dirty); + } + + while (size > 0) + { + if (file->pos >= file->nbytes) + { + /* Try to load more data into buffer. */ + file->curOffset += file->pos; + file->pos = 0; + file->nbytes = 0; + + if (!is_transient) + BufFileLoadBuffer((BufFile *) file); + else + BufFileLoadBufferTransient((TransientBufFile *) file); + + if (file->nbytes <= 0) + break; /* no more data available */ + } + + nthistime = file->nbytes - file->pos; + if (nthistime > size) + nthistime = size; + Assert(nthistime > 0); + + memcpy(ptr, file->buffer.data + file->pos, nthistime); + + file->pos += nthistime; + ptr = (void *) ((char *) ptr + nthistime); + size -= nthistime; + nread += nthistime; + } + + return nread; +} + +/* + * BufFileWriteCommon + * + * Functionality needed by both BufFileWrite() and BufFileWriteTransient(). + */ +static size_t +BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, + bool is_transient) +{ + size_t nwritten = 0; + size_t nthistime; + + Assert(!file->readOnly); + + while (size > 0) + { + if (file->pos >= BLCKSZ) + { + /* Buffer full, dump it out */ + if (file->dirty) + { + if (!is_transient) + BufFileDumpBuffer((BufFile *) file); + else + BufFileDumpBufferTransient((TransientBufFile *) file); + + if (file->dirty) + break; /* I/O error */ + } + else + { + Assert(!is_transient); + + /* Hmm, went directly from reading to writing? */ + file->curOffset += file->pos; + file->pos = 0; + file->nbytes = 0; + } + } + + nthistime = BLCKSZ - file->pos; + if (nthistime > size) + nthistime = size; + Assert(nthistime > 0); + + memcpy(file->buffer.data + file->pos, ptr, nthistime); + + file->dirty = true; + file->pos += nthistime; + if (file->nbytes < file->pos) + file->nbytes = file->pos; + ptr = (void *) ((char *) ptr + nthistime); + size -= nthistime; + nwritten += nthistime; + } + + return nwritten; +} diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 0a3ad3a188..b923a182a4 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -912,8 +912,6 @@ typedef enum WAIT_EVENT_RELATION_MAP_READ, WAIT_EVENT_RELATION_MAP_SYNC, WAIT_EVENT_RELATION_MAP_WRITE, - WAIT_EVENT_REORDER_BUFFER_READ, - WAIT_EVENT_REORDER_BUFFER_WRITE, WAIT_EVENT_REORDER_LOGICAL_MAPPING_READ, WAIT_EVENT_REPLICATION_SLOT_READ, WAIT_EVENT_REPLICATION_SLOT_RESTORE_SYNC, diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index 735e6d32b8..01fc91a8d0 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -387,10 +387,6 @@ struct ReorderBuffer MemoryContext tup_context; XLogRecPtr current_restart_decoding_lsn; - - /* buffer for disk<->memory conversions */ - char *outbuf; - Size outbufsize; }; diff --git a/src/include/storage/buffile.h b/src/include/storage/buffile.h index 1fba404fe2..625fa0014c 100644 --- a/src/include/storage/buffile.h +++ b/src/include/storage/buffile.h @@ -28,9 +28,13 @@ #include "storage/sharedfileset.h" -/* BufFile is an opaque type whose details are not known outside buffile.c. */ +/* + * BufFile and TransientBufFile are opaque types whose details are not known + * outside buffile.c. + */ typedef struct BufFile BufFile; +typedef struct TransientBufFile TransientBufFile; /* * prototypes for functions in buffile.c @@ -51,4 +55,11 @@ extern void BufFileExportShared(BufFile *file); extern BufFile *BufFileOpenShared(SharedFileSet *fileset, const char *name); extern void BufFileDeleteShared(SharedFileSet *fileset, const char *name); +extern TransientBufFile *BufFileOpenTransient(const char *path, int fileFlags); +extern void BufFileCloseTransient(TransientBufFile *file); +extern size_t BufFileReadTransient(TransientBufFile *file, void *ptr, + size_t size); +extern size_t BufFileWriteTransient(TransientBufFile *file, void *ptr, + size_t size); + #endif /* BUFFILE_H */ -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0011-Make-buffile.c-aware-of-encryption.patch From 98f42482f8e2b78dd8368a35d63944bb83c02293 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:01 +0200 Subject: [PATCH 11/17] Make buffile.c aware of encryption. This includes temporary files as well as the transient files introduced by this patch series. One particular problem encountered here is that the whole multiple of encryption block needs to be written, backend needs to know how many bytes at the end of file segment is just padding. The backend can easily remember it for normal temporary file, but for shared temporary file also this information needs to be shared. Currently we append 4-byte word at the end of the segment which informs on length of the useful data. Should this information be shared via shared memory instead? No padding would be needed if we used stream cipher, but that's not safe, README.encryption) --- src/backend/storage/file/buffile.c | 1047 +++++++++++++++++++++++++++++++++++- 1 file changed, 1021 insertions(+), 26 deletions(-) diff --git a/src/backend/storage/file/buffile.c b/src/backend/storage/file/buffile.c index b31324627a..308ceb01b7 100644 --- a/src/backend/storage/file/buffile.c +++ b/src/backend/storage/file/buffile.c @@ -44,18 +44,25 @@ #include #include "commands/tablespace.h" +#include "common/sha2.h" #include "executor/instrument.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/fd.h" #include "storage/buffile.h" #include "storage/buf_internals.h" +#include "storage/encryption.h" +#include "utils/datetime.h" #include "utils/resowner.h" /* * We break BufFiles into gigabyte-sized segments, regardless of RELSEG_SIZE. * The reason is that we'd like large BufFiles to be spread across multiple * tablespaces when available. + * + * An integer value indicating the number of useful bytes in the segment is + * appended to each segment of if the file is both shared and encrypted, see + * BufFile.useful. */ #define MAX_PHYSICAL_FILESIZE 0x40000000 #define BUFFILE_SEG_SIZE (MAX_PHYSICAL_FILESIZE / BLCKSZ) @@ -82,6 +89,27 @@ typedef struct BufFileCommon bool append; /* should new data be appended to the end? */ + /* + * If the file is encrypted, only the whole buffer can be loaded / dumped + * --- see BufFileLoadBuffer() for more info --- whether it's space is + * used up or not. Therefore we need to keep track of the actual on-disk + * size buffer of each component file, as it would be if there was no + * encryption. + * + * List would make coding simpler, however it would not be good for + * performance. Random access is important here. + */ + off_t *useful; + + /* + * The "useful" array may need to be expanded independent from + * extendBufFile() (i.e. earlier than the buffer gets dumped), so store + * the number of elements separate from numFiles. + * + * Always 1 for TransientBufFile. + */ + int nuseful; + PGAlignedBlock buffer; } BufFileCommon; @@ -98,12 +126,26 @@ struct BufFile /* all files except the last have length exactly MAX_PHYSICAL_FILESIZE */ File *files; /* palloc'd array with numFiles entries */ + /* + * Segment number is used to compute encryption tweak so we must remember + * the original numbers of segments if the file is encrypted and if it was + * passed as target to BufFileAppend() at least once. If this field is + * NULL, ->curFile is used to compute the tweak. + */ + off_t *segnos; + bool isInterXact; /* keep open over transactions? */ SharedFileSet *fileset; /* space for segment files if shared */ const char *name; /* name of this BufFile if shared */ /* + * Per-PID identifier if the file is encrypted and not shared. Used for + * tweak computation. + */ + uint32 number; + + /* * resowner is the ResourceOwner to use for underlying temp files. (We * don't need to remember the memory context we're using explicitly, * because after creation we only repalloc our arrays larger.) @@ -130,16 +172,22 @@ static BufFile *makeBufFile(File firstfile); static void extendBufFile(BufFile *file); static void BufFileLoadBuffer(BufFile *file); static void BufFileDumpBuffer(BufFile *file); +static void BufFileDumpBufferEncrypted(BufFile *file); static int BufFileFlush(BufFile *file); static File MakeNewSharedSegment(BufFile *file, int segment); +static void BufFileTweak(char *tweak, BufFileCommon *file, bool is_transient); +static void ensureUsefulArraySize(BufFileCommon *file, int required); +static void BufFileAppendMetadata(BufFile *target, BufFile *source); + static void BufFileLoadBufferTransient(TransientBufFile *file); static void BufFileDumpBufferTransient(TransientBufFile *file); static size_t BufFileReadCommon(BufFileCommon *file, void *ptr, size_t size, bool is_transient); static size_t BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, - bool is_transient); + bool is_transient); +static void BufFileUpdateUsefulLength(BufFileCommon *file, bool is_transient); /* * Create BufFile and perform the common initialization. @@ -160,6 +208,25 @@ makeBufFileCommon(int nfiles) file->isInterXact = false; file->resowner = CurrentResourceOwner; + if (data_encrypted) + { + fcommon->useful = (off_t *) palloc0(sizeof(off_t) * nfiles); + fcommon->nuseful = nfiles; + + file->segnos = NULL; + + /* + * The unused (trailing) part of the buffer should not contain + * undefined data: if we encrypt such a buffer and flush it to disk, + * the encrypted form of that "undefined part" can get zeroed due to + * seek and write beyond EOF. If such a buffer gets loaded and + * decrypted, the change of the undefined part to zeroes can affect + * the valid part if it does not end at block boundary. By setting the + * whole buffer to zeroes we ensure that the unused part of the buffer + * always contains zeroes. + */ + MemSet(fcommon->buffer.data, 0, BLCKSZ); + } return file; } @@ -205,6 +272,19 @@ extendBufFile(BufFile *file) file->files = (File *) repalloc(file->files, (file->numFiles + 1) * sizeof(File)); + + if (data_encrypted) + { + ensureUsefulArraySize(&file->common, file->numFiles + 1); + + if (file->segnos) + { + file->segnos = (off_t *) repalloc(file->segnos, + (file->numFiles + 1) * sizeof(off_t)); + file->segnos[file->numFiles] = file->numFiles; + } + } + file->files[file->numFiles] = pfile; file->numFiles++; } @@ -227,6 +307,8 @@ BufFileCreateTemp(bool interXact) BufFile *file; File pfile; + static uint32 counter_temp = 0; + /* * Ensure that temp tablespaces are set up for OpenTemporaryFile to use. * Possibly the caller will have done this already, but it seems useful to @@ -242,6 +324,10 @@ BufFileCreateTemp(bool interXact) Assert(pfile >= 0); file = makeBufFile(pfile); + + file->number = counter_temp; + counter_temp = (counter_temp + 1) % INT_MAX; + file->isInterXact = interXact; return file; @@ -345,6 +431,7 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name) files[nfiles] = SharedFileSetOpen(fileset, segment_name); if (files[nfiles] <= 0) break; + ++nfiles; CHECK_FOR_INTERRUPTS(); @@ -361,7 +448,51 @@ BufFileOpenShared(SharedFileSet *fileset, const char *name) segment_name, name))); file = makeBufFileCommon(nfiles); + + /* + * Shared encrypted segment should, at its end, contain information on the + * number of useful bytes in the last buffer. + */ + if (data_encrypted) + { + off_t pos; + int i; + + for (i = 0; i < nfiles; i++) + { + int nbytes; + File segment = files[i]; + + pos = FileSize(segment) - sizeof(off_t); + + /* + * The word must immediately follow the last buffer of the + * segment. + */ + if (pos <= 0 || pos % BLCKSZ != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not find padding info in BufFile \"%s\": %m", + name))); + + nbytes = FileRead(segment, (char *) &file->common.useful[i], + sizeof(off_t), pos, WAIT_EVENT_BUFFILE_READ); + if (nbytes != sizeof(off_t)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read padding info from BufFile \"%s\": %m", + name))); + Assert(file->common.useful[i] > 0); + + CHECK_FOR_INTERRUPTS(); + } + } + file->files = files; + + if (data_encrypted) + file->common.nuseful = nfiles; + file->common.readOnly = true; /* Can't write to files opened this way */ file->fileset = fileset; file->name = pstrdup(name); @@ -440,6 +571,14 @@ BufFileClose(BufFile *file) FileClose(file->files[i]); /* release the buffer space */ pfree(file->files); + + if (data_encrypted) + { + if (file->segnos) + pfree(file->segnos); + pfree(file->common.useful); + } + pfree(file); } @@ -456,6 +595,13 @@ BufFileLoadBuffer(BufFile *file) File thisfile; /* + * Only whole multiple of ENCRYPTION_BLOCK can be encrypted / decrypted, + */ + Assert((file->common.curOffset % BLCKSZ == 0 && + file->common.curOffset % ENCRYPTION_BLOCK == 0) || + !data_encrypted); + + /* * Advance to next component file if necessary and possible. */ if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE && @@ -466,6 +612,20 @@ BufFileLoadBuffer(BufFile *file) } /* + * See makeBufFileCommon(). + * + * Actually here we only handle the case of FileRead() returning zero + * bytes below. In contrast, if the buffer contains any data but it's not + * full, it should already have the trailing zeroes (encrypted) on disk. + * And as the encrypted buffer is always loaded in its entirety (i.e. EOF + * should only appear at buffer boundary if the data is encrypted), all + * unused bytes of the buffer should eventually be zeroes after + * decryption. + */ + if (data_encrypted) + MemSet(file->common.buffer.data, 0, BLCKSZ); + + /* * Read whatever we can get, up to a full bufferload. */ thisfile = file->files[file->common.curFile]; @@ -478,6 +638,54 @@ BufFileLoadBuffer(BufFile *file) file->common.nbytes = 0; /* we choose not to advance curOffset here */ + if (data_encrypted && file->common.nbytes > 0) + { + char tweak[TWEAK_SIZE]; + int nbytes = file->common.nbytes; + + /* + * The encrypted component file can only consist of whole number of + * our encryption units. (Only the whole buffers are dumped / loaded.) + * The only exception is that we're at the end of segment file and + * found the word indicating the number of useful bytes in the + * segment. This can only happen for shared file. + */ + if (nbytes % BLCKSZ != 0) + { + Assert(nbytes == sizeof(off_t) && file->fileset != NULL); + + /* + * This metadata his hidden to caller, so all he needs to know + * that there's no real data at the end of the file. + */ + file->common.nbytes = 0; + return; + } + + /* Decrypt the whole block at once. */ + BufFileTweak(tweak, &file->common, false); + decrypt_block(file->common.buffer.data, + file->common.buffer.data, + BLCKSZ, + tweak, + false); + +#ifdef USE_ASSERT_CHECKING + + /* + * The unused part of the buffer which we've read from disk and + * decrypted should only contain zeroes, as explained in front of the + * MemSet() call. + */ + { + int i; + + for (i = file->common.nbytes; i < BLCKSZ; i++) + Assert(file->common.buffer.data[i] == 0); + } +#endif /* USE_ASSERT_CHECKING */ + } + if (file->common.nbytes > 0) pgBufferUsage.temp_blks_read++; } @@ -561,6 +769,169 @@ BufFileDumpBuffer(BufFile *file) } /* + * BufFileDumpBufferEncrypted + * + * Encrypt buffer and dump it. The functionality is sufficiently different + * from BufFileDumpBuffer to be implemented as a separate function. The most + * notable difference is that no loop is needed here. + */ +static void +BufFileDumpBufferEncrypted(BufFile *file) +{ + char tweak[TWEAK_SIZE]; + int bytestowrite; + File thisfile; + + /* + * Caller's responsibility. + */ + Assert(file->common.pos <= file->common.nbytes); + + /* + * See comments in BufFileLoadBuffer(); + */ + Assert((file->common.curOffset % BLCKSZ == 0 && + file->common.curOffset % ENCRYPTION_BLOCK == 0)); + + /* + * Advance to next component file if necessary and possible. + */ + if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE) + { + while (file->common.curFile + 1 >= file->numFiles) + extendBufFile(file); + file->common.curFile++; + file->common.curOffset = 0L; + } + + /* + * This condition plus the alignment of curOffset to BLCKSZ (checked + * above) ensure that the encrypted buffer never crosses component file + * boundary. + */ + StaticAssertStmt((MAX_PHYSICAL_FILESIZE % BLCKSZ) == 0, + "BLCKSZ is not whole multiple of MAX_PHYSICAL_FILESIZE"); + + /* + * Encrypted data is dumped all at once. + * + * Unlike BufFileDumpBuffer(), we don't have to check here how much bytes + * is available in the segment. According to the assertions above, + * currOffset should be lower than MAX_PHYSICAL_FILESIZE by non-zero + * multiple of BLCKSZ. + */ + bytestowrite = BLCKSZ; + + /* + * The amount of data encrypted must be a multiple of ENCRYPTION_BLOCK. We + * meet this condition simply by encrypting the whole buffer. + * + * XXX Alternatively we could encrypt only as few encryption blocks that + * encompass file->common.nbyte bytes, but then we'd have to care how many + * blocks should be decrypted: decryption of the unencrypted trailing + * zeroes produces garbage, which can be a problem if lseek() created + * "holes" in the file. Such a hole should be read as a sequence of + * zeroes. + */ + BufFileTweak(tweak, &file->common, false); + + encrypt_block(file->common.buffer.data, + encrypt_buf.data, + BLCKSZ, + tweak, + false); + + thisfile = file->files[file->common.curFile]; + bytestowrite = FileWrite(thisfile, + encrypt_buf.data, + bytestowrite, + file->common.curOffset, + WAIT_EVENT_BUFFILE_WRITE); + if (bytestowrite <= 0 || bytestowrite != BLCKSZ) + return; /* failed to write */ + + file->common.curOffset += bytestowrite; + pgBufferUsage.temp_blks_written++; + + /* + * The number of useful bytes needs to be written at the end of each + * encrypted segment of a shared file so that the other backends know + * how many bytes of the last buffer are useful. + */ + if (file->fileset != NULL) + { + off_t useful; + + /* + * nuseful may be increased earlier than numFiles but not later, so + * the corresponding entry should already exist in ->useful. + */ + Assert(file->common.curFile < file->common.nuseful); + + /* + * The number of useful bytes in the current segment file. + */ + useful = file->common.useful[file->common.curFile]; + + /* + * Have we dumped the last buffer of the segment, i.e. the one that + * can contain padding? + */ + if (file->common.curOffset >= useful) + { + int bytes_extra; + + /* + * Write the number of useful bytes in the segment. + * + * Do not increase curOffset afterwards. Thus we ensure that the + * next buffer appended will overwrite the "useful" value just + * written, instead of being appended to it. + */ + bytes_extra = FileWrite(file->files[file->common.curFile], + (char *) &useful, + sizeof(useful), + file->common.curOffset, + WAIT_EVENT_BUFFILE_WRITE); + if (bytes_extra != sizeof(useful)) + return; /* failed to write */ + } + } + file->common.dirty = false; + + if (file->common.pos >= BLCKSZ) + { + Assert(file->common.pos == BLCKSZ); + + /* + * curOffset points to the beginning of the next buffer, so just reset + * pos and nbytes. + */ + file->common.pos = 0; + file->common.nbytes = 0; + + /* See makeBufFile() */ + MemSet(file->common.buffer.data, 0, BLCKSZ); + } + else + { + /* + * Move curOffset to the beginning of the just-written buffer and + * preserve pos. + */ + file->common.curOffset -= BLCKSZ; + + /* + * At least pos bytes should be written even if the first change since + * now appears at pos == nbytes, but in fact the whole buffer will be + * written regardless pos. This is the price we pay for the choosing + * BLCKSZ as the I/O unit for encrypted data. + */ + file->common.nbytes = BLCKSZ; + } +} + +/* * BufFileRead * * Like fread() except we assume 1-byte element size. @@ -592,7 +963,10 @@ BufFileFlush(BufFile *file) { if (file->common.dirty) { - BufFileDumpBuffer(file); + if (!data_encrypted) + BufFileDumpBuffer(file); + else + BufFileDumpBufferEncrypted(file); if (file->common.dirty) return EOF; } @@ -688,9 +1062,43 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) return EOF; /* Seek is OK! */ file->common.curFile = newFile; - file->common.curOffset = newOffset; - file->common.pos = 0; - file->common.nbytes = 0; + if (!data_encrypted) + { + file->common.curOffset = newOffset; + file->common.pos = 0; + file->common.nbytes = 0; + } + else + { + /* + * Offset of an encrypted buffer must be a multiple of BLCKSZ. + */ + file->common.pos = newOffset % BLCKSZ; + file->common.curOffset = newOffset - file->common.pos; + + /* + * BufFileLoadBuffer() will set nbytes iff it can read something. + */ + file->common.nbytes = 0; + + /* + * Load and decrypt the existing part of the buffer. + */ + BufFileLoadBuffer(file); + if (file->common.nbytes == 0) + { + /* + * The data requested is not in the file, but this is not an + * error. + */ + return 0; + } + + /* + * The whole buffer should have been loaded. + */ + Assert(file->common.nbytes == BLCKSZ); + } return 0; } @@ -721,6 +1129,123 @@ BufFileSeekBlock(BufFile *file, long blknum) SEEK_SET); } +static void +BufFileTweak(char *tweak, BufFileCommon *file, bool is_transient) +{ + off_t block; + + /* + * The unused bytes should always be defined. + */ + memset(tweak, 0, TWEAK_SIZE); + + if (!is_transient) + { + BufFile *tmpfile = (BufFile *) file; + pid_t pid; + uint32 number; + int curFile; + + if (tmpfile->fileset) + { + pid = tmpfile->fileset->creator_pid; + number = tmpfile->fileset->number; + } + else + { + pid = MyProcPid; + number = tmpfile->number; + } + + curFile = file->curFile; + + /* + * If the file was produced by BufFileAppend(), we need the original + * curFile, as it was used originally for encryption. + */ + if (tmpfile->segnos) + curFile = tmpfile->segnos[curFile]; + + block = curFile * BUFFILE_SEG_SIZE + file->curOffset / BLCKSZ; + + StaticAssertStmt(sizeof(pid) + sizeof(number) + sizeof(block) <= + TWEAK_SIZE, + "tweak components do not fit into TWEAK_SIZE"); + + /* + * The tweak consists of PID of the owning backend (the leader backend + * in the case of parallel query processing), number within the PID + * and block number. + * + * XXX Additional flag would be handy to distinguish local file from + * shared one. Since there's no more room within TWEAK_SIZE, should we + * use the highest bit in one of the existing components (preferably + * other than pid so that tweaks of different processes do not become + * identical)? + */ + *((pid_t *) tweak) = pid; + tweak += sizeof(pid_t); + *((uint32 *) tweak) = number; + tweak += sizeof(number); + *((off_t *) tweak) = block; + } + else + { + TransientBufFile *transfile = (TransientBufFile *) file; + pg_sha256_ctx sha_ctx; + unsigned char sha[PG_SHA256_DIGEST_LENGTH]; +#define BUF_FILE_PATH_HASH_LEN 8 + + /* + * For transient file we can't use any field of TransientBufFile + * because this info gets lost if the file is closed and reopened. + * Hash of the file path string is an easy way to get "persistent" + * tweak value, however usual hashes do not fit into TWEAK_SIZE. The + * hash portion that we can store is actually even smaller because + * block number needs to be stored too. Even though we only use part + * of the hash, the tweak we finally use for data encryption / + * decryption should not be predictable because the tweak we compute + * here is further processed, see cbc_essi_preprocess_tweak(). + */ + pg_sha256_init(&sha_ctx); + pg_sha256_update(&sha_ctx, + (uint8 *) transfile->path, + strlen(transfile->path)); + pg_sha256_final(&sha_ctx, sha); + + StaticAssertStmt(BUF_FILE_PATH_HASH_LEN + sizeof(block) <= TWEAK_SIZE, + "tweak components do not fit into TWEAK_SIZE"); + memcpy(tweak, sha, BUF_FILE_PATH_HASH_LEN); + tweak += BUF_FILE_PATH_HASH_LEN; + block = file->curOffset / BLCKSZ; + *((off_t *) tweak) = block; + } +} + +/* + * Make sure that BufFile.useful array has the required size. + */ +static void +ensureUsefulArraySize(BufFileCommon *file, int required) +{ + /* + * Does the array already have enough space? + */ + if (required <= file->nuseful) + return; + + /* + * It shouldn't be possible to jump beyond the end of the last segment, + * i.e. skip more than 1 segment. + */ + Assert(file->nuseful + 1 == required); + + file->useful = (off_t *) + repalloc(file->useful, required * sizeof(off_t)); + file->useful[file->nuseful] = 0L; + file->nuseful++; +} + #ifdef NOT_USED /* * BufFileTellBlock --- block-oriented tell @@ -752,14 +1277,32 @@ BufFileSize(BufFile *file) Assert(file->fileset != NULL); - /* Get the size of the last physical file. */ - lastFileSize = FileSize(file->files[file->numFiles - 1]); - if (lastFileSize < 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not determine size of temporary file \"%s\" from BufFile \"%s\": %m", - FilePathName(file->files[file->numFiles - 1]), - file->name))); + if (!data_encrypted) + { + /* Get the size of the last physical file. */ + lastFileSize = FileSize(file->files[file->numFiles - 1]); + if (lastFileSize < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not determine size of temporary file \"%s\" from BufFile \"%s\": %m", + FilePathName(file->files[file->numFiles - 1]), + file->name))); + } + else + { + /* + * "useful" should be initialized even for shared file, see + * BufFileOpenShared(). + */ + Assert(file->common.useful != NULL && + file->common.nuseful >= file->numFiles); + + /* + * The number of useful bytes in the segment is what caller is + * interested in. + */ + lastFileSize = file->common.useful[file->common.nuseful - 1]; + } return ((file->numFiles - 1) * (int64) MAX_PHYSICAL_FILESIZE) + lastFileSize; @@ -803,12 +1346,80 @@ BufFileAppend(BufFile *target, BufFile *source) repalloc(target->files, sizeof(File) * newNumFiles); for (i = target->numFiles; i < newNumFiles; i++) target->files[i] = source->files[i - target->numFiles]; + + if (data_encrypted) + BufFileAppendMetadata(target, source); + target->numFiles = newNumFiles; return startBlock; } /* + * Add encryption-specific metadata of the source file to the target. + */ +static void +BufFileAppendMetadata(BufFile *target, BufFile *source) +{ + int newNumFiles = target->numFiles + source->numFiles; + int newNUseful = target->common.nuseful + source->common.nuseful; + int i; + + /* + * XXX As the typical use case is that parallel workers expose file to the + * leader, can we expect both target and source to have been exported, + * i.e. flushed? In such a case "nuseful" would have to be equal to + * "numFiles" for both input files and the code could get a bit + * simpler. It seems that at least source should be flushed, as + * source->readOnly is expected to be true above. + */ + target->common.useful = (off_t *) + repalloc(target->common.useful, sizeof(off_t) * newNUseful); + + for (i = target->common.nuseful; i < newNUseful; i++) + target->common.useful[i] = source->common.useful[i - target->common.nuseful]; + target->common.nuseful = newNUseful; + + /* + * File segments can appear at different position due to concatenation, so + * make sure we remember the original positions for the sake of encryption + * tweak. + */ + if (target->segnos == NULL) + { + /* + * If the target does not have the array yet, allocate it for both + * target and source and initialize the target part. + */ + target->segnos = (off_t *) palloc(newNumFiles * sizeof(off_t)); + for (i = 0; i < target->numFiles; i++) + target->segnos[i] = i; + } + else + { + /* + * Use the existing target part and add space for the source part. + */ + target->segnos = (off_t *) repalloc(target->segnos, + newNumFiles * sizeof(off_t)); + } + + /* + * The source segment number either equals to (0-based) index of the + * segment, or to an element of an already existing array. + */ + for (i = target->numFiles; i < newNumFiles; i++) + { + off_t segno = i - target->numFiles; + + if (source->segnos == NULL) + target->segnos[i] = segno; + else + target->segnos[i] = source->segnos[segno]; + } +} + +/* * Open TransientBufFile at given path or create one if it does not * exist. User will be allowed either to write to the file or to read from it, * according to fileFlags, but not both. @@ -829,6 +1440,28 @@ BufFileOpenTransient(const char *path, int fileFlags) /* Check whether user wants read or write access. */ readOnly = (fileFlags & O_WRONLY) == 0; + if (data_encrypted) + { + /* + * In the encryption case, even if user will only be allowed to write, + * internally we also need to read, see below. + */ + fileFlags &= ~O_WRONLY; + fileFlags |= O_RDWR; + + /* + * We can only emulate the append behavior by setting curOffset to + * file size because if the underlying file was opened in append mode, + * we could not rewrite the old value of file->common.useful[0] with + * data. + */ + if (fileFlags & O_APPEND) + { + append = true; + fileFlags &= ~O_APPEND; + } + } + /* * Append mode for read access is not useful, so don't bother implementing * it. @@ -861,6 +1494,8 @@ BufFileOpenTransient(const char *path, int fileFlags) fcommon->readOnly = readOnly; fcommon->append = append; fcommon->curFile = 0; + fcommon->useful = (off_t *) palloc0(sizeof(off_t)); + fcommon->nuseful = 1; file->path = pstrdup(path); file->fd = fd; @@ -881,6 +1516,56 @@ BufFileOpenTransient(const char *path, int fileFlags) else fcommon->curOffset = 0L; + /* + * Encrypted transient file should, at its end, contain information on the + * number of useful bytes in the last buffer. + */ + if (data_encrypted) + { + off_t pos = size; + int nbytes; + + /* No metadata in an empty file. */ + if (pos == 0) + return file; + + pos -= sizeof(off_t); + + /* + * The word must immediately follow the last buffer of the segment. + */ + if (pos < 0 || pos % BLCKSZ != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not find padding info in TransientBufFile \"%s\": %m", + path))); + + errno = 0; + nbytes = pg_pread(file->fd, + (char *) &fcommon->useful[0], + sizeof(off_t), + pos); + if (nbytes != sizeof(off_t)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read padding info from TransientBufFile \"%s\": %m", + path))); + Assert(fcommon->useful[0] > 0); + + if (fcommon->append) + { + off_t useful = fcommon->useful[0]; + + /* + * If new buffer should be added, make sure it will end up + * immediately after the last complete one, and also that the next + * write position follows the last valid byte. + */ + fcommon->pos = useful % BLCKSZ; + fcommon->curOffset = useful - fcommon->pos; + } + } + return file; } @@ -912,6 +1597,8 @@ BufFileCloseTransient(TransientBufFile *file) (errcode_for_file_access(), errmsg("could not close file \"%s\": %m", file->path))); + if (data_encrypted) + pfree(file->common.useful); pfree(file->path); pfree(file); } @@ -928,6 +1615,9 @@ BufFileLoadBufferTransient(TransientBufFile *file) Assert(!file->common.dirty); Assert(file->common.pos == 0 && file->common.nbytes == 0); + /* See comments in BufFileLoadBuffer(). */ + if (data_encrypted) + MemSet(file->common.buffer.data, 0, BLCKSZ); retry: /* @@ -952,6 +1642,54 @@ retry: return; } /* we choose not to advance offset here */ + + if (data_encrypted && file->common.nbytes > 0) + { + char tweak[TWEAK_SIZE]; + int nbytes = file->common.nbytes; + + /* + * The encrypted file can only consist of whole number of our + * encryption units. (Only the whole buffers are dumped / loaded.) The + * only exception is that we're at the end of segment file and found + * the word indicating the number of useful bytes in the segment. This + * can only happen for shared file. + */ + if (nbytes % BLCKSZ != 0) + { + Assert(nbytes == sizeof(off_t)); + + /* + * This metadata his hidden to caller, so all he needs to know + * that there's no real data at the end of the file. + */ + file->common.nbytes = 0; + return; + } + + /* Decrypt the whole block at once. */ + BufFileTweak(tweak, &file->common, true); + decrypt_block(file->common.buffer.data, + file->common.buffer.data, + BLCKSZ, + tweak, + false); + +#ifdef USE_ASSERT_CHECKING + + /* + * The unused part of the buffer which we've read from disk and + * decrypted should only contain zeroes, as explained in front of the + * MemSet() call. + */ + { + int i; + + for (i = file->common.nbytes; i < BLCKSZ; i++) + Assert(file->common.buffer.data[i] == 0); + } +#endif /* USE_ASSERT_CHECKING */ + } } /* @@ -960,7 +1698,9 @@ retry: static void BufFileDumpBufferTransient(TransientBufFile *file) { - int nwritten; + int bytestowrite, + nwritten; + char *write_ptr; /* This function should only be needed during write access ... */ Assert(!file->common.readOnly); @@ -969,12 +1709,33 @@ BufFileDumpBufferTransient(TransientBufFile *file) Assert(file->common.dirty); Assert(file->common.nbytes > 0); + if (!data_encrypted) + { + write_ptr = file->common.buffer.data; + bytestowrite = file->common.nbytes; + } + else + { + char tweak[TWEAK_SIZE]; + + /* + * Encrypt the whole buffer, see comments in BufFileDumpBuffer(). + */ + BufFileTweak(tweak, &file->common, true); + encrypt_block(file->common.buffer.data, + encrypt_buf.data, + BLCKSZ, + tweak, + false); + write_ptr = encrypt_buf.data; + bytestowrite = BLCKSZ; + } retry: errno = 0; pgstat_report_wait_start(WAIT_EVENT_BUFFILE_WRITE); nwritten = pg_pwrite(file->fd, - file->common.buffer.data, - file->common.nbytes, + write_ptr, + bytestowrite, file->common.curOffset); pgstat_report_wait_end(); @@ -993,6 +1754,43 @@ retry: return; /* failed to write */ } + file->common.curOffset += nwritten; + + if (data_encrypted) + { + off_t useful; + + /* + * The number of useful bytes in file. + */ + useful = file->common.useful[0]; + + /* + * Have we dumped the last buffer of the segment, i.e. the one that + * can contain padding? + */ + if (file->common.curOffset >= useful) + { + int bytes_extra; + + /* + * Write the number of useful bytes in the file + * + * Do not increase curOffset afterwards. Thus we ensure that the + * next buffer appended will overwrite the "useful" value just + * written, instead of being appended to it. + */ + pgstat_report_wait_start(WAIT_EVENT_BUFFILE_WRITE); + bytes_extra = pg_pwrite(file->fd, + (char *) &useful, + sizeof(useful), + file->common.curOffset); + pgstat_report_wait_end(); + if (bytes_extra != sizeof(useful)) + return; /* failed to write */ + } + } + file->common.dirty = false; file->common.pos = 0; @@ -1046,21 +1844,110 @@ BufFileReadCommon(BufFileCommon *file, void *ptr, size_t size, { if (file->pos >= file->nbytes) { - /* Try to load more data into buffer. */ - file->curOffset += file->pos; - file->pos = 0; + /* + * Neither read nor write nor seek should leave pos greater than + * nbytes, regardless the data is encrypted or not. pos can only + * be greater if nbytes is zero --- this situation can be caused + * by BufFileSeek(). + */ + Assert(file->pos == file->nbytes || file->nbytes == 0); + + /* + * The Assert() above implies that pos is a whole multiple of + * BLCKSZ, so curOffset has meet the same encryption-specific + * requirement too. + */ + Assert(file->curOffset % BLCKSZ == 0 || !data_encrypted); + file->nbytes = 0; + /* Try to load more data into buffer. */ + if (!data_encrypted || file->pos % BLCKSZ == 0) + { + file->curOffset += file->pos; + file->pos = 0; - if (!is_transient) - BufFileLoadBuffer((BufFile *) file); + if (!is_transient) + BufFileLoadBuffer((BufFile *) file); + else + BufFileLoadBufferTransient((TransientBufFile *) file); + + if (file->nbytes <= 0) + break; /* no more data available */ + } else - BufFileLoadBufferTransient((TransientBufFile *) file); + { + int nbytes_orig = file->nbytes; + + /* + * Given that BLCKSZ is the I/O unit for encrypted data (see + * comments in BufFileDumpBuffer), we cannot add pos to + * curOffset because that would make it point outside block + * boundary. The only thing we can do is to reload the whole + * buffer and see if more data is eventually there than the + * previous load has fetched. + */ + if (!is_transient) + BufFileLoadBuffer((BufFile *) file); + else + BufFileLoadBufferTransient((TransientBufFile *) file); - if (file->nbytes <= 0) - break; /* no more data available */ + Assert(file->nbytes >= nbytes_orig); + if (file->nbytes == nbytes_orig) + break; /* no more data available */ + } } nthistime = file->nbytes - file->pos; + + /* + * The buffer can contain trailing zeroes because BLCKSZ is the I/O + * unit for encrypted data. These are not available for reading. + */ + if (data_encrypted) + { + off_t useful = file->useful[file->curFile]; + + /* + * The criterion is whether the useful data ends within the + * currently loaded buffer. + */ + if (useful < file->curOffset + BLCKSZ) + { + int avail; + + /* + * Compute the number of bytes available in the current + * buffer. + */ + avail = useful - file->curOffset; + Assert(avail >= 0); + + /* + * An empty buffer can exist, e.g. after a seek to the end of + * the last component file. + */ + if (avail == 0) + break; + + /* + * Seek beyond the current EOF, which was not followed by + * write, could have resulted in position outside the useful + * data + */ + if (file->pos > avail) + break; + + nthistime = avail - file->pos; + Assert(nthistime >= 0); + + /* + * Have we reached the end of the valid data? + */ + if (nthistime == 0) + break; + } + } + if (nthistime > size) nthistime = size; Assert(nthistime > 0); @@ -1098,7 +1985,12 @@ BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, if (file->dirty) { if (!is_transient) - BufFileDumpBuffer((BufFile *) file); + { + if (!data_encrypted) + BufFileDumpBuffer((BufFile *)file); + else + BufFileDumpBufferEncrypted((BufFile *) file); + } else BufFileDumpBufferTransient((TransientBufFile *) file); @@ -1109,11 +2001,28 @@ BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, { Assert(!is_transient); - /* Hmm, went directly from reading to writing? */ + /* + * Hmm, went directly from reading to writing? + * + * As pos should be exactly BLCKSZ, there is nothing special + * to do about data_encrypted, except for zeroing the buffer. + */ + Assert(file->pos == BLCKSZ); + file->curOffset += file->pos; file->pos = 0; file->nbytes = 0; + + /* See makeBufFile() */ + if (data_encrypted) + MemSet(file->buffer.data, 0, BLCKSZ); } + + /* + * If curOffset changed above, it should still meet the assumption + * that buffer is the I/O unit for encrypted data. + */ + Assert(file->curOffset % BLCKSZ == 0 || !data_encrypted); } nthistime = BLCKSZ - file->pos; @@ -1127,6 +2036,14 @@ BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, file->pos += nthistime; if (file->nbytes < file->pos) file->nbytes = file->pos; + + /* + * Remember how many bytes of the file are valid - the rest is + * padding. + */ + if (data_encrypted) + BufFileUpdateUsefulLength(file, is_transient); + ptr = (void *) ((char *) ptr + nthistime); size -= nthistime; nwritten += nthistime; @@ -1134,3 +2051,81 @@ BufFileWriteCommon(BufFileCommon *file, void *ptr, size_t size, return nwritten; } + +/* + * Update information about the effective length of the file. + */ +static void +BufFileUpdateUsefulLength(BufFileCommon *file, bool is_transient) +{ + off_t new_useful; + int fileno; + + if (is_transient) + + { + /* + * Transient file is a single file on the disk, so the whole thing is + * pretty simple. + */ + fileno = 0; + + new_useful = file->curOffset + file->pos; + if (new_useful > file->useful[fileno]) + file->useful[fileno] = new_useful; + } + else + { + fileno = file->curFile; + + /* + * curFile does not necessarily correspond to the offset: it can still + * have the initial value if BufFileSeek() skipped the previous file + * w/o dumping anything of it. While curFile will be fixed during the + * next dump, we need valid fileno now. + */ + if (file->curOffset >= MAX_PHYSICAL_FILESIZE) + { + /* + * Even BufFileSeek() should not allow curOffset to become more + * than MAX_PHYSICAL_FILESIZE (if caller passes higher offset, + * curFile gets increased instead). + */ + Assert(file->curOffset == MAX_PHYSICAL_FILESIZE); + + fileno++; + } + + /* + * fileno can now point to a segment that does not exist on disk yet. + */ + ensureUsefulArraySize(file, fileno + 1); + + /* + * Update the number of useful bytes in the underlying component file + * if we've added any useful data. + */ + new_useful = file->curOffset; + + /* + * Make sure the offset is relative to the correct component file + * (segment). If the write just crossed segment boundary, curOffset + * can still point at the end of the previous segment, and so + * new_useful is also relative to the start of that previous + * segment. Make sure it's relative to the current (fileno) segment. + */ + if (file->curOffset % MAX_PHYSICAL_FILESIZE == 0) + new_useful %= MAX_PHYSICAL_FILESIZE; + + /* Finalize the offset. */ + new_useful += file->pos; + + /* + * Adjust the number of useful bytes in the file if needed. This has + * to happen immediately, independent from BufFileDumpBuffer(), so + * that BufFileRead() works correctly anytime. + */ + if (new_useful > file->useful[fileno]) + file->useful[fileno] = new_useful; + } +} -- 2.13.7 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=v04-0012-Add-tests-for-buffile.c.patch From 4d1502a37c329c28977c84281cdec3a000ba30de Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:02 +0200 Subject: [PATCH 12/17] Add tests for buffile.c. Since the encryption adds complexity to the buffile.c, regression tests seem like a good idea. In order to be able to test corner cases that deal with segment boudaries, this patch makes the segment size configurable - the tests would be too expensive with 1GB segments. The new GUC is marked with DEVELOPER_OPTIONS, DBA is not supposed to use it during normal operation. --- src/backend/storage/file/buffile.c | 73 ++--- src/backend/utils/misc/guc.c | 34 +- src/include/storage/buffile.h | 19 ++ src/test/modules/buffile/Makefile | 21 ++ src/test/modules/buffile/README | 11 + src/test/modules/buffile/buffile--1.0.sql | 54 ++++ src/test/modules/buffile/buffile.c | 429 ++++++++++++++++++++++++++ src/test/modules/buffile/buffile.control | 5 + src/test/modules/buffile/expected/test_01.out | 61 ++++ src/test/modules/buffile/expected/test_02.out | 48 +++ src/test/modules/buffile/expected/test_03.out | 27 ++ src/test/modules/buffile/expected/test_04.out | 84 +++++ src/test/modules/buffile/expected/test_05.out | 33 ++ src/test/modules/buffile/expected/test_06.out | 43 +++ src/test/modules/buffile/expected/test_07.out | 41 +++ src/test/modules/buffile/expected/test_08.out | 41 +++ src/test/modules/buffile/expected/test_09.out | 39 +++ src/test/modules/buffile/expected/test_10.out | 76 +++++ src/test/modules/buffile/expected/test_11.out | 34 ++ src/test/modules/buffile/expected/test_12.out | 16 + src/test/modules/buffile/expected/test_13.out | 86 ++++++ src/test/modules/buffile/sql/test_01.sql | 17 + src/test/modules/buffile/sql/test_02.sql | 13 + src/test/modules/buffile/sql/test_03.sql | 7 + src/test/modules/buffile/sql/test_04.sql | 25 ++ src/test/modules/buffile/sql/test_05.sql | 8 + src/test/modules/buffile/sql/test_06.sql | 15 + src/test/modules/buffile/sql/test_07.sql | 13 + src/test/modules/buffile/sql/test_08.sql | 13 + src/test/modules/buffile/sql/test_09.sql | 9 + src/test/modules/buffile/sql/test_10.sql | 25 ++ src/test/modules/buffile/sql/test_11.sql | 9 + src/test/modules/buffile/sql/test_12.sql | 7 + src/test/modules/buffile/sql/test_13.sql | 25 ++ 34 files changed, 1420 insertions(+), 41 deletions(-) create mode 100644 src/test/modules/buffile/Makefile create mode 100644 src/test/modules/buffile/README create mode 100644 src/test/modules/buffile/buffile--1.0.sql create mode 100644 src/test/modules/buffile/buffile.c create mode 100644 src/test/modules/buffile/buffile.control create mode 100644 src/test/modules/buffile/expected/test_01.out create mode 100644 src/test/modules/buffile/expected/test_02.out create mode 100644 src/test/modules/buffile/expected/test_03.out create mode 100644 src/test/modules/buffile/expected/test_04.out create mode 100644 src/test/modules/buffile/expected/test_05.out create mode 100644 src/test/modules/buffile/expected/test_06.out create mode 100644 src/test/modules/buffile/expected/test_07.out create mode 100644 src/test/modules/buffile/expected/test_08.out create mode 100644 src/test/modules/buffile/expected/test_09.out create mode 100644 src/test/modules/buffile/expected/test_10.out create mode 100644 src/test/modules/buffile/expected/test_11.out create mode 100644 src/test/modules/buffile/expected/test_12.out create mode 100644 src/test/modules/buffile/expected/test_13.out create mode 100644 src/test/modules/buffile/sql/test_01.sql create mode 100644 src/test/modules/buffile/sql/test_02.sql create mode 100644 src/test/modules/buffile/sql/test_03.sql create mode 100644 src/test/modules/buffile/sql/test_04.sql create mode 100644 src/test/modules/buffile/sql/test_05.sql create mode 100644 src/test/modules/buffile/sql/test_06.sql create mode 100644 src/test/modules/buffile/sql/test_07.sql create mode 100644 src/test/modules/buffile/sql/test_08.sql create mode 100644 src/test/modules/buffile/sql/test_09.sql create mode 100644 src/test/modules/buffile/sql/test_10.sql create mode 100644 src/test/modules/buffile/sql/test_11.sql create mode 100644 src/test/modules/buffile/sql/test_12.sql create mode 100644 src/test/modules/buffile/sql/test_13.sql diff --git a/src/backend/storage/file/buffile.c b/src/backend/storage/file/buffile.c index 308ceb01b7..998dabd134 100644 --- a/src/backend/storage/file/buffile.c +++ b/src/backend/storage/file/buffile.c @@ -56,16 +56,11 @@ #include "utils/resowner.h" /* - * We break BufFiles into gigabyte-sized segments, regardless of RELSEG_SIZE. - * The reason is that we'd like large BufFiles to be spread across multiple - * tablespaces when available. - * - * An integer value indicating the number of useful bytes in the segment is - * appended to each segment of if the file is both shared and encrypted, see - * BufFile.useful. + * The functions bellow actually use integer constants so that the size can be + * controlled by GUC. This is useful for development and regression tests. */ -#define MAX_PHYSICAL_FILESIZE 0x40000000 -#define BUFFILE_SEG_SIZE (MAX_PHYSICAL_FILESIZE / BLCKSZ) +int buffile_max_filesize = MAX_PHYSICAL_FILESIZE; +int buffile_seg_blocks = BUFFILE_SEG_BLOCKS(MAX_PHYSICAL_FILESIZE); /* * Fields that both BufFile and TransientBufFile structures need. It must be @@ -123,7 +118,7 @@ struct BufFile BufFileCommon common; /* Common fields, see above. */ int numFiles; /* number of physical files in set */ - /* all files except the last have length exactly MAX_PHYSICAL_FILESIZE */ + /* all files except the last have length exactly buffile_max_filesize */ File *files; /* palloc'd array with numFiles entries */ /* @@ -291,7 +286,7 @@ extendBufFile(BufFile *file) /* * Create a BufFile for a new temporary file (which will expand to become - * multiple temporary files if more than MAX_PHYSICAL_FILESIZE bytes are + * multiple temporary files if more than buffile_max_filesize bytes are * written to it). * * If interXact is true, the temp file will not be automatically deleted @@ -595,7 +590,7 @@ BufFileLoadBuffer(BufFile *file) File thisfile; /* - * Only whole multiple of ENCRYPTION_BLOCK can be encrypted / decrypted, + * Only whole multiple of ENCRYPTION_BLOCK can be encrypted / decrypted. */ Assert((file->common.curOffset % BLCKSZ == 0 && file->common.curOffset % ENCRYPTION_BLOCK == 0) || @@ -604,7 +599,7 @@ BufFileLoadBuffer(BufFile *file) /* * Advance to next component file if necessary and possible. */ - if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE && + if (file->common.curOffset >= buffile_max_filesize && file->common.curFile + 1 < file->numFiles) { file->common.curFile++; @@ -715,7 +710,7 @@ BufFileDumpBuffer(BufFile *file) /* * Advance to next component file if necessary and possible. */ - if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE) + if (file->common.curOffset >= buffile_max_filesize) { while (file->common.curFile + 1 >= file->numFiles) extendBufFile(file); @@ -727,7 +722,7 @@ BufFileDumpBuffer(BufFile *file) * Determine how much we need to write into this file. */ bytestowrite = file->common.nbytes - wpos; - availbytes = MAX_PHYSICAL_FILESIZE - file->common.curOffset; + availbytes = buffile_max_filesize - file->common.curOffset; if ((off_t) bytestowrite > availbytes) bytestowrite = (int) availbytes; @@ -758,7 +753,7 @@ BufFileDumpBuffer(BufFile *file) { file->common.curFile--; Assert(file->common.curFile >= 0); - file->common.curOffset += MAX_PHYSICAL_FILESIZE; + file->common.curOffset += buffile_max_filesize; } /* @@ -796,7 +791,7 @@ BufFileDumpBufferEncrypted(BufFile *file) /* * Advance to next component file if necessary and possible. */ - if (file->common.curOffset >= MAX_PHYSICAL_FILESIZE) + if (file->common.curOffset >= buffile_max_filesize) { while (file->common.curFile + 1 >= file->numFiles) extendBufFile(file); @@ -809,15 +804,14 @@ BufFileDumpBufferEncrypted(BufFile *file) * above) ensure that the encrypted buffer never crosses component file * boundary. */ - StaticAssertStmt((MAX_PHYSICAL_FILESIZE % BLCKSZ) == 0, - "BLCKSZ is not whole multiple of MAX_PHYSICAL_FILESIZE"); + Assert((buffile_max_filesize % BLCKSZ) == 0); /* * Encrypted data is dumped all at once. * * Unlike BufFileDumpBuffer(), we don't have to check here how much bytes * is available in the segment. According to the assertions above, - * currOffset should be lower than MAX_PHYSICAL_FILESIZE by non-zero + * currOffset should be lower than buffile_max_filesize by non-zero * multiple of BLCKSZ. */ bytestowrite = BLCKSZ; @@ -1021,7 +1015,7 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) { if (--newFile < 0) return EOF; - newOffset += MAX_PHYSICAL_FILESIZE; + newOffset += buffile_max_filesize; } if (newFile == file->common.curFile && newOffset >= file->common.curOffset && @@ -1050,13 +1044,13 @@ BufFileSeek(BufFile *file, int fileno, off_t offset, int whence) if (newFile == file->numFiles && newOffset == 0) { newFile--; - newOffset = MAX_PHYSICAL_FILESIZE; + newOffset = buffile_max_filesize; } - while (newOffset > MAX_PHYSICAL_FILESIZE) + while (newOffset > buffile_max_filesize) { if (++newFile >= file->numFiles) return EOF; - newOffset -= MAX_PHYSICAL_FILESIZE; + newOffset -= buffile_max_filesize; } if (newFile >= file->numFiles) return EOF; @@ -1124,8 +1118,8 @@ int BufFileSeekBlock(BufFile *file, long blknum) { return BufFileSeek(file, - (int) (blknum / BUFFILE_SEG_SIZE), - (off_t) (blknum % BUFFILE_SEG_SIZE) * BLCKSZ, + (int) (blknum / buffile_seg_blocks), + (off_t) (blknum % buffile_seg_blocks) * BLCKSZ, SEEK_SET); } @@ -1166,7 +1160,7 @@ BufFileTweak(char *tweak, BufFileCommon *file, bool is_transient) if (tmpfile->segnos) curFile = tmpfile->segnos[curFile]; - block = curFile * BUFFILE_SEG_SIZE + file->curOffset / BLCKSZ; + block = curFile * buffile_seg_blocks + file->curOffset / BLCKSZ; StaticAssertStmt(sizeof(pid) + sizeof(number) + sizeof(block) <= TWEAK_SIZE, @@ -1258,7 +1252,7 @@ BufFileTellBlock(BufFile *file) long blknum; blknum = (file->common.curOffset + file->common.pos) / BLCKSZ; - blknum += file->common.curFile * BUFFILE_SEG_SIZE; + blknum += file->common.curFile * buffile_seg_blocks; return blknum; } @@ -1304,7 +1298,7 @@ BufFileSize(BufFile *file) lastFileSize = file->common.useful[file->common.nuseful - 1]; } - return ((file->numFiles - 1) * (int64) MAX_PHYSICAL_FILESIZE) + + return ((file->numFiles - 1) * (int64) buffile_max_filesize) + lastFileSize; } @@ -1317,11 +1311,10 @@ BufFileSize(BufFile *file) * called here first. Resource owners for source and target must match, * too. * - * This operation works by manipulating lists of segment files, so the - * file content is always appended at a MAX_PHYSICAL_FILESIZE-aligned - * boundary, typically creating empty holes before the boundary. These - * areas do not contain any interesting data, and cannot be read from by - * caller. + * This operation works by manipulating lists of segment files, so the file + * content is always appended at a buffile_max_filesize-aligned boundary, + * typically creating empty holes before the boundary. These areas do not + * contain any interesting data, and cannot be read from by caller. * * Returns the block number within target where the contents of source * begins. Caller should apply this as an offset when working off block @@ -1330,7 +1323,7 @@ BufFileSize(BufFile *file) long BufFileAppend(BufFile *target, BufFile *source) { - long startBlock = target->numFiles * BUFFILE_SEG_SIZE; + long startBlock = target->numFiles * buffile_seg_blocks; int newNumFiles = target->numFiles + source->numFiles; int i; @@ -2084,14 +2077,14 @@ BufFileUpdateUsefulLength(BufFileCommon *file, bool is_transient) * w/o dumping anything of it. While curFile will be fixed during the * next dump, we need valid fileno now. */ - if (file->curOffset >= MAX_PHYSICAL_FILESIZE) + if (file->curOffset >= buffile_max_filesize) { /* * Even BufFileSeek() should not allow curOffset to become more - * than MAX_PHYSICAL_FILESIZE (if caller passes higher offset, + * than buffile_max_filesize (if caller passes higher offset, * curFile gets increased instead). */ - Assert(file->curOffset == MAX_PHYSICAL_FILESIZE); + Assert(file->curOffset == buffile_max_filesize); fileno++; } @@ -2114,8 +2107,8 @@ BufFileUpdateUsefulLength(BufFileCommon *file, bool is_transient) * new_useful is also relative to the start of that previous * segment. Make sure it's relative to the current (fileno) segment. */ - if (file->curOffset % MAX_PHYSICAL_FILESIZE == 0) - new_useful %= MAX_PHYSICAL_FILESIZE; + if (file->curOffset % buffile_max_filesize == 0) + new_useful %= buffile_max_filesize; /* Finalize the offset. */ new_useful += file->pos; diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 440120635e..879c8fd816 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -70,9 +70,10 @@ #include "replication/syncrep.h" #include "replication/walreceiver.h" #include "replication/walsender.h" +#include "storage/buffile.h" #include "storage/bufmgr.h" -#include "storage/encryption.h" #include "storage/dsm_impl.h" +#include "storage/encryption.h" #include "storage/standby.h" #include "storage/fd.h" #include "storage/large_object.h" @@ -189,6 +190,8 @@ static const char *show_tcp_keepalives_idle(void); static const char *show_tcp_keepalives_interval(void); static const char *show_tcp_keepalives_count(void); static const char *show_tcp_user_timeout(void); +static bool check_buffile_max_filesize(int *newval, void **extra, GucSource source); +static void assign_buffile_max_filesize(int newval, void *extra); static bool check_maxconnections(int *newval, void **extra, GucSource source); static bool check_max_worker_processes(int *newval, void **extra, GucSource source); static bool check_autovacuum_max_workers(int *newval, void **extra, GucSource source); @@ -3209,6 +3212,18 @@ static struct config_int ConfigureNamesInt[] = NULL, assign_tcp_user_timeout, show_tcp_user_timeout }, + { + /* Not for general use */ + {"buffile_max_filesize", PGC_SUSET, DEVELOPER_OPTIONS, + gettext_noop("Maximum size of BufFile segment."), + gettext_noop("This makes testing of some corner cases easier, especially when read or write crosses segment boundary."), + GUC_NOT_IN_SAMPLE | GUC_UNIT_BYTE + }, + &buffile_max_filesize, + MAX_PHYSICAL_FILESIZE, BLCKSZ, MAX_PHYSICAL_FILESIZE, + check_buffile_max_filesize, assign_buffile_max_filesize, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL @@ -11301,6 +11316,23 @@ show_tcp_user_timeout(void) } static bool +check_buffile_max_filesize(int *newval, void **extra, GucSource source) +{ + if (*newval % BLCKSZ != 0) + { + GUC_check_errdetail("The value must be whole multiple of BLCKSZ."); + return false; + } + return true; +} + +static void +assign_buffile_max_filesize(int newval, void *extra) +{ + buffile_seg_blocks = BUFFILE_SEG_BLOCKS(newval); +} + +static bool check_maxconnections(int *newval, void **extra, GucSource source) { if (*newval + autovacuum_max_workers + 1 + diff --git a/src/include/storage/buffile.h b/src/include/storage/buffile.h index 625fa0014c..977942f4b3 100644 --- a/src/include/storage/buffile.h +++ b/src/include/storage/buffile.h @@ -37,6 +37,25 @@ typedef struct BufFile BufFile; typedef struct TransientBufFile TransientBufFile; /* + * We break BufFiles into gigabyte-sized segments, regardless of RELSEG_SIZE. + * The reason is that we'd like large BufFiles to be spread across multiple + * tablespaces when available. + * + * An integer value indicating the number of useful bytes in the segment is + * appended to each segment of if the file is both shared and encrypted, see + * BufFile.useful. + */ +#define MAX_PHYSICAL_FILESIZE 0x40000000 + +/* Express segment size in the number of blocks. */ +#define BUFFILE_SEG_BLOCKS(phys) ((phys) / BLCKSZ) + +/* GUC to control size of the file segment. */ +extern int buffile_max_filesize; +/* Segment size in blocks, derived from the above. */ +extern int buffile_seg_blocks; + +/* * prototypes for functions in buffile.c */ diff --git a/src/test/modules/buffile/Makefile b/src/test/modules/buffile/Makefile new file mode 100644 index 0000000000..f0f163c45e --- /dev/null +++ b/src/test/modules/buffile/Makefile @@ -0,0 +1,21 @@ +PG_CONFIG ?= pg_config +MODULE_big = buffile_test +OBJS = buffile.o $(WIN32RES) +PGFILEDESC = "buffile_test" + +EXTENSION = buffile +DATA = buffile--1.0.sql + +REGRESS = test_01 test_02 test_03 test_04 test_05 test_06 test_07 test_08 \ +test_09 test_10 test_11 test_12 test_13 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/buffile +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/buffile/README b/src/test/modules/buffile/README new file mode 100644 index 0000000000..9c80787797 --- /dev/null +++ b/src/test/modules/buffile/README @@ -0,0 +1,11 @@ +This extension was written to check if changes introduced due to cluster +encryption do not break buffile.c. + +Caution: To make the test cheaper, it was decided to adjust the segment sizeq, +see + +#define MAX_PHYSICAL_FILESIZE (4 * BLCKSZ) + +in buffile.c. BLCKSZ is 8192 (the default). All the tests rely on this +value. So if you haven't compiled Postgres with this value, the tests will +create 1 GB files and they will fail. diff --git a/src/test/modules/buffile/buffile--1.0.sql b/src/test/modules/buffile/buffile--1.0.sql new file mode 100644 index 0000000000..7c168f3514 --- /dev/null +++ b/src/test/modules/buffile/buffile--1.0.sql @@ -0,0 +1,54 @@ +CREATE FUNCTION buffile_create() +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_create' +LANGUAGE C; + +CREATE FUNCTION buffile_close() +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_close' +LANGUAGE C; + +CREATE FUNCTION buffile_write(text) +RETURNS bigint +AS 'MODULE_PATHNAME', 'buffile_write' +LANGUAGE C; + +CREATE FUNCTION buffile_read(bigint) +RETURNS bytea +AS 'MODULE_PATHNAME', 'buffile_read' +LANGUAGE C; + +CREATE FUNCTION buffile_seek(int, bigint) +RETURNS int +AS 'MODULE_PATHNAME', 'buffile_seek' +LANGUAGE C; + +CREATE FUNCTION buffile_assert_fileno(int) +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_assert_fileno' +LANGUAGE C; + +CREATE FUNCTION buffile_test_shared() +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_test_shared' +LANGUAGE C; + +CREATE FUNCTION buffile_test_shared_append() +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_test_shared_append' +LANGUAGE C; + +CREATE FUNCTION buffile_open_transient(text, bool, bool) +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_open_transient' +LANGUAGE C; + +CREATE FUNCTION buffile_close_transient() +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_close_transient' +LANGUAGE C; + +CREATE FUNCTION buffile_delete_file(text) +RETURNS void +AS 'MODULE_PATHNAME', 'buffile_delete_file' +LANGUAGE C; diff --git a/src/test/modules/buffile/buffile.c b/src/test/modules/buffile/buffile.c new file mode 100644 index 0000000000..fe0680771a --- /dev/null +++ b/src/test/modules/buffile/buffile.c @@ -0,0 +1,429 @@ +#include + +#include "postgres.h" +#include "fmgr.h" +#include "lib/stringinfo.h" +#include "storage/buffile.h" +#include "utils/array.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/memutils.h" +#include "utils/resowner.h" + +PG_MODULE_MAGIC; + +/* + * To cope with files that span multiple segments w/o wasting resources, use + * the smallest possible segment size. The test scripts need to set + * buffile_max_filesize (GUC) accordingly. + */ +#define MAX_PHYSICAL_FILESIZE_TEST (4 * BLCKSZ) + +static BufFile *bf = NULL; +static TransientBufFile *bft = NULL; + +static void check_file(void); + +extern Datum buffile_create_transient(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_create); +Datum +buffile_create(PG_FUNCTION_ARGS) +{ + MemoryContext old_cxt; + ResourceOwner old_ro; + + if (bf != NULL) + elog(ERROR, "file already exists"); + + old_cxt = MemoryContextSwitchTo(TopMemoryContext); + + /* + * Make sure the file is not deleted across function calls. + */ + old_ro = CurrentResourceOwner; + CurrentResourceOwner = TopTransactionResourceOwner; + + bf = BufFileCreateTemp(false); + + CurrentResourceOwner = old_ro; + MemoryContextSwitchTo(old_cxt); + + PG_RETURN_VOID(); +} + +extern Datum buffile_close(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_close); +Datum +buffile_close(PG_FUNCTION_ARGS) +{ + if (bf == NULL) + elog(ERROR, "there's no file to close"); + + BufFileClose(bf); + bf = NULL; + + PG_RETURN_VOID(); +} + +extern Datum buffile_write(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_write); +Datum +buffile_write(PG_FUNCTION_ARGS) +{ + Datum d = PG_GETARG_DATUM(0); + char *s = TextDatumGetCString(d); + size_t res; + + if (bf) + res = BufFileWrite(bf, s, strlen(s)); + else if (bft) + res = BufFileWriteTransient(bft, s, strlen(s)); + else + elog(ERROR, "No file is open"); + + PG_RETURN_INT64(res); +} + +extern Datum buffile_read(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_read); +Datum +buffile_read(PG_FUNCTION_ARGS) +{ + int64 size = PG_GETARG_INT64(0); + StringInfo buf = makeStringInfo(); + size_t res_size; + bytea *result; + + enlargeStringInfo(buf, size); + + if (bf) + res_size = BufFileRead(bf, buf->data, size); + else if (bft) + res_size = BufFileReadTransient(bft, buf->data, size); + else + elog(ERROR, "No file is open"); + + buf->len = res_size; + + result = DatumGetByteaPP(DirectFunctionCall1(bytearecv, + PointerGetDatum(buf))); + PG_RETURN_BYTEA_P(result); +} + +extern Datum buffile_seek(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_seek); +Datum +buffile_seek(PG_FUNCTION_ARGS) +{ + int32 fileno = PG_GETARG_INT32(0); + int64 offset = PG_GETARG_INT64(1); + int32 res; + + check_file(); + res = BufFileSeek(bf, fileno, offset, SEEK_SET); + + PG_RETURN_INT32(res); +} + +extern Datum buffile_assert_fileno(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_assert_fileno); +Datum +buffile_assert_fileno(PG_FUNCTION_ARGS) +{ + int32 fileno_expected = PG_GETARG_INT32(0); + int32 fileno; + off_t offset; + + check_file(); + BufFileTell(bf, &fileno, &offset); + + if (fileno != fileno_expected) + { + /* + * Bring the backend down so that the following tests have no chance + * to create the 1GB files. + */ + elog(FATAL, "file number does not match"); + } + + PG_RETURN_VOID(); +} + +static void +check_file(void) +{ + if (bf == NULL) + elog(ERROR, "the file is not opened"); +} + +/* + * This test is especially important for shared encrypted files, see the + * comments below. + */ +extern Datum buffile_test_shared(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_test_shared); +Datum +buffile_test_shared(PG_FUNCTION_ARGS) +{ + dsm_segment *seg; + SharedFileSet *fileset; + BufFile *bf_1, + *bf_2; + char *data_1, + *data_2, + *data; + Size chunk_size_1, + chunk_size_2; + int fileno, + i; + off_t offset, + res, + total_size; + + /* + * The size is not important, we actually do not need the shared memory. + * The segment is only needed to initialize the fileset. + */ + seg = dsm_create(1024, 0); + + /* + * The fileset must survive error handling, so that dsm_detach works fine. + * (The typical use case is that the fileset is in shared memory.) + */ + fileset = (SharedFileSet *) MemoryContextAlloc(TopTransactionContext, + sizeof(SharedFileSet)); + SharedFileSetInit(fileset, seg); + + bf_1 = BufFileCreateShared(fileset, "file_1"); + + /* + * Write more data than the buffer size, so that we can check that the + * number of "useful bytes" word is only appended at the end of the + * segment, not after each buffer. + */ + chunk_size_1 = BLCKSZ + 256; + data_1 = (char *) palloc(chunk_size_1); + memset(data_1, 1, chunk_size_1); + if (BufFileWrite(bf_1, data_1, chunk_size_1) != chunk_size_1) + elog(ERROR, "Failed to write data"); + pfree(data_1); + + /* + * Enforce buffer flush (The BufFileFlush() function is not exported). + * Thus the "useful bytes" metadata should appear at the current end the + * first file segment. The next write will have to seek back to overwrite + * the metadata. + */ + BufFileTell(bf_1, &fileno, &offset); + if (BufFileSeek(bf_1, 0, 0, SEEK_SET) != 0) + elog(ERROR, "seek failed"); + if (BufFileSeek(bf_1, fileno, offset, SEEK_SET) != 0) + elog(ERROR, "seek failed"); + + /* + * Write another chunk that does not fit into the first segment file. Thus + * the "useful bytes" metadata should appear at the end of both segments. + */ + chunk_size_2 = 3 * BLCKSZ; + data_2 = (char *) palloc(chunk_size_2); + memset(data_2, 1, chunk_size_2); + if (BufFileWrite(bf_1, data_2, chunk_size_2) != chunk_size_2) + elog(ERROR, "Failed to write data"); + pfree(data_2); + BufFileClose(bf_1); + + /* + * The word indicating the number of "useful bytes" (i.e. the actual data + * w/o padding to buffer size) is stored at the end of each segment file. + * Check that this metadata is read correctly. + */ + bf_2 = BufFileOpenShared(fileset, "file_1"); + total_size = BufFileSize(bf_2); + if (total_size != (chunk_size_1 + chunk_size_2)) + elog(ERROR, "Incorrect file size: %zu", total_size); + + data = (char *) palloc(total_size); + res = BufFileRead(bf_2, data, total_size); + if (res != total_size) + elog(ERROR, "Incorrect chunk size read: %zu", res); + for (i = 0; i < total_size; i++) + if (data[i] != 1) + elog(ERROR, "Unexpected data read from the file"); + pfree(data); + BufFileClose(bf_2); + + dsm_detach(seg); + + PG_RETURN_VOID(); +} + + +/* + * Test BufFileAppend(). + */ +extern Datum buffile_test_shared_append(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_test_shared_append); +Datum +buffile_test_shared_append(PG_FUNCTION_ARGS) +{ + dsm_segment *seg; + SharedFileSet *fileset; + BufFile *bf_1, + *bf_2, + *bf_3; + char *data; + Size chunk_size; + int i; + off_t res, + total_size; + + seg = dsm_create(1024, 0); + + fileset = (SharedFileSet *) MemoryContextAlloc(TopTransactionContext, + sizeof(SharedFileSet)); + SharedFileSetInit(fileset, seg); + + /* + * XXX Does the chunk size matter much? + */ + chunk_size = 8; + data = (char *) palloc(chunk_size); + memset(data, 1, chunk_size); + + bf_1 = BufFileCreateShared(fileset, "file_1"); + if (BufFileWrite(bf_1, data, chunk_size) != chunk_size) + elog(ERROR, "Failed to write data"); + + bf_2 = BufFileCreateShared(fileset, "file_2"); + if (BufFileWrite(bf_2, data, chunk_size) != chunk_size) + elog(ERROR, "Failed to write data"); + + /* + * Make sure it's read-only so that BufFileAppend() can accept it as + * source. + */ + BufFileClose(bf_2); + bf_2 = BufFileOpenShared(fileset, "file_2"); + + bf_3 = BufFileCreateShared(fileset, "file_3"); + if (BufFileWrite(bf_3, data, chunk_size) != chunk_size) + elog(ERROR, "Failed to write data"); + BufFileClose(bf_3); + bf_3 = BufFileOpenShared(fileset, "file_3"); + + BufFileAppend(bf_1, bf_2); + BufFileAppend(bf_1, bf_3); + + total_size = BufFileSize(bf_1); + + /* + * The result should contain complete segments of bf_1 and bf_2 and the + * valid part of bf_3. + */ + if (total_size != (2 * MAX_PHYSICAL_FILESIZE_TEST + chunk_size)) + elog(ERROR, "Incorrect total size of the appended data: %zu", + total_size); + + /* + * Check that data of the 2nd segment was decrypted correctly. + */ + if (BufFileSeek(bf_1, 1, 0, SEEK_SET) != 0) + elog(ERROR, "seek failed"); + res = BufFileRead(bf_1, data, chunk_size); + if (res != chunk_size) + elog(ERROR, "Incorrect chunk size read: %zu", res); + for (i = 0; i < chunk_size; i++) + if (data[i] != 1) + elog(ERROR, "Unexpected data read from the file"); + + /* + * And the same for the 3rd segment. + * + * TODO Reuse the code above by putting it into a function. + */ + if (BufFileSeek(bf_1, 2, 0, SEEK_SET) != 0) + elog(ERROR, "seek failed"); + res = BufFileRead(bf_1, data, chunk_size); + if (res != chunk_size) + elog(ERROR, "Incorrect chunk size read: %zu", res); + for (i = 0; i < chunk_size; i++) + if (data[i] != 1) + elog(ERROR, "Unexpected data read from the file"); + + BufFileClose(bf_1); + dsm_detach(seg); + PG_RETURN_VOID(); +} + +extern Datum buffile_open_transient(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_open_transient); +Datum +buffile_open_transient(PG_FUNCTION_ARGS) +{ + MemoryContext old_cxt; + Datum d = PG_GETARG_DATUM(0); + char *path = TextDatumGetCString(d); + bool write_only = PG_GETARG_BOOL(1); + bool append = PG_GETARG_BOOL(2); + int flags = O_CREAT | PG_BINARY; + + if (bft != NULL) + elog(ERROR, "file already exists"); + + if (write_only) + flags |= O_WRONLY; + if (append) + flags |= O_APPEND; + + old_cxt = MemoryContextSwitchTo(TopMemoryContext); + + /* + * Make sure the file is not deleted across function calls. + */ + bft = BufFileOpenTransient(path, flags); + + MemoryContextSwitchTo(old_cxt); + + PG_RETURN_VOID(); +} + +extern Datum buffile_close_transient(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_close_transient); +Datum +buffile_close_transient(PG_FUNCTION_ARGS) +{ + if (bft == NULL) + elog(ERROR, "there's no file to close"); + + BufFileCloseTransient(bft); + bft = NULL; + + PG_RETURN_VOID(); +} + +extern Datum buffile_delete_file(PG_FUNCTION_ARGS); + +PG_FUNCTION_INFO_V1(buffile_delete_file); +Datum +buffile_delete_file(PG_FUNCTION_ARGS) +{ + Datum d = PG_GETARG_DATUM(0); + char *path = TextDatumGetCString(d); + + if (bft != NULL) + elog(ERROR, "the file is still open"); + + PathNameDeleteTemporaryFile(path, true); + + PG_RETURN_VOID(); +} diff --git a/src/test/modules/buffile/buffile.control b/src/test/modules/buffile/buffile.control new file mode 100644 index 0000000000..8472c5a348 --- /dev/null +++ b/src/test/modules/buffile/buffile.control @@ -0,0 +1,5 @@ +# buffile_test extension +comment = 'buffile_test' +default_version = '1.0' +module_pathname = '$libdir/buffile_test' +relocatable = true diff --git a/src/test/modules/buffile/expected/test_01.out b/src/test/modules/buffile/expected/test_01.out new file mode 100644 index 0000000000..c1a4f5ac7b --- /dev/null +++ b/src/test/modules/buffile/expected/test_01.out @@ -0,0 +1,61 @@ +-- This is the first test, so make sure the test extension is there. +CREATE EXTENSION IF NOT EXISTS buffile; +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +SELECT buffile_seek(0, 1); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('abc'); + buffile_write +--------------- + 3 +(1 row) + +SELECT buffile_seek(0, 0); + buffile_seek +-------------- + 0 +(1 row) + +-- Check that the trailing zeroes are not fetched. +SELECT buffile_read(16); + buffile_read +-------------- + \x00616263 +(1 row) + +-- Adjust the number of useful bytes. +SELECT buffile_write('abc'); + buffile_write +--------------- + 3 +(1 row) + +-- ... and check again. +SELECT buffile_seek(0, 0); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(16); + buffile_read +------------------ + \x00616263616263 +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_02.out b/src/test/modules/buffile/expected/test_02.out new file mode 100644 index 0000000000..f783d0cb24 --- /dev/null +++ b/src/test/modules/buffile/expected/test_02.out @@ -0,0 +1,48 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +SELECT buffile_seek(0, 8189); + buffile_seek +-------------- + 0 +(1 row) + +-- Initialize the last 3 positions of the first buffer and the initial 3 +-- positions of the 2nd buffer. +SELECT buffile_write('abcdef'); + buffile_write +--------------- + 6 +(1 row) + +SELECT buffile_seek(0, 0); + buffile_seek +-------------- + 0 +(1 row) + +-- Read the first buffer. +SELECT length(buffile_read(8192)); + length +-------- + 8192 +(1 row) + +-- Only 3 bytes of the 2nd buffer should be fetched. +SELECT length(buffile_read(8192)); + length +-------- + 3 +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_03.out b/src/test/modules/buffile/expected/test_03.out new file mode 100644 index 0000000000..e899fa3b38 --- /dev/null +++ b/src/test/modules/buffile/expected/test_03.out @@ -0,0 +1,27 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Read from an empty file. +SELECT buffile_seek(0, 8); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(16); + buffile_read +-------------- + \x +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_04.out b/src/test/modules/buffile/expected/test_04.out new file mode 100644 index 0000000000..1f8eeabe48 --- /dev/null +++ b/src/test/modules/buffile/expected/test_04.out @@ -0,0 +1,84 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Write something near the end of the first buffer, but leave some trailing +-- space. +SELECT buffile_seek(0, 8184); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('abcd'); + buffile_write +--------------- + 4 +(1 row) + +-- Leave the 2nd buffer empty, as well as a few leading bytes. Thus we should +-- get a hole that spans the whole 2nd buffer as well as a few adjacent bytes +-- on each side. +SELECT buffile_seek(0, 2 * 8192 + 4); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('efgh'); + buffile_write +--------------- + 4 +(1 row) + +-- Check the initial part of the hole, which crosses the boundary of the 1st +-- and the 2nd buffer. +SELECT buffile_seek(0, 8184); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(16); + buffile_read +------------------------------------ + \x61626364000000000000000000000000 +(1 row) + +-- Check the trailing part of the whole, which crosses the boundary of the 2nd +-- and the 3rd buffer. +SELECT buffile_seek(0, 2 * 8192 - 8); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(16); + buffile_read +------------------------------------ + \x00000000000000000000000065666768 +(1 row) + +-- Check that the hole contains nothing but zeroes. +SELECT buffile_seek(0, 8192 - 4); + buffile_seek +-------------- + 0 +(1 row) + +SELECT btrim(buffile_read(8192 + 8), '\x00'); + btrim +------- + \x +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_05.out b/src/test/modules/buffile/expected/test_05.out new file mode 100644 index 0000000000..6a73147711 --- /dev/null +++ b/src/test/modules/buffile/expected/test_05.out @@ -0,0 +1,33 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Seek does not extend the file if it's not followed by write. +SELECT buffile_seek(0, 1); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_seek(0, 0); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(2); + buffile_read +-------------- + \x +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_06.out b/src/test/modules/buffile/expected/test_06.out new file mode 100644 index 0000000000..d406bda9e6 --- /dev/null +++ b/src/test/modules/buffile/expected/test_06.out @@ -0,0 +1,43 @@ +-- This test shows that the if first component file (segment) stays empty, +-- read stops prematurely even if it starts within that segment, although it'd +-- otherwise receive some data from the following one. +BEGIN; +-- Neither disk space nor time needs to be wasted. +SET buffile_max_filesize TO 8192; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +SELECT buffile_seek(0, 8192); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('a'); + buffile_write +--------------- + 1 +(1 row) + +SELECT buffile_seek(0, 8191); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(2); + buffile_read +-------------- + \x +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_07.out b/src/test/modules/buffile/expected/test_07.out new file mode 100644 index 0000000000..6f2ee3d2d9 --- /dev/null +++ b/src/test/modules/buffile/expected/test_07.out @@ -0,0 +1,41 @@ +BEGIN; +-- Use a small segment, not to waste disk space and time. +SET buffile_max_filesize TO 8192; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Write data at component file boundary and try to read it. +SELECT buffile_seek(0, 8192); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('abcd'); + buffile_write +--------------- + 4 +(1 row) + +SELECT buffile_seek(0, 8192); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(8); + buffile_read +-------------- + \x61626364 +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_08.out b/src/test/modules/buffile/expected/test_08.out new file mode 100644 index 0000000000..6ffcb9dcd6 --- /dev/null +++ b/src/test/modules/buffile/expected/test_08.out @@ -0,0 +1,41 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Neither disk space nor time needs to be wasted. +SET buffile_max_filesize TO 8192; +-- Write data across component file boundary and try to read it. +SELECT buffile_seek(0, 8190); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('abcd'); + buffile_write +--------------- + 4 +(1 row) + +SELECT buffile_seek(0, 8190); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(8); + buffile_read +-------------- + \x61626364 +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_09.out b/src/test/modules/buffile/expected/test_09.out new file mode 100644 index 0000000000..b9d325b676 --- /dev/null +++ b/src/test/modules/buffile/expected/test_09.out @@ -0,0 +1,39 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Write data across buffer boundary and try to read it. +SELECT buffile_seek(0, 8190); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('abcd'); + buffile_write +--------------- + 4 +(1 row) + +SELECT buffile_seek(0, 8190); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(8); + buffile_read +-------------- + \x61626364 +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_10.out b/src/test/modules/buffile/expected/test_10.out new file mode 100644 index 0000000000..8e457fdcda --- /dev/null +++ b/src/test/modules/buffile/expected/test_10.out @@ -0,0 +1,76 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +-- Write some data at the end of the buffer. +SELECT buffile_seek(0, 8188); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_write('abcd'); + buffile_write +--------------- + 4 +(1 row) + +SELECT buffile_seek(0, 8189); + buffile_seek +-------------- + 0 +(1 row) + +-- Enforce flush with the write position not at the end of the buffer. This is +-- special by not moving curOffset to the next buffer. +SELECT buffile_read(1); + buffile_read +-------------- + \x62 +(1 row) + +-- Therefore the next writes should eventually affect the original data. (Here +-- we also test going directly from read to write and vice versa.) +SELECT buffile_write('x'); + buffile_write +--------------- + 1 +(1 row) + +SELECT buffile_read(1); + buffile_read +-------------- + \x64 +(1 row) + +-- Start a new buffer, i.e. force flushing of the previous one. +SELECT buffile_write('z'); + buffile_write +--------------- + 1 +(1 row) + +-- Check that the 'x' and 'y' letters are in the first buffer, not in the +-- 2nd. (We read enough data to find any non-zero bytes in the 2nd buffer.) +SELECT buffile_seek(0, 8188); + buffile_seek +-------------- + 0 +(1 row) + +SELECT buffile_read(4 + 8192); + buffile_read +-------------- + \x616278647a +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_11.out b/src/test/modules/buffile/expected/test_11.out new file mode 100644 index 0000000000..c6804d08af --- /dev/null +++ b/src/test/modules/buffile/expected/test_11.out @@ -0,0 +1,34 @@ +BEGIN; +SELECT buffile_create(); + buffile_create +---------------- + +(1 row) + +SELECT buffile_write('abcd'); + buffile_write +--------------- + 4 +(1 row) + +-- Seek beyond EOF not followed by write. +SELECT buffile_seek(0, 5); + buffile_seek +-------------- + 0 +(1 row) + +-- Nothing should be fetched. +SELECT buffile_read(8); + buffile_read +-------------- + \x +(1 row) + +SELECT buffile_close(); + buffile_close +--------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/expected/test_12.out b/src/test/modules/buffile/expected/test_12.out new file mode 100644 index 0000000000..fe507a8e58 --- /dev/null +++ b/src/test/modules/buffile/expected/test_12.out @@ -0,0 +1,16 @@ +-- Neither disk space nor time needs to be wasted. +-- +-- Here the tests are designed for segments of multiple buffers. +SET buffile_max_filesize TO 32768; +SELECT buffile_test_shared(); + buffile_test_shared +--------------------- + +(1 row) + +SELECT buffile_test_shared_append(); + buffile_test_shared_append +---------------------------- + +(1 row) + diff --git a/src/test/modules/buffile/expected/test_13.out b/src/test/modules/buffile/expected/test_13.out new file mode 100644 index 0000000000..7ba50a9c9d --- /dev/null +++ b/src/test/modules/buffile/expected/test_13.out @@ -0,0 +1,86 @@ +-- Use transaction block so that the file does not closed automatically at +-- command boundary. +BEGIN; +SELECT buffile_open_transient('trans1', true, false); + buffile_open_transient +------------------------ + +(1 row) + +SELECT buffile_write('01234567'); + buffile_write +--------------- + 8 +(1 row) + +SELECT buffile_close_transient(); + buffile_close_transient +------------------------- + +(1 row) + +-- Open for reading. +SELECT buffile_open_transient('trans1', false, false); + buffile_open_transient +------------------------ + +(1 row) + +SELECT length(buffile_read(65536)); + length +-------- + 8 +(1 row) + +SELECT buffile_close_transient(); + buffile_close_transient +------------------------- + +(1 row) + +-- Open for writing in append mode. +SELECT buffile_open_transient('trans1', true, true); + buffile_open_transient +------------------------ + +(1 row) + +-- Add BLCKSZ bytes, so that buffer boundary is crossed. +SELECT buffile_write(repeat('x', 8192)); + buffile_write +--------------- + 8192 +(1 row) + +SELECT buffile_close_transient(); + buffile_close_transient +------------------------- + +(1 row) + +-- Open for reading and verify the valid part. +SELECT buffile_open_transient('trans1', false, false); + buffile_open_transient +------------------------ + +(1 row) + +SELECT length(buffile_read(65536)); + length +-------- + 8200 +(1 row) + +SELECT buffile_close_transient(); + buffile_close_transient +------------------------- + +(1 row) + +SELECT buffile_delete_file('trans1'); + buffile_delete_file +--------------------- + +(1 row) + +COMMIT; diff --git a/src/test/modules/buffile/sql/test_01.sql b/src/test/modules/buffile/sql/test_01.sql new file mode 100644 index 0000000000..92331dd0de --- /dev/null +++ b/src/test/modules/buffile/sql/test_01.sql @@ -0,0 +1,17 @@ +-- This is the first test, so make sure the test extension is there. +CREATE EXTENSION IF NOT EXISTS buffile; + +BEGIN; +SELECT buffile_create(); +SELECT buffile_seek(0, 1); +SELECT buffile_write('abc'); +SELECT buffile_seek(0, 0); +-- Check that the trailing zeroes are not fetched. +SELECT buffile_read(16); +-- Adjust the number of useful bytes. +SELECT buffile_write('abc'); +-- ... and check again. +SELECT buffile_seek(0, 0); +SELECT buffile_read(16); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_02.sql b/src/test/modules/buffile/sql/test_02.sql new file mode 100644 index 0000000000..35d34722d7 --- /dev/null +++ b/src/test/modules/buffile/sql/test_02.sql @@ -0,0 +1,13 @@ +BEGIN; +SELECT buffile_create(); +SELECT buffile_seek(0, 8189); +-- Initialize the last 3 positions of the first buffer and the initial 3 +-- positions of the 2nd buffer. +SELECT buffile_write('abcdef'); +SELECT buffile_seek(0, 0); +-- Read the first buffer. +SELECT length(buffile_read(8192)); +-- Only 3 bytes of the 2nd buffer should be fetched. +SELECT length(buffile_read(8192)); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_03.sql b/src/test/modules/buffile/sql/test_03.sql new file mode 100644 index 0000000000..a95391f7c3 --- /dev/null +++ b/src/test/modules/buffile/sql/test_03.sql @@ -0,0 +1,7 @@ +BEGIN; +SELECT buffile_create(); +-- Read from an empty file. +SELECT buffile_seek(0, 8); +SELECT buffile_read(16); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_04.sql b/src/test/modules/buffile/sql/test_04.sql new file mode 100644 index 0000000000..64e8d39f94 --- /dev/null +++ b/src/test/modules/buffile/sql/test_04.sql @@ -0,0 +1,25 @@ +BEGIN; +SELECT buffile_create(); +-- Write something near the end of the first buffer, but leave some trailing +-- space. +SELECT buffile_seek(0, 8184); +SELECT buffile_write('abcd'); +-- Leave the 2nd buffer empty, as well as a few leading bytes. Thus we should +-- get a hole that spans the whole 2nd buffer as well as a few adjacent bytes +-- on each side. +SELECT buffile_seek(0, 2 * 8192 + 4); +SELECT buffile_write('efgh'); +-- Check the initial part of the hole, which crosses the boundary of the 1st +-- and the 2nd buffer. +SELECT buffile_seek(0, 8184); +SELECT buffile_read(16); +-- Check the trailing part of the whole, which crosses the boundary of the 2nd +-- and the 3rd buffer. +SELECT buffile_seek(0, 2 * 8192 - 8); +SELECT buffile_read(16); +-- Check that the hole contains nothing but zeroes. +SELECT buffile_seek(0, 8192 - 4); +SELECT btrim(buffile_read(8192 + 8), '\x00'); + +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_05.sql b/src/test/modules/buffile/sql/test_05.sql new file mode 100644 index 0000000000..5fd642e558 --- /dev/null +++ b/src/test/modules/buffile/sql/test_05.sql @@ -0,0 +1,8 @@ +BEGIN; +SELECT buffile_create(); +-- Seek does not extend the file if it's not followed by write. +SELECT buffile_seek(0, 1); +SELECT buffile_seek(0, 0); +SELECT buffile_read(2); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_06.sql b/src/test/modules/buffile/sql/test_06.sql new file mode 100644 index 0000000000..35aa48857c --- /dev/null +++ b/src/test/modules/buffile/sql/test_06.sql @@ -0,0 +1,15 @@ +-- This test shows that the if first component file (segment) stays empty, +-- read stops prematurely even if it starts within that segment, although it'd +-- otherwise receive some data from the following one. +BEGIN; + +-- Neither disk space nor time needs to be wasted. +SET buffile_max_filesize TO 8192; + +SELECT buffile_create(); +SELECT buffile_seek(0, 8192); +SELECT buffile_write('a'); +SELECT buffile_seek(0, 8191); +SELECT buffile_read(2); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_07.sql b/src/test/modules/buffile/sql/test_07.sql new file mode 100644 index 0000000000..06057c5a1b --- /dev/null +++ b/src/test/modules/buffile/sql/test_07.sql @@ -0,0 +1,13 @@ +BEGIN; + +-- Use a small segment, not to waste disk space and time. +SET buffile_max_filesize TO 8192; + +SELECT buffile_create(); +-- Write data at component file boundary and try to read it. +SELECT buffile_seek(0, 8192); +SELECT buffile_write('abcd'); +SELECT buffile_seek(0, 8192); +SELECT buffile_read(8); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_08.sql b/src/test/modules/buffile/sql/test_08.sql new file mode 100644 index 0000000000..2736dc22df --- /dev/null +++ b/src/test/modules/buffile/sql/test_08.sql @@ -0,0 +1,13 @@ +BEGIN; +SELECT buffile_create(); + +-- Neither disk space nor time needs to be wasted. +SET buffile_max_filesize TO 8192; + +-- Write data across component file boundary and try to read it. +SELECT buffile_seek(0, 8190); +SELECT buffile_write('abcd'); +SELECT buffile_seek(0, 8190); +SELECT buffile_read(8); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_09.sql b/src/test/modules/buffile/sql/test_09.sql new file mode 100644 index 0000000000..cc7060932e --- /dev/null +++ b/src/test/modules/buffile/sql/test_09.sql @@ -0,0 +1,9 @@ +BEGIN; +SELECT buffile_create(); +-- Write data across buffer boundary and try to read it. +SELECT buffile_seek(0, 8190); +SELECT buffile_write('abcd'); +SELECT buffile_seek(0, 8190); +SELECT buffile_read(8); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_10.sql b/src/test/modules/buffile/sql/test_10.sql new file mode 100644 index 0000000000..63af760d9f --- /dev/null +++ b/src/test/modules/buffile/sql/test_10.sql @@ -0,0 +1,25 @@ +BEGIN; +SELECT buffile_create(); +-- Write some data at the end of the buffer. +SELECT buffile_seek(0, 8188); +SELECT buffile_write('abcd'); +SELECT buffile_seek(0, 8189); +-- Enforce flush with the write position not at the end of the buffer. This is +-- special by not moving curOffset to the next buffer. +SELECT buffile_read(1); + +-- Therefore the next writes should eventually affect the original data. (Here +-- we also test going directly from read to write and vice versa.) +SELECT buffile_write('x'); +SELECT buffile_read(1); + +-- Start a new buffer, i.e. force flushing of the previous one. +SELECT buffile_write('z'); + +-- Check that the 'x' and 'y' letters are in the first buffer, not in the +-- 2nd. (We read enough data to find any non-zero bytes in the 2nd buffer.) +SELECT buffile_seek(0, 8188); +SELECT buffile_read(4 + 8192); + +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_11.sql b/src/test/modules/buffile/sql/test_11.sql new file mode 100644 index 0000000000..94300d253f --- /dev/null +++ b/src/test/modules/buffile/sql/test_11.sql @@ -0,0 +1,9 @@ +BEGIN; +SELECT buffile_create(); +SELECT buffile_write('abcd'); +-- Seek beyond EOF not followed by write. +SELECT buffile_seek(0, 5); +-- Nothing should be fetched. +SELECT buffile_read(8); +SELECT buffile_close(); +COMMIT; diff --git a/src/test/modules/buffile/sql/test_12.sql b/src/test/modules/buffile/sql/test_12.sql new file mode 100644 index 0000000000..017050ef5b --- /dev/null +++ b/src/test/modules/buffile/sql/test_12.sql @@ -0,0 +1,7 @@ +-- Neither disk space nor time needs to be wasted. +-- +-- Here the tests are designed for segments of multiple buffers. +SET buffile_max_filesize TO 32768; + +SELECT buffile_test_shared(); +SELECT buffile_test_shared_append(); diff --git a/src/test/modules/buffile/sql/test_13.sql b/src/test/modules/buffile/sql/test_13.sql new file mode 100644 index 0000000000..bcabf7bfe6 --- /dev/null +++ b/src/test/modules/buffile/sql/test_13.sql @@ -0,0 +1,25 @@ +-- Use transaction block so that the file does not closed automatically at +-- command boundary. +BEGIN; +SELECT buffile_open_transient('trans1', true, false); +SELECT buffile_write('01234567'); +SELECT buffile_close_transient(); + +-- Open for reading. +SELECT buffile_open_transient('trans1', false, false); +SELECT length(buffile_read(65536)); +SELECT buffile_close_transient(); + +-- Open for writing in append mode. +SELECT buffile_open_transient('trans1', true, true); +-- Add BLCKSZ bytes, so that buffer boundary is crossed. +SELECT buffile_write(repeat('x', 8192)); +SELECT buffile_close_transient(); + +-- Open for reading and verify the valid part. +SELECT buffile_open_transient('trans1', false, false); +SELECT length(buffile_read(65536)); +SELECT buffile_close_transient(); + +SELECT buffile_delete_file('trans1'); +COMMIT; -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0013-Make-database-OID-constant-for-postgres-database.patch From 02df324bdcbaf4ff475025b11c37afea9c86623b Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:02 +0200 Subject: [PATCH 13/17] Make database OID constant for postgres database. Later in the patch series we teach pg_upgrade to preserve OIDs of relfilenodes, tablespaces and databases. To ensure the last, we need to make the OID of the "postgres" database constant because pg_upgrade can drop it in the new cluster and create it again. --- src/backend/commands/dbcommands.c | 21 +++++++++++++++++---- src/include/catalog/pg_database.h | 14 ++++++++++++++ 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 7ea783d096..4ffb6dccf4 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -513,11 +513,24 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) */ pg_database_rel = table_open(DatabaseRelationId, RowExclusiveLock); - do + /* The "postgres" database has fixed OID. */ + if (strcmp(dbname, "postgres") == 0) { - dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, - Anum_pg_database_oid); - } while (check_db_file_conflict(dboid)); + dboid = PostgresDbOid; + + if (check_db_file_conflict(dboid)) + ereport(ERROR, + (errcode(ERRCODE_DUPLICATE_DATABASE), + errmsg("database \"%s\" already exists", dbname))); + } + else + { + do + { + dboid = GetNewOidWithIndex(pg_database_rel, DatabaseOidIndexId, + Anum_pg_database_oid); + } while (check_db_file_conflict(dboid)); + } /* * Insert a new tuple into pg_database. This establishes our ownership of diff --git a/src/include/catalog/pg_database.h b/src/include/catalog/pg_database.h index 06fea45f53..ebb17e3051 100644 --- a/src/include/catalog/pg_database.h +++ b/src/include/catalog/pg_database.h @@ -80,4 +80,18 @@ CATALOG(pg_database,1262,DatabaseRelationId) BKI_SHARED_RELATION BKI_ROWTYPE_OID */ typedef FormData_pg_database *Form_pg_database; +/* + * The "postgres" database has reserved pg_database(oid) because initially + * it's below FirstNormalObjectId, and thus pg_upgrade cannot guarantee that + * the same OID will be assigned in the new cluster (unlike TemplateDbOid, + * which has always OID=1). Since database OID is used to construct encryption + * initialization vector (tweak), different OID would break data decryption in + * the new cluster. + * + * The easiest way to avoid collision with any system OID is to use the + * maximum system OID. initdb won't reach this value and GetNewObjectId() will + * always skip that. + */ +#define PostgresDbOid (FirstNormalObjectId - 1) + #endif /* PG_DATABASE_H */ -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0014-Teach-pg_upgrade-to-preserve-OIDs-of-relfilenode-dat.patch From c9a9d7f731a7e103f2346d52d3c6deae0c4d7045 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:02 +0200 Subject: [PATCH 14/17] Teach pg_upgrade to preserve OIDs of relfilenode, database and tablespace. Since these OIDs are used to construct encryption initialization vector, we need to keep them the same in the new cluster, otherwise relation files need to be "reencrypted" during the update and the --link mode must be disabled for encrypted cluster. --- src/backend/catalog/heap.c | 71 +++++++++++++++++++---- src/backend/catalog/index.c | 1 + src/backend/commands/dbcommands.c | 11 ++++ src/backend/commands/tablespace.c | 14 ++++- src/backend/utils/adt/pg_upgrade_support.c | 55 ++++++++++++++++++ src/bin/pg_dump/pg_dump.c | 90 ++++++++++++++++++++++++++---- src/bin/pg_dump/pg_dumpall.c | 3 + src/bin/pg_upgrade/Makefile | 2 +- src/bin/pg_upgrade/pg_upgrade.c | 14 +++-- src/include/catalog/binary_upgrade.h | 7 +++ src/include/catalog/pg_proc.dat | 20 +++++++ 11 files changed, 257 insertions(+), 31 deletions(-) diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 3b8c8b193a..0aef214cae 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -91,7 +91,9 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_heap_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; Oid binary_upgrade_next_toast_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; static void AddNewRelationTuple(Relation pg_class_desc, Relation new_rel_desc, @@ -366,17 +368,60 @@ heap_create(const char *relname, break; } - /* - * Decide whether to create storage. If caller passed a valid relfilenode, - * storage is already created, so don't do it here. Also don't create it - * for relkinds without physical storage. - */ - if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) - create_storage = false; + /* Decide whether to create storage. */ + if (!IsBinaryUpgrade) + { + /* + * If caller passed a valid relfilenode, storage is already created, + * so don't do it here. Also don't create it for relkinds without + * physical storage. + */ + if (!RELKIND_HAS_STORAGE(relkind) || OidIsValid(relfilenode)) + create_storage = false; + else + { + create_storage = true; + relfilenode = relid; + } + } else { + /* Override relfilenode? */ + if (relkind == RELKIND_RELATION || relkind == RELKIND_SEQUENCE || + relkind == RELKIND_MATVIEW) + { + if (!OidIsValid(binary_upgrade_next_heap_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("relfilenode value not set when in binary upgrade mode"))); + + relfilenode = binary_upgrade_next_heap_pg_class_relfilenode; + binary_upgrade_next_heap_pg_class_relfilenode = InvalidOid; + } + else if (relkind == RELKIND_TOASTVALUE) + { + if (!OidIsValid(binary_upgrade_next_toast_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("toast relfilenode value not set when in binary upgrade mode"))); + relfilenode = binary_upgrade_next_toast_pg_class_relfilenode; + binary_upgrade_next_toast_pg_class_relfilenode = InvalidOid; + } + else if (relkind == RELKIND_INDEX) + { + if (!OidIsValid(binary_upgrade_next_index_pg_class_relfilenode)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_class index relfilenode value not set when in binary upgrade mode"))); + relfilenode = binary_upgrade_next_index_pg_class_relfilenode; + binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; + } + + /* + * The storage is needed for the following DDLs, although it will + * eventually be overwritten with that of the old cluster. + */ create_storage = true; - relfilenode = relid; } /* @@ -1149,9 +1194,6 @@ heap_create_with_catalog(const char *relname, /* * Allocate an OID for the relation, unless we were told what to use. - * - * The OID will be the relfilenode as well, so make sure it doesn't - * collide with either pg_class OIDs or existing physical files. */ if (!OidIsValid(relid)) { @@ -1179,8 +1221,15 @@ heap_create_with_catalog(const char *relname, binary_upgrade_next_toast_pg_class_oid = InvalidOid; } else + { + /* + * The OID will be the relfilenode as well, so make sure it + * doesn't collide with either pg_class OIDs or existing physical + * files. + */ relid = GetNewRelFileNode(reltablespace, pg_class_desc, relpersistence); + } } /* diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index bb60b23093..1566d03dcc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -83,6 +83,7 @@ /* Potentially set by pg_upgrade_support functions */ Oid binary_upgrade_next_index_pg_class_oid = InvalidOid; +Oid binary_upgrade_next_index_pg_class_relfilenode = InvalidOid; /* * Pointer-free representation of variables used when reindexing system diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c index 4ffb6dccf4..692bd40907 100644 --- a/src/backend/commands/dbcommands.c +++ b/src/backend/commands/dbcommands.c @@ -92,6 +92,7 @@ static void remove_dbtablespaces(Oid db_id); static bool check_db_file_conflict(Oid db_id); static int errdetail_busy_db(int notherbackends, int npreparedxacts); +Oid binary_upgrade_next_pg_database_oid = InvalidOid; /* * CREATE DATABASE @@ -523,6 +524,16 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt) (errcode(ERRCODE_DUPLICATE_DATABASE), errmsg("database \"%s\" already exists", dbname))); } + else if (IsBinaryUpgrade) + { + if (!OidIsValid(binary_upgrade_next_pg_database_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_database OID value not set when in binary upgrade mode"))); + + dboid = binary_upgrade_next_pg_database_oid; + binary_upgrade_next_pg_database_oid = InvalidOid; + } else { do diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 84efb414d8..dcbecc7196 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -89,6 +89,7 @@ char *default_tablespace = NULL; char *temp_tablespaces = NULL; +Oid binary_upgrade_next_pg_tablespace_oid = InvalidOid; static void create_tablespace_directories(const char *location, const Oid tablespaceoid); @@ -336,7 +337,18 @@ CreateTableSpace(CreateTableSpaceStmt *stmt) MemSet(nulls, false, sizeof(nulls)); - tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, + if (IsBinaryUpgrade) + { + if (!OidIsValid(binary_upgrade_next_pg_tablespace_oid)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("pg_tablespace OID value not set when in binary upgrade mode"))); + + tablespaceoid = binary_upgrade_next_pg_tablespace_oid; + binary_upgrade_next_pg_tablespace_oid = InvalidOid; + } + else + tablespaceoid = GetNewOidWithIndex(rel, TablespaceOidIndexId, Anum_pg_tablespace_oid); values[Anum_pg_tablespace_oid - 1] = ObjectIdGetDatum(tablespaceoid); values[Anum_pg_tablespace_spcname - 1] = diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index 99db5ba389..d2e7c0d7eb 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -30,6 +30,28 @@ do { \ } while (0) Datum +binary_upgrade_set_next_pg_database_oid(PG_FUNCTION_ARGS) +{ + Oid dboid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_database_oid = dboid; + + PG_RETURN_VOID(); +} + +Datum +binary_upgrade_set_next_pg_tablespace_oid(PG_FUNCTION_ARGS) +{ + Oid tbspoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_pg_tablespace_oid = tbspoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_type_oid(PG_FUNCTION_ARGS) { Oid typoid = PG_GETARG_OID(0); @@ -74,6 +96,17 @@ binary_upgrade_set_next_heap_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_heap_pg_class_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_heap_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -85,6 +118,17 @@ binary_upgrade_set_next_index_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_index_pg_class_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_index_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) { Oid reloid = PG_GETARG_OID(0); @@ -96,6 +140,17 @@ binary_upgrade_set_next_toast_pg_class_oid(PG_FUNCTION_ARGS) } Datum +binary_upgrade_set_next_toast_pg_class_relfilenode(PG_FUNCTION_ARGS) +{ + Oid nodeoid = PG_GETARG_OID(0); + + CHECK_IS_BINARY_UPGRADE; + binary_upgrade_next_toast_pg_class_relfilenode = nodeoid; + + PG_RETURN_VOID(); +} + +Datum binary_upgrade_set_next_pg_enum_oid(PG_FUNCTION_ARGS) { Oid enumoid = PG_GETARG_OID(0); diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 806fc78f04..912bfcb8f3 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -48,6 +48,7 @@ #include "catalog/pg_attribute_d.h" #include "catalog/pg_cast_d.h" #include "catalog/pg_class_d.h" +#include "catalog/pg_database.h" #include "catalog/pg_default_acl_d.h" #include "catalog/pg_largeobject_d.h" #include "catalog/pg_largeobject_metadata_d.h" @@ -280,7 +281,7 @@ static void binary_upgrade_set_type_oids_by_type_oid(Archive *fout, bool force_array_type); static bool binary_upgrade_set_type_oids_by_rel_oid(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_rel_oid); -static void binary_upgrade_set_pg_class_oids(Archive *fout, +static void binary_upgrade_set_pg_class_oids_and_relfilenodes(Archive *fout, PQExpBuffer upgrade_buffer, Oid pg_class_oid, bool is_index); static void binary_upgrade_extension_member(PQExpBuffer upgrade_buffer, @@ -2630,6 +2631,7 @@ dumpDatabase(Archive *fout) PQExpBuffer dbQry = createPQExpBuffer(); PQExpBuffer delQry = createPQExpBuffer(); PQExpBuffer creaQry = createPQExpBuffer(); + PQExpBuffer setDBIdQry = createPQExpBuffer(); PQExpBuffer labelq = createPQExpBuffer(); PGconn *conn = GetConnection(fout); PGresult *res; @@ -2800,6 +2802,37 @@ dumpDatabase(Archive *fout) qdatname = pg_strdup(fmtId(datname)); /* + * Make sure that pg_upgrade does not change database OID. Don't care + * about "postgres" database, backend will assign it fixed OID + * anyway. ("template1" has fixed OID too but the value 1 should not + * collide with any other OID so backend pays no attention to it.) + */ + if (dopt->binary_upgrade && dbCatId.oid != PostgresDbOid) + { + appendPQExpBufferStr(setDBIdQry, "\n-- For binary upgrade, must preserve pg_database oid\n"); + appendPQExpBuffer(setDBIdQry, + "SELECT pg_catalog.binary_upgrade_set_next_pg_database_oid('%u'::pg_catalog.oid);\n", + dbCatId.oid); + + /* + * Need a separate entry, otherwise the command will be run in the + * same transaction as the CREATE DATABASE command, which is not + * allowed. + */ + ArchiveEntry(fout, + dbCatId, /* catalog ID */ + dbDumpId, /* dump ID */ + ARCHIVE_OPTS(.tag = datname, + .owner = dba, + .description = "SET_DB_OID", + .section = SECTION_PRE_DATA, + .createStmt = setDBIdQry->data, + .dropStmt = NULL)); + + destroyPQExpBuffer(setDBIdQry); + } + + /* * Prepare the CREATE DATABASE command. We must specify encoding, locale, * and tablespace since those can't be altered later. Other DB properties * are left to the DATABASE PROPERTIES entry, so that they can be applied @@ -4391,36 +4424,57 @@ binary_upgrade_set_type_oids_by_rel_oid(Archive *fout, return toast_set; } +/* + * The function handles both oid and relfilenode since they are related to one + * another. It'd be inefficient if we ran two similar queries for each + * relation instead of just one. + */ static void -binary_upgrade_set_pg_class_oids(Archive *fout, - PQExpBuffer upgrade_buffer, Oid pg_class_oid, - bool is_index) +binary_upgrade_set_pg_class_oids_and_relfilenodes(Archive *fout, + PQExpBuffer upgrade_buffer, + Oid pg_class_oid, + bool is_index) { PQExpBuffer upgrade_query = createPQExpBuffer(); PGresult *upgrade_res; + Oid pg_class_relfilenode; Oid pg_class_reltoastrelid; + Oid pg_class_relfilenode_toast; Oid pg_index_indexrelid; + Oid pg_class_relfilenode_toast_idx; appendPQExpBuffer(upgrade_query, - "SELECT c.reltoastrelid, i.indexrelid " + "SELECT c.relfilenode, c.reltoastrelid, ct.relfilenode AS relfilenode_toast, i.indexrelid, cti.relfilenode AS relfilenode_toast_index " "FROM pg_catalog.pg_class c LEFT JOIN " "pg_catalog.pg_index i ON (c.reltoastrelid = i.indrelid AND i.indisvalid) " + "LEFT JOIN pg_catalog.pg_class ct ON (c.reltoastrelid = ct.oid) " + "LEFT JOIN pg_catalog.pg_class AS cti ON (i.indexrelid = cti.oid) " "WHERE c.oid = '%u'::pg_catalog.oid;", pg_class_oid); upgrade_res = ExecuteSqlQueryForSingleRow(fout, upgrade_query->data); + pg_class_relfilenode = atooid(PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relfilenode"))); pg_class_reltoastrelid = atooid(PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "reltoastrelid"))); + pg_class_relfilenode_toast = atooid(PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relfilenode_toast"))); pg_index_indexrelid = atooid(PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "indexrelid"))); + pg_class_relfilenode_toast_idx = atooid(PQgetvalue(upgrade_res, 0, PQfnumber(upgrade_res, "relfilenode_toast_index"))); appendPQExpBufferStr(upgrade_buffer, - "\n-- For binary upgrade, must preserve pg_class oids\n"); + "\n-- For binary upgrade, must preserve pg_class oids and relfilenodes\n"); if (!is_index) { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + + /* Not every relation has storage. */ + if (OidIsValid(pg_class_relfilenode)) + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_heap_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode); + /* only tables have toast tables, not indexes */ if (OidIsValid(pg_class_reltoastrelid)) { @@ -4436,17 +4490,28 @@ binary_upgrade_set_pg_class_oids(Archive *fout, appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_reltoastrelid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_toast_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode_toast); /* every toast table has an index */ appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_index_indexrelid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode_toast_idx); } } else + { appendPQExpBuffer(upgrade_buffer, "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_oid('%u'::pg_catalog.oid);\n", pg_class_oid); + appendPQExpBuffer(upgrade_buffer, + "SELECT pg_catalog.binary_upgrade_set_next_index_pg_class_relfilenode('%u'::pg_catalog.oid);\n", + pg_class_relfilenode); + } appendPQExpBufferChar(upgrade_buffer, '\n'); @@ -11008,7 +11073,7 @@ dumpCompositeType(Archive *fout, TypeInfo *tyinfo) binary_upgrade_set_type_oids_by_type_oid(fout, q, tyinfo->dobj.catId.oid, false); - binary_upgrade_set_pg_class_oids(fout, q, tyinfo->typrelid, false); + binary_upgrade_set_pg_class_oids_and_relfilenodes(fout, q, tyinfo->typrelid, false); } qtypname = pg_strdup(fmtId(tyinfo->dobj.name)); @@ -15507,7 +15572,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) appendPQExpBuffer(delq, "DROP VIEW %s;\n", qualrelname); if (dopt->binary_upgrade) - binary_upgrade_set_pg_class_oids(fout, q, + binary_upgrade_set_pg_class_oids_and_relfilenodes(fout, q, tbinfo->dobj.catId.oid, false); appendPQExpBuffer(q, "CREATE VIEW %s", qualrelname); @@ -15583,7 +15648,7 @@ dumpTableSchema(Archive *fout, TableInfo *tbinfo) appendPQExpBuffer(delq, "DROP %s %s;\n", reltypename, qualrelname); if (dopt->binary_upgrade) - binary_upgrade_set_pg_class_oids(fout, q, + binary_upgrade_set_pg_class_oids_and_relfilenodes(fout, q, tbinfo->dobj.catId.oid, false); appendPQExpBuffer(q, "CREATE %s%s %s", @@ -16337,7 +16402,7 @@ dumpIndex(Archive *fout, IndxInfo *indxinfo) int nstatvals; if (dopt->binary_upgrade) - binary_upgrade_set_pg_class_oids(fout, q, + binary_upgrade_set_pg_class_oids_and_relfilenodes(fout, q, indxinfo->dobj.catId.oid, true); /* Plain secondary index */ @@ -16559,7 +16624,7 @@ dumpConstraint(Archive *fout, ConstraintInfo *coninfo) coninfo->dobj.name); if (dopt->binary_upgrade) - binary_upgrade_set_pg_class_oids(fout, q, + binary_upgrade_set_pg_class_oids_and_relfilenodes(fout, q, indxinfo->dobj.catId.oid, true); appendPQExpBuffer(q, "ALTER TABLE ONLY %s\n", @@ -16971,12 +17036,13 @@ dumpSequence(Archive *fout, TableInfo *tbinfo) if (dopt->binary_upgrade) { - binary_upgrade_set_pg_class_oids(fout, query, + binary_upgrade_set_pg_class_oids_and_relfilenodes(fout, query, tbinfo->dobj.catId.oid, false); binary_upgrade_set_type_oids_by_rel_oid(fout, query, tbinfo->dobj.catId.oid); } + if (tbinfo->is_identity_sequence) { TableInfo *owning_tab = findTableByOid(tbinfo->owning_tab); diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c index 158c0c74b2..d072f28956 100644 --- a/src/bin/pg_dump/pg_dumpall.c +++ b/src/bin/pg_dump/pg_dumpall.c @@ -1259,6 +1259,9 @@ dumpTablespaces(PGconn *conn) /* needed for buildACLCommands() */ fspcname = pg_strdup(fmtId(spcname)); + appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n"); + appendPQExpBuffer(buf, "SELECT pg_catalog.binary_upgrade_set_next_pg_tablespace_oid('%u'::pg_catalog.oid);\n", spcoid); + appendPQExpBuffer(buf, "CREATE TABLESPACE %s", fspcname); appendPQExpBuffer(buf, " OWNER %s", fmtId(spcowner)); diff --git a/src/bin/pg_upgrade/Makefile b/src/bin/pg_upgrade/Makefile index 2cb829acd5..0eaf869119 100644 --- a/src/bin/pg_upgrade/Makefile +++ b/src/bin/pg_upgrade/Makefile @@ -11,7 +11,7 @@ OBJS = check.o controldata.o dump.o exec.o file.o function.o info.o \ option.o parallel.o pg_upgrade.o relfilenode.o server.o \ tablespace.o util.o version.o $(WIN32RES) -override CPPFLAGS := -DDLSUFFIX=\"$(DLSUFFIX)\" -I$(srcdir) -I$(libpq_srcdir) $(CPPFLAGS) +override CPPFLAGS := -DDLSUFFIX=\"$(DLSUFFIX)\" -I$(srcdir) -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS) LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) all: pg_upgrade diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index ff78929707..336df7378f 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -15,12 +15,14 @@ * oids are the same between old and new clusters. This is important * because toast oids are stored as toast pointers in user tables. * - * While pg_class.oid and pg_class.relfilenode are initially the same - * in a cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM - * FULL. In the new cluster, pg_class.oid and pg_class.relfilenode will - * be the same and will match the old pg_class.oid value. Because of - * this, old/new pg_class.relfilenode values will not match if CLUSTER, - * REINDEX, or VACUUM FULL have been performed in the old cluster. + * While pg_class.oid and pg_class.relfilenode are initially the same in a + * cluster, they can diverge due to CLUSTER, REINDEX, or VACUUM FULL. We + * control assignments of pg_class.relfilenode because relfilenode is used to + * derive initialization vector for data encryption. If relfilenode changed + * during upgrade, we'd be unable to decrypt the data in the new cluster. + * + * pg_database.oid and pg_tablespace.oid are also preserved because they + * participate in the encryption initialization vector. * * We control all assignments of pg_type.oid because these oids are stored * in user composite type values. diff --git a/src/include/catalog/binary_upgrade.h b/src/include/catalog/binary_upgrade.h index 2927b7a4d3..cffafbc219 100644 --- a/src/include/catalog/binary_upgrade.h +++ b/src/include/catalog/binary_upgrade.h @@ -14,14 +14,21 @@ #ifndef BINARY_UPGRADE_H #define BINARY_UPGRADE_H +extern PGDLLIMPORT Oid binary_upgrade_next_pg_database_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_pg_tablespace_oid; + extern PGDLLIMPORT Oid binary_upgrade_next_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_array_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_type_oid; extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_index_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_oid; +extern PGDLLIMPORT Oid binary_upgrade_next_toast_pg_class_relfilenode; +extern PGDLLIMPORT Oid binary_upgrade_next_heap_pg_class_relfilenode; extern PGDLLIMPORT Oid binary_upgrade_next_pg_enum_oid; extern PGDLLIMPORT Oid binary_upgrade_next_pg_authid_oid; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 29f0944774..bcb78168d3 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10080,6 +10080,26 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '4142', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_heap_pg_class_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_heap_pg_class_relfilenode' }, +{ oid => '4191', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_index_pg_class_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_index_pg_class_relfilenode' }, +{ oid => '4192', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_toast_pg_class_relfilenode', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_toast_pg_class_relfilenode' }, +{ oid => '4193', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_database_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_database_oid' }, +{ oid => '4194', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_next_pg_tablespace_oid', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid', + prosrc => 'binary_upgrade_set_next_pg_tablespace_oid' } # conversion functions { oid => '4300', -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0015-Make-pg_upgrade-aware-of-encryption.patch From 33b4699fd753965689e0d0d63881ea689b68c2f5 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:02 +0200 Subject: [PATCH 15/17] Make pg_upgrade aware of encryption. This includes pg_resetwal utility becaue pg_upgrade calls it. --- src/bin/pg_resetwal/Makefile | 10 +++++-- src/bin/pg_resetwal/pg_resetwal.c | 51 +++++++++++++++++++++++++++++++++-- src/bin/pg_upgrade/controldata.c | 39 +++++++++++++++++++++++++++ src/bin/pg_upgrade/dump.c | 2 +- src/bin/pg_upgrade/exec.c | 30 +++++++++++++++++++-- src/bin/pg_upgrade/option.c | 10 ++++++- src/bin/pg_upgrade/parallel.c | 7 ++--- src/bin/pg_upgrade/pg_upgrade.c | 57 +++++++++++++++++++++++++++++++++------ src/bin/pg_upgrade/pg_upgrade.h | 9 ++++++- src/bin/pg_upgrade/server.c | 39 +++++++++++++++++++++++++-- src/fe_utils/encryption.c | 25 ++++++++++------- src/include/fe_utils/encryption.h | 1 + 12 files changed, 249 insertions(+), 31 deletions(-) diff --git a/src/bin/pg_resetwal/Makefile b/src/bin/pg_resetwal/Makefile index 2a3835691f..faebfda27e 100644 --- a/src/bin/pg_resetwal/Makefile +++ b/src/bin/pg_resetwal/Makefile @@ -15,13 +15,19 @@ subdir = src/bin/pg_resetwal top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -OBJS= pg_resetwal.o $(WIN32RES) +override CPPFLAGS := -DFRONTEND $(CPPFLAGS) +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) + +OBJS= pg_resetwal.o encryption.o $(WIN32RES) all: pg_resetwal pg_resetwal: $(OBJS) | submake-libpgport $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) +encryption.c: % : $(top_srcdir)/src/backend/storage/file/% + rm -f $@ && $(LN_S) $< . + install: all installdirs $(INSTALL_PROGRAM) pg_resetwal$(X) '$(DESTDIR)$(bindir)/pg_resetwal$(X)' @@ -32,7 +38,7 @@ uninstall: rm -f '$(DESTDIR)$(bindir)/pg_resetwal$(X)' clean distclean maintainer-clean: - rm -f pg_resetwal$(X) $(OBJS) + rm -f pg_resetwal$(X) $(OBJS) encryption.c rm -rf tmp_check check: diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index ff0f8ea5e7..43921132d4 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -54,11 +54,12 @@ #include "common/file_perm.h" #include "common/logging.h" #include "common/restricted_token.h" +#include "fe_utils/encryption.h" +#include "storage/encryption.h" #include "storage/large_object.h" #include "pg_getopt.h" #include "getopt_long.h" - static ControlFileData ControlFile; /* pg_control values */ static XLogSegNo newXlogSegNo; /* new XLOG segment # */ static bool guessed = false; /* T if we had to guess at any values */ @@ -135,7 +136,7 @@ main(int argc, char *argv[]) } - while ((c = getopt_long(argc, argv, "c:D:e:fl:m:no:O:x:", long_options, NULL)) != -1) + while ((c = getopt_long(argc, argv, "c:D:e:fK:l:m:no:O:x:", long_options, NULL)) != -1) { switch (c) { @@ -277,6 +278,12 @@ main(int argc, char *argv[]) } break; +#ifdef USE_ENCRYPTION + case 'K': + encryption_key_command = strdup(optarg); + break; +#endif /* USE_ENCRYPTION */ + case 'l': if (strspn(optarg, "01234567890ABCDEFabcdef") != XLOG_FNAME_LEN) { @@ -408,6 +415,28 @@ main(int argc, char *argv[]) XLogFromFileName(log_fname, &minXlogTli, &minXlogSegNo, WalSegSz); /* + * If the data is encrypted, we also might need to encrypt the XLOG record + * below. + */ + if (ControlFile.data_cipher > PG_CIPHER_NONE && !noupdate) + { + if (encryption_key_command) + run_encryption_key_command(encryption_key, DataDir); + else + { + /* + * If executed by pg_upgrade, we don't want pg_resetwal to run the + * encryption key command (possibly interactive application) w/o + * access to terminal. + */ + read_encryption_key_fe(stdin); + } + + setup_encryption(); + data_encrypted = true; + } + + /* * Also look at existing segment files to set up newXlogSegNo */ FindEndOfXLOG(); @@ -814,6 +843,12 @@ PrintControlValues(bool guessed) (ControlFile.float8ByVal ? _("by value") : _("by reference"))); printf(_("Data page checksum version: %u\n"), ControlFile.data_checksum_version); + if (ControlFile.data_cipher > PG_CIPHER_NONE) + printf(_("Data encryption fingerprint: %08X%08X%08X%08X\n"), + htonl(((uint32 *) ControlFile.encryption_verification)[0]), + htonl(((uint32 *) ControlFile.encryption_verification)[1]), + htonl(((uint32 *) ControlFile.encryption_verification)[2]), + htonl(((uint32 *) ControlFile.encryption_verification)[3])); } @@ -1164,6 +1199,14 @@ WriteEmptyXLOG(void) FIN_CRC32C(crc); record->xl_crc = crc; + if (data_encrypted) + { + char tweak[TWEAK_SIZE]; + + XLogEncryptionTweak(tweak, page->xlp_tli, newXlogSegNo, 0); + encrypt_block(buffer.data, buffer.data, XLOG_BLCKSZ, tweak, true); + } + /* Write the first page */ XLogFilePath(path, ControlFile.checkPointCopy.ThisTimeLineID, newXlogSegNo, WalSegSz); @@ -1224,6 +1267,10 @@ usage(void) printf(_(" [-D, --pgdata=]DATADIR data directory\n")); printf(_(" -e, --epoch=XIDEPOCH set next transaction ID epoch\n")); printf(_(" -f, --force force update to be done\n")); +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command\n" + " command that returns encryption key\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -l, --next-wal-file=WALFILE set minimum starting location for new WAL\n")); printf(_(" -m, --multixact-ids=MXID,MXID set next and oldest multitransaction ID\n")); printf(_(" -n, --dry-run no update, just show what would be done\n")); diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c index 38236415be..dc15b979bc 100644 --- a/src/bin/pg_upgrade/controldata.c +++ b/src/bin/pg_upgrade/controldata.c @@ -485,6 +485,33 @@ get_control_data(ClusterInfo *cluster, bool live_check) cluster->controldata.data_checksum_version = str2uint(p); got_data_checksum_version = true; } + else if ((p = strstr(bufin, "encryption fingerprint")) != NULL) + { + int i; + + p = strchr(p, ':'); + + if (p == NULL || strlen(p) <= 1) + pg_fatal("%d: controldata retrieval problem\n", __LINE__); + + cluster->controldata.data_encrypted = true; + + /* Skip the colon and any whitespace after it */ + p = strchr(p, ':'); + if (p == NULL || strlen(p) <= 1) + pg_fatal("%d: controldata retrieval problem\n", __LINE__); + p = strpbrk(p, "01234567890ABCDEF"); + if (p == NULL || strlen(p) <= 1) + pg_fatal("%d: controldata retrieval problem\n", __LINE__); + + /* Make sure it looks like a valid finerprint */ + if (strspn(p, "0123456789ABCDEF") != 32) + pg_fatal("%d: controldata retrieval problem\n", __LINE__); + + for (i = 0; i < ENCRYPTION_SAMPLE_SIZE; i++) + sscanf(p + 2 * i, "%2hhx", + cluster->controldata.encryption_verification + i); + } } pclose(output); @@ -669,6 +696,18 @@ check_control_data(ControlData *oldctrl, pg_fatal("old cluster uses data checksums but the new one does not\n"); else if (oldctrl->data_checksum_version != newctrl->data_checksum_version) pg_fatal("old and new cluster pg_controldata checksum versions do not match\n"); + + if (oldctrl->data_encrypted && !newctrl->data_encrypted) + pg_fatal("old cluster is encrypted, but the new one is not\n"); + else if (!oldctrl->data_encrypted && newctrl->data_encrypted) + pg_fatal("old cluster is not encrypted, but the new one is\n"); + else if (oldctrl->data_encrypted) + { + if (memcmp(oldctrl->encryption_verification, + newctrl->encryption_verification, + ENCRYPTION_SAMPLE_SIZE) != 0) + pg_fatal("encryption of the new cluster is not compatible with encryption of the old one\n"); + } } diff --git a/src/bin/pg_upgrade/dump.c b/src/bin/pg_upgrade/dump.c index c1429fe4bf..6fcf9126a4 100644 --- a/src/bin/pg_upgrade/dump.c +++ b/src/bin/pg_upgrade/dump.c @@ -22,7 +22,7 @@ generate_old_dump(void) prep_status("Creating dump of global objects"); /* run new pg_dumpall binary for globals */ - exec_prog(UTILITY_LOG_FILE, NULL, true, true, + exec_prog(UTILITY_LOG_FILE, NULL, true, true, NULL, "\"%s/pg_dumpall\" %s --globals-only --quote-all-identifiers " "--binary-upgrade %s -f %s", new_cluster.bindir, cluster_conn_opts(&old_cluster), diff --git a/src/bin/pg_upgrade/exec.c b/src/bin/pg_upgrade/exec.c index 0363309328..cd6c68bdd9 100644 --- a/src/bin/pg_upgrade/exec.c +++ b/src/bin/pg_upgrade/exec.c @@ -74,11 +74,17 @@ get_bin_version(ClusterInfo *cluster) * If the command fails, an error message is optionally written to the specified * log_file, and the program optionally exits. * + * If encryption_key is passed, popen() is used and the key is sent to stdin + * of the command. + * * The code requires it be called first from the primary thread on Windows. + * + * TODO Consolidate the use of popen() with Windows. Or use only popen()? */ bool exec_prog(const char *log_file, const char *opt_log_file, - bool report_error, bool exit_on_error, const char *fmt,...) + bool report_error, bool exit_on_error, unsigned char *encryption_key, + const char *fmt,...) { int result = 0; int written; @@ -170,7 +176,27 @@ exec_prog(const char *log_file, const char *opt_log_file, /* see comment above */ if (mainThreadId == GetCurrentThreadId()) #endif - result = system(cmd); + { + if (!encryption_key) + result = system(cmd); + else + { + FILE *fp; + int i; + + fp = popen(cmd, "w"); + if (fp == NULL) + pg_fatal("Failed to execute \"%s\"\n", cmd); + + /* Send the key. */ + for (i = 0; i < ENCRYPTION_KEY_LENGTH; i++) + fprintf(fp, "%.2x", encryption_key[i]); + fputc('\n', fp); + + if (pclose(fp)) + pg_fatal("\"%s\" returned non-zero code\n", cmd); + } + } if (result != 0 && report_error) { diff --git a/src/bin/pg_upgrade/option.c b/src/bin/pg_upgrade/option.c index 73f395f2a3..e2bb0e7c8e 100644 --- a/src/bin/pg_upgrade/option.c +++ b/src/bin/pg_upgrade/option.c @@ -56,6 +56,7 @@ parseCommandLine(int argc, char *argv[]) {"socketdir", required_argument, NULL, 's'}, {"verbose", no_argument, NULL, 'v'}, {"clone", no_argument, NULL, 1}, + {"encryption-key-command", required_argument, NULL, 'K'}, {NULL, 0, NULL, 0} }; @@ -101,7 +102,7 @@ parseCommandLine(int argc, char *argv[]) if (os_user_effective_id == 0) pg_fatal("%s: cannot be run as root\n", os_info.progname); - while ((option = getopt_long(argc, argv, "d:D:b:B:cj:ko:O:p:P:rs:U:v", + while ((option = getopt_long(argc, argv, "d:D:b:B:cj:K:ko:O:p:P:rs:U:v", long_options, &optindex)) != -1) { switch (option) @@ -209,6 +210,10 @@ parseCommandLine(int argc, char *argv[]) user_opts.transfer_mode = TRANSFER_MODE_CLONE; break; + case 'K': + encryption_key_command = pg_strdup(optarg); + break; + default: fprintf(stderr, _("Try \"%s --help\" for more information.\n"), os_info.progname); @@ -309,6 +314,9 @@ usage(void) printf(_(" -v, --verbose enable verbose internal logging\n")); printf(_(" -V, --version display version information, then exit\n")); printf(_(" --clone clone instead of copying files to new cluster\n")); +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command command that returns encryption key\n\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -?, --help show this help, then exit\n")); printf(_("\n" "Before running pg_upgrade you must:\n" diff --git a/src/bin/pg_upgrade/parallel.c b/src/bin/pg_upgrade/parallel.c index 80ab1b8609..be3fd51f15 100644 --- a/src/bin/pg_upgrade/parallel.c +++ b/src/bin/pg_upgrade/parallel.c @@ -79,7 +79,7 @@ parallel_exec_prog(const char *log_file, const char *opt_log_file, if (user_opts.jobs <= 1) /* exit_on_error must be true to allow jobs */ - exec_prog(log_file, opt_log_file, true, true, "%s", cmd); + exec_prog(log_file, opt_log_file, true, true, NULL, "%s", cmd); else { /* parallel */ @@ -122,7 +122,8 @@ parallel_exec_prog(const char *log_file, const char *opt_log_file, child = fork(); if (child == 0) /* use _exit to skip atexit() functions */ - _exit(!exec_prog(log_file, opt_log_file, true, true, "%s", cmd)); + _exit(!exec_prog(log_file, opt_log_file, true, true, NULL, "%s", + cmd)); else if (child < 0) /* fork failed */ pg_fatal("could not create worker process: %s\n", strerror(errno)); @@ -160,7 +161,7 @@ win32_exec_prog(exec_thread_arg *args) { int ret; - ret = !exec_prog(args->log_file, args->opt_log_file, true, true, "%s", args->cmd); + ret = !exec_prog(args->log_file, args->opt_log_file, true, true, NULL, "%s", args->cmd); /* terminates thread */ return ret; diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index 336df7378f..c464f90cf9 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -73,6 +73,12 @@ char *output_files[] = { NULL }; +/* + * Declare these locally so we don't have to link storage/file/encryption.c + * here. + */ +bool encryption_setup_done = false; +unsigned char encryption_key[ENCRYPTION_KEY_LENGTH]; int main(int argc, char **argv) @@ -94,6 +100,20 @@ main(int argc, char **argv) adjust_data_dir(&old_cluster); adjust_data_dir(&new_cluster); + /* + * The encryption key is needed to start the clusters. + */ + if (encryption_key_command) + { + /* + * Both clusters should have the same KDF parameters, so we can pass + * pgdata of any one. + */ + run_encryption_key_command(encryption_key, old_cluster.pgdata); + + encryption_setup_done = true; + } + setup(argv[0], &live_check); output_check_banner(live_check); @@ -139,6 +159,16 @@ main(int argc, char **argv) copy_xact_xlog_xid(); + if (encryption_setup_done) + { + /* + * Copy KDF file so that the old cluster encryption password works for + * the new cluster. + */ + read_kdf_file(old_cluster.pgdata); + write_kdf_file(new_cluster.pgdata); + } + /* New now using xids of the old system */ /* -- NEW -- */ @@ -170,13 +200,15 @@ main(int argc, char **argv) */ prep_status("Setting next OID for new cluster"); exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, "\"%s/pg_resetwal\" -o %u \"%s\"", - new_cluster.bindir, old_cluster.controldata.chkpnt_nxtoid, + new_cluster.bindir, + old_cluster.controldata.chkpnt_nxtoid, new_cluster.pgdata); check_ok(); prep_status("Sync data directory to disk"); - exec_prog(UTILITY_LOG_FILE, NULL, true, true, + exec_prog(UTILITY_LOG_FILE, NULL, true, true, NULL, "\"%s/initdb\" --sync-only \"%s\"", new_cluster.bindir, new_cluster.pgdata); check_ok(); @@ -270,7 +302,7 @@ prepare_new_cluster(void) * --analyze so autovacuum doesn't update statistics later */ prep_status("Analyzing all rows in the new cluster"); - exec_prog(UTILITY_LOG_FILE, NULL, true, true, + exec_prog(UTILITY_LOG_FILE, NULL, true, true, NULL, "\"%s/vacuumdb\" %s --all --analyze %s", new_cluster.bindir, cluster_conn_opts(&new_cluster), log_opts.verbose ? "--verbose" : ""); @@ -283,7 +315,7 @@ prepare_new_cluster(void) * counter later. */ prep_status("Freezing all rows in the new cluster"); - exec_prog(UTILITY_LOG_FILE, NULL, true, true, + exec_prog(UTILITY_LOG_FILE, NULL, true, true, NULL, "\"%s/vacuumdb\" %s --all --freeze %s", new_cluster.bindir, cluster_conn_opts(&new_cluster), log_opts.verbose ? "--verbose" : ""); @@ -304,7 +336,7 @@ prepare_new_globals(void) */ prep_status("Restoring global objects in the new cluster"); - exec_prog(UTILITY_LOG_FILE, NULL, true, true, + exec_prog(UTILITY_LOG_FILE, NULL, true, true, NULL, "\"%s/psql\" " EXEC_PSQL_ARGS " %s -f \"%s\"", new_cluster.bindir, cluster_conn_opts(&new_cluster), GLOBALS_DUMP_FILE); @@ -351,6 +383,7 @@ create_new_objects(void) NULL, true, true, + NULL, "\"%s/pg_restore\" %s %s --exit-on-error --verbose " "--dbname postgres \"%s\"", new_cluster.bindir, @@ -448,7 +481,7 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir) prep_status("Copying old %s to new server", old_subdir); - exec_prog(UTILITY_LOG_FILE, NULL, true, true, + exec_prog(UTILITY_LOG_FILE, NULL, true, true, NULL, #ifndef WIN32 "cp -Rf \"%s\" \"%s\"", #else @@ -475,15 +508,20 @@ copy_xact_xlog_xid(void) /* set the next transaction id and epoch of the new cluster */ prep_status("Setting next transaction ID and epoch for new cluster"); exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, "\"%s/pg_resetwal\" -f -x %u \"%s\"", - new_cluster.bindir, old_cluster.controldata.chkpnt_nxtxid, + new_cluster.bindir, + old_cluster.controldata.chkpnt_nxtxid, new_cluster.pgdata); exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, "\"%s/pg_resetwal\" -f -e %u \"%s\"", - new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch, + new_cluster.bindir, + old_cluster.controldata.chkpnt_nxtepoch, new_cluster.pgdata); /* must reset commit timestamp limits also */ exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, "\"%s/pg_resetwal\" -f -c %u,%u \"%s\"", new_cluster.bindir, old_cluster.controldata.chkpnt_nxtxid, @@ -510,6 +548,7 @@ copy_xact_xlog_xid(void) * counters here and the oldest multi present on system. */ exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, "\"%s/pg_resetwal\" -O %u -m %u,%u \"%s\"", new_cluster.bindir, old_cluster.controldata.chkpnt_nxtmxoff, @@ -538,6 +577,7 @@ copy_xact_xlog_xid(void) * next=MaxMultiXactId, but multixact.c can cope with that just fine. */ exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, "\"%s/pg_resetwal\" -m %u,%u \"%s\"", new_cluster.bindir, old_cluster.controldata.chkpnt_nxtmulti + 1, @@ -549,6 +589,7 @@ copy_xact_xlog_xid(void) /* now reset the wal archives in the new cluster */ prep_status("Resetting WAL archives"); exec_prog(UTILITY_LOG_FILE, NULL, true, true, + encryption_setup_done ? encryption_key : NULL, /* use timeline 1 to match controldata and no WAL history file */ "\"%s/pg_resetwal\" -l 00000001%s \"%s\"", new_cluster.bindir, old_cluster.controldata.nextxlogfile + 8, diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index 5d31750d86..e89f6c08a0 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -10,7 +10,10 @@ #include #include +#include "fe_utils/encryption.h" #include "libpq-fe.h" +#include "storage/relfilenode.h" +#include "storage/encryption.h" /* Use port in the private/dynamic port number range */ #define DEF_PGUPORT 50432 @@ -227,6 +230,8 @@ typedef struct bool date_is_int; bool float8_pass_by_value; bool data_checksum_version; + bool data_encrypted; + uint8 encryption_verification[ENCRYPTION_SAMPLE_SIZE]; } ControlData; /* @@ -367,7 +372,9 @@ void generate_old_dump(void); #define EXEC_PSQL_ARGS "--echo-queries --set ON_ERROR_STOP=on --no-psqlrc --dbname=template1" bool exec_prog(const char *log_file, const char *opt_log_file, - bool report_error, bool exit_on_error, const char *fmt,...) pg_attribute_printf(5, 6); + bool report_error, bool exit_on_error, + unsigned char *encryption_key, const char *fmt,...) + pg_attribute_printf(6, 7); void verify_directories(void); bool pid_lock_file_exists(const char *datadir); diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c index 6ad4b14e16..93cb518cc5 100644 --- a/src/bin/pg_upgrade/server.c +++ b/src/bin/pg_upgrade/server.c @@ -252,6 +252,41 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error) cluster->pgopts ? cluster->pgopts : "", socket_string); /* + * If encryption key needs to be sent, run a separate process now and let + * it send the password to the postmaster. We cannot send the key later + * in the current process because the exec_prog call below blocks until + * the postmaster succeeds or fails to start (and it will definitely fail + * if it receives no key). + */ + if (encryption_setup_done) + { +#ifndef WIN32 + pid_t sender; + + sender = fork(); + if (sender == 0) + { + char port_str[6]; + + snprintf(port_str, sizeof(port_str), "%d", cluster->port); + + /* in child process */ + send_key_to_postmaster(cluster->sockdir, port_str, + encryption_key); + exit(EXIT_SUCCESS); + } + else if (sender < 0) + { + pg_fatal("could not create key sender process"); + exit(EXIT_FAILURE); + } +#else + /* TODO */ + #error "W32 not implemented yet" +#endif + } + + /* * Don't throw an error right away, let connecting throw the error because * it might supply a reason for the failure. */ @@ -260,7 +295,7 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error) (strcmp(SERVER_LOG_FILE, SERVER_START_LOG_FILE) != 0) ? SERVER_LOG_FILE : NULL, - report_and_exit_on_error, false, + report_and_exit_on_error, false, NULL, "%s", cmd); /* Did it fail and we are just testing if the server could be started? */ @@ -336,7 +371,7 @@ stop_postmaster(bool in_atexit) else return; /* no cluster running */ - exec_prog(SERVER_STOP_LOG_FILE, NULL, !in_atexit, !in_atexit, + exec_prog(SERVER_STOP_LOG_FILE, NULL, !in_atexit, !in_atexit, NULL, "\"%s/pg_ctl\" -w -D \"%s\" -o \"%s\" %s stop", cluster->bindir, cluster->pgconfig, cluster->pgopts ? cluster->pgopts : "", diff --git a/src/fe_utils/encryption.c b/src/fe_utils/encryption.c index 134b3bde9b..2b4a6fadad 100644 --- a/src/fe_utils/encryption.c +++ b/src/fe_utils/encryption.c @@ -259,8 +259,6 @@ run_encryption_key_command(unsigned char *encryption_key, char *data_dir) FILE *fp; char cmd[MAXPGPATH]; char *sp, *dp, *endp; - char *buf; - int read_len, i, c; Assert(encryption_key_command != NULL && strlen(encryption_key_command) > 0); @@ -312,15 +310,25 @@ run_encryption_key_command(unsigned char *encryption_key, char *data_dir) exit(EXIT_FAILURE); } + /* Read the key. */ + read_encryption_key_fe(fp); + + pclose(fp); +} + +/* + * Frontend counterpart of read_encryption_key(). + */ +void +read_encryption_key_fe(FILE *f) +{ + char *buf; + int read_len, i, c; + buf = (char *) palloc(ENCRYPTION_KEY_CHARS); - /* - * Read the key. This is very similar to backend's read_encryption_key() - * but there seems to be no straightforward way to call the function from - * here. - */ read_len = 0; - while ((c = fgetc(fp)) != EOF && c != '\n') + while ((c = fgetc(f)) != EOF && c != '\n') { if (read_len >= ENCRYPTION_KEY_CHARS) { @@ -349,7 +357,6 @@ run_encryption_key_command(unsigned char *encryption_key, char *data_dir) } pfree(buf); - pclose(fp); } /* diff --git a/src/include/fe_utils/encryption.h b/src/include/fe_utils/encryption.h index da302fe494..2c6032df07 100644 --- a/src/include/fe_utils/encryption.h +++ b/src/include/fe_utils/encryption.h @@ -22,5 +22,6 @@ extern void derive_key_from_password(unsigned char *encryption_key, const char *password, int len); extern void run_encryption_key_command(unsigned char *encryption_key, char *data_dir); +extern void read_encryption_key_fe(FILE *f); extern bool send_key_to_postmaster(const char *host, const char *port, const unsigned char *encryption_Key); -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0016-Teach-the-remaining-frontend-application-about-encry.patch From 26dc3cd48f082de47198e0975fbb7f1f971e2896 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:02 +0200 Subject: [PATCH 16/17] Teach the remaining frontend application about encryption. pg_checksums, pg_rewind, pg_waldump --- src/bin/pg_checksums/Makefile | 9 +- src/bin/pg_checksums/pg_checksums.c | 158 +++++++++++++++++++++++++++++++++--- src/bin/pg_rewind/Makefile | 9 +- src/bin/pg_rewind/parsexlog.c | 9 ++ src/bin/pg_rewind/pg_rewind.c | 57 ++++++++++++- src/bin/pg_waldump/Makefile | 9 +- src/bin/pg_waldump/pg_waldump.c | 90 +++++++++++++++++--- 7 files changed, 309 insertions(+), 32 deletions(-) diff --git a/src/bin/pg_checksums/Makefile b/src/bin/pg_checksums/Makefile index 278b7a0f2e..74e66ea0b7 100644 --- a/src/bin/pg_checksums/Makefile +++ b/src/bin/pg_checksums/Makefile @@ -15,13 +15,19 @@ subdir = src/bin/pg_checksums top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -OBJS= pg_checksums.o $(WIN32RES) +OBJS= encryption.o pg_checksums.o $(WIN32RES) + +override CPPFLAGS := -DFRONTEND $(CPPFLAGS) +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) all: pg_checksums pg_checksums: $(OBJS) | submake-libpgport $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) +encryption.c: % : $(top_srcdir)/src/backend/storage/file/% + rm -f $@ && $(LN_S) $< . + install: all installdirs $(INSTALL_PROGRAM) pg_checksums$(X) '$(DESTDIR)$(bindir)/pg_checksums$(X)' @@ -34,6 +40,7 @@ uninstall: clean distclean maintainer-clean: rm -f pg_checksums$(X) $(OBJS) rm -rf tmp_check + rm -f encryption.c check: $(prove_check) diff --git a/src/bin/pg_checksums/pg_checksums.c b/src/bin/pg_checksums/pg_checksums.c index b591fcc864..587132faf9 100644 --- a/src/bin/pg_checksums/pg_checksums.c +++ b/src/bin/pg_checksums/pg_checksums.c @@ -20,15 +20,18 @@ #include #include "access/xlog_internal.h" +#include "catalog/pg_tablespace_d.h" #include "common/controldata_utils.h" #include "common/file_perm.h" #include "common/file_utils.h" #include "common/logging.h" +#include "fe_utils/encryption.h" #include "getopt_long.h" #include "pg_getopt.h" #include "storage/bufpage.h" #include "storage/checksum.h" #include "storage/checksum_impl.h" +#include "storage/encryption.h" static int64 files = 0; @@ -80,6 +83,10 @@ usage(void) printf(_(" -c, --check check data checksums (default)\n")); printf(_(" -d, --disable disable data checksums\n")); printf(_(" -e, --enable enable data checksums\n")); +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command=COMMAND\n" + " command that returns encryption key\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -f, --filenode=FILENODE check only relation with specified filenode\n")); printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n")); printf(_(" -P, --progress show progress information\n")); @@ -167,7 +174,8 @@ skipfile(const char *fn) } static void -scan_file(const char *fn, BlockNumber segmentno) +scan_file(const char *fn, Oid relnode, BlockNumber segmentno, + ForkNumber forkno, Oid tbspace, Oid db) { PGAlignedBlock buf; PageHeader header = (PageHeader) buf.data; @@ -193,6 +201,7 @@ scan_file(const char *fn, BlockNumber segmentno) { uint16 csum; int r = read(f, buf.data, BLCKSZ); + char tweak[TWEAK_SIZE]; if (r == 0) break; @@ -204,6 +213,18 @@ scan_file(const char *fn, BlockNumber segmentno) } blocks++; + if (data_encrypted) + { + RelFileNode node; + + node.spcNode = tbspace; + node.dbNode = db; + node.relNode = relnode; + + mdtweak(tweak, &node, forkno, blockno); + decrypt_block(buf.data, buf.data, BLCKSZ, tweak, false); + } + /* New pages have no checksum yet */ if (PageIsNew(header)) continue; @@ -232,6 +253,9 @@ scan_file(const char *fn, BlockNumber segmentno) exit(1); } + if (data_encrypted) + encrypt_block(buf.data, buf.data, BLCKSZ, tweak, false); + /* Write block with checksum */ if (write(f, buf.data, BLCKSZ) != BLCKSZ) { @@ -262,9 +286,13 @@ scan_file(const char *fn, BlockNumber segmentno) * all the items which have checksums is computed and returned back * to the caller without operating on the files. This is used to compile * the total size of the data directory for progress reports. + * + * If db is a valid pointer, *db contains database OID. If it's NULL, the + * database OID needs to be recognized, possibly by recursive call. */ static int64 -scan_directory(const char *basedir, const char *subdir, bool sizeonly) +scan_directory(const char *basedir, const char *subdir, bool sizeonly, + Oid tbspace, Oid *db) { int64 dirsize = 0; char path[MAXPGPATH]; @@ -282,6 +310,9 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly) { char fn[MAXPGPATH]; struct stat st; + bool tbsp_identified = false; + Oid database; + bool db_identified = false; if (strcmp(de->d_name, ".") == 0 || strcmp(de->d_name, "..") == 0) @@ -305,12 +336,58 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly) pg_log_error("could not stat file \"%s\": %m", fn); exit(1); } + + if (tbspace != InvalidOid && db == NULL) + { + /* + * If tablespace is passed by caller or identified by upper call + * of this function, and if caller could not identify the database + * OID, try to do it now. + */ + if (strcmp(de->d_name, TABLESPACE_VERSION_DIRECTORY) == 0) + { + /* + * Major-version-specific tablespace subdirectory needs no + * special attention, we'll recurse into it below. + */ + } + else + { + errno = 0; + database = strtol(de->d_name, NULL, 10); + if (errno != 0 || database == InvalidOid) + { + fprintf(stderr, _("%s: invalid database oid \"%s\"\n"), + progname, de->d_name); + exit(1); + } + db_identified = true; + } + } + else if (tbspace == InvalidOid) + { + /* + * This entry should be a direct subdirectory of pg_tblspc, so the + * name is supposedly tablespace oid. + */ + errno = 0; + tbspace = strtol(de->d_name, NULL, 10); + if (errno != 0 || tbspace == InvalidOid) + { + fprintf(stderr, _("%s: invalid tablespace oid \"%s\"\n"), + progname, de->d_name); + exit(1); + } + tbsp_identified = true; + } + if (S_ISREG(st.st_mode)) { char fnonly[MAXPGPATH]; char *forkpath, *segmentpath; BlockNumber segmentno = 0; + ForkNumber forkno; if (skipfile(de->d_name)) continue; @@ -337,10 +414,16 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly) forkpath = strchr(fnonly, '_'); if (forkpath != NULL) + { *forkpath++ = '\0'; + forkno = forkname_to_number(forkpath); + } + else + forkno = MAIN_FORKNUM; + if (only_filenode && strcmp(only_filenode, fnonly) != 0) - /* filenode not to be included */ + /* Relfilenode not to be included */ continue; dirsize += st.st_size; @@ -350,14 +433,31 @@ scan_directory(const char *basedir, const char *subdir, bool sizeonly) * the items in the data folder. */ if (!sizeonly) - scan_file(fn, segmentno); + { + Oid relnode = atoi(fnonly); + + scan_file(fn, relnode, segmentno, forkno, tbspace, *db); + } } #ifndef WIN32 else if (S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)) #else else if (S_ISDIR(st.st_mode) || pgwin32_is_junction(fn)) #endif - dirsize += scan_directory(path, de->d_name, sizeonly); + + /* + * If database OID is not passed by caller, use the one we + * identified ourselves. + */ + dirsize = scan_directory(path, de->d_name, sizeonly, tbspace, + db_identified ? &database : db); + + /* + * If tablespace is not passed by caller, forget what we found out so + * that next directory entry is examined in the same way. + */ + if (tbsp_identified) + tbspace = InvalidOid; } closedir(dir); return dirsize; @@ -371,6 +471,9 @@ main(int argc, char *argv[]) {"pgdata", required_argument, NULL, 'D'}, {"disable", no_argument, NULL, 'd'}, {"enable", no_argument, NULL, 'e'}, +#ifdef USE_ENCRYPTION + {"encryption-key-command", required_argument, NULL, 'K'}, +#endif /* USE_ENCRYPTION */ {"filenode", required_argument, NULL, 'f'}, {"no-sync", no_argument, NULL, 'N'}, {"progress", no_argument, NULL, 'P'}, @@ -401,7 +504,7 @@ main(int argc, char *argv[]) } } - while ((c = getopt_long(argc, argv, "cD:deNPf:v", long_options, &option_index)) != -1) + while ((c = getopt_long(argc, argv, "cD:deNK:Pf:v", long_options, &option_index)) != -1) { switch (c) { @@ -422,6 +525,12 @@ main(int argc, char *argv[]) } only_filenode = pstrdup(optarg); break; +#ifdef USE_ENCRYPTION + case 'K': + encryption_key_command = pg_strdup(optarg); + data_encrypted = true; + break; +#endif /* USE_ENCRYPTION */ case 'N': do_sync = false; break; @@ -483,6 +592,24 @@ main(int argc, char *argv[]) exit(1); } + if (ControlFile->data_cipher > PG_CIPHER_NONE) + { + if (encryption_key_command == NULL) + { +#ifdef USE_ENCRYPTION + fprintf(stderr, _("%s: please specify command to retrieve encryption key\n"), + progname); +#else + fprintf(stderr, _("%s: compile postgres with --with-openssl to use encryption\n"), + progname); +#endif /* USE_ENCRYPTION */ + exit(1); + } + + run_encryption_key_command(encryption_key, DataDir); + setup_encryption(); + } + if (ControlFile->pg_control_version != PG_CONTROL_VERSION) { pg_log_error("cluster is not compatible with this version of pg_checksums"); @@ -533,6 +660,8 @@ main(int argc, char *argv[]) /* Operate on all files if checking or enabling checksums */ if (mode == PG_MODE_CHECK || mode == PG_MODE_ENABLE) { + Oid db_shared = 0; + /* * If progress status information is requested, we need to scan the * directory tree twice: once to know how much total data needs to be @@ -540,14 +669,19 @@ main(int argc, char *argv[]) */ if (showprogress) { - total_size = scan_directory(DataDir, "global", true); - total_size += scan_directory(DataDir, "base", true); - total_size += scan_directory(DataDir, "pg_tblspc", true); + total_size = scan_directory(DataDir, "global", true, + GLOBALTABLESPACE_OID, &db_shared); + total_size += scan_directory(DataDir, "base", true, + DEFAULTTABLESPACE_OID, NULL); + total_size += scan_directory(DataDir, "pg_tblspc", true, + InvalidOid, NULL); } - (void) scan_directory(DataDir, "global", false); - (void) scan_directory(DataDir, "base", false); - (void) scan_directory(DataDir, "pg_tblspc", false); + (void) scan_directory(DataDir, "global", false, GLOBALTABLESPACE_OID, + &db_shared); + (void) scan_directory(DataDir, "base", false, DEFAULTTABLESPACE_OID, + NULL); + (void) scan_directory(DataDir, "pg_tblspc", false, InvalidOid, NULL); if (showprogress) { diff --git a/src/bin/pg_rewind/Makefile b/src/bin/pg_rewind/Makefile index 859d3abc41..e41d2035f4 100644 --- a/src/bin/pg_rewind/Makefile +++ b/src/bin/pg_rewind/Makefile @@ -16,10 +16,10 @@ top_builddir = ../../.. include $(top_builddir)/src/Makefile.global override CPPFLAGS := -I$(libpq_srcdir) -DFRONTEND $(CPPFLAGS) -LDFLAGS_INTERNAL += $(libpq_pgport) +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) OBJS = pg_rewind.o parsexlog.o xlogreader.o datapagemap.o timeline.o \ - fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o \ + fetch.o file_ops.o copy_fetch.o libpq_fetch.o filemap.o encryption.o\ $(WIN32RES) EXTRA_CLEAN = xlogreader.c @@ -32,6 +32,9 @@ pg_rewind: $(OBJS) | submake-libpq submake-libpgport xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/% rm -f $@ && $(LN_S) $< . +encryption.c: % : $(top_srcdir)/src/backend/storage/file/% + rm -f $@ && $(LN_S) $< . + install: all installdirs $(INSTALL_PROGRAM) pg_rewind$(X) '$(DESTDIR)$(bindir)/pg_rewind$(X)' @@ -42,7 +45,7 @@ uninstall: rm -f '$(DESTDIR)$(bindir)/pg_rewind$(X)' clean distclean maintainer-clean: - rm -f pg_rewind$(X) $(OBJS) xlogreader.c + rm -f pg_rewind$(X) $(OBJS) xlogreader.c encryption.c rm -rf tmp_check check: diff --git a/src/bin/pg_rewind/parsexlog.c b/src/bin/pg_rewind/parsexlog.c index 287af60c4e..0624c4c5ef 100644 --- a/src/bin/pg_rewind/parsexlog.c +++ b/src/bin/pg_rewind/parsexlog.c @@ -320,6 +320,15 @@ SimpleXLogPageRead(XLogReaderState *xlogreader, XLogRecPtr targetPagePtr, return -1; } + if (data_encrypted) + { + char tweak[TWEAK_SIZE]; + + XLogEncryptionTweak(tweak, targetHistory[private->tliIndex].tli, + xlogreadsegno, targetPageOff); + decrypt_block(readBuf, readBuf, XLOG_BLCKSZ, tweak, true); + } + Assert(targetSegNo == xlogreadsegno); *pageTLI = targetHistory[private->tliIndex].tli; diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index d378053de4..58c192adad 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -27,8 +27,10 @@ #include "common/file_perm.h" #include "common/file_utils.h" #include "common/restricted_token.h" +#include "fe_utils/encryption.h" #include "getopt_long.h" #include "storage/bufpage.h" +#include "storage/encryption.h" static void usage(const char *progname); @@ -75,6 +77,10 @@ usage(const char *progname) printf(_(" -D, --target-pgdata=DIRECTORY existing data directory to modify\n")); printf(_(" --source-pgdata=DIRECTORY source data directory to synchronize with\n")); printf(_(" --source-server=CONNSTR source server to synchronize with\n")); +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command=COMMAND\n" + " command that returns encryption key\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -n, --dry-run stop before modifying anything\n")); printf(_(" -N, --no-sync do not wait for changes to be written\n" " safely to disk\n")); @@ -99,6 +105,9 @@ main(int argc, char **argv) {"no-sync", no_argument, NULL, 'N'}, {"progress", no_argument, NULL, 'P'}, {"debug", no_argument, NULL, 3}, +#ifdef USE_ENCRYPTION + {"encryption-key-command", required_argument, NULL, 'K'}, +#endif /* USE_ENCRYPTION */ {NULL, 0, NULL, 0} }; int option_index; @@ -134,7 +143,7 @@ main(int argc, char **argv) } } - while ((c = getopt_long(argc, argv, "D:nNP", long_options, &option_index)) != -1) + while ((c = getopt_long(argc, argv, "D:K:nNP", long_options, &option_index)) != -1) { switch (c) { @@ -169,6 +178,12 @@ main(int argc, char **argv) case 2: /* --source-server */ connstr_source = pg_strdup(optarg); break; +#ifdef USE_ENCRYPTION + case 4: /* --encryption-key-command */ + case 'K': + encryption_key_command = strdup(optarg); + break; +#endif /* USE_ENCRYPTION */ } } @@ -248,6 +263,28 @@ main(int argc, char **argv) sanityChecks(); /* + * Setup encryption if it's obvious that we'll have to deal with encrypted + * XLOG. + */ + if (ControlFile_target.data_cipher > PG_CIPHER_NONE) + { + if (encryption_key_command == NULL) + { + pg_log_error("-K option must be passed for encrypted cluster"); + exit(EXIT_FAILURE); + } + + /* + * It should not matter whether we pass the source or target data + * directory. It should have been checked earlier that both clusters + * are encrypted using the same key. + */ + run_encryption_key_command(encryption_key, datadir_source); + setup_encryption(); + data_encrypted = true; + } + + /* * If both clusters are already on the same timeline, there's nothing to * do. */ @@ -446,6 +483,24 @@ sanityChecks(void) ControlFile_source.state != DB_SHUTDOWNED && ControlFile_source.state != DB_SHUTDOWNED_IN_RECOVERY) pg_fatal("source data directory must be shut down cleanly"); + + /* + * Since slave receives XLOG stream encrypted by master, handling + * differently encrypted clusters is not the typical use case for + * pg_rewind. Yet we should check the encryption. + */ + if (ControlFile_source.data_cipher > PG_CIPHER_NONE || + ControlFile_target.data_cipher > PG_CIPHER_NONE) + { + if (ControlFile_source.data_cipher != + ControlFile_target.data_cipher) + pg_fatal("source and target server must be both unencrypted or both encrypted\n"); + + if (memcmp(ControlFile_source.encryption_verification, + ControlFile_target.encryption_verification, + ENCRYPTION_SAMPLE_SIZE)) + pg_fatal("both source and target server must use the same encryption key"); + } } /* diff --git a/src/bin/pg_waldump/Makefile b/src/bin/pg_waldump/Makefile index 135979cef3..9eb1c2f01d 100644 --- a/src/bin/pg_waldump/Makefile +++ b/src/bin/pg_waldump/Makefile @@ -7,10 +7,11 @@ subdir = src/bin/pg_waldump top_builddir = ../../.. include $(top_builddir)/src/Makefile.global -OBJS = pg_waldump.o compat.o xlogreader.o rmgrdesc.o \ +OBJS = pg_waldump.o compat.o xlogreader.o rmgrdesc.o encryption.o \ $(RMGRDESCOBJS) $(WIN32RES) override CPPFLAGS := -DFRONTEND $(CPPFLAGS) +LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) RMGRDESCSOURCES = $(sort $(notdir $(wildcard $(top_srcdir)/src/backend/access/rmgrdesc/*desc.c))) RMGRDESCOBJS = $(patsubst %.c,%.o,$(RMGRDESCSOURCES)) @@ -27,6 +28,9 @@ xlogreader.c: % : $(top_srcdir)/src/backend/access/transam/% $(RMGRDESCSOURCES): % : $(top_srcdir)/src/backend/access/rmgrdesc/% rm -f $@ && $(LN_S) $< . +encryption.c: % : $(top_srcdir)/src/backend/storage/file/% + rm -f $@ && $(LN_S) $< . + install: all installdirs $(INSTALL_PROGRAM) pg_waldump$(X) '$(DESTDIR)$(bindir)/pg_waldump$(X)' @@ -37,7 +41,8 @@ uninstall: rm -f '$(DESTDIR)$(bindir)/pg_waldump$(X)' clean distclean maintainer-clean: - rm -f pg_waldump$(X) $(OBJS) $(RMGRDESCSOURCES) xlogreader.c + rm -f pg_waldump$(X) $(OBJS) $(RMGRDESCSOURCES) xlogreader.c \ +encryption.c rm -rf tmp_check check: diff --git a/src/bin/pg_waldump/pg_waldump.c b/src/bin/pg_waldump/pg_waldump.c index b95d467805..4df84efb01 100644 --- a/src/bin/pg_waldump/pg_waldump.c +++ b/src/bin/pg_waldump/pg_waldump.c @@ -22,6 +22,8 @@ #include "access/transam.h" #include "common/fe_memutils.h" #include "common/logging.h" +#include "fe_utils/encryption.h" +#include "storage/encryption.h" #include "getopt_long.h" #include "rmgrdesc.h" @@ -194,27 +196,59 @@ search_directory(const char *directory, const char *fname) PGAlignedXLogBlock buf; int r; - r = read(fd, buf.data, XLOG_BLCKSZ); - if (r == XLOG_BLCKSZ) + if (data_encrypted) { - XLogLongPageHeader longhdr = (XLogLongPageHeader) buf.data; - - WalSegSz = longhdr->xlp_seg_size; + /* + * Segment size affects calculation of segNo and thus also the + * encryption tweak, so we cannot get the size from the header + * until the page is decrypted. Here we need to take more + * expensive approach and really check the file size. + */ + WalSegSz = (int) lseek(fd, 0, SEEK_END); + if (WalSegSz <= 0) + fatal_error("Could not determine size of WAL segment \"%s\"", fname); + /* + * Verification of the file size is the only useful thing we can + * do. If anything else is wrong, the XLOG reader should find out + * after decryption. + */ if (!IsValidWalSegSize(WalSegSz)) - fatal_error(ngettext("WAL segment size must be a power of two between 1 MB and 1 GB, but the WAL file \"%s\" header specifies %d byte", - "WAL segment size must be a power of two between 1 MB and 1 GB, but the WAL file \"%s\" header specifies %d bytes", + fatal_error(ngettext("WAL segment size must be a power of two between 1 MB and 1 GB, but size of the WAL file \"%s\" is %d byte", + "WAL segment size must be a power of two between 1 MB and 1 GB, but size of the WAL file \"%s\" is %d bytes", WalSegSz), fname, WalSegSz); } else { - if (errno != 0) - fatal_error("could not read file \"%s\": %s", - fname, strerror(errno)); + r = read(fd, buf.data, XLOG_BLCKSZ); + if (r == XLOG_BLCKSZ) + { + XLogLongPageHeader longhdr; + + longhdr = (XLogLongPageHeader) buf.data; + + if (longhdr->std.xlp_magic != XLOG_PAGE_MAGIC) + fatal_error(gettext("WAL segment \"%s\" has an incorrect magic number. If it's encrypted, use the -K option to pass encryption credentials"), + fname); + + WalSegSz = longhdr->xlp_seg_size; + + if (!IsValidWalSegSize(WalSegSz)) + fatal_error(ngettext("WAL segment size must be a power of two between 1 MB and 1 GB, but the WAL file \"%s\" header specifies %d byte", + "WAL segment size must be a power of two between 1 MB and 1 GB, but the WAL file \"%s\" header specifies %d bytes", + WalSegSz), + fname, WalSegSz); + } else - fatal_error("could not read file \"%s\": read %d of %zu", - fname, r, (Size) XLOG_BLCKSZ); + { + if (errno != 0) + fatal_error("could not read file \"%s\": %s", + fname, strerror(errno)); + else + fatal_error("could not read file \"%s\": read %d of %zu", + fname, r, (Size) XLOG_BLCKSZ); + } } close(fd); return true; @@ -444,6 +478,19 @@ XLogDumpReadPage(XLogReaderState *state, XLogRecPtr targetPagePtr, int reqLen, XLogDumpXLogRead(private->inpath, private->timeline, targetPagePtr, readBuff, count); + if (data_encrypted) + { + char tweak[TWEAK_SIZE]; + XLogSegNo readSegNo; + uint32 readSegOff; + + XLByteToSeg(targetPagePtr, readSegNo, WalSegSz); + readSegOff = targetPagePtr % WalSegSz; + + XLogEncryptionTweak(tweak, private->timeline, readSegNo, readSegOff); + decrypt_block(readBuff, readBuff, count, tweak, true); + } + return count; } @@ -792,6 +839,10 @@ usage(void) printf(_(" -b, --bkp-details output detailed information about backup blocks\n")); printf(_(" -e, --end=RECPTR stop reading at WAL location RECPTR\n")); printf(_(" -f, --follow keep retrying after reaching end of WAL\n")); +#ifdef USE_ENCRYPTION + printf(_(" -K, --encryption-key-command=COMMAND\n" + " command that returns encryption key\n")); +#endif /* USE_ENCRYPTION */ printf(_(" -n, --limit=N number of records to display\n")); printf(_(" -p, --path=PATH directory in which to find log segment files or a\n" " directory with a ./pg_wal that contains such files\n" @@ -827,6 +878,7 @@ main(int argc, char **argv) {"end", required_argument, NULL, 'e'}, {"follow", no_argument, NULL, 'f'}, {"help", no_argument, NULL, '?'}, + {"encryption-key-command", required_argument, NULL, 'K'}, {"limit", required_argument, NULL, 'n'}, {"path", required_argument, NULL, 'p'}, {"rmgr", required_argument, NULL, 'r'}, @@ -884,7 +936,7 @@ main(int argc, char **argv) goto bad_argument; } - while ((option = getopt_long(argc, argv, "be:fn:p:r:s:t:x:z", + while ((option = getopt_long(argc, argv, "be:fK:n:p:r:s:t:x:z", long_options, &optindex)) != -1) { switch (option) @@ -904,6 +956,12 @@ main(int argc, char **argv) case 'f': config.follow = true; break; +#ifdef USE_ENCRYPTION + case 'K': + encryption_key_command = pg_strdup(optarg); + data_encrypted = true; + break; +#endif /* USE_ENCRYPTION */ case 'n': if (sscanf(optarg, "%d", &config.stop_after_records) != 1) { @@ -1099,6 +1157,12 @@ main(int argc, char **argv) /* done with argument parsing, do the actual work */ + if (data_encrypted) + { + run_encryption_key_command(encryption_key, NULL); + setup_encryption(); + } + /* we have everything we need, start reading */ xlogreader_state = XLogReaderAllocate(WalSegSz, XLogDumpReadPage, &private); -- 2.13.7 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v04-0017-User-and-development-documentation.patch From 96bd043f2f61e0378674216eb6e4c02ec5f4a963 Mon Sep 17 00:00:00 2001 From: Antonin Houska Date: Fri, 5 Jul 2019 16:24:02 +0200 Subject: [PATCH 17/17] User and development documentation. --- doc/src/sgml/config.sgml | 14 ++ doc/src/sgml/encryption.sgml | 95 ++++++++++++++ doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/installation.sgml | 14 +- doc/src/sgml/postgres.sgml | 1 + doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/initdb.sgml | 11 ++ doc/src/sgml/ref/pg_checksums.sgml | 13 +- doc/src/sgml/ref/pg_ctl-ref.sgml | 11 ++ doc/src/sgml/ref/pg_keytool.sgml | 121 +++++++++++++++++ doc/src/sgml/ref/pg_resetwal.sgml | 11 ++ doc/src/sgml/ref/pg_rewind.sgml | 11 ++ doc/src/sgml/ref/pg_waldump.sgml | 11 ++ doc/src/sgml/ref/pgupgrade.sgml | 31 +++++ doc/src/sgml/reference.sgml | 1 + src/backend/storage/file/README.encryption | 202 +++++++++++++++++++++++++++++ 16 files changed, 542 insertions(+), 7 deletions(-) create mode 100644 doc/src/sgml/encryption.sgml create mode 100644 doc/src/sgml/ref/pg_keytool.sgml create mode 100644 src/backend/storage/file/README.encryption diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index c91e3e1550..5d64736a03 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -9004,6 +9004,20 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' + + data_encryption (boolean) + + data_encryption configuration parameter + + + + + Reports whether data encryption is enabled for this cluster. + See for more information. + + + + data_directory_mode (integer) diff --git a/doc/src/sgml/encryption.sgml b/doc/src/sgml/encryption.sgml new file mode 100644 index 0000000000..5913883bc0 --- /dev/null +++ b/doc/src/sgml/encryption.sgml @@ -0,0 +1,95 @@ + + + + Transparent Cluster Encryption + + + Cluster encryption can be used if DBA cannot or does not + want to rely on the filesystem in terms of data confidentiality. If this + feature is enabled, PostgreSQL encrypts data + (both relations and write-ahead-log) when writing it to disk, and decrypts + when reading it. The encryption is transparent, so applications see no + difference between encrypted and unencrypted cluster. + + + + To create encrypted cluster, use the option to pass the + utility path to the command that will retrieve + the encryption key. For example: + +$ initdb -D /usr/local/pgsql/data -K /usr/local/pgsql/fetch_key_cmd + + Here /usr/local/pgsql/fetch_key_cmd is an executable + file that writes the encryption key to its standard output and returns + zero. The simplest command looks like: + + +echo <your key> + + + + + The encryption key is expected in hexadecimal format, two characters + (hexadecimal digits) per + byte. Since PostgreSQL currently uses key + of length 32 bytes (256 bits), the expected length of the key string + is 64 characters. For example: + + 882fb7c12e80280fd664c69d2d636913e86c381ba487c82f77653c0fac8ffc69 + + + + + If the DBA wants to enter password, he can use + utility to derive ecryption key from + it. For example, if the option looks like in the + following example, the DBA will be asked for password interactively: + + +$ initdb -K 'read -sp "Cluster encryption password: " PGENCRPWD; echo $PGENCRPWD +| pg_keytool -D %D -w' -D data + + + + + + + As long as the application is aware of the data directory, you can + put %D into the command and the application will replace + it with the actual directory path. + + + + Likewise, the same command can be passed to in + order to start the cluster: + + +$ pg_ctl -K 'read -sp "Cluster encryption password: " PGENCRPWD; echo $PGENCRPWD +| pg_keytool -D %D -w' -D data start + + + + + + If encryption is enabled, must be + turned on, otherwise the server refuses to start. This is because the + encryption introduces dependencies between data within a page, and thus + server crash during disk write can result in more serious damage of the page + than it would do without encryption. The whole page needs to be retrieved + from WAL in such a case to ensure reliable recovery. + + + + Once the PostgreSQL server is running, client + applications should recognize no difference from an unencrypted cluster, + except that variable is set. + + + + Since WAL is encrypted, any replication solution based on log shipping + () assume that all standby servers are + encrypted using the same key as their primary server. On the other hand, + allows replication between encrypted + and unencrypted clusters, or between clusters encrypted with different keys. + + diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index 3da2365ea9..d21dabf03b 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -48,6 +48,7 @@ + diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml index 449386243b..a2daf33e6f 100644 --- a/doc/src/sgml/installation.sgml +++ b/doc/src/sgml/installation.sgml @@ -245,8 +245,9 @@ su - postgres You need OpenSSL, if you want to support - encrypted client connections. OpenSSL is - also required for random number generation on platforms that do not + on-disk data encryption or encrypted client + connections. OpenSSL is also required for + random number generation on platforms that do not have /dev/urandom (except Windows). The minimum version required is 0.9.8. @@ -835,10 +836,11 @@ su - postgres - Build with support for SSL (encrypted) - connections. This requires the OpenSSL - package to be installed. configure will check - for the required header files and libraries to make sure that + Build with support for on-disk data encryption + or SSL (encrypted) connections. This requires + the OpenSSL package to be + installed. configure will check for the + required header files and libraries to make sure that your OpenSSL installation is sufficient before proceeding. diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 3e115f1c76..b0d35600b2 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -163,6 +163,7 @@ &wal; &logical-replication; &jit; + &encryption; ®ress; diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 8d91f3529e..a60b02e23f 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -206,6 +206,7 @@ Complete list of usable sgml source files in this directory. + diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml index 7fc3152c6d..ce19151521 100644 --- a/doc/src/sgml/ref/initdb.sgml +++ b/doc/src/sgml/ref/initdb.sgml @@ -209,6 +209,17 @@ PostgreSQL documentation + + + + + + Encrypt the cluster data using a key retrieved from the command + specified here. See for details. + + + + diff --git a/doc/src/sgml/ref/pg_checksums.sgml b/doc/src/sgml/ref/pg_checksums.sgml index 33706d1d97..7a2cae5e57 100644 --- a/doc/src/sgml/ref/pg_checksums.sgml +++ b/doc/src/sgml/ref/pg_checksums.sgml @@ -106,7 +106,18 @@ PostgreSQL documentation Only validate checksums in the relation with filenode - filenode. + filenode. + + + + + + + + + + If the cluster is encrypted, run this command to retrieve the + encryption key. See for details. diff --git a/doc/src/sgml/ref/pg_ctl-ref.sgml b/doc/src/sgml/ref/pg_ctl-ref.sgml index e31275a04e..48966eb58e 100644 --- a/doc/src/sgml/ref/pg_ctl-ref.sgml +++ b/doc/src/sgml/ref/pg_ctl-ref.sgml @@ -293,6 +293,17 @@ PostgreSQL documentation + + + + + If the cluster is encrypted, run this command to retrieve the + encryption key. See for details. + + + + + diff --git a/doc/src/sgml/ref/pg_keytool.sgml b/doc/src/sgml/ref/pg_keytool.sgml new file mode 100644 index 0000000000..3c15910885 --- /dev/null +++ b/doc/src/sgml/ref/pg_keytool.sgml @@ -0,0 +1,121 @@ + + + + + pg_keysetup + + + + pg_keytool + 1 + Application + + + + pg_keytool + produce cluster encryption key, possibly deriving it from a + passphrase + + + + + pg_keysetup + option + + + + + Description + + pg_keytool reads encryption key or password from + the standard input. If it received password, it runs key + derivation function on it in order to derive the key. The + encryption key is written either to the standard output or sent to via + frontend/backend protocol to postmaster process which is being started up. + + + + One common use case is that pg_keytool is used + within encryption key command, see the examples in + . In this case it sends the key to the + standard output. + + + + The other use case is that PostgreSQL is + started other way than using , typically + via systemd or a custom script. In this + case pg_keytool can be used to send the key to + specified TCP/IP port or to an Unix socket, on which the server + listens. This is the same port or socket to which applications will + eventually connect, but when the encrypted cluster starts up, it only + accepts the key. + + + + + Options + + + + + + + + Specifies the directory where the database cluster is + stored. Different directory generally means that a different key will + be derived from the same password. + + + + + + + + + + Specifies the host name of the machine on which the server is + listening. If the value begins with a slash, it is used as the + directory for the Unix-domain socket. + + + + + + + + + Specifies the TCP port or the local Unix-domain socket file + extension on which the server is listening for + connections. Defaults to the value of the PGPORT + environment variable or, if not set, to the port specified at + compile time, usually 5432. + + + + + + + + + Send the key to PostgreSQL server rather + than to standard output. + + + + + + + + + Expect password on standard input rather than encryption key. + + + + + + + diff --git a/doc/src/sgml/ref/pg_resetwal.sgml b/doc/src/sgml/ref/pg_resetwal.sgml index 8a9e22d050..613d14e1cc 100644 --- a/doc/src/sgml/ref/pg_resetwal.sgml +++ b/doc/src/sgml/ref/pg_resetwal.sgml @@ -100,6 +100,17 @@ PostgreSQL documentation + + + + + If the cluster is encrypted, run this command to retrieve the + encryption key. See for details. + + + + + diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index 4d91eeb0ff..0e4f79b652 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -140,6 +140,17 @@ PostgreSQL documentation + + + + + If the cluster is encrypted, run this command to retrieve the + encryption key. See for details. + + + + + diff --git a/doc/src/sgml/ref/pg_waldump.sgml b/doc/src/sgml/ref/pg_waldump.sgml index 329c10e430..ca1e4e1468 100644 --- a/doc/src/sgml/ref/pg_waldump.sgml +++ b/doc/src/sgml/ref/pg_waldump.sgml @@ -101,6 +101,17 @@ PostgreSQL documentation + + + + + If the cluster is encrypted, run this command to retrieve the + encryption key. See for details. + + + + + diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index 82886760f1..0c04990d51 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -120,6 +120,37 @@ + + + + + If the clusters are encrypted, run this command to retrieve the + encryption key. This is the same command that was passed to + for cluster creation. For the upgrade to + be successful, both old and new clusters must use the same key. + Therefore, unless the command contains the %D + string, it does not matter which cluster the command comes from. + + + + + If you are using to derive the + encryption key (see ), + make sure the points to + the old cluster. If it pointed to the new + one then the new cluster would derive a different key from the same + password and thus would fail to start + the new cluster. + + + + + See for details on the command. + + + + + use hard links instead of copying files to the new diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index cef09dd38b..5eed489f27 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -279,6 +279,7 @@ &pgChecksums; &pgControldata; &pgCtl; + &pgKeytool; &pgResetwal; &pgRewind; &pgtestfsync; diff --git a/src/backend/storage/file/README.encryption b/src/backend/storage/file/README.encryption new file mode 100644 index 0000000000..5ffd587cf5 --- /dev/null +++ b/src/backend/storage/file/README.encryption @@ -0,0 +1,202 @@ +src/backend/storage/file/README.encryption + +Transparent Cluster Encryption +============================== + +When creating a new cluster (instance), user can choose to have his data +encrypted on disk. If this feature is active, data is encrypted before it's +written to disk and decrypted after it has been read, however the data is +unencrypted in memory (data-at-rest encryption). The following characteristics +should be considered by anyone who is interested in the feature: + + 1. The encryption is transparent from application's point of view. + + 2. A single key is used to encrypt the whole cluster. + +The full instance encryption feature helps to ensure data confidentiality, +especially when user cannot rely on confidentiality on filesystem level. On +the other hand, it does not ensure data integrity, i.e. it does not help to +detect whether an adversary wrote his cipher data to the disk. The block +cipher methods generally do not protect data integrity, and it'd probably be +hard anyway because we encrypt postgres data pages (typically of size 8 kB) +separate from each other. If the attacker only changes part of the page, this +can be detected if data checksums feature is enabled. And specifically for +XLOG, each XLOG record has its own checksum. + +Since the data is stored on a disk, we naturally base our approach on "Disk +encryption theory" [1]. For each kind of file we use the AES cipher in the +appropriate mode of operation. The AES cipher itself encrypts / decrypts +individual blocks ("encryption blocks") of 16 bytes (128 bits), while the mode +of operation defines the rules how to apply the cipher to data which spans +multiple encryption blocks. + +Relations +--------- + +AES cipher in CBC mode [2] is used to encrypt relation files, one relation +block (page) at a time. The important characteristic of this mode is that, if +a single encryption block changes, the next run of the encryption produces an +encrypted page where not only that block is different, but also all the +following blocks. Furthermore, as the postgres page starts with LSN, even if +only the last encryption block of the page changes, the whole cipher text of +the page will be different after the next encryption. + +This propagation of changes across encryption blocks makes it harder for +adversary to deduce what happens in the database. Without the propagation he +might be able to see data changes with higher granularity. For example, if +only the beginning of index page changes, it gives some information on the +keys inserted. + +On the other hand, the same change propagation can cause problems if only a +part of a new version of encrypted page is written to disk (torn page +write). If such a page is decrypted during crash recovery, part of the plain +text can become garbage. This can affect even those parts of the page that +haven't been modified recently, and therefore XLOG records to fix those parts +are not available. Therefore, if the cluster encryption is enabled, +full_page_writes configuration variable must be set too, otherwise the server +refuses to start. With this setting, the damage described here can be fixed by +retrieving the full-page-image (FPI) from XLOG and applying it. + +However, even if full_page_writes is set, postgres can still avoid writing FPI +XLOG record if only hint bit(s) changed on a page and if wal_log_hints +configuration variable is off. The problem with encryption is that even a +single bit change is propagated to all the following encryption blocks during +the next encryption of the page. Again, if only part of the new cipher text is +written, the remaining portion of the page becomes garbage after decryption, +and we might not be able to fix it during recovery unless we have the FPI in +the XLOG. So if encryption is enabled, postgres behaves as if wal_log_hints +was always set. + +XLOG +---- + +The specific problem of XLOG is that record must not be changed once it has +been flushed to disk. However if we used a block cipher, and if a new XLOG +record started in the same encryption block in which the previous record ends, +that encryption block would become completely different after the next run of +encryption of the containing page. Torn write of such a block (e.g. if memory +page boundary crosses the encryption block) is likely to make decryption +produce garbage, which will also appear in the already-flushed record. + +Therefore we encrypt XLOG using a stream cipher, or rather block cipher in +stream mode of operation. Stream cipher uses XOR operation to combine a "key +stream" with the input stream, and it does not matter if the length of the +input stream is aligned to any value. In particular, the CTR mode [3] was +chosen because it allows for both read and write operations to be +parallelized. + +The XLOG is encrypted / decrypted one XLOG page (typically 8 kB) at a time, so +rather than a single stream, the encrypted XLOG is actually a sequence of +per-page streams. + +Since stream ciphers are susceptible to "reused key attack" [4], we must +ensure that the unused part of the last XLOG page (filled with zeroes) is +never encrypted. + +Temporary files +--------------- + +BufFileWrite() and BufFileRead() functions (see buffile.c) hide the encryption +/ decryption from caller. The encryption / decryption processes one buffer at +a time so that the buffers can be retrieved independent from each other. + +If the encryption is enabled, the following requirements need to be taken into +account: + +1. The file buffer cannot be positioned at arbitrary offset of the file. If +the encryption routine starts at some position of the file, decryption must +not start elsewhere because there's no way to determine which initialization +vector was used internally for the corresponding encryption blocks (of 16 +bytes) during encryption. It makes sense to position the buffer at file offset +that is whole multiple of buffer size. + +2. In general, the useful (written) data does not fill whole multiple of +encryption blocks, but we must write the whole blocks for decryption to +succeed. This implies that we need to fill the unused part of the last block +with zeroes and also remember the amount of useful bytes in the segment file. +(In fact we align the segment file size to file buffers instead of encryption +blocks, which probably makes the implementation a bit simpler.) + +Stream cipher might seem like a solution of the padding problem, but we cannot +use it here because parts of the temporary file can be rewritten. That would +expose the temporary file to "reused key attack" [4]. + +Auxiliary files +--------------- + +To store other kinds of data encrypted than the ones above, developers are +advised to use BufFileWriteTransient() and BufFileReadTransient() functions +(also located in buffile.c). These are especially useful if some data +structure should be written to an encrypted file and user does not want to +care whether the next write position is at encryption block boundary. + +Wherever the write starts, BufFileWriteTransient() first reads the whole +encryption block that contains the start positions, then decrypts it, adds the +new data to it, encrypts the whole block and writes it back to the +file. Likewise, BufFileReadTransient() ensures that only the whole encryption +blocks are read from disk and decrypted, whether the next read position is at +block boundary or not. + +Serialization of data changes during logical decoding (reorderbuffer.c) is the +typical use case for this API. + +Initialization vector (IV), encryption tweak +-------------------------------------------- + +Besides the input data and the key, both block and stream cipher used for the +postgres cluster encryption require an initialization vector. It should make +analysis of the encrypted data more difficult. Also according to [1], the same +data should be encrypted differently if located elsewhere on the disk. Term +"encryption tweak" is more common in the context of disk encryption. + +When encrypting relations, each encryption unit (i.e. page) has an unique +tweak, which consists of RelFileNode structure, buffer number and fork +number. As a consequence, we need to decrypt page using the existing tweak and +encrypt it with a new tweak if copying it from one relation to +another. Typically this happens when a new database is being created from +template database. + +To avoid "reencryption" during pg_upgrade, all the OIDs contained in the +RelFileNode structure (tablespace OID, database OID and relation file OID) +must be preserved. If pg_upgrade had to reencrypt files, it could not be used +with the --link option. + +XLOG encryption tweak consists of timeline, segment number and offset at which +the XLOG page starts in the segment. The "reencryption" takes place when XLOG +page is copied from one timeline to another, typicially at the end of +recovery. + +As for temporary files, PID of the owning backend, file / fileset number and +block number (where block is of the same size as a relation page) within the +file provide sufficient uniqueness, so we use these to generate the tweak. + +Auxiliary files can be closed and reopened by another backend, so there's no +PID strictly associated with them. Therefore we generate the tweak by hashing +the file path, and appending block number to the hash. + +Replication +----------- + +During streaming replication, the walsender process sends the XLOG encrypted +and walreceiver just writes it. Decryption is performed before the slave +cluster tries to apply the changes. That implies that the same encryption key +must be used both master and slave. If it should be possible someday to stream +the XLOG unencrypted. One implication is that pg_basebackup should then also +receive the relation data unencrypted. + +As for logical replication, the only change introduced by this feature is that +the XLOG has to be decrypted before the contained data changes can be +decoded. The data changes are transferred to the subscribing database / +cluster unencrypted, so both master and slave can use different encryption +keys. + +References +---------- + +[1] https://en.wikipedia.org/wiki/Disk_encryption_theory + +[2] https://en.wikipedia.org/wiki/Disk_encryption_theory#Cipher-block_chaining_(CBC) + +[3] https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_(CTR) + +[4] https://en.wikipedia.org/wiki/Stream_cipher_attacks#Reused_key_attack -- 2.13.7 --=-=-=--