public inbox for [email protected]help / color / mirror / Atom feed
Moving forward with TDE 9+ messages / 4 participants [nested] [flat]
* Moving forward with TDE @ 2022-10-24 16:29 David Christensen <[email protected]> 2022-11-03 14:09 ` Re: Moving forward with TDE Aleksander Alekseev <[email protected]> 2022-11-15 19:07 ` Re: Moving forward with TDE Jacob Champion <[email protected]> 0 siblings, 2 replies; 9+ messages in thread From: David Christensen @ 2022-10-24 16:29 UTC (permalink / raw) To: pgsql-hackers; +Cc: Stephen Frost <[email protected]> Hi -hackers, Working with Stephen, I am attempting to pick up some of the work that was left off with TDE and the key management infrastructure. I have rebased Bruce's KMS/TDE patches as they existed on the https://wiki.postgresql.org/wiki/Transparent_Data_Encryption wiki page, which are enclosed in this email. I would love to open a discussion about how to move forward and get some of these features built out. The historical threads here are quite long and complicated; is there a "current state" other than the wiki that reflects the general thinking on this feature? Any major developments in direction that would not be reflected in the code from May 2021? Thanks, David Attachments: [application/octet-stream] 0001-cfe-01-doc_over_master-squash-commit.patch (10.1K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/2-0001-cfe-01-doc_over_master-squash-commit.patch) download | inline diff: From a12fd01c86e1768310a91640cdd467e979bebd94 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:43 -0400 Subject: [PATCH 01/12] cfe-01-doc_over_master squash commit --- doc/src/sgml/database-encryption.sgml | 149 ++++++++++++++++++++++++++ doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/installation.sgml | 2 +- doc/src/sgml/monitoring.sgml | 13 +++ doc/src/sgml/postgres.sgml | 1 + 5 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 doc/src/sgml/database-encryption.sgml diff --git a/doc/src/sgml/database-encryption.sgml b/doc/src/sgml/database-encryption.sgml new file mode 100644 index 0000000000..c4377ada64 --- /dev/null +++ b/doc/src/sgml/database-encryption.sgml @@ -0,0 +1,149 @@ +<!-- doc/src/sgml/database-encryption.sgml --> + +<chapter id="cluster-file-encryption"> + <title>Cluster File Encryption</title> + + <indexterm zone="cluster-file-encryption"> + <primary>Cluster File Encryption</primary> + </indexterm> + + <para> + The purpose of cluster file encryption is to prevent users with read + access on the directories used to store database files and write-ahead + log files from being able to access the data stored in those files. + For example, when using cluster file encryption, users who have read + access to the cluster directories for backup purposes will not be able + to decrypt the data stored in these files. Read-only access for a group + of users can be enabled using the <application>initdb</application> + <option>--allow-group-access</option> option. Cluster file encryption + also provides data-at-rest security, protecting users from data loss + should the physical storage media be stolen or improperly erased before + disposal. + </para> + + <para> + Cluster file encryption does not protect against unauthorized file + system writes. Such writes can allow data decryption if used to weaken + the system's security and the weakened system is later supplied with + the externally-stored cluster encryption key. This also does not always + detect if users with write access remove or modify database files. + </para> + + <para> + This also does not protect against users who have read access to database + process memory because all in-memory data pages and data encryption keys + are stored unencrypted in memory. Therefore, an attacker who is able + to read memory can read the data encryption keys and decrypt the entire + cluster. The Postgres operating system user and the operating system + administrator, e.g., the <literal>root</literal> user, have such access. + </para> + + <sect1 id="cluster-encryption-keys"> + <title>Keys</title> + + <para> + Cluster file encryption uses two levels of encryption — an upper + key which encrypts lower-level keys. The upper-level key is often + referred to as a Key Encryption Key (<acronym>KEK</acronym>). This key + is <emphasis>not</emphasis> stored in the file system, but provided at + <command>initdb</command> time and each time the server is started. This + key can be easily changed via <command>pg_alterckey</command> without + requiring any changes to the the data files or <command>WAL</command> + files. + </para> + + <para> + The lower level keys are data encryption keys, specifically for relations + and <acronym>WAL</acronym>. The relation key is used to encrypt database + heap and index files. The WAL key is used to encrypt write-ahead log + (WAL) files. Two different keys are used so that primary and standby + servers can use different relation keys, but the same WAL key, so that + these keys can (in a future release) be rotated by switching the + primary to the standby and then changing the WAL key. Eventually, + encryption will be able to added to non-encrypted clusters by creating + encrypted replicas and switching over to them. + </para> + + <para> + Postgres stores the data encryption (lower-level) keys in the data + directory encrypted (wrapped) by key encryption (upper-level) key. + Though the data encryption keys technically exist in the file system, + the key encryption key does not, so the data encryption keys are + securely stored. Data encryption keys are used to security encrypt + other database files. + </para> + </sect1> + + <sect1 id="cluster-encryption-initialization"> + <title>Initialization</title> + + <para> + Cluster file encryption is enabled when + <productname>PostgreSQL</productname> is built + with <literal>--with-openssl</literal> and <xref + linkend="app-initdb-cluster-key-command"/> is specified + during <command>initdb</command>. The cluster key + provided by the <option>--cluster-key-command</option> + option during <command>initdb</command> and the one generated + by <xref linkend="guc-cluster-key-command"/> in the + <filename>postgresql.conf</filename> must match for the database + cluster to start. Note that the cluster key command + passed to <command>initdb</command> must return a key of + 64 hexadecimal characters. For example: +<programlisting> +initdb -D dbname --cluster-key-command='ckey_passphrase.sh' +</programlisting> + Cluster file encryption does not support a <varname>wal_level</varname> + of <literal>minimal</literal>. + </para> + </sect1> + + <sect1 id="cluster-encryption-operation"> + <title>Operation</title> + + <para> + During the <command>initdb</command> process, if + <option>--cluster-key-command</option> is specified, two data-level + encryption keys are created. These two keys are then encrypted with + the key encryption key (KEK) supplied by the cluster key command before + being stored in the database directory. The key or passphrase that + derives the key must be supplied from the terminal or stored in a + trusted key store, such as key vault software or a hardware security + module. + </para> + + <para> + If the <productname>PostgreSQL</productname> server has + been initialized to require a cluster key, each time the + server starts the <filename>postgresql.conf</filename> + <varname>cluster_key_command</varname> command will be executed + and the cluster key retrieved. The data encryption keys in the + <filename>pg_cryptokeys</filename> directory will then be decrypted + using the supplied key and integrity-checked to ensure it matches the + initdb-supplied key. (If this check fails, the server will refuse + to start.) The cluster encryption key will then be removed from + system memory. The decrypted data encryption keys will remain in + shared memory until the server is stopped. + </para> + + <para> + The data encryption keys are randomly generated and can be 128, 192, + or 256-bits in length, depending on whether <literal>AES128</literal>, + <literal>AES192</literal>, or <literal>AES256</literal> is specified. + They are encrypted by the key encryption key (KEK) using Advanced + Encryption Standard (<acronym>AES256</acronym>) encryption in Key + Wrap Padded Mode, which also provides KEK authentication; see <ulink + url="https://tools.ietf.org/html/rfc5649">RFC 5649</ulink>. While + 128-bit encryption is sufficient for most sites, 256-bit encryption + is thought to be more immune to future quantum cryptographic attacks. + </para> + + <para>. + If you prefer to create the random keys on your own, you can create + a empty directory with a <filename>pg_cryptokeys/live</filename> + subdirectory, generate the keys there using your tools. and use the + <command>initdb</command> <option>--copy-encryption-keys</option> + to copy those keys into the newly-created cluster. + </para> + </sect1> +</chapter> diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index de450cd661..efeb542cf7 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -49,6 +49,7 @@ <!ENTITY wal SYSTEM "wal.sgml"> <!ENTITY logical-replication SYSTEM "logical-replication.sgml"> <!ENTITY jit SYSTEM "jit.sgml"> +<!ENTITY database-encryption SYSTEM "database-encryption.sgml"> <!-- programmer's guide --> <!ENTITY bgworker SYSTEM "bgworker.sgml"> diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml index 319c7e6966..b67d3b7ffb 100644 --- a/doc/src/sgml/installation.sgml +++ b/doc/src/sgml/installation.sgml @@ -1011,7 +1011,7 @@ build-postgresql: <listitem> <para> Build with support for <acronym>SSL</acronym> (encrypted) - connections. The only <replaceable>LIBRARY</replaceable> + connections and cluster file encryption. The only <replaceable>LIBRARY</replaceable> supported is <option>openssl</option>. This requires the <productname>OpenSSL</productname> package to be installed. <filename>configure</filename> will check for the required diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index e5d622d514..3e3e1adea6 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1340,6 +1340,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser <entry><literal>DataFileWrite</literal></entry> <entry>Waiting for a write to a relation data file.</entry> </row> + <row> + <entry><literal>KeyFileRead</literal></entry> + <entry>Waiting for a read of the wrapped data encryption keys.</entry> + </row> + <row> + <entry><literal>KeyFileWrite</literal></entry> + <entry>Waiting for a write of the wrapped data encryption keys.</entry> + </row> + <row> + <entry><literal>KeyFileSync</literal></entry> + <entry>Waiting for changes to the wrapped data encryption keys to reach + durable storage.</entry> + </row> <row> <entry><literal>LockFileAddToDataDirRead</literal></entry> <entry>Waiting for a read while adding a line to the data directory lock diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 73439c049e..d99b24605f 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -171,6 +171,7 @@ break is not needed in a wider output rendering. &wal; &logical-replication; &jit; + &database-encryption; ®ress; </part> -- 2.37.0 (Apple Git-136) [application/octet-stream] 0002-cfe-02-internaldoc_over_cfe-01-doc-squash-commit.patch (11.4K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/3-0002-cfe-02-internaldoc_over_cfe-01-doc-squash-commit.patch) download | inline diff: From 2971ab6e1fc42102f3b1f8be72bfe304c2faef63 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:45 -0400 Subject: [PATCH 02/12] cfe-02-internaldoc_over_cfe-01-doc squash commit --- src/backend/crypto/README | 231 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 src/backend/crypto/README diff --git a/src/backend/crypto/README b/src/backend/crypto/README new file mode 100644 index 0000000000..be5e5557ba --- /dev/null +++ b/src/backend/crypto/README @@ -0,0 +1,231 @@ +Cluster File Encryption +======================= + +This directory contains support functions and sample scripts to be used +for cluster file encryption. + +Architecture +------------ + +Fundamentally, cluster file encryption must store data in a file system +in such a way that the keys required to decrypt the file system data can +only be accessed using somewhere outside of the file system itself. The +external requirement can be someone typing in a passphrase, getting a +key from a key management server (KMS), or decrypting a key stored in +the file system using a hardware security module (HSM). The current +architecture supports all of these methods, and includes sample scripts +for them. + +The simplest method for accessing data keys using some external +requirement would be to retrieve all data encryption keys from a KMS. +However, retrieved keys would still need to be verified as valid. This +method also introduces unacceptable complexity for simpler use-cases, +like user-supplied passphrases or HSM usage. External key rotation +would also be very hard since it would require re-encrypting all the +file system data with the new externally-stored keys. + +For these reason, a two-tiered architecture is used, which uses two +types of encryption keys: a key encryption key (KEK) and data encryption +keys (DEK). The KEK should not be present unencrypted in the file system +--- it should be supplied the user, stored externally (e.g., in a KMS) +or stored in the file system encrypted with a HSM (e.g., PIV device). +The DEK is used to encrypt database files and is stored in the same file +system as the database but is encrypted using the KEK. Because the DEK +is encrypted, its storage in the file system is no more of a security +weakness and the storage of the encrypted database files in the same +file system. + +Implementation +-------------- + +To enable cluster file encryption, the initdb option +--cluster-key-command must be used, which specifies a command to +retrieve the KEK. initdb records the cluster_key_command in +postgresql.conf. Every time the KEK is needed, the command is run and +must return 64 hex characters which are decoded into the KEK. The +command is called twice during initdb, and every time the server starts. +initdb also sets the encryption method in controldata during server +bootstrap. + +initdb runs "postgres --boot", which calls function +kmgr.c::BootStrapKmgr(), which calls the cluster key command. The +cluster key command returns a KEK which is used to encrypt random bytes +for each DEK and writes them to the file system by +kmgr.c::KmgrWriteCryptoKeys() (unless --copy-encryption-keys is used). +Currently the DEK files are 0 and 1 and are stored in +$PGDATA/pg_cryptokeys/live. The wrapped DEK files use Key Wrapping with +Padding which verifies the validity of the KEK. + +initdb also does a non-boot backend start which calls +kmgr.c::InitializeKmgr(), which calls the cluster key command a second +time. This decrypts/unwraps the DEK keys and stores them in the shared +memory structure KmgrShmem. This step also happens every time the server +starts. Later patches will use the keys stored in KmgrShmem to +encrypt/decrypt database files. KmgrShmem is erased via +explicit_bzero() on server shutdown. + +Limitations +----------- + +There doesn't seem to be a reasonable way to detect all malicious data +modification or key extraction if a user has write permission on the +files in PGDATA. It might be possible to limit the key extraction risk +if postgresql.auto.conf were able to be moved to a directory outside of +PGDATA, and if postmaster.opts could be moved or ignored when cluster +file encryption is used. (This file is used by pg_ctl restart.) + +It doesn't appear possible to detect all malicious writes --- even if +you add message authentication code (MAC) checks to encrypted files, +modifying non-encrypted files could still affect encrypted ones, e.g., +modifying files in pg_xact could affect how heap rows are interpreted. +Basically you would need to encrypt all files, and at that point you +might as well just use an encrypted file system. There also doesn't seem +to be a way to prevent key extraction if someone has read permission on +postgres process memory. + +Initialization Vector +--------------------- + +Nonce means "number used once". An Initialization Vector (IV) is a +specific type of nonce. That is, unique but not necessarily random or +secret, as specified by the NIST +(https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf). +To generate unique IVs, the NIST recommends two methods: + + The first method is to apply the forward cipher function, under + the same key that is used for the encryption of the plaintext, + to a nonce. The nonce must be a data block that is unique to + each execution of the encryption operation. For example, the + nonce may be a counter, as described in Appendix B, or a message + number. The second method is to generate a random data block + using a FIPS-approved random number generator. + +We will use the first method to generate IVs. That is, select nonce +carefully and use a cipher with the key to make it unique enough to use +as an IV. The nonce selection for buffer encryption and WAL encryption +are described below. + +If the IV was used more than once with the same key (and we only use one +data encryption key), changes in the unencrypted data would be visible +in the encrypted data. + +IV for Heap/Index Encryption +- - - - - - - - - - - - - - + +To create the 16-byte IV needed by AES for each page version, we will +use the page LSN (8 bytes) and page number (4 bytes). In the remaining +four bytes, one bit will be used to indicate if the LSN is WAL (real) or +fake (see below). The LSN is ideal for use in the IV because it is +always increasing, and is changed every time a page is updated. The +same LSN is never used for two relations with different page contents. + +However, the same LSN can be used in multiple pages in the same relation +--- this can happen when a heap update expires an old tuple and adds a +new tuple to another page. By adding the page number to the IV, we keep +the IV unique. + +By not using the database id in the IV, CREATE DATABASE can copy the +heap/index files from the old database to a new one without +decryption/encryption. Both page copies are valid. Once a database +changes its pages, it gets new LSNs, and hence new IV. Using only the +LSN and page number also avoids requiring pg_upgrade to preserve +database oids, tablespace oids, and relfilenodes. + +As part of WAL logging, every change of a WAL-logged page gets a new +LSN, and therefore a new IV automatically. + +However, the LSN must then be visible on encrypted pages, so we will not +encrypt the LSN on the page. We will also not encrypt the CRC so +pg_checksums can still check pages offline without access to the keys. + +Non-Permanent Relations +- - - - - - - - - - - - + +To avoid the overhead of generating WAL for non-permanent (unlogged and +temporary) relations, we assign fake LSNs that are derived from a +counter via xlog.c::GetFakeLSNForUnloggedRel(). (GiST also uses this +counter for LSNs.) We also set a bit in the IV so the use of the same +value for WAL (real) and fake LSNs will still generate unique IVs. Only +main forks are encrypted, not init, vm, or fsm files. + +In the code, we need to identify if a page uses WAL or fake LSNs in +four places, when: + +1. Reading a page from the file system and decrypting +2. Setting the WAL or fake LSN on a page +3. Hint bits changes requiring new LSNs for the encryption IV +4. Encrypting and writing a page to the file system + +For all these case, we have access to the fork number and either the +relation's persistence state or the buffer state. If it is a "main" +fork and the relation persistence state is RELPERSISTENCE_PERMANENT, or +if it is an "init" fork, we use a real LSN. If it is a main fork and +RELPERSISTENCE_PERMANENT is false, we use a fake LSN. The buffer state +BM_PERMANENT is true if the relation is PERMANENT or is an init fork. + +Init Forks +- - - - - + +Init forks for unlogged relations get permanent LSNs because unlogged +relation creation is WAL logged/crash safe, even though the relation's +contents are not. When the init fork is copied to represent an empty +relation during crash recovery, it becomes a non-permanent page and must +be successfully decrypted as such. Therefore, when it is copied, its +LSN is changed to e fake LSN and then encrypted. This prevents a real +LSN from being encrypted with the fake nonce bit. + +LSN Assignment, GiST, & Non-Permanent Relations +- - - - - - - - - - - - - - - - - - - - - - - - + +LSN assignment has to be slightly modified for encryption. In normal, +non-encryption mode, LSNs are assigned to pages following these rules: + +1. During GiST builds, some pages are assigned fixed LSNs (GistBuildLSN) + +2. During GiST builds, non-permanent pages not assigned fixed LSNs in +#1 are assigned fake LSNs, via gistutil.c::gistGetFakeLSN(). + +3. All other permanent pages are assigned WAL-based LSNs based on the +WAL position of their WAL records. + +4. All other non-permanent pages have LSNs of zero. + +When encryption is enabled: + +1. During GiST builds, permanent pages are assigned WAL-based LSNs +generated by xloginsert.c::LSNForEncryption(). + +2. During GiST builds, non-permanent pages are assigned fake LSNs. +(No constant LSNs are used in #1 or #2.) + +3. same as #3 above + +4. All other non-permanent pages are assigned fake LSNs before page +encryption. + +When switching to an encrypted replica from a non-encrypted primary, +GiST indexes will be using fixed LSNs for permanent tables, so it is +recommended to rebuild GiST indexes. Non-permanent relations are not +replicated, so they are not an issue. + +Hint Bits +- - - - - + +For hint bit changes, the LSN normally doesn't change, which is a +problem. By enabling wal_log_hints, you get full page writes to the WAL +after the first hint bit change of the checkpoint. This is useful for +two reasons. First, it generates a new LSN, which is needed for the IV +to be secure. Second, full page images protect against torn pages, +which is an even bigger requirement for encryption because the new LSN +is re-encrypting the entire page, not just the hint bit changes. You +can safely lose the hint bit changes, but you need to use the same LSN +to decrypt the entire page, so a torn page with an LSN change cannot be +decrypted. To prevent this, wal_log_hints guarantees that the +pre-hint-bit version (and previous LSN version) of the page is restored. + +However, if a hint-bit-modified page is written to the file system +during a checkpoint, and there is a later hint bit change switching the +same page from clean to dirty during the same checkpoint, we need a new +LSN, and wal_log_hints doesn't give us a new LSN here. The fix for this +is to update the page LSN by writing a dummy WAL record via +xloginsert.c::LSNForEncryption() in such cases. -- 2.37.0 (Apple Git-136) [application/octet-stream] 0003-cfe-03-scripts_over_cfe-02-internaldoc-squash-commit.patch (13.8K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/4-0003-cfe-03-scripts_over_cfe-02-internaldoc-squash-commit.patch) download | inline diff: From 28ba21f1d8e9cbffe4bf7e6fea07e65ed8061112 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:47 -0400 Subject: [PATCH 03/12] cfe-03-scripts_over_cfe-02-internaldoc squash commit --- src/backend/Makefile | 15 +++- src/backend/crypto/ckey_aws.sh.sample | 53 +++++++++++++ src/backend/crypto/ckey_direct.sh.sample | 39 ++++++++++ src/backend/crypto/ckey_passphrase.sh.sample | 35 +++++++++ src/backend/crypto/ckey_piv_nopin.sh.sample | 68 ++++++++++++++++ src/backend/crypto/ckey_piv_pin.sh.sample | 81 ++++++++++++++++++++ src/backend/crypto/ssl_passphrase.sh.sample | 35 +++++++++ 7 files changed, 325 insertions(+), 1 deletion(-) create mode 100644 src/backend/crypto/ckey_aws.sh.sample create mode 100644 src/backend/crypto/ckey_direct.sh.sample create mode 100644 src/backend/crypto/ckey_passphrase.sh.sample create mode 100644 src/backend/crypto/ckey_piv_nopin.sh.sample create mode 100644 src/backend/crypto/ckey_piv_pin.sh.sample create mode 100644 src/backend/crypto/ssl_passphrase.sh.sample diff --git a/src/backend/Makefile b/src/backend/Makefile index 181c217fae..1a5239b822 100644 --- a/src/backend/Makefile +++ b/src/backend/Makefile @@ -198,6 +198,12 @@ endif $(INSTALL_DATA) $(srcdir)/libpq/pg_hba.conf.sample '$(DESTDIR)$(datadir)/pg_hba.conf.sample' $(INSTALL_DATA) $(srcdir)/libpq/pg_ident.conf.sample '$(DESTDIR)$(datadir)/pg_ident.conf.sample' $(INSTALL_DATA) $(srcdir)/utils/misc/postgresql.conf.sample '$(DESTDIR)$(datadir)/postgresql.conf.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_aws.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_aws.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_direct.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_direct.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_passphrase.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_passphrase.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_piv_nopin.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_nopin.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_piv_pin.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_pin.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ssl_passphrase.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ssl_passphrase.sh.sample' ifeq ($(with_llvm), yes) install-bin: install-postgres-bitcode @@ -223,6 +229,7 @@ endif installdirs: $(MKDIR_P) '$(DESTDIR)$(bindir)' '$(DESTDIR)$(datadir)' + $(MKDIR_P) '$(DESTDIR)$(datadir)' '$(DESTDIR)$(datadir)/auth_commands' ifeq ($(PORTNAME), cygwin) ifeq ($(MAKE_DLL), true) $(MKDIR_P) '$(DESTDIR)$(libdir)' @@ -262,7 +269,13 @@ endif $(MAKE) -C utils uninstall-data rm -f '$(DESTDIR)$(datadir)/pg_hba.conf.sample' \ '$(DESTDIR)$(datadir)/pg_ident.conf.sample' \ - '$(DESTDIR)$(datadir)/postgresql.conf.sample' + '$(DESTDIR)$(datadir)/postgresql.conf.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_aws.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_direct.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_passphrase.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_nopin.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_pin.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ssl_passphrase.sh.sample' ifeq ($(with_llvm), yes) $(call uninstall_llvm_module,postgres) endif diff --git a/src/backend/crypto/ckey_aws.sh.sample b/src/backend/crypto/ckey_aws.sh.sample new file mode 100644 index 0000000000..d9bee53132 --- /dev/null +++ b/src/backend/crypto/ckey_aws.sh.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# This uses the AWS Secrets Manager using the AWS CLI and OpenSSL. +# This stores the AWS secret Id in $DIR. +# Do not create any file with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -ne 1 ] && echo "cluster_key_command usage: $0 \"%d\"" 1>&2 && exit 1 +# No need for %R or -R since we are not prompting + +DIR="$1" +[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1 +[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1 + +# File containing the id of the AWS secret +AWS_ID_FILE="$DIR/aws-secret.id" + + +# ---------------------------------------------------------------------- + + +# Create an AWS Secrets Manager secret? +if [ ! -e "$AWS_ID_FILE" ] +then # The 'postgres' operating system user must have permission to + # access the AWS CLI + + # The epoch-time/directory/hostname combination is unique + HASH=$(echo -n "$(date '+%s')$DIR$(hostname)" | sha1sum | cut -d' ' -f1) + AWS_SECRET_ID="Postgres-cluster-key-$HASH" + + # Use stdin to avoid passing the secret on the command line + openssl rand -hex 32 | + aws secretsmanager create-secret \ + --name "$AWS_SECRET_ID" \ + --description "Postgres cluster file encryption on $(hostname)" \ + --secret-string 'file:///dev/stdin' \ + --output text > /dev/null + if [ "$?" -ne 0 ] + then echo 'cluster key generation failed' 1>&2 + exit 1 + fi + + echo "$AWS_SECRET_ID" > "$AWS_ID_FILE" +fi + +if ! aws secretsmanager get-secret-value \ + --secret-id "$(cat "$AWS_ID_FILE")" \ + --output text +then echo 'cluster key retrieval failed' 1>&2 + exit 1 +fi | awk -F'\t' 'NR == 1 {print $4}' + +exit 0 diff --git a/src/backend/crypto/ckey_direct.sh.sample b/src/backend/crypto/ckey_direct.sh.sample new file mode 100644 index 0000000000..492defcffe --- /dev/null +++ b/src/backend/crypto/ckey_direct.sh.sample @@ -0,0 +1,39 @@ +#!/bin/sh + +# This uses a 64-character hex key supplied by the user. +# If OpenSSL is installed, you can generate a pseudo-random key by running: +# openssl rand -hex 32 +# To get a true random key, run: +# wget -q -O - 'https://www.random.org/cgi-bin/randbyte?nbytes=32&format=h' | tr -d ' \n'; echo +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 1 ] && echo "cluster_key_command usage: $0 %R [%p]" 1>&2 && exit 1 +# Supports environment variable PROMPT + +FD="$1" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 + +[ "$2" ] && PROMPT="$2" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter cluster key as 64 hexadecimal characters: ' + +stty -echo <&"$FD" + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" +read KEY <&"$FD" + +stty echo <&"$FD" + +if [ "$(expr "$KEY" : '[0-9a-fA-F]*$')" -ne 64 ] +then echo 'invalid; must be 64 hexadecimal characters' 1>&2 + exit 1 +fi + +echo "$KEY" + +exit 0 diff --git a/src/backend/crypto/ckey_passphrase.sh.sample b/src/backend/crypto/ckey_passphrase.sh.sample new file mode 100644 index 0000000000..a5d837b45e --- /dev/null +++ b/src/backend/crypto/ckey_passphrase.sh.sample @@ -0,0 +1,35 @@ +#!/bin/sh + +# This uses a passphrase supplied by the user. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 1 ] && echo "cluster_key_command usage: $0 %R [\"%p\"]" 1>&2 && exit 1 + +FD="$1" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 +# Supports environment variable PROMPT + +[ "$2" ] && PROMPT="$2" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter cluster passphrase: ' + +stty -echo <&"$FD" + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" +read PASS <&"$FD" + +stty echo <&"$FD" + +if [ ! "$PASS" ] +then echo 'invalid: empty passphrase' 1>&2 + exit 1 +fi + +echo "$PASS" | sha256sum | cut -d' ' -f1 + +exit 0 diff --git a/src/backend/crypto/ckey_piv_nopin.sh.sample b/src/backend/crypto/ckey_piv_nopin.sh.sample new file mode 100644 index 0000000000..e90a579dea --- /dev/null +++ b/src/backend/crypto/ckey_piv_nopin.sh.sample @@ -0,0 +1,68 @@ +#!/bin/sh + +# This uses the public/private keys on a PIV device, like a CAC or Yubikey. +# It uses a PIN stored in a file. +# It uses OpenSSL with PKCS11 enabled via OpenSC. +# This stores the cluster encryption key encrypted with the PIV public +# key in $DIR. This is technically a three-level encryption +# architecture, with the third level requiring the PIV and PIN. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -ne 1 ] && echo "cluster_key_command usage: $0 \"%d\"" 1>&2 && exit 1 +# Supports environment variable PIV_PIN_FILE +# No need for %R or -R since we are not prompting for a PIN + +DIR="$1" +[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1 +[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1 + +# Set these here or pass in as environment variables. +# File that stores the PIN to unlock the PIV +#PIV_PIN_FILE='' +# PIV slot 3 is the "Key Management" slot, so we use '0:3' +PIV_SLOT='0:3' + +# File containing the cluster key encrypted with the PIV_SLOT's public key +KEY_FILE="$DIR/pivpass.key" + + +# ---------------------------------------------------------------------- + +[ ! "$PIV_PIN_FILE" ] && echo 'PIV_PIN_FILE undefined' 1>&2 && exit 1 +[ ! -e "$PIV_PIN_FILE" ] && echo "$PIV_PIN_FILE does not exist" 1>&2 && exit 1 +[ -d "$PIV_PIN_FILE" ] && echo "$PIV_PIN_FILE is a directory" 1>&2 && exit 1 + +[ ! "$KEY_FILE" ] && echo 'KEY_FILE undefined' 1>&2 && exit 1 +[ -d "$KEY_FILE" ] && echo "$KEY_FILE is a directory" 1>&2 && exit 1 + +# Create a cluster key encrypted with the PIV_SLOT's public key? +if [ ! -e "$KEY_FILE" ] +then # The 'postgres' operating system user must have permission to + # access the PIV device. + + openssl rand -hex 32 | + if ! openssl rsautl -engine pkcs11 -keyform engine -encrypt \ + -inkey "$PIV_SLOT" -passin file:"$PIV_PIN_FILE" -out "$KEY_FILE" + then echo 'cluster key generation failed' 1>&2 + exit 1 + fi + + # Warn the user to save the cluster key in a safe place + cat 1>&2 <<END + +WARNING: The PIV device can be locked and require a reset if too many PIN +attempts fail. It is recommended to run this command manually and save +the cluster key in a secure location for possible recovery. +END + +fi + +# Decrypt the cluster key encrypted with the PIV_SLOT's public key +if ! openssl rsautl -engine pkcs11 -keyform engine -decrypt \ + -inkey "$PIV_SLOT" -passin file:"$PIV_PIN_FILE" -in "$KEY_FILE" +then echo 'cluster key decryption failed' 1>&2 + exit 1 +fi + +exit 0 diff --git a/src/backend/crypto/ckey_piv_pin.sh.sample b/src/backend/crypto/ckey_piv_pin.sh.sample new file mode 100644 index 0000000000..e693ac31ba --- /dev/null +++ b/src/backend/crypto/ckey_piv_pin.sh.sample @@ -0,0 +1,81 @@ +#!/bin/sh + +# This uses the public/private keys on a PIV device, like a CAC or Yubikey. +# It requires a user-entered PIN. +# It uses OpenSSL with PKCS11 enabled via OpenSC. +# This stores the cluster encryption key encrypted with the PIV public +# key in $DIR. This is technically a three-level encryption +# architecture, with the third level requiring the PIV and PIN. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 2 ] && echo "cluster_key_command usage: $0 \"%d\" %R [\"%p\"]" 1>&2 && exit 1 +# Supports environment variable PROMPT + +DIR="$1" +[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1 +[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1 + +FD="$2" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 + +[ "$3" ] && PROMPT="$3" + +# PIV slot 3 is the "Key Management" slot, so we use '0:3' +PIV_SLOT='0:3' + +# File containing the cluster key encrypted with the PIV_SLOT's public key +KEY_FILE="$DIR/pivpass.key" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter PIV PIN: ' + +stty -echo <&"$FD" + +# Create a cluster key encrypted with the PIV_SLOT's public key? +if [ ! -e "$KEY_FILE" ] +then echo 1>&"$FD" + echo -n "$PROMPT" 1>&"$FD" + + # The 'postgres' operating system user must have permission to + # access the PIV device. + + openssl rand -hex 32 | + # 'engine "pkcs11" set.' message confuses prompting + if ! openssl rsautl -engine pkcs11 -keyform engine -encrypt \ + -inkey "$PIV_SLOT" -passin fd:"$FD" -out "$KEY_FILE" 2>&1 + then stty echo <&"$FD" + echo 'cluster key generation failed' 1>&2 + exit 1 + fi | grep -v 'engine "pkcs11" set\.' + + echo 1>&"$FD" + + # Warn the user to save the cluster key in a safe place + cat 1>&"$FD" <<END + +WARNING: The PIV can be locked and require a reset if too many PIN +attempts fail. It is recommended to run this command manually and save +the cluster key in a secure location for possible recovery. +END + +fi + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" + +# Decrypt the cluster key encrypted with the PIV_SLOT's public key +if ! openssl rsautl -engine pkcs11 -keyform engine -decrypt \ + -inkey "$PIV_SLOT" -passin fd:"$FD" -in "$KEY_FILE" 2>&1 +then stty echo <&"$FD" + echo 'cluster key retrieval failed' 1>&2 + exit 1 +fi | grep -v 'engine "pkcs11" set\.' + +echo 1>&"$FD" + +stty echo <&"$FD" + +exit 0 diff --git a/src/backend/crypto/ssl_passphrase.sh.sample b/src/backend/crypto/ssl_passphrase.sh.sample new file mode 100644 index 0000000000..efbf5c0720 --- /dev/null +++ b/src/backend/crypto/ssl_passphrase.sh.sample @@ -0,0 +1,35 @@ +#!/bin/sh + +# This uses a passphrase supplied by the user. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 1 ] && echo "ssl_passphrase_command usage: $0 %R [\"%p\"]" 1>&2 && exit 1 + +FD="$1" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 +# Supports environment variable PROMPT + +[ "$2" ] && PROMPT="$2" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter cluster passphrase: ' + +stty -echo <&"$FD" + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" +read PASS <&"$FD" + +stty echo <&"$FD" + +if [ ! "$PASS" ] +then echo 'invalid: empty passphrase' 1>&2 + exit 1 +fi + +echo "$PASS" + +exit 0 -- 2.37.0 (Apple Git-136) [application/octet-stream] 0004-cfe-04-common_over_cfe-03-scripts-squash-commit.patch (32.8K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/5-0004-cfe-04-common_over_cfe-03-scripts-squash-commit.patch) download | inline diff: From 64d6a89405aebe7683e5a37feab94039a3a3c274 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:49 -0400 Subject: [PATCH 04/12] cfe-04-common_over_cfe-03-scripts squash commit --- src/common/Makefile | 3 + src/common/cipher.c | 98 +++++++ src/common/cipher_openssl.c | 419 ++++++++++++++++++++++++++++ src/common/kmgr_utils.c | 465 ++++++++++++++++++++++++++++++++ src/include/common/cipher.h | 74 +++++ src/include/common/kmgr_utils.h | 94 +++++++ src/include/utils/wait_event.h | 3 + src/tools/msvc/Mkvcbuild.pm | 4 +- 8 files changed, 1159 insertions(+), 1 deletion(-) create mode 100644 src/common/cipher.c create mode 100644 src/common/cipher_openssl.c create mode 100644 src/common/kmgr_utils.c create mode 100644 src/include/common/cipher.h create mode 100644 src/include/common/kmgr_utils.h diff --git a/src/common/Makefile b/src/common/Makefile index e9af7346c9..9f63b01609 100644 --- a/src/common/Makefile +++ b/src/common/Makefile @@ -62,6 +62,7 @@ OBJS_COMMON = \ ip.o \ jsonapi.o \ keywords.o \ + kmgr_utils.o \ kwlookup.o \ link-canary.o \ md5_common.o \ @@ -83,11 +84,13 @@ OBJS_COMMON = \ ifeq ($(with_ssl),openssl) OBJS_COMMON += \ + cipher_openssl.o \ protocol_openssl.o \ cryptohash_openssl.o \ hmac_openssl.o else OBJS_COMMON += \ + cipher.o \ cryptohash.o \ hmac.o \ md5.o \ diff --git a/src/common/cipher.c b/src/common/cipher.c new file mode 100644 index 0000000000..7aa3ea6138 --- /dev/null +++ b/src/common/cipher.c @@ -0,0 +1,98 @@ +/*------------------------------------------------------------------------- + * + * cipher.c + * Shared frontend/backend for cryptographic functions + * + * This is the set of in-core functions used when there are no other + * alternative options like OpenSSL. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/common/cipher.c + * + *------------------------------------------------------------------------- + */ + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/cipher.h" + +static void cipher_failure(void) pg_attribute_noreturn(); + + +PgCipherCtx * +pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc) +{ + cipher_failure(); + return NULL; /* keep compiler quiet */ +} + +void +pg_cipher_ctx_free(PgCipherCtx *ctx) +{ + cipher_failure(); +} + +int +pg_cipher_blocksize(PgCipherCtx *ctx) +{ + cipher_failure(); + return -1; /* keep compiler quiet */ +} + +bool +pg_cipher_encrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *plaintext, + const int inlen, unsigned char *ciphertext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *outtag, const int taglen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +bool +pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *ciphertext, + const int inlen, unsigned char *plaintext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *intag, const int taglen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +bool +pg_cipher_keywrap(PgCipherCtx *ctx, const unsigned char *plaintext, + const int inlen, unsigned char *ciphertext, int *outlen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +bool +pg_cipher_keyunwrap(PgCipherCtx *ctx, const unsigned char *ciphertext, + const int inlen, unsigned char *plaintext, int *outlen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +static void +cipher_failure(void) +{ +#ifndef FRONTEND + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"), + errhint("Compile with --with-openssl to use this feature.")))); +#else + fprintf(stderr, _("cluster file encryption is not supported because OpenSSL is not supported by this build")); + exit(1); +#endif +} diff --git a/src/common/cipher_openssl.c b/src/common/cipher_openssl.c new file mode 100644 index 0000000000..a03fbe435f --- /dev/null +++ b/src/common/cipher_openssl.c @@ -0,0 +1,419 @@ +/*------------------------------------------------------------------------- + * cipher_openssl.c + * Cryptographic function using OpenSSL + * + * This contains the common low-level functions needed in both frontend and + * backend, for implement the database encryption. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/common/cipher_openssl.c + * + *------------------------------------------------------------------------- + */ +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/cipher.h" +#include <openssl/conf.h> +#include <openssl/err.h> +#include <openssl/evp.h> +#include <openssl/ssl.h> + +/* + * prototype for the EVP functions that return an algorithm, e.g. + * EVP_aes_128_gcm(). + */ +typedef const EVP_CIPHER *(*ossl_EVP_cipher_func) (void); + +static ossl_EVP_cipher_func get_evp_aes_gcm(int klen); +static EVP_CIPHER_CTX *ossl_cipher_ctx_create(int cipher, unsigned char *key, + int klen, bool enc); + +/* + * Return a newly created cipher context. 'cipher' specifies cipher algorithm + * by identifier like PG_CIPHER_XXX. + */ +PgCipherCtx * +pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc) +{ + PgCipherCtx *ctx = NULL; + + if (cipher > PG_MAX_CIPHER_ID) + return NULL; + + ctx = ossl_cipher_ctx_create(cipher, key, klen, enc); + + return ctx; +} + +void +pg_cipher_ctx_free(PgCipherCtx *ctx) +{ + EVP_CIPHER_CTX_free(ctx); +} + +int +pg_cipher_blocksize(PgCipherCtx *ctx) +{ + Assert(ctx); + + return EVP_CIPHER_CTX_block_size(ctx); +} + +/* + * Encryption routine to encrypt data provided. + * + * ctx is the encryption context which must have been created previously. + * + * plaintext is the data we are going to encrypt + * inlen is the length of the data to encrypt + * + * ciphertext is the encrypted result + * outlen is the encrypted length + * + * iv is the IV to use. + * ivlen is the IV length to use. + * + * outtag is the resulting tag. + * taglen is the length of the tag. + */ +bool +pg_cipher_encrypt(PgCipherCtx *ctx, int cipher, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *outtag, const int taglen) +{ + int len; + int enclen; + + Assert(ctx != NULL); + + /* + * Here we are setting the IV for the context which was passed in. Note + * that we signal to OpenSSL that we are configuring a new value for the + * context by passing in 'NULL' for the 2nd ('type') parameter. + */ + + /* + * We don't use GCM mode, but it has a MAC, so we support it and test it + * in case we need it later. XXX is this correct for GCM and CTR? + */ + /* Set the GCM IV length first */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, ivlen, NULL)) + return false; + + /* Set the IV for this encryption. */ + if (!EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv)) + return false; + + /* + * This is the function which is actually performing the encryption for + * us. + */ + if (!EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, inlen)) + return false; + + enclen = len; + + /* Finalize the encryption, which could add more to output. */ + if (!EVP_EncryptFinal_ex(ctx, ciphertext + enclen, &len)) + return false; + + *outlen = enclen + len; + + /* + * Once all of the encryption has been completed we grab the tag. + */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, taglen, outtag)) + return false; + + return true; +} + +/* + * Decryption routine + * + * ctx is the encryption context which must have been created previously. + * + * ciphertext is the data we are going to decrypt + * inlen is the length of the data to decrypt + * + * plaintext is the decrypted result + * outlen is the decrypted length + * + * iv is the IV to use. + * ivlen is the length of the IV. + * + * intag is the tag to use to verify. + * taglen is the length of the tag. + */ +bool +pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *intag, const int taglen) +{ + int declen; + int len; + + /* + * Here we are setting the IV for the context which was passed in. Note + * that we signal to OpenSSL that we are configuring a new value for the + * context by passing in 'NULL' for the 2nd ('type') parameter. + */ + + /* XXX is this correct for GCM and CTR? */ + /* Set the GCM IV length first */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, ivlen, NULL)) + return false; + + /* Set the IV for this decryption. */ + if (!EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv)) + return false; + + /* + * This is the function which is actually performing the decryption for + * us. + */ + if (!EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, inlen)) + return false; + + declen = len; + + /* Set the expected tag value. */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, taglen, intag)) + return false; + + /* + * Finalize the decryption, which could add more to output, this is also + * the step which checks the tag and we MUST fail if this indicates an + * invalid tag! + */ + if (!EVP_DecryptFinal_ex(ctx, plaintext + declen, &len)) + return false; + + *outlen = declen + len; + + return true; +} + +/* + * Routine to perform key wrapping for data provided. + * + * ctx is the encryption context which must have been created previously. + * + * plaintext is the data/key we are going to encrypt/wrap + * inlen is the length of the data + * + * ciphertext is the wrapped result + * outlen is the encrypted length (will be larger than input!) + */ +bool +pg_cipher_keywrap(PgCipherCtx *ctx, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen) +{ + int len; + int enclen; + + Assert(ctx != NULL); + + /* + * This is the function which is actually performing the encryption for + * us. + */ + if (!EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, inlen)) + return false; + + enclen = len; + + /* Finalize the encryption, which could add more to output. */ + if (!EVP_EncryptFinal_ex(ctx, ciphertext + enclen, &len)) + return false; + + *outlen = enclen + len; + + return true; +} + +/* + * Routine to perform key unwrapping of the data provided. + * + * ctx is the encryption context which must have been created previously. + * + * ciphertext is the wrapped key we are going to unwrap + * inlen is the length of the data to decrypt/unwrap + * + * plaintext is the decrypted result + * outlen is the decrypted length (will be smaller than input!) + */ +bool +pg_cipher_keyunwrap(PgCipherCtx *ctx, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen) +{ + int declen; + int len; + + /* + * This is the function which is actually performing the decryption for + * us. + */ + if (!EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, inlen)) + return false; + + declen = len; + + /* + * Finalize the decryption, which could add more to output, this is also + * the step which checks the tag and we MUST fail if this indicates an + * invalid result! + */ + if (!EVP_DecryptFinal_ex(ctx, plaintext + declen, &len)) + return false; + + *outlen = declen + len; + + return true; +} + +/* + * Returns the correct GCM cipher functions for OpenSSL based + * on the key length requested. + */ +static ossl_EVP_cipher_func +get_evp_aes_gcm(int klen) +{ + switch (klen) + { + case PG_AES128_KEY_LEN: + return EVP_aes_128_gcm; + case PG_AES192_KEY_LEN: + return EVP_aes_192_gcm; + case PG_AES256_KEY_LEN: + return EVP_aes_256_gcm; + default: + return NULL; + } +} + +/* + * Returns the correct KWP cipher functions for OpenSSL based + * on the key length requested. + */ +static ossl_EVP_cipher_func +get_evp_aes_kwp(int klen) +{ + switch (klen) + { + case PG_AES128_KEY_LEN: + return EVP_aes_128_wrap_pad; + case PG_AES192_KEY_LEN: + return EVP_aes_192_wrap_pad; + case PG_AES256_KEY_LEN: + return EVP_aes_256_wrap_pad; + default: + return NULL; + } +} + +/* + * Returns the correct CTR cipher functions for OpenSSL based + * on the key length requested. + */ +static ossl_EVP_cipher_func +get_evp_aes_ctr(int klen) +{ + switch (klen) + { + case PG_AES128_KEY_LEN: + return EVP_aes_128_ctr; + case PG_AES192_KEY_LEN: + return EVP_aes_192_ctr; + case PG_AES256_KEY_LEN: + return EVP_aes_256_ctr; + default: + return NULL; + } +} + +/* + * Initialize and return an EVP_CIPHER_CTX. Returns NULL if the given + * cipher algorithm is not supported or on failure. + */ +static EVP_CIPHER_CTX * +ossl_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc) +{ + EVP_CIPHER_CTX *ctx; + ossl_EVP_cipher_func func; + int ret; + + ctx = EVP_CIPHER_CTX_new(); + + /* + * We currently only support AES GCM but others could be added in the + * future. + */ + switch (cipher) + { + case PG_CIPHER_AES_GCM: + func = get_evp_aes_gcm(klen); + if (!func) + goto failed; + break; + case PG_CIPHER_AES_KWP: + func = get_evp_aes_kwp(klen); + + /* + * Since wrapping will produce more output then input, and we have + * to be ready for that, OpenSSL requires that we explicitly + * enable wrapping for the context. + */ + EVP_CIPHER_CTX_set_flags(ctx, EVP_CIPHER_CTX_FLAG_WRAP_ALLOW); + if (!func) + goto failed; + break; + case PG_CIPHER_AES_CTR: + func = get_evp_aes_ctr(klen); + if (!func) + goto failed; + break; + default: + goto failed; + } + + /* + * We create the context here based on the cipher requested and the + * provided key. Note that the IV will be provided in the actual + * encryption call through another EVP_EncryptInit_ex call- this is fine + * as long as 'type' is passed in as NULL! + */ + if (enc) + ret = EVP_EncryptInit_ex(ctx, (const EVP_CIPHER *) func(), NULL, key, NULL); + else + ret = EVP_DecryptInit_ex(ctx, (const EVP_CIPHER *) func(), NULL, key, NULL); + + if (!ret) + goto failed; + + /* Set the key length based on the key length requested. */ + if (!EVP_CIPHER_CTX_set_key_length(ctx, klen)) + goto failed; + + return ctx; + +failed: + EVP_CIPHER_CTX_free(ctx); + return NULL; +} diff --git a/src/common/kmgr_utils.c b/src/common/kmgr_utils.c new file mode 100644 index 0000000000..6c50cb093b --- /dev/null +++ b/src/common/kmgr_utils.c @@ -0,0 +1,465 @@ +/*------------------------------------------------------------------------- + * + * kmgr_utils.c + * Shared frontend/backend for cluster file encryption + * + * These functions handle reading the wrapped DEK files from the + * file system, and wrapping and unwrapping them. It also handles + * running the cluster_key_command. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/common/kmgr_utils.c + * + *------------------------------------------------------------------------- + */ + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include <unistd.h> +#include <sys/stat.h> +#include <fcntl.h> + +#ifdef FRONTEND +#include "common/logging.h" +#endif +#include "common/cryptohash.h" +#include "common/file_perm.h" +#include "common/hex.h" +#include "common/string.h" +#include "crypto/kmgr.h" +#include "lib/stringinfo.h" +#include "storage/fd.h" + +#ifndef FRONTEND +#include "pgstat.h" +#include "storage/fd.h" +#endif + +#define KMGR_PROMPT_MSG "Enter authentication needed to generate the cluster key: " + +#ifdef FRONTEND +static FILE *open_pipe_stream(const char *command); +static int close_pipe_stream(FILE *file); +#endif + +static void read_wrapped_data_key(const char *cryptoKeyDir, uint32 id, unsigned char **key_p, int *key_len); + +/* + * We need this array in frontend and backend code, so define it here. + * You can only add to the end of this array since the array index is + * stored in pg_control. + */ +encryption_method encryption_methods[NUM_ENCRYPTION_METHODS] = { + {"", 0}, + {"AES128", 128}, + {"AES192", 192}, + {"AES256", 256} +}; + +/* This maps wrapped key filesnames to their array slots */ +char *wkey_filenames[KMGR_NUM_DATA_KEYS] = { + "relation", + "wal" +}; + + +/* + * Wrap the given CryptoKey. + * + * Returns true and writes encrypted/wrapped/padded data to 'out', and the length + * of the result to outlen, if successful. + * + * Otherwise returns false. The caller must allocate sufficient space + * for cipher data calculated by using KmgrSizeOfCipherText(). Please note that + * this function modifies 'out' data even on failure. + */ +bool +kmgr_wrap_data_key(PgCipherCtx *ctx, CryptoKey *in, unsigned char *out, int *outlen) +{ + Assert(ctx && in && out); + + if (!pg_cipher_keywrap(ctx, (unsigned char *) in, sizeof(CryptoKey), out, outlen)) + return false; + + return true; +} + +/* + * Decrypt the given data. Return true and set plain text data to `out` if + * successful. Otherwise return false. The caller must allocate sufficient + * space for cipher data calculated by using KmgrSizeOfPlainText(). Please + * note that this function modifies 'out' data even on failure. + */ +bool +kmgr_unwrap_data_key(PgCipherCtx *ctx, unsigned char *in, int inlen, CryptoKey *out) +{ + int outlen; + + Assert(ctx && in && out); + + if (!pg_cipher_keyunwrap(ctx, in, inlen, (unsigned char *) out, &outlen)) + return false; + + Assert(outlen == sizeof(CryptoKey)); + + return true; +} + +/* + * Verify the correctness of the given cluster key by unwrapping the given keys. + * If the given cluster key is correct we set unwrapped keys to out_keys and return + * true. Otherwise return false. Please note that this function changes the + * contents of out_keys even on failure. Both in_keys and out_keys must be the + * same length. + */ +bool +kmgr_verify_cluster_key(unsigned char *cluster_key, + unsigned char **in_keys, int *key_lens, CryptoKey *out_keys) +{ + PgCipherCtx *ctx; + + /* Create decryption context with the KEK. */ + ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, cluster_key, + KMGR_CLUSTER_KEY_LEN, false); + + /* unwrap each DEK */ + for (int i = 0; i < KMGR_NUM_DATA_KEYS; i++) + { + if (!kmgr_unwrap_data_key(ctx, in_keys[i], key_lens[i], &(out_keys[i]))) + { + /* The cluster key is not correct */ + pg_cipher_ctx_free(ctx); + return false; + } + explicit_bzero(in_keys[i], key_lens[i]); + } + + /* The cluster key is correct, free the cipher context */ + pg_cipher_ctx_free(ctx); + + return true; +} + +/* + * Run cluster key command. + * + * Substitute %d for directory, %p for prompt, %R for file descriptor. + * + * The result will be put in buffer buf, which is of size "size". + * The return value is the length of the actual result. + */ +int +kmgr_run_cluster_key_command(char *cluster_key_command, char *buf, + int size, char *dir, int terminal_fd) +{ + StringInfoData command; + const char *sp; + FILE *fh; + int pclose_rc; + size_t len = 0; + + buf[0] = '\0'; + + Assert(size > 0); + + /* + * Build the command to be executed. + */ + initStringInfo(&command); + + for (sp = cluster_key_command; *sp; sp++) + { + if (*sp == '%') + { + switch (sp[1]) + { + /* directory */ + case 'd': + { + char *nativePath; + + sp++; + + /* + * This needs to use a placeholder to not modify the + * input with the conversion done via + * make_native_path(). + */ + nativePath = pstrdup(dir); + make_native_path(nativePath); + appendStringInfoString(&command, nativePath); + pfree(nativePath); + break; + } + /* prompt string */ + case 'p': + sp++; + appendStringInfoString(&command, KMGR_PROMPT_MSG); + break; + /* file descriptor number */ + case 'R': + { + char fd_str[20]; + + if (terminal_fd == -1) + { +#ifdef FRONTEND + pg_fatal("cluster key command referenced %%R, but --authprompt not specified"); +#else + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("cluster key command referenced %%R, but --authprompt not specified"))); +#endif + } + + sp++; + snprintf(fd_str, sizeof(fd_str), "%d", terminal_fd); + appendStringInfoString(&command, fd_str); + break; + } + /* literal "%" */ + case '%': + /* convert %% to a single % */ + sp++; + appendStringInfoChar(&command, *sp); + break; + default: + /* otherwise treat the % as not special */ + appendStringInfoChar(&command, *sp); + break; + } + } + else + { + appendStringInfoChar(&command, *sp); + } + } + +#ifdef FRONTEND + fh = open_pipe_stream(command.data); + if (fh == NULL) + { + pg_fatal("could not execute command \"%s\": %m", + command.data); + } +#else + fh = OpenPipeStream(command.data, "r"); + if (fh == NULL) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not execute command \"%s\": %m", + command.data))); +#endif + + if (!fgets(buf, size, fh)) + { + if (ferror(fh)) + { +#ifdef FRONTEND + pg_fatal("could not read from command \"%s\": %m", + command.data); +#else + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read from command \"%s\": %m", + command.data))); +#endif + } + } + +#ifdef FRONTEND + pclose_rc = close_pipe_stream(fh); +#else + pclose_rc = ClosePipeStream(fh); +#endif + + if (pclose_rc == -1) + { +#ifdef FRONTEND + pg_fatal("could not close pipe to external command: %m"); +#else + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close pipe to external command: %m"))); +#endif + } + else if (pclose_rc != 0) + { +#ifdef FRONTEND + pg_fatal("command \"%s\" failed", command.data); +#else + ereport(ERROR, + (errcode_for_file_access(), + errmsg("command \"%s\" failed", + command.data), + errdetail_internal("%s", wait_result_to_str(pclose_rc)))); +#endif + } + + /* strip trailing newline and carriage returns */ + len = pg_strip_crlf(buf); + + pfree(command.data); + + return len; +} + +#ifdef FRONTEND +static FILE * +open_pipe_stream(const char *command) +{ + FILE *res; + +#ifdef WIN32 + size_t cmdlen = strlen(command); + char *buf; + int save_errno; + + buf = malloc(cmdlen + 2 + 1); + if (buf == NULL) + { + errno = ENOMEM; + return NULL; + } + buf[0] = '"'; + memcpy(&buf[1], command, cmdlen); + buf[cmdlen + 1] = '"'; + buf[cmdlen + 2] = '\0'; + + res = _popen(buf, "r"); + + save_errno = errno; + free(buf); + errno = save_errno; +#else + res = popen(command, "r"); +#endif /* WIN32 */ + return res; +} + +static int +close_pipe_stream(FILE *file) +{ +#ifdef WIN32 + return _pclose(file); +#else + return pclose(file); +#endif /* WIN32 */ +} +#endif /* FRONTEND */ + +/* + * Reads the keys at path. + * + * This routine simply reads in the raw encrypted/wrapped keys; + * it does not handle any decryption, see kmgr_key_unwrap(). + * + * For each key returned, the key and key length are returned + * in the keys and key_lens arrays respectfully. + * + * Note that keys and key_lens must be allocated before calling + * this function as arrays of at least KMGR_NUM_DATA_KEYS length. + */ +void +kmgr_read_wrapped_data_keys(const char *path, unsigned char **keys, int *key_lens) +{ +/* StaticAssertStmt(lengthof(wkey_filenames) == KMGR_NUM_DATA_KEYS, + "wkey_filenames[] must match KMGR_NUM_DATA_KEYS"); +*/ + StaticAssertStmt(1 == 1, + "wkey_filenames[] must match KMGR_NUM_DATA_KEYS"); + + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + read_wrapped_data_key(path, id, &(keys[id]), &(key_lens[id])); + + return; +} + +/* Read a wrapped DEK file */ +static void +read_wrapped_data_key(const char *cryptoKeyDir, uint32 id, unsigned char **key_p, int *key_len) +{ + char path[MAXPGPATH]; + int fd; + int r; + struct stat st; + + CryptoKeyFilePath(path, cryptoKeyDir, id); + +#ifndef FRONTEND + if ((fd = OpenTransientFile(path, O_RDONLY | PG_BINARY)) == -1) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\" for reading: %m", + path))); + else if (fstat(fd, &st)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + path))); +#else + if ((fd = open(path, O_RDONLY | PG_BINARY, 0)) == -1) + pg_fatal("could not open file \"%s\" for reading: %m", + path); + else if (fstat(fd, &st)) + pg_fatal("could not stat file \"%s\": %m", + path); +#endif + + *key_len = st.st_size; + +#ifndef FRONTEND + pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_READ); +#endif + + *key_p = (unsigned char *) palloc0(*key_len); + + /* Get key bytes */ + r = read(fd, *key_p, *key_len); + if (r != *key_len) + { + if (r < 0) + { +#ifndef FRONTEND + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", path))); +#else + pg_fatal("could not read file \"%s\": %m", path); +#endif + } + else + { +#ifndef FRONTEND + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("could not read file \"%s\": read %d of %u", + path, r, *key_len))); +#else + pg_fatal("could not read file \"%s\": read %d of %u", + path, r, *key_len); +#endif + } + } + +#ifndef FRONTEND + pgstat_report_wait_end(); +#endif + +#ifndef FRONTEND + if (CloseTransientFile(fd) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", + path))); +#else + if (close(fd) != 0) + pg_fatal("could not close file \"%s\": %m", path); +#endif +} diff --git a/src/include/common/cipher.h b/src/include/common/cipher.h new file mode 100644 index 0000000000..da97ca3776 --- /dev/null +++ b/src/include/common/cipher.h @@ -0,0 +1,74 @@ +/*------------------------------------------------------------------------- + * + * cipher.h + * Declarations for cryptographic functions + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/common/cipher.h + * + *------------------------------------------------------------------------- + */ +#ifndef PG_CIPHER_H +#define PG_CIPHER_H + +#ifdef USE_OPENSSL +#include <openssl/evp.h> +#include <openssl/conf.h> +#include <openssl/err.h> +#endif + +/* + * Supported symmetric encryption algorithm. These identifiers are passed + * to pg_cipher_ctx_create() function, and then actual encryption + * implementations need to initialize their context of the given encryption + * algorithm. + */ +#define PG_CIPHER_AES_GCM 0 +#define PG_CIPHER_AES_CTR 1 +#define PG_CIPHER_AES_KWP 2 +#define PG_MAX_CIPHER_ID 2 + +/* AES128/192/256 various length definitions */ +#define PG_AES128_KEY_LEN (128 / 8) +#define PG_AES192_KEY_LEN (192 / 8) +#define PG_AES256_KEY_LEN (256 / 8) + +/* + * The encrypted data is a series of blocks of size. Initialization + * vector(IV) is the same size of cipher block. + */ +#define PG_AES_BLOCK_SIZE 16 +#define PG_AES_IV_SIZE (PG_AES_BLOCK_SIZE) + +#ifdef USE_OPENSSL +typedef EVP_CIPHER_CTX PgCipherCtx; +#else +typedef void PgCipherCtx; +#endif + +extern PgCipherCtx *pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, + bool enc); + +extern void pg_cipher_ctx_free(PgCipherCtx *ctx); +extern bool pg_cipher_encrypt(PgCipherCtx *ctx, int cipher, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *tag, const int taglen); +extern bool pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *intag, const int taglen); + +extern bool pg_cipher_keywrap(PgCipherCtx *ctx, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen); +extern bool pg_cipher_keyunwrap(PgCipherCtx *ctx, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen); + +extern int pg_cipher_blocksize(PgCipherCtx *ctx); + +#endif /* PG_CIPHER_H */ diff --git a/src/include/common/kmgr_utils.h b/src/include/common/kmgr_utils.h new file mode 100644 index 0000000000..ca264db728 --- /dev/null +++ b/src/include/common/kmgr_utils.h @@ -0,0 +1,94 @@ +/*------------------------------------------------------------------------- + * + * kmgr_utils.h + * Declarations for utility function for file encryption key + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/common/kmgr_utils.h + * + *------------------------------------------------------------------------- + */ +#ifndef KMGR_UTILS_H +#define KMGR_UTILS_H + +#include "common/cipher.h" + +/* Current version number */ +#define KMGR_VERSION 1 + +/* + * Directories where cluster file encryption keys reside within PGDATA. + */ +#define KMGR_DIR "pg_cryptokeys" +#define KMGR_DIR_PID KMGR_DIR"/pg_alterckey.pid" +#define LIVE_KMGR_DIR KMGR_DIR"/live" +/* used during cluster key rotation */ +#define NEW_KMGR_DIR KMGR_DIR"/new" +#define OLD_KMGR_DIR KMGR_DIR"/old" + +/* CryptoKey file name is keys id */ +#define CryptoKeyFilePath(path, dir, id) \ + snprintf((path), MAXPGPATH, "%s/%s.wkey", (dir), (wkey_filenames[id])) + +/* + * Identifiers of internal keys. + */ +#define KMGR_KEY_ID_REL 0 +#define KMGR_KEY_ID_WAL 1 +#define KMGR_NUM_DATA_KEYS 2 + +/* We always, today, use a 256-bit AES key. */ +#define KMGR_CLUSTER_KEY_LEN PG_AES256_KEY_LEN + +/* double for hex format, plus some for spaces, \r,\n, and null byte */ +#define ALLOC_KMGR_CLUSTER_KEY_LEN (KMGR_CLUSTER_KEY_LEN * 2 + 10 + 2 + 1) + +/* Maximum length of key the key manager can store */ +#define KMGR_MAX_KEY_LEN 256 +#define KMGR_MAX_KEY_LEN_BYTES KMGR_MAX_KEY_LEN / 8 + + +/* + * Cryptographic key data structure. + * + * This is the structure we use to write out the encrypted keys and + * which we use to store the keys in shared memory. + * + * Note that wrapping this structure results in an encrypted byte + * string which is what we actually write and then read back in. + * + * klen is the key length in bytes + * key is the encryption key of klen length + */ +typedef struct CryptoKey +{ + int klen; /* key length in bytes */ + unsigned char key[KMGR_MAX_KEY_LEN_BYTES]; +} CryptoKey; + +/* Encryption method array */ +typedef struct encryption_method +{ + const char *name; + const int bit_length; +} encryption_method; + +#define NUM_ENCRYPTION_METHODS 4 +#define DISABLED_ENCRYPTION_METHOD 0 +#define DEFAULT_ENABLED_ENCRYPTION_METHOD 1 + +extern encryption_method encryption_methods[NUM_ENCRYPTION_METHODS]; +extern char *wkey_filenames[KMGR_NUM_DATA_KEYS]; + +extern bool kmgr_wrap_data_key(PgCipherCtx *ctx, CryptoKey *in, unsigned char *out, int *outlen); +extern bool kmgr_unwrap_data_key(PgCipherCtx *ctx, unsigned char *in, int inlen, CryptoKey *out); +extern bool kmgr_verify_cluster_key(unsigned char *cluster_key, + unsigned char **in_keys, int *klens, CryptoKey *out_keys); +extern int kmgr_run_cluster_key_command(char *cluster_key_command, + char *buf, int size, char *dir, + int terminal_fd); +extern void kmgr_read_wrapped_data_keys(const char *path, unsigned char **keys, + int *key_lens); + +#endif /* KMGR_UTILS_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index 6f2d5612e0..4b049f2ab4 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -180,6 +180,9 @@ typedef enum WAIT_EVENT_DATA_FILE_WRITE, WAIT_EVENT_DSM_ALLOCATE, WAIT_EVENT_DSM_FILL_ZERO_WRITE, + WAIT_EVENT_KEY_FILE_READ, + WAIT_EVENT_KEY_FILE_WRITE, + WAIT_EVENT_KEY_FILE_SYNC, WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ, WAIT_EVENT_LOCK_FILE_ADDTODATADIR_SYNC, WAIT_EVENT_LOCK_FILE_ADDTODATADIR_WRITE, diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm index 83a3e40425..7c448183f4 100644 --- a/src/tools/msvc/Mkvcbuild.pm +++ b/src/tools/msvc/Mkvcbuild.pm @@ -136,19 +136,21 @@ sub mkvcbuild archive.c base64.c checksum_helper.c compression.c config_info.c controldata_utils.c d2s.c encnames.c exec.c f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c - keywords.c kwlookup.c link-canary.c md5_common.c + keywords.c kmgr_utils.c kwlookup.c link-canary.c md5_common.c pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c username.c wait_error.c wchar.c); if ($solution->{options}->{openssl}) { + push(@pgcommonallfiles, 'cipher_openssl.c'); push(@pgcommonallfiles, 'cryptohash_openssl.c'); push(@pgcommonallfiles, 'hmac_openssl.c'); push(@pgcommonallfiles, 'protocol_openssl.c'); } else { + push(@pgcommonallfiles, 'cipher.c'); push(@pgcommonallfiles, 'cryptohash.c'); push(@pgcommonallfiles, 'hmac.c'); push(@pgcommonallfiles, 'md5.c'); -- 2.37.0 (Apple Git-136) [application/octet-stream] 0005-cfe-05-crypto_over_cfe-04-common-squash-commit.patch (23.8K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/6-0005-cfe-05-crypto_over_cfe-04-common-squash-commit.patch) download | inline diff: From 41c49693ab4df45d40ebdd62a227e6732e7fec9a Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:51 -0400 Subject: [PATCH 05/12] cfe-05-crypto_over_cfe-04-common squash commit --- doc/src/sgml/config.sgml | 120 ++++++++-- src/backend/crypto/Makefile | 18 ++ src/backend/crypto/kmgr.c | 438 ++++++++++++++++++++++++++++++++++++ src/common/kmgr_utils.c | 3 +- src/include/crypto/kmgr.h | 25 ++ 5 files changed, 588 insertions(+), 16 deletions(-) create mode 100644 src/backend/crypto/Makefile create mode 100644 src/backend/crypto/kmgr.c create mode 100644 src/include/crypto/kmgr.h diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 6c649336e1..69ee55a568 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1558,19 +1558,31 @@ include_dir 'conf.d' mechanism is used. </para> <para> - The command must print the passphrase to the standard output and exit - with code 0. In the parameter value, <literal>%p</literal> is - replaced by a prompt string. (Write <literal>%%</literal> for a - literal <literal>%</literal>.) Note that the prompt string will - probably contain whitespace, so be sure to quote adequately. A single - newline is stripped from the end of the output if present. + The command must print the passphrase to the standard output + and exit with code 0. It can prompt from the terminal if + <option>--authprompt</option> is used. In the parameter + value, <literal>%R</literal> is replaced by a file descriptor + number opened to the terminal that started the server. A file + descriptor is only available if enabled at server start via + <option>-R</option>. If <literal>%R</literal> is specified and + no file descriptor is available, the server will not start. Value + <literal>%p</literal> is replaced by a pre-defined prompt string. + (Write <literal>%%</literal> for a literal <literal>%</literal>.) + Note that the prompt string will probably contain whitespace, + so be sure to quote its use adequately. Newlines are stripped + from the end of the output if present. </para> + <para> - The command does not actually have to prompt the user for a - passphrase. It can read it from a file, obtain it from a keychain - facility, or similar. It is up to the user to make sure the chosen - mechanism is adequately secure. + Sample scripts can be found in + <filename>$SHAREDIR/auth_commands</filename>, + where <literal>$SHAREDIR</literal> means the + <productname>PostgreSQL</productname> installation's shared-data + directory, often <filename>/usr/local/share/postgresql</filename> + (use <command>pg_config --sharedir</command> to determine it if + you're not sure). </para> + <para> This parameter can only be set in the <filename>postgresql.conf</filename> file or on the server command line. @@ -1592,10 +1604,12 @@ include_dir 'conf.d' parameter is off (the default), then <varname>ssl_passphrase_command</varname> will be ignored during a reload and the SSL configuration will not be reloaded if a passphrase - is needed. That setting is appropriate for a command that requires a - TTY for prompting, which might not be available when the server is - running. Setting this parameter to on might be appropriate if the - passphrase is obtained from a file, for example. + is needed. This setting is appropriate for a command that requires a + terminal for prompting, which will likely not be available when the server is + running. (<option>--authprompt</option> closes the terminal file + descriptor soon after server start.) Setting this parameter on + might be appropriate, for example, if the passphrase is obtained + from a file. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> @@ -2794,7 +2808,9 @@ include_dir 'conf.d' <varname>max_wal_senders</varname> is non-zero. Note that changing <varname>wal_level</varname> to <literal>minimal</literal> makes previous base backups unusable - for point-in-time recovery and standby servers. + for point-in-time recovery and standby servers, which may + lead to data loss. Cluster file encryption also does not support + <varname>wal_level</varname> <literal>minimal</literal>. </para> <para> In <literal>logical</literal> level, the same information is logged as @@ -8483,6 +8499,64 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; </variablelist> </sect1> + <sect1 id="runtime-config-encryption"> + <title>Cluster File Encryption</title> + + <variablelist> + <varlistentry id="guc-cluster-key-command" xreflabel="cluster_key_command"> + <term><varname>cluster_key_command</varname> (<type>string</type>) + <indexterm> + <primary><varname>cluster_key_command</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + This option specifies an external command to obtain the cluster-level + key for cluster file encryption during server initialization and + server start. + </para> + <para> + The command must print the cluster key to the standard + output as 64 hexadecimal characters, and exit with code 0. + The command can prompt for the passphrase or PIN from the + terminal if <option>--authprompt</option> is used. In the + parameter value, <literal>%R</literal> is replaced by a file + descriptor number opened to the terminal that started the server. + A file descriptor is only available if enabled at server start + via <option>-R</option>. If <literal>%R</literal> is specified + and no file descriptor is available, the server will not start. + Value <literal>%p</literal> is replaced by a pre-defined + prompt string. Value <literal>%d</literal> is replaced by the + directory containing the keys; this is useful if the command + must create files with the keys, e.g., to store a cluster-level + key encrypted by a key stored in a hardware security module. + (Write <literal>%%</literal> for a literal <literal>%</literal>.) + Note that the prompt string will probably contain whitespace, + so be sure to quote its use adequately. Newlines are stripped + from the end of the output if present. + </para> + + <para> + Sample script can be found in + <filename>$SHAREDIR/auth_commands</filename>, + where <literal>$SHAREDIR</literal> means the + <productname>PostgreSQL</productname> installation's shared-data + directory, often <filename>/usr/local/share/postgresql</filename> + (use <command>pg_config --sharedir</command> to determine it if + you're not sure). + </para> + + <para> + This parameter can only be set by + <application>initdb</application>, in the + <filename>postgresql.conf</filename> file, or on the server + command line. + </para> + </listitem> + </varlistentry> + </variablelist> + </sect1> + <sect1 id="runtime-config-client"> <title>Client Connection Defaults</title> @@ -10508,6 +10582,22 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' </listitem> </varlistentry> + <varlistentry id="guc-file-encryption-method" xreflabel="file_encryption_method"> + <term><varname>file_encryption_method</varname> (<type>boolean</type>) + <indexterm> + <primary>Cluster file encryption method</primary> + </indexterm> + </term> + <listitem> + <para> + Reports the cluster file + encryption method. See <xref + linkend="app-initdb-cluster-key-command"/> for more + information. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-data-directory-mode" xreflabel="data_directory_mode"> <term><varname>data_directory_mode</varname> (<type>integer</type>) <indexterm> diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile new file mode 100644 index 0000000000..c27362029d --- /dev/null +++ b/src/backend/crypto/Makefile @@ -0,0 +1,18 @@ +#------------------------------------------------------------------------- +# +# Makefile +# Makefile for src/backend/crypto +# +# IDENTIFICATION +# src/backend/crypto/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/crypto +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = \ + kmgr.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c new file mode 100644 index 0000000000..4b74961323 --- /dev/null +++ b/src/backend/crypto/kmgr.c @@ -0,0 +1,438 @@ +/*------------------------------------------------------------------------- + * + * kmgr.c + * Cluster file encryption routines + * + * Cluster file encryption is enabled if user requests it during initdb. + * During bootstrap, we generate data encryption keys, wrap them with the + * cluster-level key, and store them into each file located at KMGR_DIR. + * During startup, we decrypt all internal keys and load them to the shared + * memory. Internal keys in the shared memory are read-only. The wrapping + * and unwrapping key routines require the OpenSSL library. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/crypto/kmgr.c + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include <sys/stat.h> +#include <unistd.h> + +#include "funcapi.h" +#include "miscadmin.h" +#include "pgstat.h" + +#include "access/xlog.h" +#include "common/file_perm.h" +#include "common/kmgr_utils.h" +#include "common/sha2.h" +#include "access/xlog.h" +#include "common/controldata_utils.h" +#include "crypto/kmgr.h" +#include "postmaster/postmaster.h" +#include "storage/copydir.h" +#include "storage/fd.h" +#include "storage/ipc.h" +#include "storage/shmem.h" +#include "utils/builtins.h" +#include "utils/memutils.h" +/* Struct stores file encryption keys in plaintext format */ +typedef struct KmgrShmemData +{ + CryptoKey intlKeys[KMGR_NUM_DATA_KEYS]; +} KmgrShmemData; +static KmgrShmemData *KmgrShmem; + +/* GUC variables */ +char *cluster_key_command = NULL; + +CryptoKey bootstrap_keys[KMGR_NUM_DATA_KEYS]; + +extern char *bootstrap_old_key_datadir; +extern int bootstrap_file_encryption_method; + +static void bzeroKmgrKeys(int status, Datum arg); +static void KmgrWriteCryptoKeys(const char *dir, unsigned char **keys, int *key_lens); +static CryptoKey *generate_crypto_key(int len); + +/* + * This function must be called ONCE during initdb. It creates the DEK + * files wrapped with the KEK supplied by kmgr_run_cluster_key_command(). + * There is also an option for the keys to be copied from another cluster. + */ +void +BootStrapKmgr(void) +{ + char live_path[MAXPGPATH]; + unsigned char *keys_wrap[KMGR_NUM_DATA_KEYS]; + int key_lens[KMGR_NUM_DATA_KEYS]; + char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN]; + int cluster_key_hex_len; + unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN]; + + if (!FileEncryptionEnabled) + return; + +#ifndef USE_OPENSSL + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"), + errhint("Compile with --with-openssl to use this feature.")))); +#endif + + /* + * There are too many optimizations for wal_level=minimal that don't set + * LSNs for permanent tables, so just disallow it. + */ + if (!XLogIsNeeded()) + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported with a wal_level of \"minimal\"")))); + + snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR); + + /* + * Copy cluster file encryption keys from an old cluster? This is useful + * for pg_upgrade upgrades where the copied database files are already + * encrypted using the old cluster's DEK keys. + */ + if (bootstrap_old_key_datadir != NULL) + { + char old_key_dir[MAXPGPATH]; + + snprintf(old_key_dir, sizeof(old_key_dir), "%s/%s", + bootstrap_old_key_datadir, LIVE_KMGR_DIR); + copydir(old_key_dir, LIVE_KMGR_DIR, true); + } + /* create an empty directory */ + else + { + if (mkdir(LIVE_KMGR_DIR, pg_dir_create_mode) < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not create cluster file encryption directory \"%s\": %m", + LIVE_KMGR_DIR))); + } + + /* + * Get key encryption key (KEK) from the cluster_key command. The cluster + * key command might need to check for the existence of files in the live + * directory, e.g., PIV, so run this _after_ copying the directory in + * place. + */ + cluster_key_hex_len = kmgr_run_cluster_key_command(cluster_key_command, + cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + live_path, terminal_fd); + + /* decode supplied hex */ + if (hex_decode(cluster_key_hex, cluster_key_hex_len, + (char *) cluster_key) != + KMGR_CLUSTER_KEY_LEN) + ereport(ERROR, + (errmsg("cluster key must be %d hexadecimal characters", + KMGR_CLUSTER_KEY_LEN * 2))); + + /* We are not in copy mode? Generate new cluster file encryption keys. */ + if (bootstrap_old_key_datadir == NULL) + { + unsigned char *bootstrap_keys_wrap[KMGR_NUM_DATA_KEYS]; + int key_lens[KMGR_NUM_DATA_KEYS]; + PgCipherCtx *cluster_key_ctx; + + /* Create KEK encryption context */ + cluster_key_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, cluster_key, + KMGR_CLUSTER_KEY_LEN, true); + if (!cluster_key_ctx) + elog(ERROR, "could not initialize encryption context"); + + /* Wrap data encryption keys (DEK) using the key encryption key (KEK) */ + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + { + CryptoKey *key; + + /* generate a DEK */ + key = generate_crypto_key( + encryption_methods[bootstrap_file_encryption_method].bit_length / 8); + + /* output generated random string as hex, for testing */ + { + char str[MAXPGPATH]; + int out_len; + + out_len = hex_encode((char *) (key->key), key->klen, + str); + str[out_len] = '\0'; + } + + bootstrap_keys_wrap[id] = palloc0(KMGR_MAX_KEY_LEN_BYTES + + pg_cipher_blocksize(cluster_key_ctx)); + + /* wrap DEK with KEK */ + if (!kmgr_wrap_data_key(cluster_key_ctx, key, bootstrap_keys_wrap[id], &(key_lens[id]))) + { + pg_cipher_ctx_free(cluster_key_ctx); + elog(ERROR, "failed to wrap data encryption key"); + } + + /* remove DEK from memory */ + explicit_bzero(key, sizeof(CryptoKey)); + } + + /* Write data encryption keys to the disk */ + KmgrWriteCryptoKeys(LIVE_KMGR_DIR, bootstrap_keys_wrap, key_lens); + + pg_cipher_ctx_free(cluster_key_ctx); + } + + /* + * We are either decrypting keys we copied from an old cluster, or + * decrypting keys we just wrote above --- either way, we decrypt them + * here and store them in a file-scoped variable for use in later + * encrypting during bootstrap mode. + */ + + /* Get the crypto keys from the live directory */ + kmgr_read_wrapped_data_keys(LIVE_KMGR_DIR, keys_wrap, key_lens); + + if (!kmgr_verify_cluster_key(cluster_key, keys_wrap, key_lens, bootstrap_keys)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("supplied cluster key does not match expected cluster_key"))); + + /* bzero DEK on exit */ + on_proc_exit(bzeroKmgrKeys, 0); + + /* bzero KEK */ + explicit_bzero(cluster_key_hex, cluster_key_hex_len); + explicit_bzero(cluster_key, KMGR_CLUSTER_KEY_LEN); +} + +/* Report shared-memory space needed by KmgrShmem */ +Size +KmgrShmemSize(void) +{ + if (!FileEncryptionEnabled) + return 0; + + return MAXALIGN(sizeof(KmgrShmemData)); +} + +/* Allocate and initialize key manager memory */ +void +KmgrShmemInit(void) +{ + bool found; + + if (!FileEncryptionEnabled) + return; + + KmgrShmem = (KmgrShmemData *) ShmemInitStruct("File encryption key manager", + KmgrShmemSize(), &found); + + /* bzero DEK on exit */ + on_shmem_exit(bzeroKmgrKeys, 0); +} + +/* + * Get cluster key and verify it, then get the data encryption keys. + * This function is called by postmaster at startup time. + */ +void +InitializeKmgr(void) +{ + unsigned char *keys_wrap[KMGR_NUM_DATA_KEYS]; + int key_lens[KMGR_NUM_DATA_KEYS]; + char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN]; + int cluster_key_hex_len; + struct stat buffer; + char live_path[MAXPGPATH]; + unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN]; + +#ifndef USE_OPENSSL + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"), + errhint("Compile with --with-openssl to use this feature.")))); +#endif + + /* + * There are too many optimizations for wal_level=minimal that don't set + * LSNs for permanent tables, so just disallow it. + */ + if (!XLogIsNeeded()) + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported with a wal_level of \"minimal\"")))); + + elog(DEBUG1, "starting up cluster file encryption manager"); + + if (stat(KMGR_DIR, &buffer) != 0 || !S_ISDIR(buffer.st_mode)) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster file encryption directory %s is missing", KMGR_DIR)))); + + if (stat(KMGR_DIR_PID, &buffer) == 0) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster had a pg_alterckey failure that needs repair or pg_alterckey is running"), + errhint("Run pg_alterckey --repair or wait for it to complete.")))); + + /* + * We want OLD deleted since it allows access to the data encryption keys + * using the old cluster key. If NEW exists, it means either the new + * directory is partly written, or NEW wasn't renamed to LIVE --- in + * either case, it needs to be repaired. See src/bin/pg_alterckey/README + * for more details. + */ + if (stat(OLD_KMGR_DIR, &buffer) == 0 || stat(NEW_KMGR_DIR, &buffer) == 0) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster had a pg_alterckey failure that needs repair"), + errhint("Run pg_alterckey --repair.")))); + + /* If OLD, NEW, and LIVE do not exist, there is a serious problem. */ + if (stat(LIVE_KMGR_DIR, &buffer) != 0 || !S_ISDIR(buffer.st_mode)) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster has no data encryption keys")))); + + /* Get the cluster key (KEK) */ + snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR); + cluster_key_hex_len = kmgr_run_cluster_key_command(cluster_key_command, + cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + live_path, terminal_fd); + + /* decode supplied hex */ + if (hex_decode(cluster_key_hex, cluster_key_hex_len, + (char *) cluster_key) != + KMGR_CLUSTER_KEY_LEN) + ereport(ERROR, + (errmsg("cluster key must be %d hexadecimal characters", + KMGR_CLUSTER_KEY_LEN * 2))); + + /* Load wrapped DEKs from their files into an array */ + kmgr_read_wrapped_data_keys(LIVE_KMGR_DIR, keys_wrap, key_lens); + + /* + * Verify cluster key and store the unwrapped data encryption keys in + * shared memory. + */ + if (!kmgr_verify_cluster_key(cluster_key, keys_wrap, key_lens, KmgrShmem->intlKeys)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("supplied cluster key does not match expected cluster key"))); + + /* Check that retrieved key lengths match controldata length. */ + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + if (KmgrShmem->intlKeys[id].klen * 8 != + encryption_methods[GetFileEncryptionMethod()].bit_length) + { + char path[MAXPGPATH]; + + CryptoKeyFilePath(path, DataDir, id); + + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("data encryption key %s of length %d does not match controldata key length %d", + path, KmgrShmem->intlKeys[id].klen * 8, + encryption_methods[GetFileEncryptionMethod()].bit_length))); + } + + /* bzero KEK */ + explicit_bzero(cluster_key_hex, cluster_key_hex_len); + explicit_bzero(cluster_key, KMGR_CLUSTER_KEY_LEN); +} + +static void +bzeroKmgrKeys(int status, Datum arg) +{ + if (IsBootstrapProcessingMode()) + explicit_bzero(bootstrap_keys, sizeof(bootstrap_keys)); + else + explicit_bzero(KmgrShmem->intlKeys, sizeof(KmgrShmem->intlKeys)); +} + +/* return requested DEK */ +const CryptoKey * +KmgrGetKey(int id) +{ + Assert(id < KMGR_NUM_DATA_KEYS); + + return (const CryptoKey *) (IsBootstrapProcessingMode() ? + &(bootstrap_keys[id]) : &(KmgrShmem->intlKeys[id])); +} + +/* Generate a DEK inside a CryptoKey */ +static CryptoKey * +generate_crypto_key(int len) +{ + CryptoKey *newkey; + + Assert(len <= KMGR_MAX_KEY_LEN); + newkey = (CryptoKey *) palloc0(sizeof(CryptoKey)); + + newkey->klen = len; + + if (!pg_strong_random(newkey->key, len)) + elog(ERROR, "failed to generate new file encryption key"); + + return newkey; +} + +/* + * Write the DEKs to the disk. + */ +static void +KmgrWriteCryptoKeys(const char *dir, unsigned char **keys, int *key_lens) +{ + elog(DEBUG2, "writing data encryption keys wrapped using the cluster key"); + + for (int i = 0; i < KMGR_NUM_DATA_KEYS; i++) + { + int fd; + char path[MAXPGPATH]; + + CryptoKeyFilePath(path, dir, i); + + if ((fd = BasicOpenFile(path, O_RDWR | O_CREAT | O_EXCL | PG_BINARY)) < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\": %m", + path))); + + errno = 0; + pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_WRITE); + if (write(fd, keys[i], key_lens[i]) != key_lens[i]) + { + /* if write didn't set errno, assume problem is no disk space */ + if (errno == 0) + errno = ENOSPC; + + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not write file \"%s\": %m", + path))); + } + pgstat_report_wait_end(); + + pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_SYNC); + if (pg_fsync(fd) != 0) + ereport(PANIC, + (errcode_for_file_access(), + errmsg("could not fsync file \"%s\": %m", + path))); + pgstat_report_wait_end(); + + if (close(fd) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", + path))); + } +} diff --git a/src/common/kmgr_utils.c b/src/common/kmgr_utils.c index 6c50cb093b..92b6441186 100644 --- a/src/common/kmgr_utils.c +++ b/src/common/kmgr_utils.c @@ -30,11 +30,12 @@ #endif #include "common/cryptohash.h" #include "common/file_perm.h" -#include "common/hex.h" #include "common/string.h" #include "crypto/kmgr.h" #include "lib/stringinfo.h" #include "storage/fd.h" +#include "postgres.h" +#include "utils/builtins.h" #ifndef FRONTEND #include "pgstat.h" diff --git a/src/include/crypto/kmgr.h b/src/include/crypto/kmgr.h new file mode 100644 index 0000000000..ec341c70b9 --- /dev/null +++ b/src/include/crypto/kmgr.h @@ -0,0 +1,25 @@ +/*------------------------------------------------------------------------- + * + * kmgr.h + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/crypto/kmgr.h + * + *------------------------------------------------------------------------- + */ +#ifndef KMGR_H +#define KMGR_H + +#include "common/kmgr_utils.h" + +/* GUC parameters */ +extern char *cluster_key_command; + +extern Size KmgrShmemSize(void); +extern void KmgrShmemInit(void); +extern void BootStrapKmgr(void); +extern void InitializeKmgr(void); +extern const CryptoKey *KmgrGetKey(int id); + +#endif /* KMGR_H */ -- 2.37.0 (Apple Git-136) [application/octet-stream] 0006-cfe-06-backend_over_cfe-05-crypto-squash-commit.patch (21.7K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/7-0006-cfe-06-backend_over_cfe-05-crypto-squash-commit.patch) download | inline diff: From 208d4b7ececc75ef93bdb9b948349ba6f0b69363 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:53 -0400 Subject: [PATCH 06/12] cfe-06-backend_over_cfe-05-crypto squash commit --- doc/src/sgml/storage.sgml | 5 +++ src/backend/Makefile | 2 +- src/backend/access/transam/xlog.c | 33 +++++++++++++++++++ src/backend/backup/basebackup.c | 5 +++ src/backend/bootstrap/bootstrap.c | 29 +++++++++++++++- src/backend/crypto/kmgr.c | 3 ++ src/backend/libpq/be-secure-common.c | 14 ++++++++ src/backend/main/main.c | 3 ++ src/backend/postmaster/postmaster.c | 13 +++++++- src/backend/storage/ipc/ipci.c | 3 ++ src/backend/storage/lmgr/lwlocknames.txt | 1 + src/backend/tcop/postgres.c | 28 +++++++++++++++- src/backend/utils/activity/wait_event.c | 9 +++++ src/backend/utils/misc/guc_tables.c | 25 ++++++++++++++ src/backend/utils/misc/pg_controldata.c | 11 +++++-- src/backend/utils/misc/postgresql.conf.sample | 5 +++ src/include/access/xlog.h | 5 +++ src/include/catalog/pg_control.h | 5 ++- src/include/postmaster/postmaster.h | 2 ++ src/include/utils/guc_tables.h | 1 + 20 files changed, 194 insertions(+), 8 deletions(-) diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index e5b9f3f1ff..f9dbf04735 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -77,6 +77,11 @@ Item <entry>Subdirectory containing transaction commit timestamp data</entry> </row> +<row> + <entry><filename>pg_cryptokeys</filename></entry> + <entry>Subdirectory containing file encryption keys</entry> +</row> + <row> <entry><filename>pg_dynshmem</filename></entry> <entry>Subdirectory containing files used by the dynamic shared memory diff --git a/src/backend/Makefile b/src/backend/Makefile index 1a5239b822..4b8cdd1b44 100644 --- a/src/backend/Makefile +++ b/src/backend/Makefile @@ -22,7 +22,7 @@ SUBDIRS = access backup bootstrap catalog parser commands executor \ main nodes optimizer partitioning port postmaster \ regex replication rewrite \ statistics storage tcop tsearch utils $(top_builddir)/src/timezone \ - jit + jit crypto include $(srcdir)/common.mk diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index dea978a962..de92e88d81 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -69,6 +69,7 @@ #include "catalog/pg_database.h" #include "common/controldata_utils.h" #include "common/file_utils.h" +#include "crypto/kmgr.h" #include "executor/instrument.h" #include "miscadmin.h" #include "pg_trace.h" @@ -76,6 +77,7 @@ #include "port/atomics.h" #include "port/pg_iovec.h" #include "postmaster/bgwriter.h" +#include "postmaster/postmaster.h" #include "postmaster/startup.h" #include "postmaster/walwriter.h" #include "replication/logical.h" @@ -109,6 +111,7 @@ #include "utils/varlena.h" extern uint32 bootstrap_data_checksum_version; +extern int bootstrap_file_encryption_method; /* timeline ID to be used when bootstrapping */ #define BootstrapTimeLineID 1 @@ -3898,6 +3901,7 @@ InitControlFile(uint64 sysidentifier) ControlFile->wal_log_hints = wal_log_hints; ControlFile->track_commit_timestamp = track_commit_timestamp; ControlFile->data_checksum_version = bootstrap_data_checksum_version; + ControlFile->file_encryption_method = bootstrap_file_encryption_method; } static void @@ -4185,6 +4189,12 @@ ReadControlFile(void) /* Make the initdb settings visible as GUC variables, too */ SetConfigOption("data_checksums", DataChecksumsEnabled() ? "yes" : "no", PGC_INTERNAL, PGC_S_DYNAMIC_DEFAULT); + + StaticAssertStmt(lengthof(encryption_methods) == NUM_ENCRYPTION_METHODS, + "encryption_methods[] must match NUM_ENCRYPTION_METHODS"); + SetConfigOption("file_encryption_method", + encryption_methods[ControlFile->file_encryption_method].name, + PGC_INTERNAL, PGC_S_OVERRIDE); } /* @@ -4227,6 +4237,21 @@ DataChecksumsEnabled(void) return (ControlFile->data_checksum_version > 0); } +/* + * Is cluster file encryption enabled? + */ +int +GetFileEncryptionMethod(void) +{ + if (IsBootstrapProcessingMode()) + return bootstrap_file_encryption_method; + else + { + Assert(ControlFile != NULL); + return ControlFile->file_encryption_method; + } +} + /* * Returns a fake LSN for unlogged relations. * @@ -4806,6 +4831,14 @@ BootStrapXLOG(void) /* some additional ControlFile fields are set in WriteControlFile() */ WriteControlFile(); + BootStrapKmgr(); + + if (terminal_fd != -1) + { + close(terminal_fd); + terminal_fd = -1; + } + /* Bootstrap the commit log, too */ BootStrapCLOG(); BootStrapCommitTs(); diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 74fb529380..35e955e036 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -24,6 +24,7 @@ #include "backup/basebackup_target.h" #include "commands/defrem.h" #include "common/compression.h" +#include "common/kmgr_utils.h" #include "common/file_perm.h" #include "lib/stringinfo.h" #include "miscadmin.h" @@ -129,6 +130,10 @@ struct exclude_list_item */ static const char *const excludeDirContents[] = { + /* Skip temporary crypto key directories */ + NEW_KMGR_DIR, + OLD_KMGR_DIR, + /* * Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped * because extensions like pg_stat_statements store data there. diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 58247d826d..9fc09b34de 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -33,6 +33,7 @@ #include "miscadmin.h" #include "nodes/makefuncs.h" #include "pg_getopt.h" +#include "postmaster/postmaster.h" /* TODO: verify we need this still */ #include "storage/bufmgr.h" #include "storage/bufpage.h" #include "storage/condition_variable.h" @@ -46,6 +47,8 @@ #include "utils/relmapper.h" uint32 bootstrap_data_checksum_version = 0; /* No checksum */ +int bootstrap_file_encryption_method = DISABLED_ENCRYPTION_METHOD; +char *bootstrap_old_key_datadir = NULL; /* disabled */ static void CheckerModeMain(void); @@ -221,7 +224,7 @@ BootstrapModeMain(int argc, char *argv[], bool check_only) argv++; argc--; - while ((flag = getopt(argc, argv, "B:c:d:D:Fkr:X:-:")) != -1) + while ((flag = getopt(argc, argv, "B:c:d:D:FkK:r:R:u:X:-:")) != -1) { switch (flag) { @@ -250,9 +253,33 @@ BootstrapModeMain(int argc, char *argv[], bool check_only) case 'k': bootstrap_data_checksum_version = PG_DATA_CHECKSUM_VERSION; break; + case 'K': + { + int i; + + /* method 0/disabled cannot be specified */ + for (i = DISABLED_ENCRYPTION_METHOD + 1; + i < NUM_ENCRYPTION_METHODS; i++) + if (pg_strcasecmp(optarg, encryption_methods[i].name) == 0) + { + bootstrap_file_encryption_method = i; + break; + } + if (i == NUM_ENCRYPTION_METHODS) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid encryption method specified"))); + } + break; case 'r': strlcpy(OutputFileName, optarg, MAXPGPATH); break; + case 'R': + terminal_fd = atoi(optarg); + break; + case 'u': + bootstrap_old_key_datadir = pstrdup(optarg); + break; case 'X': { int WalSegSz = strtoul(optarg, NULL, 0); diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c index 4b74961323..a83effbca8 100644 --- a/src/backend/crypto/kmgr.c +++ b/src/backend/crypto/kmgr.c @@ -253,6 +253,9 @@ InitializeKmgr(void) char live_path[MAXPGPATH]; unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN]; + if (!FileEncryptionEnabled) + return; + #ifndef USE_OPENSSL ereport(ERROR, (errcode(ERRCODE_CONFIG_FILE_ERROR), diff --git a/src/backend/libpq/be-secure-common.c b/src/backend/libpq/be-secure-common.c index 2719ce1403..9357c80ea6 100644 --- a/src/backend/libpq/be-secure-common.c +++ b/src/backend/libpq/be-secure-common.c @@ -22,6 +22,7 @@ #include <sys/stat.h> #include <unistd.h> +#include "postmaster/postmaster.h" #include "common/string.h" #include "libpq/libpq.h" #include "storage/fd.h" @@ -61,6 +62,19 @@ run_ssl_passphrase_command(const char *prompt, bool is_server_start, char *buf, appendStringInfoString(&command, prompt); p++; break; + case 'R': + { + char fd_str[20]; + + if (terminal_fd == -1) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("ssl_passphrase_command referenced %%R, but -R not specified"))); + p++; + snprintf(fd_str, sizeof(fd_str), "%d", terminal_fd); + appendStringInfoString(&command, fd_str); + break; + } case '%': appendStringInfoChar(&command, '%'); p++; diff --git a/src/backend/main/main.c b/src/backend/main/main.c index f5da4260a1..ddf3fcb6fb 100644 --- a/src/backend/main/main.c +++ b/src/backend/main/main.c @@ -343,6 +343,7 @@ help(const char *progname) #endif printf(_(" -N MAX-CONNECT maximum number of allowed connections\n")); printf(_(" -p PORT port number to listen on\n")); + printf(_(" -R fd prompt for the cluster key\n")); printf(_(" -s show statistics after each query\n")); printf(_(" -S WORK-MEM set amount of memory for sorts (in kB)\n")); printf(_(" -V, --version output version information, then exit\n")); @@ -371,7 +372,9 @@ help(const char *progname) printf(_(" --boot selects bootstrapping mode (must be first argument)\n")); printf(_(" --check selects check mode (must be first argument)\n")); printf(_(" DBNAME database name (mandatory argument in bootstrapping mode)\n")); + printf(_(" -K LEN enable cluster file encryption with specified key bit length\n")); printf(_(" -r FILENAME send stdout and stderr to given file\n")); + printf(_(" -u DATADIR copy encryption keys from datadir\n")); printf(_("\nPlease read the documentation for the complete list of run-time\n" "configuration settings and how to set them on the command line or in\n" diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 30fb576ac3..aec9b094bd 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -98,6 +98,7 @@ #include "common/ip.h" #include "common/pg_prng.h" #include "common/string.h" +#include "crypto/kmgr.h" #include "lib/ilist.h" #include "libpq/auth.h" #include "libpq/libpq.h" @@ -231,6 +232,7 @@ static int SendStop = false; /* still more option variables */ bool EnableSSL = false; +int terminal_fd = -1; int PreAuthDelay = 0; int AuthenticationTimeout = 60; @@ -697,7 +699,7 @@ PostmasterMain(int argc, char *argv[]) * tcop/postgres.c (the option sets should not conflict) and with the * common help() function in main/main.c. */ - while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:W:-:")) != -1) + while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:R:S:sTt:W:-:")) != -1) { switch (opt) { @@ -788,6 +790,10 @@ PostmasterMain(int argc, char *argv[]) /* only used by single-user backend */ break; + case 'R': + terminal_fd = atoi(optarg); + break; + case 'S': SetConfigOption("work_mem", optarg, PGC_POSTMASTER, PGC_S_ARGV); break; @@ -1347,6 +1353,11 @@ PostmasterMain(int argc, char *argv[]) pfree(rawstring); } + InitializeKmgr(); + + if (terminal_fd != -1) + close(terminal_fd); + /* * check that we have some socket to listen on */ diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index b204ecdbc3..061da11a58 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -25,6 +25,7 @@ #include "access/xlogprefetcher.h" #include "access/xlogrecovery.h" #include "commands/async.h" +#include "crypto/kmgr.h" #include "miscadmin.h" #include "pgstat.h" #include "postmaster/autovacuum.h" @@ -142,6 +143,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, SyncScanShmemSize()); size = add_size(size, AsyncShmemSize()); size = add_size(size, StatsShmemSize()); + size = add_size(size, KmgrShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -294,6 +296,7 @@ CreateSharedMemoryAndSemaphores(void) SyncScanShmemInit(); AsyncShmemInit(); StatsShmemInit(); + KmgrShmemInit(); #ifdef EXEC_BACKEND diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt index 6c7cf6c295..56ac32407f 100644 --- a/src/backend/storage/lmgr/lwlocknames.txt +++ b/src/backend/storage/lmgr/lwlocknames.txt @@ -53,3 +53,4 @@ XactTruncationLock 44 # 45 was XactTruncationLock until removal of BackendRandomLock WrapLimitsVacuumLock 46 NotifyQueueTailLock 47 +KmgrFileLock 48 diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index a9a1851c94..5970ec97a5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -35,6 +35,7 @@ #include "commands/async.h" #include "commands/prepare.h" #include "common/pg_prng.h" +#include "crypto/kmgr.h" #include "jit/jit.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" @@ -3718,7 +3719,7 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx, * postmaster/postmaster.c (the option sets should not conflict) and with * the common help() function in main/main.c. */ - while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:v:W:-:")) != -1) + while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:R:S:sTt:v:W:-:")) != -1) { switch (flag) { @@ -3810,6 +3811,19 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx, strlcpy(OutputFileName, optarg, MAXPGPATH); break; + case 'R': + terminal_fd = atoi(optarg); + if (terminal_fd == -1) + { + /* + * Allow file descriptor closing to be bypassed via -1. + * We just duplicate sterr. This is useful for + * single-user mode. + */ + terminal_fd = dup(2); + } + break; + case 'S': SetConfigOption("work_mem", optarg, ctx, gucsource); break; @@ -4055,6 +4069,18 @@ PostgresMain(const char *dbname, const char *username) SetProcessingMode(InitProcessing); + /* + * Initialize kmgr for cluster encryption. Since kmgr needs to attach to + * shared memory the initialization must be called after BaseInit(). + */ + if (!IsUnderPostmaster) + { + InitializeKmgr(); + + if (terminal_fd != -1) + close(terminal_fd); + } + /* * Set up signal handlers. (InitPostmasterChild or InitStandaloneProcess * has already set up BlockSig and made that the active signal mask.) diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 92f24a6c9b..6f7953accb 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -591,6 +591,15 @@ pgstat_get_wait_io(WaitEventIO w) case WAIT_EVENT_DSM_FILL_ZERO_WRITE: event_name = "DSMFillZeroWrite"; break; + case WAIT_EVENT_KEY_FILE_READ: + event_name = "KeyFileRead"; + break; + case WAIT_EVENT_KEY_FILE_WRITE: + event_name = "KeyFileWrite"; + break; + case WAIT_EVENT_KEY_FILE_SYNC: + event_name = "KeyFileSync"; + break; case WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ: event_name = "LockFileAddToDataDirRead"; break; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 05ab087934..402f1aa8f0 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -40,6 +40,7 @@ #include "commands/trigger.h" #include "commands/user.h" #include "commands/vacuum.h" +#include "crypto/kmgr.h" #include "jit/jit.h" #include "libpq/auth.h" #include "libpq/libpq.h" @@ -565,6 +566,7 @@ static char *recovery_target_string; static char *recovery_target_xid_string; static char *recovery_target_name_string; static char *recovery_target_lsn_string; +static char *file_encryption_method_str; /* should be static, but commands/variable.c needs to get at this */ char *role_string; @@ -687,6 +689,8 @@ const char *const config_group_names[] = gettext_noop("Statistics / Monitoring"), /* STATS_CUMULATIVE */ gettext_noop("Statistics / Cumulative Query and Index Statistics"), + /* ENCRYPTION */ + gettext_noop("Encryption"), /* AUTOVACUUM */ gettext_noop("Autovacuum"), /* CLIENT_CONN_STATEMENT */ @@ -4405,6 +4409,27 @@ struct config_string ConfigureNamesString[] = NULL, NULL, NULL }, + { + {"cluster_key_command", PGC_SIGHUP, ENCRYPTION, + gettext_noop("Command to obtain cluster key for cluster file encryption."), + NULL + }, + &cluster_key_command, + "", + NULL, NULL, NULL + }, + + { + {"file_encryption_method", PGC_INTERNAL, PRESET_OPTIONS, + gettext_noop("Shows the cluster file encryption method."), + NULL, + GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE + }, + &file_encryption_method_str, + "", + NULL, NULL, NULL + }, + { {"application_name", PGC_USERSET, LOGGING_WHAT, gettext_noop("Sets the application name to be reported in statistics and logs."), diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 781f8b8758..d5fbf8f7df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -263,8 +263,8 @@ pg_control_recovery(PG_FUNCTION_ARGS) Datum pg_control_init(PG_FUNCTION_ARGS) { - Datum values[11]; - bool nulls[11]; + Datum values[12]; + bool nulls[12]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -274,7 +274,7 @@ pg_control_init(PG_FUNCTION_ARGS) * Construct a tuple descriptor for the result row. This must match this * function's pg_proc entry! */ - tupdesc = CreateTemplateTupleDesc(11); + tupdesc = CreateTemplateTupleDesc(12); TupleDescInitEntry(tupdesc, (AttrNumber) 1, "max_data_alignment", INT4OID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 2, "database_block_size", @@ -297,6 +297,8 @@ pg_control_init(PG_FUNCTION_ARGS) BOOLOID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 11, "data_page_checksum_version", INT4OID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 12, "file_encryption_method", + INT4OID, -1, 0); tupdesc = BlessTupleDesc(tupdesc); /* read the control file */ @@ -338,6 +340,9 @@ pg_control_init(PG_FUNCTION_ARGS) values[10] = Int32GetDatum(ControlFile->data_checksum_version); nulls[10] = false; + values[11] = Int32GetDatum(ControlFile->file_encryption_method); + nulls[11] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 868d21c351..96b5c869e1 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -659,6 +659,11 @@ # autovacuum, -1 means use # vacuum_cost_limit +#------------------------------------------------------------------------------ +# ENCRYPTION +#------------------------------------------------------------------------------ + +#cluster_key_command = '' #------------------------------------------------------------------------------ # CLIENT CONNECTION DEFAULTS diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 1fbd48fbda..8f219fbacc 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -13,6 +13,7 @@ #include "access/xlogbackup.h" #include "access/xlogdefs.h" +#include "common/kmgr_utils.h" #include "datatype/timestamp.h" #include "lib/stringinfo.h" #include "nodes/pg_list.h" @@ -224,6 +225,10 @@ extern XLogRecPtr GetXLogWriteRecPtr(void); extern uint64 GetSystemIdentifier(void); extern char *GetMockAuthenticationNonce(void); extern bool DataChecksumsEnabled(void); + +#define FileEncryptionEnabled (GetFileEncryptionMethod() != DISABLED_ENCRYPTION_METHOD) +extern int GetFileEncryptionMethod(void); + extern XLogRecPtr GetFakeLSNForUnloggedRel(void); extern Size XLOGShmemSize(void); extern void XLOGShmemInit(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 06368e2366..db7de77287 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -22,7 +22,7 @@ /* Version identifier for this pg_control format */ -#define PG_CONTROL_VERSION 1300 +#define PG_CONTROL_VERSION 1400 /* Nonce key length, see below */ #define MOCK_AUTH_NONCE_LEN 32 @@ -226,6 +226,9 @@ typedef struct ControlFileData */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; + /* File encryption method; index into encryption_methods[]. */ + int file_encryption_method; + /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h index 90e333ccd2..deac25e115 100644 --- a/src/include/postmaster/postmaster.h +++ b/src/include/postmaster/postmaster.h @@ -31,6 +31,8 @@ extern PGDLLIMPORT char *bonjour_name; extern PGDLLIMPORT bool restart_after_crash; extern PGDLLIMPORT bool remove_temp_files_after_crash; +extern int terminal_fd; + #ifdef WIN32 extern PGDLLIMPORT HANDLE PostmasterHandle; #else diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h index 51f7f09fde..1446b334b1 100644 --- a/src/include/utils/guc_tables.h +++ b/src/include/utils/guc_tables.h @@ -86,6 +86,7 @@ enum config_group PROCESS_TITLE, STATS_MONITORING, STATS_CUMULATIVE, + ENCRYPTION, AUTOVACUUM, CLIENT_CONN_STATEMENT, CLIENT_CONN_LOCALE, -- 2.37.0 (Apple Git-136) [application/octet-stream] 0007-cfe-07-bin_over_cfe-06-backend-squash-commit.patch (33.3K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/8-0007-cfe-07-bin_over_cfe-06-backend-squash-commit.patch) download | inline diff: From e60810874bb68af7a84cebdb8ba1674e2ce2239f Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:56 -0400 Subject: [PATCH 07/12] cfe-07-bin_over_cfe-06-backend squash commit --- doc/src/sgml/ref/initdb.sgml | 47 +++++++++ doc/src/sgml/ref/pg_ctl-ref.sgml | 14 +++ doc/src/sgml/ref/pgupgrade.sgml | 20 +++- doc/src/sgml/ref/postgres-ref.sgml | 13 +++ src/bin/initdb/initdb.c | 125 ++++++++++++++++++++++-- src/bin/pg_controldata/pg_controldata.c | 3 + src/bin/pg_ctl/pg_ctl.c | 59 +++++++++-- src/bin/pg_resetwal/pg_resetwal.c | 3 + src/bin/pg_rewind/filemap.c | 8 ++ src/bin/pg_upgrade/check.c | 34 +++++++ src/bin/pg_upgrade/controldata.c | 41 +++++++- src/bin/pg_upgrade/file.c | 2 + src/bin/pg_upgrade/option.c | 7 +- src/bin/pg_upgrade/pg_upgrade.h | 4 + src/bin/pg_upgrade/server.c | 2 +- 15 files changed, 361 insertions(+), 21 deletions(-) diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml index 8158896298..06b10fbaaf 100644 --- a/doc/src/sgml/ref/initdb.sgml +++ b/doc/src/sgml/ref/initdb.sgml @@ -186,6 +186,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry id="app-initdb-cluster-key-command" xreflabel="cluster key command"> + <term><option>-c <replaceable class="parameter">command</replaceable></option></term> + <term><option>--cluster-key-command=<replaceable class="parameter">command</replaceable></option></term> + <listitem> + <para> + This option specifies an external command to obtain the cluster-level + key for cluster file encryption during server initialization and + server start; see <xref linkend="guc-cluster-key-command"/> for details. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-D <replaceable class="parameter">directory</replaceable></option></term> <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term> @@ -257,6 +269,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry id="app-initdb-file-encryption-method" + xreflabel="file encryption"> + <term><option>-K <replaceable class="parameter">encryption_method</replaceable></option></term> + <term><option>--file-encryption-method=<replaceable class="parameter">encryption_method</replaceable></option></term> + <listitem> + <para> + Specifies the cluster file encryption method. The + value values are <literal>AES128</literal> (the default), <literal>AES192</literal>, and <literal>AES256</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--locale=<replaceable>locale</replaceable></option></term> <listitem> @@ -344,6 +368,17 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem> + <para> + Allows the <option>--cluster-key-command</option> command + to prompt for a passphrase or PIN. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-S</option></term> <term><option>--sync-only</option></term> @@ -369,6 +404,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-u <replaceable>datadir</replaceable></option></term> + <term><option>--copy-encryption-keys=<replaceable>datadir</replaceable></option></term> + <listitem> + <para> + Copies cluster file encryption keys from another cluster; required + when using <application>pg_upgrade</application> on a cluster + with cluster file encryption enabled. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-U <replaceable class="parameter">username</replaceable></option></term> <term><option>--username=<replaceable class="parameter">username</replaceable></option></term> diff --git a/doc/src/sgml/ref/pg_ctl-ref.sgml b/doc/src/sgml/ref/pg_ctl-ref.sgml index 46906966eb..e3705c724e 100644 --- a/doc/src/sgml/ref/pg_ctl-ref.sgml +++ b/doc/src/sgml/ref/pg_ctl-ref.sgml @@ -38,6 +38,7 @@ PostgreSQL documentation <arg choice="opt"><option>-s</option></arg> <arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg> <arg choice="opt"><option>-p</option> <replaceable>path</replaceable></arg> + <arg choice="opt"><option>-R</option></arg> <arg choice="opt"><option>-c</option></arg> </cmdsynopsis> @@ -72,6 +73,7 @@ PostgreSQL documentation <arg choice="opt"><option>-t</option> <replaceable>seconds</replaceable></arg> <arg choice="opt"><option>-s</option></arg> <arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg> + <arg choice="opt"><option>-R</option></arg> <arg choice="opt"><option>-c</option></arg> </cmdsynopsis> @@ -373,6 +375,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem> + <para> + Allows <option>ssl_passphrase_command</option> or + <option>cluster_key_command</option> to prompt for a passphrase + or PIN. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-s</option></term> <term><option>--silent</option></term> diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index 8f7a3025c3..4362e551c9 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -182,6 +182,15 @@ PostgreSQL documentation </para></listitem> </varlistentry> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem><para>allows <option>ssl_passphrase_command</option> or + <option>cluster_key_command</option> to prompt for a passphrase + or PIN. + </para></listitem> + </varlistentry> + <varlistentry> <term><option>-s</option> <replaceable>dir</replaceable></term> <term><option>--socketdir=</option><replaceable>dir</replaceable></term> @@ -309,7 +318,9 @@ make prefix=/usr/local/pgsql.new install Again, use compatible <command>initdb</command> flags that match the old cluster. Many prebuilt installers do this step automatically. There is no need to - start the new cluster. + start the new cluster. If upgrading a cluster that uses + cluster file encryption, the <command>initdb</command> option + <option>--copy-encryption-keys</option> must be specified. </para> </step> @@ -842,6 +853,13 @@ psql --username=postgres --file=script.sql postgres is down. </para> + <para> + If the old cluster uses file encryption, the new cluster must use + the same keys, so <command>pg_upgrade</command> copies them to the + new cluster. It is necessary to initialize the new cluster with + the same <varname>cluster_key_command</varname> and the same + file encryption method. + </para> </refsect1> <refsect1> diff --git a/doc/src/sgml/ref/postgres-ref.sgml b/doc/src/sgml/ref/postgres-ref.sgml index 55a3f6c69d..f0fa52ebdc 100644 --- a/doc/src/sgml/ref/postgres-ref.sgml +++ b/doc/src/sgml/ref/postgres-ref.sgml @@ -305,6 +305,19 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-R <replaceable class="parameter">file-descriptor</replaceable></option></term> + <listitem> + <para> + Makes <command>postgres</command> prompt for a passphrase or PIN + from the specified open numeric file descriptor. The descriptor + is closed after the key is read. The file descriptor number + <literal>-1</literal> duplicates standard error for the terminal; + this is useful for single-user mode. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-s</option></term> <listitem> diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index f61a043055..8c7e1e39c5 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -68,6 +68,7 @@ #include "catalog/pg_database_d.h" /* pgrminclude ignore */ #include "common/file_perm.h" #include "common/file_utils.h" +#include "common/kmgr_utils.h" #include "common/logging.h" #include "common/pg_prng.h" #include "common/restricted_token.h" @@ -147,11 +148,16 @@ static bool noclean = false; static bool noinstructions = false; static bool do_sync = true; static bool sync_only = false; +static bool pass_terminal_fd = false; +static char *term_fd_opt = NULL; +static int file_encryption_method = DISABLED_ENCRYPTION_METHOD; static bool show_setting = false; static bool data_checksums = false; static char *xlog_dir = NULL; static char *str_wal_segment_size_mb = NULL; static int wal_segment_size_mb; +static char *cluster_key_cmd = NULL; +static char *old_key_datadir = NULL; /* internal vars */ @@ -214,6 +220,7 @@ static const char *const subdirs[] = { "global", "pg_wal/archive_status", "pg_commit_ts", + "pg_cryptokeys", "pg_dynshmem", "pg_notify", "pg_serial", @@ -904,12 +911,13 @@ test_config_settings(void) test_buffs = MIN_BUFS_FOR_CONNS(test_conns); snprintf(cmd, sizeof(cmd), - "\"%s\" --check %s %s " + "\"%s\" --check -x0 %s %s %s " "-c max_connections=%d " "-c shared_buffers=%d " "-c dynamic_shared_memory_type=%s " "< \"%s\" > \"%s\" 2>&1", backend_exec, boot_options, extra_options, + term_fd_opt ? term_fd_opt : "", test_conns, test_buffs, dynamic_shared_memory_type, DEVNULL, DEVNULL); @@ -941,12 +949,13 @@ test_config_settings(void) } snprintf(cmd, sizeof(cmd), - "\"%s\" --check %s %s " + "\"%s\" --check %s %s %s " "-c max_connections=%d " "-c shared_buffers=%d " "-c dynamic_shared_memory_type=%s " "< \"%s\" > \"%s\" 2>&1", backend_exec, boot_options, extra_options, + term_fd_opt ? term_fd_opt : "", n_connections, test_buffs, dynamic_shared_memory_type, DEVNULL, DEVNULL); @@ -1133,6 +1142,13 @@ setup_config(void) "password_encryption = md5"); } + if (cluster_key_cmd) + { + snprintf(repltok, sizeof(repltok), "cluster_key_command = '%s'", + escape_quotes(cluster_key_cmd)); + conflines = replace_token(conflines, "#cluster_key_command = ''", repltok); + } + /* * If group access has been enabled for the cluster then it makes sense to * ensure that the log files also allow group access. Otherwise a backup @@ -1321,11 +1337,17 @@ bootstrap_template1(void) unsetenv("PGCLIENTENCODING"); snprintf(cmd, sizeof(cmd), - "\"%s\" --boot -X %d %s %s %s %s", + "\"%s\" --boot -X %d %s %s %s %s %s %s %s %s %s", backend_exec, wal_segment_size_mb * (1024 * 1024), data_checksums ? "-k" : "", - boot_options, extra_options, + cluster_key_cmd ? "-K" : "", + cluster_key_cmd ? encryption_methods[file_encryption_method].name : "", + old_key_datadir ? "-u" : "", + old_key_datadir ? old_key_datadir : "", + boot_options, + extra_options, + term_fd_opt ? term_fd_opt : "", debug ? "-d 5" : ""); @@ -2159,18 +2181,26 @@ usage(const char *progname) printf(_(" -T, --text-search-config=CFG\n" " default text search configuration\n")); printf(_(" -U, --username=NAME database superuser name\n")); - printf(_(" -W, --pwprompt prompt for a password for the new superuser\n")); + printf(_(" -W, --pwprompt prompt for the new superuser password\n")); printf(_(" -X, --waldir=WALDIR location for the write-ahead log directory\n")); printf(_(" --wal-segsize=SIZE size of WAL segments, in megabytes\n")); printf(_("\nLess commonly used options:\n")); + printf(_(" -c, --cluster-key-command=COMMAND\n" + " enable cluster file encryption and set command\n" + " to obtain the cluster key\n")); printf(_(" -d, --debug generate lots of debugging output\n")); printf(_(" --discard-caches set debug_discard_caches=1\n")); + printf(_(" -K, --file-encryption-method=METHOD\n" + " cluster file encryption method\n")); printf(_(" -L DIRECTORY where to find the input files\n")); printf(_(" -n, --no-clean do not clean up after errors\n")); printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n")); + printf(_(" -R, --authprompt prompt for a passphrase or PIN\n")); printf(_(" --no-instructions do not print instructions for next steps\n")); printf(_(" -s, --show show internal settings\n")); printf(_(" -S, --sync-only only sync database files to disk, then exit\n")); + printf(_(" -u, --copy-encryption-keys=DATADIR\n" + " copy the file encryption key from another cluster\n")); printf(_("\nOther options:\n")); printf(_(" -V, --version output version information, then exit\n")); printf(_(" -?, --help show this help, then exit\n")); @@ -2700,6 +2730,23 @@ initialize_data_directory(void) /* Top level PG_VERSION is checked by bootstrapper, so make it first */ write_version_file(NULL); + if (pass_terminal_fd) + { +#ifndef WIN32 + int terminal_fd = open("/dev/tty", O_RDWR, 0); +#else + int terminal_fd = open("CONOUT$", O_RDWR, 0); +#endif + + if (terminal_fd < 0) + { + pg_log_error(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + term_fd_opt = psprintf("-R %d", terminal_fd); + } + /* Select suitable configuration settings */ set_null_conf(); test_config_settings(); @@ -2723,8 +2770,9 @@ initialize_data_directory(void) fflush(stdout); snprintf(cmd, sizeof(cmd), - "\"%s\" %s %s template1 >%s", + "\"%s\" %s %s %s template1 >%s", backend_exec, backend_options, extra_options, + term_fd_opt ? term_fd_opt : "", DEVNULL); PG_CMD_OPEN; @@ -2802,10 +2850,14 @@ main(int argc, char *argv[]) {"waldir", required_argument, NULL, 'X'}, {"wal-segsize", required_argument, NULL, 12}, {"data-checksums", no_argument, NULL, 'k'}, + {"authprompt", no_argument, NULL, 'R'}, + {"file-encryption-method", required_argument, NULL, 'K'}, {"allow-group-access", no_argument, NULL, 'g'}, {"discard-caches", no_argument, NULL, 14}, {"locale-provider", required_argument, NULL, 15}, {"icu-locale", required_argument, NULL, 16}, + {"cluster-key-command", required_argument, NULL, 'c'}, + {"copy-encryption-keys", required_argument, NULL, 'u'}, {NULL, 0, NULL, 0} }; @@ -2847,7 +2899,7 @@ main(int argc, char *argv[]) /* process command-line options */ - while ((c = getopt_long(argc, argv, "A:dD:E:gkL:nNsST:U:WX:", long_options, &option_index)) != -1) + while ((c = getopt_long(argc, argv, "A:c:dD:E:gkK:L:nNRsST:u:U:WX:", long_options, &option_index)) != -1) { switch (c) { @@ -2893,6 +2945,28 @@ main(int argc, char *argv[]) case 'N': do_sync = false; break; + case 'R': + pass_terminal_fd = true; + break; + case 'K': + { + int i; + + /* method 0/disabled cannot be specified */ + for (i = DISABLED_ENCRYPTION_METHOD + 1; + i < NUM_ENCRYPTION_METHODS; i++) + if (pg_strcasecmp(optarg, encryption_methods[i].name) == 0) + { + file_encryption_method = i; + break; + } + if (i == NUM_ENCRYPTION_METHODS) + { + fprintf(stderr, _("invalid cluster encryption method\n")); + exit(1); + } + } + break; case 'S': sync_only = true; break; @@ -2929,6 +3003,12 @@ main(int argc, char *argv[]) case 9: pwfilename = pg_strdup(optarg); break; + case 'c': + cluster_key_cmd = pg_strdup(optarg); + break; + case 'u': + old_key_datadir = pg_strdup(optarg); + break; case 's': show_setting = true; break; @@ -3014,6 +3094,32 @@ main(int argc, char *argv[]) if (pwprompt && pwfilename) pg_fatal("password prompt and password file cannot be specified together"); +#ifndef USE_OPENSSL + if (cluster_key_cmd) + { + pg_log_error("cluster file encryption is not supported because OpenSSL is not supported by this build"); + exit(1); + } +#endif + + if (old_key_datadir != NULL && cluster_key_cmd == NULL) + { + pg_log_error("copying encryption keys requires the cluster key command to be specified"); + exit(1); + } + + if (file_encryption_method != DISABLED_ENCRYPTION_METHOD && + cluster_key_cmd == NULL) + { + pg_log_error("a file encryption method requires the cluster key command to be specified"); + exit(1); + } + + /* set the default */ + if (file_encryption_method == DISABLED_ENCRYPTION_METHOD && + cluster_key_cmd != NULL) + file_encryption_method = DEFAULT_ENABLED_ENCRYPTION_METHOD; + check_authmethod_unspecified(&authmethodlocal); check_authmethod_unspecified(&authmethodhost); @@ -3072,6 +3178,11 @@ main(int argc, char *argv[]) else printf(_("Data page checksums are disabled.\n")); + if (cluster_key_cmd) + printf(_("Cluster file encryption is enabled.\n")); + else + printf(_("Cluster file encryption is disabled.\n")); + if (pwprompt || pwfilename) get_su_pwd(); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index c390ec51ce..d25d13c992 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -25,6 +25,7 @@ #include "access/xlog_internal.h" #include "catalog/pg_control.h" #include "common/controldata_utils.h" +#include "common/kmgr_utils.h" #include "common/logging.h" #include "getopt_long.h" #include "pg_getopt.h" @@ -328,5 +329,7 @@ main(int argc, char *argv[]) ControlFile->data_checksum_version); printf(_("Mock authentication nonce: %s\n"), mock_auth_nonce_str); + printf(_("File encryption method: %s\n"), + encryption_methods[ControlFile->file_encryption_method].name); return 0; } diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c index a509fc9109..d90c07db10 100644 --- a/src/bin/pg_ctl/pg_ctl.c +++ b/src/bin/pg_ctl/pg_ctl.c @@ -74,6 +74,7 @@ typedef enum static bool do_wait = true; static int wait_seconds = DEFAULT_WAIT; static bool wait_seconds_arg = false; +static bool pass_terminal_fd = false; static bool silent_mode = false; static ShutdownMode shutdown_mode = FAST_MODE; static int sig = SIGINT; /* default */ @@ -439,7 +440,7 @@ free_readfile(char **optlines) static pid_t start_postmaster(void) { - char *cmd; + char *cmd, *term_fd_opt = NULL; #ifndef WIN32 pid_t pm_pid; @@ -467,6 +468,19 @@ start_postmaster(void) /* fork succeeded, in child */ + if (pass_terminal_fd) + { + int terminal_fd = open("/dev/tty", O_RDWR, 0); + + if (terminal_fd < 0) + { + write_stderr(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + term_fd_opt = psprintf(" -R %d", terminal_fd); + } + /* * If possible, detach the postmaster process from the launching process * group and make it a group leader, so that it doesn't get signaled along @@ -487,12 +501,14 @@ start_postmaster(void) * has the same PID as the current child process. */ if (log_file != NULL) - cmd = psprintf("exec \"%s\" %s%s < \"%s\" >> \"%s\" 2>&1", + cmd = psprintf("exec \"%s\" %s%s%s < \"%s\" >> \"%s\" 2>&1", exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL, log_file); else - cmd = psprintf("exec \"%s\" %s%s < \"%s\" 2>&1", - exec_path, pgdata_opt, post_opts, DEVNULL); + cmd = psprintf("exec \"%s\" %s%s%s < \"%s\" 2>&1", + exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL); (void) execl("/bin/sh", "/bin/sh", "-c", cmd, (char *) NULL); @@ -513,6 +529,21 @@ start_postmaster(void) PROCESS_INFORMATION pi; const char *comspec; + if (pass_terminal_fd) + { + /* Hopefully we can read and write CONOUT, see simple_prompt() XXX */ + /* Do CreateRestrictedProcess() children even inherit open file descriptors? XXX */ + int terminal_fd = open("CONOUT$", O_RDWR, 0); + + if (terminal_fd < 0) + { + write_stderr(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + term_fd_opt = psprintf(" -R %d", terminal_fd); + } + /* Find CMD.EXE location using COMSPEC, if it's set */ comspec = getenv("COMSPEC"); if (comspec == NULL) @@ -553,12 +584,14 @@ start_postmaster(void) else close(fd); - cmd = psprintf("\"%s\" /C \"\"%s\" %s%s < \"%s\" >> \"%s\" 2>&1\"", - comspec, exec_path, pgdata_opt, post_opts, DEVNULL, log_file); + cmd = psprintf("\"%s\" /C \"\"%s\" %s%s%s < \"%s\" >> \"%s\" 2>&1\"", + comspec, exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL, log_file); } else - cmd = psprintf("\"%s\" /C \"\"%s\" %s%s < \"%s\" 2>&1\"", - comspec, exec_path, pgdata_opt, post_opts, DEVNULL); + cmd = psprintf("\"%s\" /C \"\"%s\" %s%s%s < \"%s\" 2>&1\"", + comspec, exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL); if (!CreateRestrictedProcess(cmd, &pi, false)) { @@ -689,7 +722,8 @@ wait_for_postmaster_start(pid_t pm_pid, bool do_checkpoint) } else #endif - print_msg("."); + if (!pass_terminal_fd) + print_msg("."); } pg_usleep(USEC_PER_SEC / WAITS_PER_SEC); @@ -2005,6 +2039,7 @@ do_help(void) printf(_(" -o, --options=OPTIONS command line options to pass to postgres\n" " (PostgreSQL server executable) or initdb\n")); printf(_(" -p PATH-TO-POSTGRES normally not necessary\n")); + printf(_(" -R, --authprompt prompt for a passphrase or PIN\n")); printf(_("\nOptions for stop or restart:\n")); printf(_(" -m, --mode=MODE MODE can be \"smart\", \"fast\", or \"immediate\"\n")); @@ -2200,6 +2235,7 @@ main(int argc, char **argv) {"mode", required_argument, NULL, 'm'}, {"pgdata", required_argument, NULL, 'D'}, {"options", required_argument, NULL, 'o'}, + {"authprompt", no_argument, NULL, 'R'}, {"silent", no_argument, NULL, 's'}, {"timeout", required_argument, NULL, 't'}, {"core-files", no_argument, NULL, 'c'}, @@ -2272,7 +2308,7 @@ main(int argc, char **argv) /* process command-line options */ while (optind < argc) { - while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:sS:t:U:wW", + while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:RsS:t:U:wW", long_options, &option_index)) != -1) { switch (c) @@ -2324,6 +2360,9 @@ main(int argc, char **argv) case 'P': register_password = pg_strdup(optarg); break; + case 'R': + pass_terminal_fd = true; + break; case 's': silent_mode = true; break; diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 089063f471..73cb048b3e 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -52,6 +52,7 @@ #include "common/controldata_utils.h" #include "common/fe_memutils.h" #include "common/file_perm.h" +#include "common/kmgr_utils.h" #include "common/logging.h" #include "common/restricted_token.h" #include "common/string.h" @@ -772,6 +773,8 @@ PrintControlValues(bool guessed) (ControlFile.float8ByVal ? _("by value") : _("by reference"))); printf(_("Data page checksum version: %u\n"), ControlFile.data_checksum_version); + printf(_("File encryption method: %s\n"), + encryption_methods[ControlFile.file_encryption_method].name); } diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index 269ed6446e..d0b71f972a 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -28,6 +28,7 @@ #include "catalog/pg_tablespace_d.h" #include "common/hashfn.h" +#include "common/kmgr_utils.h" #include "common/string.h" #include "datapagemap.h" #include "filemap.h" @@ -106,6 +107,13 @@ static const char *excludeDirContents[] = /* Contents removed on startup, see AsyncShmemInit(). */ "pg_notify", + /* + * Skip cryptographic keys. It's generally not a good idea to copy the + * cryptographic keys from source database because these might use + * different cluster key. + */ + KMGR_DIR, + /* * Old contents are loaded for possible debugging but are not required for * normal operation, see SerialInit(). diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c index f1bc1e6886..ae7aa601a7 100644 --- a/src/bin/pg_upgrade/check.c +++ b/src/bin/pg_upgrade/check.c @@ -11,6 +11,7 @@ #include "catalog/pg_authid_d.h" #include "catalog/pg_collation.h" +#include "common/kmgr_utils.h" #include "fe_utils/string_utils.h" #include "mb/pg_wchar.h" #include "pg_upgrade.h" @@ -30,6 +31,7 @@ static void check_for_composite_data_type_usage(ClusterInfo *cluster); static void check_for_reg_data_type_usage(ClusterInfo *cluster); static void check_for_jsonb_9_4_usage(ClusterInfo *cluster); static void check_for_pg_role_prefix(ClusterInfo *cluster); +static void check_for_cluster_key_failure(ClusterInfo *cluster); static void check_for_new_tablespace_dir(ClusterInfo *new_cluster); static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster); static char *get_canonical_locale_name(int category, const char *locale); @@ -160,6 +162,9 @@ check_and_dump_old_cluster(bool live_check) if (GET_MAJOR_VERSION(old_cluster.major_version) <= 905) check_for_pg_role_prefix(&old_cluster); + if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1400) + check_for_cluster_key_failure(&old_cluster); + if (GET_MAJOR_VERSION(old_cluster.major_version) == 904 && old_cluster.controldata.cat_ver < JSONB_FORMAT_CHANGE_CAT_VER) check_for_jsonb_9_4_usage(&old_cluster); @@ -190,6 +195,9 @@ check_new_cluster(void) check_loadable_libraries(); + if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1400) + check_for_cluster_key_failure(&new_cluster); + switch (user_opts.transfer_mode) { case TRANSFER_MODE_CLONE: @@ -1466,6 +1474,32 @@ check_for_user_defined_encoding_conversions(ClusterInfo *cluster) } +/* + * check_for_cluster_key_failure() + * + * Make sure there was no unrepaired pg_alterckey failure + */ +static void +check_for_cluster_key_failure(ClusterInfo *cluster) +{ + struct stat buffer; + + if (stat (KMGR_DIR_PID, &buffer) == 0) + { + if (cluster == &old_cluster) + pg_fatal("The source cluster had a pg_alterckey failure that needs repair or\n" + "pg_alterckey is running. Run pg_alterckey --repair or wait for it\n" + "to complete.\n"); + else + pg_fatal("The target cluster had a pg_alterckey failure that needs repair or\n" + "pg_alterckey is running. Run pg_alterckey --repair or wait for it\n" + "to complete.\n"); + } + + check_ok(); +} + + /* * get_canonical_locale_name * diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c index 018cd310f7..ee6a48438b 100644 --- a/src/bin/pg_upgrade/controldata.c +++ b/src/bin/pg_upgrade/controldata.c @@ -9,12 +9,18 @@ #include "postgres_fe.h" +#include <dirent.h> #include <ctype.h> #include "pg_upgrade.h" #include "common/string.h" +#include "access/xlog_internal.h" +#include "common/controldata_utils.h" +#include "common/file_utils.h" +#include "common/kmgr_utils.h" + /* * get_control_data() * @@ -62,6 +68,7 @@ get_control_data(ClusterInfo *cluster, bool live_check) bool got_date_is_int = false; bool got_data_checksum_version = false; bool got_cluster_state = false; + int got_file_encryption_method = false; char *lc_collate = NULL; char *lc_ctype = NULL; char *lc_monetary = NULL; @@ -203,6 +210,13 @@ get_control_data(ClusterInfo *cluster, bool live_check) got_data_checksum_version = true; } + /* Only in <= 14 */ + if (GET_MAJOR_VERSION(cluster->major_version) <= 1400) + { + cluster->controldata.file_encryption_method = DISABLED_ENCRYPTION_METHOD; + got_file_encryption_method = true; + } + /* we have the result of cmd in "output". so parse it line by line now */ while (fgets(bufin, sizeof(bufin), output)) { @@ -498,6 +512,18 @@ get_control_data(ClusterInfo *cluster, bool live_check) cluster->controldata.data_checksum_version = str2uint(p); got_data_checksum_version = true; } + else if ((p = strstr(bufin, "Cluster file encryption method:")) != NULL) + { + p = strchr(p, ':'); + + if (p == NULL || strlen(p) <= 1) + pg_fatal("%d: controldata retrieval problem\n", __LINE__); + + p++; /* remove ':' char */ + /* used later for contrib check */ + cluster->controldata.file_encryption_method = atoi(p); + got_file_encryption_method = true; + } } pclose(output); @@ -566,7 +592,8 @@ get_control_data(ClusterInfo *cluster, bool live_check) !got_index || !got_toast || (!got_large_object && cluster->controldata.ctrl_ver >= LARGE_OBJECT_SIZE_PG_CONTROL_VER) || - !got_date_is_int || !got_data_checksum_version) + !got_date_is_int || !got_data_checksum_version || + !got_file_encryption_method) { if (cluster == &old_cluster) pg_log(PG_REPORT, @@ -635,6 +662,10 @@ get_control_data(ClusterInfo *cluster, bool live_check) if (!got_data_checksum_version) pg_log(PG_REPORT, " data checksum version"); + /* value added in Postgres 14 */ + if (!got_file_encryption_method) + pg_log(PG_REPORT, " file encryption method\n"); + pg_fatal("Cannot continue without required control information, terminating"); } } @@ -699,6 +730,14 @@ check_control_data(ControlData *oldctrl, pg_fatal("old cluster uses data checksums but the new one does not"); else if (oldctrl->data_checksum_version != newctrl->data_checksum_version) pg_fatal("old and new cluster pg_controldata checksum versions do not match"); + + /* + * We cannot upgrade if the old cluster file encryption method + * doesn't match the new one. + */ + if (oldctrl->file_encryption_method != newctrl->file_encryption_method) + pg_fatal("old and new clusters use different file encryption methods or\n" + "one cluster uses encryption and the other does not"); } diff --git a/src/bin/pg_upgrade/file.c b/src/bin/pg_upgrade/file.c index 079fbda838..29fd003318 100644 --- a/src/bin/pg_upgrade/file.c +++ b/src/bin/pg_upgrade/file.c @@ -11,6 +11,7 @@ #include <sys/stat.h> #include <fcntl.h> +#include <dirent.h> #ifdef HAVE_COPYFILE_H #include <copyfile.h> #endif @@ -21,6 +22,7 @@ #include "access/visibilitymapdefs.h" #include "common/file_perm.h" +#include "common/file_utils.h" #include "pg_upgrade.h" #include "storage/bufpage.h" #include "storage/checksum.h" diff --git a/src/bin/pg_upgrade/option.c b/src/bin/pg_upgrade/option.c index 3f719f1762..b6285d68ed 100644 --- a/src/bin/pg_upgrade/option.c +++ b/src/bin/pg_upgrade/option.c @@ -52,6 +52,7 @@ parseCommandLine(int argc, char *argv[]) {"check", no_argument, NULL, 'c'}, {"link", no_argument, NULL, 'k'}, {"retain", no_argument, NULL, 'r'}, + {"authprompt", no_argument, NULL, 'R'}, {"jobs", required_argument, NULL, 'j'}, {"socketdir", required_argument, NULL, 's'}, {"verbose", no_argument, NULL, 'v'}, @@ -99,7 +100,7 @@ parseCommandLine(int argc, char *argv[]) if (os_user_effective_id == 0) pg_fatal("%s: cannot be run as root", os_info.progname); - while ((option = getopt_long(argc, argv, "d:D:b:B:cj:kNo:O:p:P:rs:U:v", + while ((option = getopt_long(argc, argv, "d:D:b:B:cj:kNo:O:p:P:rRs:U:v", long_options, &optindex)) != -1) { switch (option) @@ -176,6 +177,10 @@ parseCommandLine(int argc, char *argv[]) log_opts.retain = true; break; + case 'R': + user_opts.pass_terminal_fd = true; + break; + case 's': user_opts.socketdir = pg_strdup(optarg); break; diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index 31589b0fdc..878f8e8032 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -12,6 +12,7 @@ #include "common/relpath.h" #include "libpq-fe.h" +#include "common/kmgr_utils.h" /* For now, pg_upgrade does not use common/logging.c; use our own pg_fatal */ #undef pg_fatal @@ -218,6 +219,7 @@ typedef struct bool date_is_int; bool float8_pass_by_value; uint32 data_checksum_version; + int file_encryption_method; } ControlData; /* @@ -296,6 +298,8 @@ typedef struct transferMode transfer_mode; /* copy files or link them? */ int jobs; /* number of processes/threads to use */ char *socketdir; /* directory to use for Unix sockets */ + bool ind_coll_unknown; /* mark unknown index collation versions */ + bool pass_terminal_fd; /* pass -R to pg_ctl? */ } UserOpts; typedef struct diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c index 9a1a20fb27..0775010379 100644 --- a/src/bin/pg_upgrade/server.c +++ b/src/bin/pg_upgrade/server.c @@ -239,7 +239,7 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error) * vacuumdb --freeze actually freezes the tuples. */ snprintf(cmd, sizeof(cmd), - "\"%s/pg_ctl\" -w -l \"%s/%s\" -D \"%s\" -o \"-p %d -b%s %s%s\" start", + "\"%s/pg_ctl\" -w -l \"%s/%s\" -D \"%s\" -o \"-p %d%s -b %s%s\" start", cluster->bindir, log_opts.logdir, SERVER_LOG_FILE, cluster->pgconfig, cluster->port, -- 2.37.0 (Apple Git-136) [application/octet-stream] 0008-cfe-08-pg_alterckey_over_cfe-07-bin-squash-commit.patch (30.8K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/9-0008-cfe-08-pg_alterckey_over_cfe-07-bin-squash-commit.patch) download | inline diff: From 6772e88cdecf4db53b0bbfc55d9d6b626ac7b01d Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:58 -0400 Subject: [PATCH 08/12] cfe-08-pg_alterckey_over_cfe-07-bin squash commit --- doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/pg_alterckey.sgml | 210 ++++++++ doc/src/sgml/reference.sgml | 1 + src/bin/Makefile | 1 + src/bin/pg_alterckey/.gitignore | 1 + src/bin/pg_alterckey/Makefile | 38 ++ src/bin/pg_alterckey/README | 24 + src/bin/pg_alterckey/pg_alterckey.c | 724 ++++++++++++++++++++++++++++ 8 files changed, 1000 insertions(+) create mode 100644 doc/src/sgml/ref/pg_alterckey.sgml create mode 100644 src/bin/pg_alterckey/.gitignore create mode 100644 src/bin/pg_alterckey/Makefile create mode 100644 src/bin/pg_alterckey/README create mode 100644 src/bin/pg_alterckey/pg_alterckey.c diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index e90a0e1f83..a2b96129ba 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -197,6 +197,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY dropuser SYSTEM "dropuser.sgml"> <!ENTITY ecpgRef SYSTEM "ecpg-ref.sgml"> <!ENTITY initdb SYSTEM "initdb.sgml"> +<!ENTITY pgalterckey SYSTEM "pg_alterckey.sgml"> <!ENTITY pgamcheck SYSTEM "pg_amcheck.sgml"> <!ENTITY pgarchivecleanup SYSTEM "pgarchivecleanup.sgml"> <!ENTITY pgBasebackup SYSTEM "pg_basebackup.sgml"> diff --git a/doc/src/sgml/ref/pg_alterckey.sgml b/doc/src/sgml/ref/pg_alterckey.sgml new file mode 100644 index 0000000000..867c439b8e --- /dev/null +++ b/doc/src/sgml/ref/pg_alterckey.sgml @@ -0,0 +1,210 @@ +<!-- +doc/src/sgml/ref/pg_alterckey.sgml +PostgreSQL documentation +--> + +<refentry id="app-pg_alterckey"> + <indexterm zone="app-pg_alterckey"> + <primary>pg_alterckey</primary> + </indexterm> + + <refmeta> + <refentrytitle><application>pg_alterckey</application></refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo>Application</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>pg_alterckey</refname> + <refpurpose>alter the <productname>PostgreSQL</productname> cluster key</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>pg_alterckey</command> + + <group choice="plain"> + <arg choice="plain"><option>-R</option></arg> + <arg choice="plain"><option>--authprompt</option></arg> + </group> + + <arg choice="plain"> + <replaceable class="parameter">old_cluster_key_command</replaceable> + <replaceable class="parameter">new_cluster_key_command</replaceable> + </arg> + + <arg choice="opt"> + <group choice="plain"> + <arg choice="plain"><option>-D</option></arg> + <arg choice="plain"><option>--pgdata</option></arg> + </group> + <replaceable class="parameter">datadir</replaceable> + </arg> + + </cmdsynopsis> + + <cmdsynopsis> + <command>pg_alterckey</command> + + <group choice="opt"> + <arg choice="plain"><option>-R</option></arg> + <arg choice="plain"><option>--authprompt</option></arg> + </group> + + <group choice="plain"> + <arg choice="plain"><option>-r</option></arg> + <arg choice="plain"><option>--repair</option></arg> + </group> + + <arg choice="opt"> + <group choice="opt"> + <arg choice="plain"><option>-D</option></arg> + <arg choice="plain"><option>--pgdata</option></arg> + </group> + <replaceable class="parameter">datadir</replaceable> + </arg> + + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1 id="r1-app-pg_alterckey-1"> + <title>Description</title> + <para> + <command>pg_alterckey</command> alters the cluster key used + for cluster file encryption. The cluster key is initially set + during <xref linkend="app-initdb"/>. The command can be run while the + server is running or stopped. The new password must be used the next + time the server is started. + </para> + + <para> + <command>pg_alterckey</command> changes the key encryption key + (<acronym>KEK</acronym>) which encrypts the data encryption keys; + it does not change the data encryption keys. It does this by + decrypting each data encryption key using the <replaceable + class="parameter">old_cluster_key_command</replaceable>, + re-encrypting it using the <replaceable + class="parameter">new_cluster_key_command</replaceable>, and then + writes the result back to the cluster directory. + </para> + + <para> + See the <xref linkend="app-initdb"/> documentation for how to define + the old and new passphrase commands. You can use different executables + for these commands, or you can use the same executable with different + arguments to specify retrieval of the old or new key. + </para> + + <para> + <command>pg_alterckey</command> manages data encryption keys, + which are critical to allowing Postgres to access its decrypted + data. For this reason, it is very careful to preserve these + keys in most possible failure conditions, e.g., operating system + failure during cluster encryption key rotation. + </para> + + <para> + When started, <command>pg_alterckey</command> repairs any files that + remain from previous failures before altering the cluster encryption + key. During this repair phase, <command>pg_alterckey</command> will + either roll back the cluster key or roll forward the changes that + were previously requested. The server will not start if repair is + needed, though a running server is unaffected by an unrepaired cluster + key configuration. Therefore, if <command>pg_alterckey</command> + fails for any reason, it is recommended you run the command with + <option>--repair</option> to simply roll back or forward any previous + changes. This will report if it rolled the cluster key back or forward, + and then run the command again to change the cluster key if needed. + </para> + + <para> + You can specify the data directory on the command line, or use + the environment variable <envar>PGDATA</envar>. + </para> + </refsect1> + + <refsect1> + <title>Options</title> + + <para> + <variablelist> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem> + <para> + Allows the <option>old_cluster_key_command</option> and + <option>new_cluster_key_command</option> commands + to prompt for a passphrase or PIN. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Other options: + + <variablelist> + <varlistentry> + <term><option>-V</option></term> + <term><option>--version</option></term> + <listitem> + <para> + Print the <application>pg_alterckey</application> version and exit. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><option>-?</option></term> + <term><option>--help</option></term> + <listitem> + <para> + Show help about <application>pg_alterckey</application> command line + arguments, and exit. + </para> + </listitem> + </varlistentry> + + </variablelist> + </para> + + </refsect1> + + <refsect1> + <title>Environment</title> + + <variablelist> + <varlistentry> + <term><envar>PGDATA</envar></term> + + <listitem> + <para> + Default data directory location + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><envar>PG_COLOR</envar></term> + <listitem> + <para> + Specifies whether to use color in diagnostic messages. Possible values + are <literal>always</literal>, <literal>auto</literal> and + <literal>never</literal>. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="app-initdb"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index a3b743e8c1..46f7834f72 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -241,6 +241,7 @@ </para> </partintro> + &pgalterckey; &clusterdb; &createdb; &createuser; diff --git a/src/bin/Makefile b/src/bin/Makefile index 7f9dde924e..018ee6592f 100644 --- a/src/bin/Makefile +++ b/src/bin/Makefile @@ -17,6 +17,7 @@ SUBDIRS = \ initdb \ pg_amcheck \ pg_archivecleanup \ + pg_alterckey \ pg_basebackup \ pg_checksums \ pg_config \ diff --git a/src/bin/pg_alterckey/.gitignore b/src/bin/pg_alterckey/.gitignore new file mode 100644 index 0000000000..4c4f39f2cc --- /dev/null +++ b/src/bin/pg_alterckey/.gitignore @@ -0,0 +1 @@ +/pg_alterckey diff --git a/src/bin/pg_alterckey/Makefile b/src/bin/pg_alterckey/Makefile new file mode 100644 index 0000000000..2133a4ba06 --- /dev/null +++ b/src/bin/pg_alterckey/Makefile @@ -0,0 +1,38 @@ +#------------------------------------------------------------------------- +# +# Makefile for src/bin/pg_alterckey +# +# Copyright (c) 1998-2021, PostgreSQL Global Development Group +# +# src/bin/pg_alterckey/Makefile +# +#------------------------------------------------------------------------- + +PGFILEDESC = "pg_alterckey - alter the cluster key" +PGAPPICON=win32 + +subdir = src/bin/pg_alterckey +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = \ + $(WIN32RES) \ + pg_alterckey.o + +all: pg_alterckey + +pg_alterckey: $(OBJS) | submake-libpgport + $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) + +install: all installdirs + $(INSTALL_PROGRAM) pg_alterckey$(X) '$(DESTDIR)$(bindir)/pg_alterckey$(X)' + +installdirs: + $(MKDIR_P) '$(DESTDIR)$(bindir)' + +uninstall: + rm -f '$(DESTDIR)$(bindir)/pg_alterckey$(X)' + +clean distclean maintainer-clean: + rm -f pg_alterckey$(X) $(OBJS) + rm -rf tmp_check diff --git a/src/bin/pg_alterckey/README b/src/bin/pg_alterckey/README new file mode 100644 index 0000000000..6db3739271 --- /dev/null +++ b/src/bin/pg_alterckey/README @@ -0,0 +1,24 @@ +pg_alterckey +============ + +This directory contains the code to generate the pg_alterckey binary. + +Architecture +------------ + +pg_alterckey allows altering of the cluster encryption key (key +encryption key or KEK) which is stored outside of the file system; see +src/backend/crypto/README for more details. This must be done in a +crash-safe manner since the keys are critical to reading an encrypted +cluster. The active data encryption keys (DEK) are encrypted/wrapped by +the KEK and stored in PGDATA/pg_cryptokeys/live as separate files, +currently files 0 and 1. + +This process can be interrupted at anytime; the new execution of +pg_alterckey will repair any previously interrupted execution of +pg_alterckey. + +pg_alterckey should never be run concurrently. A lock file prevents +almost all concurrent execution. pg_alterckey can be run if the +database server is running or stopped, so it can't use database locking +that is only available when the server is running. diff --git a/src/bin/pg_alterckey/pg_alterckey.c b/src/bin/pg_alterckey/pg_alterckey.c new file mode 100644 index 0000000000..7e3d4ad915 --- /dev/null +++ b/src/bin/pg_alterckey/pg_alterckey.c @@ -0,0 +1,724 @@ +/*------------------------------------------------------------------------- + * + * pg_alterckey.c + * A utility to change the cluster key (key encryption key, KEK) + * used for cluster file encryption. The KEK wrap data encryption + * keys (DEK). + * + * The theory of operation is fairly simple: + * 1. Create lock file + * 2. Retrieve current and new cluster key using the supplied + * commands. + * 3. Revert any failed alter operation. + * 4. Create a "new" directory + * 5. Unwrap each DEK in "live" using the old KEK + * 6. Wrap each DEK using the new KEK and write it to "new" + * 7. Rename "live" to "old" + * 8. Rename "new" to "live" + * 9. Remove "old" + * 10. Remove lock file + * + * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/bin/pg_alterckey/pg_alterckey.c + * + *------------------------------------------------------------------------- + */ + + +#define FRONTEND 1 + +#include "postgres_fe.h" + +#include <signal.h> +#include <unistd.h> +#include <sys/stat.h> +#include <fcntl.h> + +#include "common/file_perm.h" +#include "common/file_utils.h" +#include "common/hex.h" +#include "common/restricted_token.h" +#include "crypto/kmgr.h" +#include "common/logging.h" +#include "getopt_long.h" +#include "pg_getopt.h" + +typedef enum +{ + SUCCESS_EXIT = 0, + ERROR_EXIT, + RMDIR_EXIT, + REPAIR_EXIT +} exit_action; + +#define MAX_WRAPPED_KEY_LENGTH 64 + +static int lock_fd = -1; +static bool pass_terminal_fd = false; +int terminal_fd = -1; +static bool repair_mode = false; +static char *old_cluster_key_cmd = NULL, + *new_cluster_key_cmd = NULL; +static char old_cluster_key[KMGR_CLUSTER_KEY_LEN], + new_cluster_key[KMGR_CLUSTER_KEY_LEN]; +static CryptoKey data_key; +unsigned char in_key[MAX_WRAPPED_KEY_LENGTH], + out_key[MAX_WRAPPED_KEY_LENGTH]; +int in_klen, + out_klen; +static char top_path[MAXPGPATH], + pid_path[MAXPGPATH], + live_path[MAXPGPATH], + new_path[MAXPGPATH], + old_path[MAXPGPATH]; + +static char *DataDir = NULL; +static const char *progname; + +static void create_lockfile(void); +static void recover_failure(void); +static void retrieve_cluster_keys(void); +static void bzero_keys_and_exit(exit_action action); +static void reencrypt_data_keys(void); +static void install_new_keys(void); + +static void +usage(const char *progname) +{ + printf(_("%s changes the cluster key of a PostgreSQL database cluster.\n\n"), progname); + printf(_("Usage:\n")); + printf(_(" %s [OPTION] old_cluster_key_command new_cluster_key_command [DATADIR]\n"), progname); + printf(_(" %s [repair_option] [DATADIR]\n"), progname); + printf(_("\nOptions:\n")); + printf(_(" -R, --authprompt prompt for a passphrase or PIN\n")); + printf(_(" [-D, --pgdata=]DATADIR data directory\n")); + printf(_(" -V, --version output version information, then exit\n")); + printf(_(" -?, --help show this help, then exit\n")); + printf(_("\nRepair options:\n")); + printf(_(" -r, --repair repair previous failure\n")); + printf(_("\nIf no data directory (DATADIR) is specified, " + "the environment variable PGDATA\nis used.\n\n")); + printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT); + printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL); +} + + +int +main(int argc, char *argv[]) +{ + static struct option long_options[] = { + {"authprompt", required_argument, NULL, 'R'}, + {"repair", required_argument, NULL, 'r'}, + {"pgdata", required_argument, NULL, 'D'}, + {NULL, 0, NULL, 0} + }; + + int c; + + pg_logging_init(argv[0]); + set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_alterckey")); + progname = get_progname(argv[0]); + + if (argc > 1) + { + if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0) + { + usage(progname); + exit(0); + } + if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0) + { + puts("pg_alterckey (PostgreSQL) " PG_VERSION); + exit(0); + } + } + + /* check for -r/-R */ + while ((c = getopt_long(argc, argv, "D:rR", long_options, NULL)) != -1) + { + switch (c) + { + case 'D': + DataDir = optarg; + break; + case 'r': + repair_mode = true; + break; + case 'R': + pass_terminal_fd = true; + break; + default: + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(1); + } + } + + if (!repair_mode) + { + /* get cluster key commands */ + if (optind < argc) + old_cluster_key_cmd = argv[optind++]; + else + { + pg_log_error("missing old_cluster_key_command"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + if (optind < argc) + new_cluster_key_cmd = argv[optind++]; + else + { + pg_log_error("missing new_cluster_key_command"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + } + + if (DataDir == NULL) + { + if (optind < argc) + DataDir = argv[optind++]; + else + DataDir = getenv("PGDATA"); + + /* If no DataDir was specified, and none could be found, error out */ + if (DataDir == NULL) + { + pg_log_error("no data directory specified"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(1); + } + } + + /* Complain if any arguments remain */ + if (optind < argc) + { + pg_log_error("too many command-line arguments (first is \"%s\")", + argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + /* + * Disallow running as root because we create directories in PGDATA + */ +#ifndef WIN32 + if (geteuid() == 0) + { + pg_log_error("%s: cannot be run as root\n" + "Please log in (using, e.g., \"su\") as the " + "(unprivileged) user that will\n" + "own the server process.\n", + progname); + exit(1); + } +#endif + + get_restricted_token(); + + /* Set mask based on PGDATA permissions */ + if (!GetDataDirectoryCreatePerm(DataDir)) + { + pg_log_error("could not read permissions of directory \"%s\": %m", + DataDir); + exit(1); + } + + umask(pg_mode_mask); + + snprintf(top_path, sizeof(top_path), "%s/%s", DataDir, KMGR_DIR); + snprintf(pid_path, sizeof(pid_path), "%s/%s", DataDir, KMGR_DIR_PID); + snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR); + snprintf(new_path, sizeof(new_path), "%s/%s", DataDir, NEW_KMGR_DIR); + snprintf(old_path, sizeof(old_path), "%s/%s", DataDir, OLD_KMGR_DIR); + + /* Complain if any arguments remain */ + if (optind < argc) + { + pg_log_error("too many command-line arguments (first is \"%s\")", + argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + if (DataDir == NULL) + { + pg_log_error("no data directory specified"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(1); + } + + create_lockfile(); + + recover_failure(); + + if (!repair_mode) + { + retrieve_cluster_keys(); + reencrypt_data_keys(); + install_new_keys(); + } + +#ifndef WIN32 + /* remove file system reference to file */ + if (unlink(pid_path) < 0) + { + pg_log_error("could not delete lock file \"%s\": %m", KMGR_DIR_PID); + exit(1); + } +#endif + + close(lock_fd); + + bzero_keys_and_exit(SUCCESS_EXIT); +} + +/* Create a lock file; this prevents almost all cases of concurrent access */ +void +create_lockfile(void) +{ + struct stat buffer; + char lock_pid_str[20]; + + if (stat(top_path, &buffer) != 0 || !S_ISDIR(buffer.st_mode)) + { + pg_log_error("cluster file encryption directory \"%s\" is missing; is it enabled?", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + /* Does a lockfile exist? */ + if ((lock_fd = open(pid_path, O_RDONLY, 0)) != -1) + { + int lock_pid; + int len; + + /* read the PID */ + if ((len = read(lock_fd, lock_pid_str, sizeof(lock_pid_str) - 1)) == 0) + { + pg_log_error("cannot read pid from lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + lock_pid_str[len] = '\0'; + + if ((lock_pid = atoi(lock_pid_str)) == 0) + { + pg_log_error("invalid pid in lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + /* Is the PID running? */ + if (kill(lock_pid, 0) == 0) + { + pg_log_error("active process %d currently holds a lock on this operation, recorded in \"%s\"", + lock_pid, KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + close(lock_fd); + + if (repair_mode) + printf("old lock file removed\n"); + + /* ---------- + * pid is no longer running, so remove the lock file. + * This is not 100% safe from concurrent access, e.g.: + * + * process 1 exits and leaves stale lock file + * process 2 checks stale lock file of process 1 + * process 3 checks stale lock file of process 1 + * process 2 remove the lock file of process 1 + * process 4 creates a lock file + * process 3 remove the lock file of process 4 + * process 5 creates a lock file + * + * The sleep(2) helps with this since it reduces the likelihood + * a process that did an unlock will interfere with another unlock + * process. We could ask users to remove the lock, but that seems + * even more error-prone, especially since this might happen + * on server start. Many PG tools seem to have problems with + * concurrent access. + * ---------- + */ + unlink(pid_path); + + /* Sleep to reduce the likelihood of concurrent unlink */ + pg_usleep(2000000L); /* 2 seconds */ + } + + /* Create our own lockfile? */ +#ifndef WIN32 + lock_fd = open(pid_path, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode); +#else + /* delete on close */ + lock_fd = open(pid_path, O_RDWR | O_CREAT | O_EXCL | O_TEMPORARY, + pg_file_create_mode); +#endif + + if (lock_fd == -1) + { + if (errno == EEXIST) + pg_log_error("an active process currently holds a lock on this operation, recorded in \"%s\"", + KMGR_DIR_PID); + else + pg_log_error("unable to create lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + snprintf(lock_pid_str, sizeof(lock_pid_str), "%d\n", getpid()); + if (write(lock_fd, lock_pid_str, strlen(lock_pid_str)) != strlen(lock_pid_str)) + { + pg_log_error("could not write pid to lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } +} + +/* + * ---------- + * recover_failure + * + * A previous pg_alterckey might have failed, so it might need recovery. + * The normal operation is: + * 1. reencrypt LIVE_KMGR_DIR -> NEW_KMGR_DIR + * 2. rename KMGR_DIR -> OLD_KMGR_DIR + * 3. rename NEW_KMGR_DIR -> LIVE_KMGR_DIR + * remove OLD_KMGR_DIR + * + * There are eight possible directory configurations: + * + * LIVE_KMGR_DIR NEW_KMGR_DIR OLD_KMGR_DIR + * + * Normal: + * 0. normal X + * 1. remove new X X + * 2. install new X X + * 3. remove old X X + * + * Abnormal: + * fatal + * restore old X + * install new X + * remove old and new X X X + * + * We don't handle the abnormal cases, just report an error. + * ---------- + */ +static void +recover_failure(void) +{ + struct stat buffer; + bool is_live, + is_new, + is_old; + + is_live = !stat(live_path, &buffer); + is_new = !stat(new_path, &buffer); + is_old = !stat(old_path, &buffer); + + /* normal #0 */ + if (is_live && !is_new && !is_old) + { + if (repair_mode) + printf("repair unnecessary\n"); + return; + } + /* remove new #1 */ + else if (is_live && is_new && !is_old) + { + if (!rmtree(new_path, true)) + { + pg_log_error("unable to remove new directory \"%s\": %m", NEW_KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + printf(_("removed files created during previously aborted alter operation\n")); + return; + } + /* install new #2 */ + else if (!is_live && is_new && is_old) + { + if (rename(new_path, live_path) != 0) + { + pg_log_error("unable to rename directory \"%s\" to \"%s\": %m", + NEW_KMGR_DIR, LIVE_KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + printf(_("Installed new cluster password supplied in previous alter operation\n")); + return; + } + /* remove old #3 */ + else if (is_live && !is_new && is_old) + { + if (!rmtree(old_path, true)) + { + pg_log_error("unable to remove old directory \"%s\": %m", OLD_KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + printf(_("Removed old files invalidated during previous alter operation\n")); + return; + } + else + { + pg_log_error("cluster file encryption directory \"%s\" is in an abnormal state and cannot be processed", + KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } +} + +/* Retrieve old and new cluster keys */ +void +retrieve_cluster_keys(void) +{ + int cluster_key_len; + char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN]; + + /* + * If we have been asked to pass an open file descriptor to the user + * terminal to the commands, set one up. + */ + if (pass_terminal_fd) + { +#ifndef WIN32 + terminal_fd = open("/dev/tty", O_RDWR, 0); +#else + terminal_fd = open("CONOUT$", O_RDWR, 0); +#endif + if (terminal_fd < 0) + { + pg_log_error(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + } + + /* Get old key encryption key from the cluster key command */ + cluster_key_len = kmgr_run_cluster_key_command(old_cluster_key_cmd, + (char *) cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + live_path, terminal_fd); + if (pg_hex_decode(cluster_key_hex, cluster_key_len, + (char *) old_cluster_key, KMGR_CLUSTER_KEY_LEN) != + KMGR_CLUSTER_KEY_LEN) + { + pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); + bzero_keys_and_exit(ERROR_EXIT); + } + + /* + * Create new key directory here in case the new cluster key command needs + * it to exist. + */ + if (mkdir(new_path, pg_dir_create_mode) != 0) + { + pg_log_error("unable to create new cluster key directory \"%s\": %m", NEW_KMGR_DIR); + bzero_keys_and_exit(ERROR_EXIT); + } + + /* Get new key */ + cluster_key_len = kmgr_run_cluster_key_command(new_cluster_key_cmd, + (char *) cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + new_path, terminal_fd); + if (pg_hex_decode(cluster_key_hex, cluster_key_len, + (char *) new_cluster_key, KMGR_CLUSTER_KEY_LEN) != + KMGR_CLUSTER_KEY_LEN) + { + pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); + bzero_keys_and_exit(ERROR_EXIT); + } + + if (pass_terminal_fd) + close(terminal_fd); + + /* output newline */ + puts(""); + + if (memcmp(old_cluster_key, new_cluster_key, KMGR_CLUSTER_KEY_LEN) == 0) + { + pg_log_error("cluster keys are identical, exiting\n"); + bzero_keys_and_exit(RMDIR_EXIT); + } +} + +/* Decrypt old keys encrypted with old pass phrase and reencrypt with new one */ +void +reencrypt_data_keys(void) +{ + PgCipherCtx *old_ctx, + *new_ctx; + + old_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, + (unsigned char *) old_cluster_key, + KMGR_CLUSTER_KEY_LEN, false); + if (!old_ctx) + pg_log_error("could not initialize encryption context"); + + new_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, + (unsigned char *) new_cluster_key, + KMGR_CLUSTER_KEY_LEN, true); + if (!new_ctx) + pg_log_error("could not initialize encryption context"); + + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + { + char src_path[MAXPGPATH], + dst_path[MAXPGPATH]; + int src_fd, + dst_fd; + int len; + struct stat st; + + CryptoKeyFilePath(src_path, live_path, id); + CryptoKeyFilePath(dst_path, new_path, id); + + if ((src_fd = open(src_path, O_RDONLY | PG_BINARY, 0)) < 0) + { + pg_log_error("could not open file \"%s\": %m", src_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + if ((dst_fd = open(dst_path, O_RDWR | O_CREAT | O_TRUNC | PG_BINARY, + pg_file_create_mode)) < 0) + { + pg_log_error("could not open file \"%s\": %m", dst_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + if (fstat(src_fd, &st)) + { + pg_log_error("could not stat file \"%s\": %m", src_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + in_klen = st.st_size; + + if (in_klen > MAX_WRAPPED_KEY_LENGTH) + { + pg_log_error("invalid wrapped key length (%d) for file \"%s\"", in_klen, src_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* Read the source key */ + len = read(src_fd, in_key, in_klen); + if (len != in_klen) + { + if (len < 0) + pg_log_error("could read file \"%s\": %m", src_path); + else + pg_log_error("could read file \"%s\": read %d of %u", + src_path, len, in_klen); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* decrypt with old key */ + if (!kmgr_unwrap_data_key(old_ctx, in_key, in_klen, &data_key)) + { + pg_log_error("incorrect old key specified"); + bzero_keys_and_exit(RMDIR_EXIT); + } + + if (KMGR_MAX_KEY_LEN_BYTES + pg_cipher_blocksize(new_ctx) > MAX_WRAPPED_KEY_LENGTH) + { + pg_log_error("invalid max wrapped key length"); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* encrypt with new key */ + if (!kmgr_wrap_data_key(new_ctx, &data_key, out_key, &out_klen)) + { + pg_log_error("could not encrypt new key"); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* Write to the dest key */ + len = write(dst_fd, out_key, out_klen); + if (len != out_klen) + { + pg_log_error("could not write fie \"%s\"", dst_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + close(src_fd); + close(dst_fd); + } + + /* The cluster key is correct, free the cipher context */ + pg_cipher_ctx_free(old_ctx); + pg_cipher_ctx_free(new_ctx); +} + +/* Install new keys */ +void +install_new_keys(void) +{ + /* + * Issue fsync's so key rotation is less likely to be left in an + * inconsistent state in case of a crash during this operation. + */ + + if (rename(live_path, old_path) != 0) + { + pg_log_error("unable to rename directory \"%s\" to \"%s\": %m", + LIVE_KMGR_DIR, OLD_KMGR_DIR); + bzero_keys_and_exit(RMDIR_EXIT); + } + fsync_dir_recurse(top_path); + + if (rename(new_path, live_path) != 0) + { + pg_log_error("unable to rename directory \"%s\" to \"%s\": %m", + NEW_KMGR_DIR, LIVE_KMGR_DIR); + bzero_keys_and_exit(REPAIR_EXIT); + } + fsync_dir_recurse(top_path); + + if (!rmtree(old_path, true)) + { + pg_log_error("unable to remove old directory \"%s\": %m", OLD_KMGR_DIR); + bzero_keys_and_exit(REPAIR_EXIT); + } + fsync_dir_recurse(top_path); +} + +/* Erase memory and exit */ +void +bzero_keys_and_exit(exit_action action) +{ + explicit_bzero(old_cluster_key, sizeof(old_cluster_key)); + explicit_bzero(new_cluster_key, sizeof(new_cluster_key)); + + explicit_bzero(in_key, sizeof(in_key)); + explicit_bzero(&data_key, sizeof(data_key)); + explicit_bzero(out_key, sizeof(out_key)); + + if (action == RMDIR_EXIT) + { + if (!rmtree(new_path, true)) + pg_log_error("unable to remove new directory \"%s\": %m", NEW_KMGR_DIR); + printf("Re-running pg_alterckey to repair might be needed before the next server start\n"); + exit(1); + } + else if (action == REPAIR_EXIT) + { + unlink(pid_path); + printf("Re-running pg_alterckey to repair might be needed before the next server start\n"); + } + + /* return 0 or 1 */ + exit(action != SUCCESS_EXIT); +} -- 2.37.0 (Apple Git-136) [application/octet-stream] 0009-cfe-09-test_over_cfe-08-pg_alterckey-squash-commit.patch (114.0K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/10-0009-cfe-09-test_over_cfe-08-pg_alterckey-squash-commit.patch) download | inline diff: From 92ec48f38c405d6da82a72d9a5429f4b3a4b1fc3 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:00 -0400 Subject: [PATCH 09/12] cfe-09-test_over_cfe-08-pg_alterckey squash commit --- src/test/Makefile | 3 + src/test/README | 3 + src/test/crypto/.gitignore | 4 + src/test/crypto/KWP_AD_128.txt | 35 ++ src/test/crypto/KWP_AD_256.txt | 35 ++ src/test/crypto/KWP_AE_128.txt | 35 ++ src/test/crypto/KWP_AE_256.txt | 35 ++ src/test/crypto/Makefile | 39 +++ src/test/crypto/README | 33 ++ src/test/crypto/gcmDecrypt128.rsp | 129 +++++++ src/test/crypto/gcmDecrypt256.rsp | 129 +++++++ src/test/crypto/gcmEncryptExtIV128.rsp | 129 +++++++ src/test/crypto/gcmEncryptExtIV256.rsp | 129 +++++++ src/test/crypto/t/001_testcrypto.pl | 137 ++++++++ src/test/crypto/t/002_testkwp.pl | 126 +++++++ src/test/crypto/t/003_clusterkey.pl | 93 +++++ src/test/crypto/t/004_buffers.pl | 157 +++++++++ src/test/crypto/testcrypto.c | 458 +++++++++++++++++++++++++ 18 files changed, 1709 insertions(+) create mode 100644 src/test/crypto/.gitignore create mode 100644 src/test/crypto/KWP_AD_128.txt create mode 100644 src/test/crypto/KWP_AD_256.txt create mode 100644 src/test/crypto/KWP_AE_128.txt create mode 100644 src/test/crypto/KWP_AE_256.txt create mode 100644 src/test/crypto/Makefile create mode 100644 src/test/crypto/README create mode 100644 src/test/crypto/gcmDecrypt128.rsp create mode 100644 src/test/crypto/gcmDecrypt256.rsp create mode 100644 src/test/crypto/gcmEncryptExtIV128.rsp create mode 100644 src/test/crypto/gcmEncryptExtIV256.rsp create mode 100644 src/test/crypto/t/001_testcrypto.pl create mode 100644 src/test/crypto/t/002_testkwp.pl create mode 100644 src/test/crypto/t/003_clusterkey.pl create mode 100644 src/test/crypto/t/004_buffers.pl create mode 100644 src/test/crypto/testcrypto.c diff --git a/src/test/Makefile b/src/test/Makefile index dbd3192874..617f1dfa29 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -17,6 +17,9 @@ SUBDIRS = perl regress isolation modules authentication recovery subscription ifeq ($(with_icu),yes) SUBDIRS += icu endif +ifeq ($(with_openssl),yes) +SUBDIRS += crypto +endif ifeq ($(with_gssapi),yes) SUBDIRS += kerberos endif diff --git a/src/test/README b/src/test/README index afdc767651..0955a764e3 100644 --- a/src/test/README +++ b/src/test/README @@ -11,6 +11,9 @@ which tests get run automatically. authentication/ Tests for authentication (but see also below) +crypto/ + Tests for cluster file encryption + examples/ Demonstration programs for libpq that double as regression tests via "make check" diff --git a/src/test/crypto/.gitignore b/src/test/crypto/.gitignore new file mode 100644 index 0000000000..7b92218ad2 --- /dev/null +++ b/src/test/crypto/.gitignore @@ -0,0 +1,4 @@ +# Generated by test suite +/tmp_check/ +/log/ +/testcrypto diff --git a/src/test/crypto/KWP_AD_128.txt b/src/test/crypto/KWP_AD_128.txt new file mode 100644 index 0000000000..d77aeab719 --- /dev/null +++ b/src/test/crypto/KWP_AD_128.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AD with AES-128 cipher function' information for test-files +# Seed = 0f0c6b6602d959d22ff2305478ea477a85d196d9695abf5b445010b45ce046c501bc04e6ca4faf46137adf6eac452d97314cd91137e33fc30cc6df316117fed3 +# Generated on Fri Apr 6 14:46:45 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 1dd51f0d3a0a784174ba81b2c9f89005 +C = e1bde6d2df3b8e48ca127f97b56b5dc2672b3736cc3157c7b80a0316ef1efbdbbce19fea23da831836ccd2e002b2c1dfad206b5cec358446b8434d7f4c39e65b0e0b50897642ffc34bfb3cb3e233aa9c1058ff0d4fd48e98bc8cc3d214c06d514dd97db2278093a308f91f4ae92626d85771fb1447b36a3467fff02ac7e81ddbd0fdbcd02d1acd4f053c989ef3dcc2c01e23bc2f6090f3e8c0ba5f0082341200b1c37b99daa9cb6fec78bce3429aec5badb9fd28fdbdbdc5d53570675a9e39535b4594095658ef950ecd79a162223b60d2eb91765e022dc6e1bbdd86f1bcc280ed9df350da08a801fa16a1bf2701947acfb08f19fdfcaa1d76f466a5de2458a78fb82f6af3e1be68f405a4289f25896f4c9830005c9e895c86e67eceab0ad544856071b8d9585835b5e85a07ab01515f7ab54f98dffb4ca49a15068eefc6a01f7f52fd1adbe3631c59f6f43f79d2b4f2a691e2b30bb1d43a848dc3ee39c7f2e50f0c9deb7ab51e33bf40903ac255bb1510fd61676a6c13c3c776b8aacc6cefb95e24973ebb11192e2692dd0c6a085b58f86e11cc28ee2194988c123e3666da7339c0a4ac6afbacc83f1f100fbb39efff7cc605c9213828224a17c476395aeb9bb0a3150fb8889a8c2a494c8c526203f261642bfa69a94b86de9e6d3d932fe20fffe4bd76d502c0d437a3e1d0d8727b7a8dc0e361967109e93566326b6c517663731c4c9bdd0295d8 +P = 1a4eed4bf5b8d2e2a58f1f1277f164cc32cdadaed848f76fe634034082ff9aa1711870bf3936d01a2aa48de30de5143b9148cf56f4490f9d480dda0b672e8e17a012cd26cec3c68837bd5b2f9beb13e0110f21c6c36343e09e027f39557d1596d4ca406e3e7aa113e9bb8623106bae25f0ea23d46bc29970ba2596f83fe4f73a6f978a4d949fa7c271570a2ae5d2b50792d5ab5c43d455f359fb83c35ca3da37cd73cd66b6adce94d78ecdeabf667daa47ea70799af299e1d898ccf3fca6c42c6fff8cf2ec992f596fed4a0cdb502a00f9b5689302931d15cba691e2f8079a0411332438b714ace5234b91e4aebee8f8dda0e1968c2016fed350430a65d8d206c9436f40b79ce03083b8dc207d6960be1ce97007ed22a388ebb7b3d8f7d2b7d9f8f49731fbcb21e21db0cdd15674c795d5af2b2cd727f83e634e8c47157ed0c6873a5c9419e683f16f4a7827b444967812f9d1adb9201b89a0e66bbcf0591465f5d7036a21cdda0e10099feb819dfc37fdd3105120044dab716882d3971f312e3f4459006fd5a1eab08ff63edf6718f47ddaa37f7f40c9c372995f3aec97bc45e287b64fc8cf5559ab04a4d4d3ed482f5d61d3abd99cc87ee406da3ab9c9cd22ba3b8d191b26754aa94a2412f39e332d77fe72210adb0cbb5c96adebdbde036f1f1aaafad74a7ac2594f81efa734054e2e16dc931d49b970b81756862705fcd4 + +COUNT = 1 +K = b3fa008b5947ce58dfbd354dd01f2d43 +C = 55cd8e45138f477ce0a84f07bd28a93d7d628bb4860207a2f6dc4256bd79843e32c856a4fa831d1603699d49e6c36291b60aa80635900cc6c78cf0a2ddc457beb41782de0de03f08a064df90b41f2e98ce61185d735380403fe56b68f8343a801a14afb8a7ba79684dc2a585110da83e9a836cae1fd9e1a220dd6dc922b4f02b15ca88d43ab61e1da24a9b3cb99c4e5024ce5667f4841ca2a305b1f4c1ae9fb63d1d4dcb83870755a1a646b16c088e612d82ba2bf0e7e2fa0e8035c3baeb595f1ac9bb49b01f6f71392e217c049c0e9bd794b9aa2383cf59ee0a90f965610c65ecd629a17cba2bdf2458e3a8e1a9d219cb66eb9ec8e5226b34f95003064952523920a0b4e94ec8ecd1bdca8a65fe46ed25fd4d076e46fa62a8cde6eabc593045d17cef996ebbeca4b537f65c4f683a10baeb4c42b9867bbb49ca7ea1c5437bc114948c542cffced9bb1ebe3c946eb24ff55be89be004596ba648b264167217d267b881020b905f508e4f0e1a58eca051d56ff30d91891838c574c3de54e3feafcdf514740ddc94ba92cb85fe86033e67f14d90be7a0222e4bd1624cea8894df66a36a8e848dfe9168d8024b7ba5636afbcf6b945a53e6b2778f229af7dc2e59bebbf8bdbdfde1e21465f6b6344b13afa0e5ceac212b3b88932f21b1ae04268476597c92e64ff7c14b9ef678f10a35b56cd70ba03063f94aed97b0a6cf883d1f07facfa37b6e5b070 +P = a067ab39cede4ac6c6cb7630cba48c52a794ac8ebec037125bcd97d1a3c52a8ed64764899f9035a6944d0605a5d977172a55bbf86cd81aef5d6bafb1ac86bfa65da2b3c39bf5da94a98f7b6dbc5df16a7b38061e0665ad16b20fb6aedc9ce7f6d3497c3c55cea92e6343f21251092ef2ea307b35f999683298098bedaea847d1ccbf8bda18dc477e8d49fee4e357273396ad2245703485b97b5a7d97057bad875a3e76b67ad5adbc6ef3b8ba9a1786aa93149f0f8dd166535acbf93f1b9839754d537da3fae1ab02973427c3f353fe9aa6c5a100bf0e6ccb08dc1fdb0fc363a95c77c5758d440db0a70f0340a4c488de51e1ecb932ce2fcb2c95ea28c9f55695d97ba1765c8f11e523ae3e4e1efceb69000a192c047ab197f4840c664c035064ecc12926fd3bca0527a160b5b5a2bbaf5db11437f2c38a1c7535e87f552b9f04f2fdd309a826e4ec7708217022fb075cdfc6cc23e9301e33068caa69ef746f357b09ccc098443a3a2979a225e70be1e722e8d6fbb57d0dded2456c1d47eeb0af2241f769836026fec8fc51d97c4abbe9710a4aa5b95aaac83bee57e1333fa244ccc971b6260a9be16e31cc2fd283fec1b247a7340d149fe5309acb47c9cdb955b7bcc4df277eaf611e8af281ff0bcd64b4534309282d1b5cb14efa93141869d67ce7e418f06bb4c2feebcb7a1151aea2eb8bc2fc4dcee53de9b2fb1803490caf + +COUNT = 2 +K = 4b4c43c9de4fb4a2a7a7adafeabe2dbd +C = 6e4d08b8124f7d3e23303fac1a842014f95e3d71c438f8f1990307842796dc5e404ad81802e35c183fe000390a12c81ee684c5cf26c1d90e414cfffe6931b0f352936fcf0b31429eb5c7612cc359a15371390e518cf5c6a6bff1bb0348d14e2c39b98c9f30672ed2af1d96296df8b5567db25b9510a2083461810e119735490058ed1b46b7fdfa885041d8749f90a072b43ba49f2f51fbcda0dbf3cf99fca1d8f46330e5f6fe079d6679cfa26214c8831b782aaa023a2e0ea91050d277dab876aa6865f2bb3fc1a4a77db52f6179d5e5325993280948b6b7002b572829641d35ed3d735d8423e5b24673c4570ca25064fc2c2ad4840632536bcfaf2a7a814f3eaed92b4d501bc51c1719a0d8d8f420b66db845682bb41c88038cfedf13417143a3a701b521a9bf0bb639875a728c3b5ce6ca7e7a45bc75285c193902e6b5e7a4c6e720493d3937bf485e587bff894f70fd6165a1d0129cc673a992e0a4f5489d228a066b1df60002ec0521924f8d672cd1452fec927e58e75807b2a390256f920743fa4d0fc8f59f2469a595ef65095ca0c80adfc843e9e69b6d4a3f824af47b2bfbf2a7a6c1b650378f096f6f0bfabc752c8f279d4f45d56d09dce97962c119de3a64d83b93ea55066f24d4238a229ae86e6a7857af1d8aba823370a72fe358046049a84a70213ef31d9e77a722def8e21480e79b71299438070946bd459a7251707446c911e381 +FAIL + +COUNT = 3 +K = 96ab719a3d08df2393ebc330e151dab1 +C = d50ae797f6c3418f388a7513d693c6dd665e858767531fbccd3eb1aabe796690ec8fbb757d88b169adf5c136de50ff0f2cfdd8389f812382578aee0b0b61e13c6a2bc500640fe1585f068eee0d1fa3420220e23090e24e3248fe16f4e0c7c0e996a21b4947ddd08fd3ccc1f036651be4f48ee1ffb486cdc05911244480548221d8da1f2bc37dece080e51b2cdd1ddebf37213a4dfa1b252e567243d9cec8c89eb8db544e7c389a2e13f1b91d860df3cbcec3e85c93276c2a9a5fa080efc85e9bad3bfe2d9bb06498dd8b3720456bfabd3c69b345f6954872baa1d43b9f7ceb92ae9ad77b270d0b94c79275a48874dafb136105f5553529687b6aeeaa521790b9376c9f88ace94049235cd52c4387ad210442513dd5e07171519d58b1294fb8ac1f60ac68b8f07b418e1bb0598601ec38b9a9b137dd87d0c8a41089d17ca1c720fd0e7e3b81b85a373753bed0f5e29586f84cb29e1d88c379c965c50f6a803ddfac2e1555beb9c208a3821f53bead8f120f4ef4a1490b730a0b8a2f1869c6b985520d709bdc0e5fce44316b8aa2448a2743761bf77bdbbfdab6a721a8ec79f38f7e7321a80a2cd3a35a912eaac5eace85c4cad3c6685b88be4517cd1c20971b85bd9e8eb6e52869e014831dff7585a163f5a4dbf1d59160104da90a9cfcc8d6a0324942b40fde4319a32442d83ebbf5d7a36e9495be2ffd0e7faec1b66c96f71843750b8a051b7170 +P = 3a3b9e6de537458875e59204ef7565b6dde796e5ab11c83f7a361b8143f0f7a7eadb5b53c6efa6d199f759cad5c029004024eabbaff717bafb95646dc31a8f6063b9f8faaea650dfa8803bfa0c79091f299a55f78611c2e0d015021d6c6d3abf3d85cac306740acc144201516b787421a77c78a566c6eadc88ecdeea4ff861b6db73f7b00f0a8f62faedefa58866fb368424d7267afdf5ff1279916d2f177408d780697e1c45e58a524bb0365858d2b5a42ee2bd9e8904134d04cf071e84db8a31804aa8bebc0b28dd621360385117764178fe74b29da3ac390ac4812fdc7eedf91fce6eaae3d03163435001ce42f55982daeda5cec5deb960b35df231463cbc26267746be628c53b55f4f21ef003816eb7bfc6c710efa03d0994a1b3c8595fc9293a2c101483798034d4ee7e3d5e07bbd897c9de4b8315e53cbd1f81bdecbd59d093c844a0ed1e3e9d238707a7b893ca453745223c67756d9062152b239ceec44c436e0896a59ea9ea8cf79a93b8b759389bb5e73c5f5330e26580d9777817400166d826008be5e8c7184ae2ecf8fb9dba92af3c747c74e1534c05395f9204b5e8481fdcf4dab5ea6224a8e0ee52576d467d930c0899d31a4e288e3eecb8cb7a3be3a66c79ae93033de5d0d422a6d54ab002d1a82f3f60db97834d9fa3782dd64cbec8ddcac2216a393dc263cea2705fd072ec82dfa1ddef9c588c49f17c275 + +COUNT = 4 +K = d8c221e426109cb5911d7d6f0836f4cd +C = d853d57eaacac8096346564eccd33281ee864fb290ce91eb717fa153ca00064e033635178c59860a567215b7320fab4a72ccf716501dbc9a44d5b3d501729674987d2413cad79dece055a9b0d47ec980331f4a236b31984f5d62f9d7f58c0f3afb81fe60f266652da65d06874334be065f56096e98536bd1f2120313b0905ffe2f2c3b6de265ab7800c42be810bed18548c08f9193b02a3981a922b32b618fd9a978439ea382bf2890ad1f30d115b2319276289cf4f7a9917b0c064180e79c8644f9ac880a793b4a8ee424dff32cf2b6ca46f52ff8bd8359ed18ea8aac23e63ae337f5baea9e2ff845a5fdc0b79d5767d47c2a1a536d889f553c52696cbe91ccd2ec671a0644689bdb0f4db7e5d58c854eb539b6b4cd9214e361a216d315b1b124b43c76c703c01d3bc3142f760a399ba4887a6e326a58ecaf56fd49ae128a86cda485eedc3da80b75b171e77cade00c903c1f216eefa845dfaef660fc5ecb6791ed53765683f44da6c4ed8a9ad9e995f7d920cdee8463e79b18e8874b0a2f573b1825f8a480b1ed1245c81f4ec097bf0a0504aab9bdaef27b67d98805a7ec687c8cbcbc92ad3ce58651162a1f57f8af427ea0a111dbd6e3c7f240eb6b2360650a72b9c1c4417b1d541dfcc2a8d6ce3e8c160d8d417e4efbdce809bfe30802696bfd52a0f40be4db9be247dfd867179d82390b55180ebc6ceba0a990e3f6d32eef9dfdb946706371 +P = 92814e18dbe6e83714c4a82ba3ac3baf682a8054eb36666c9546db040d40b8613fc560d97b05265fb19ade180efeb55dfce2cc5981ca222e66b547b78a42401710535b1181674fdd426cf2b0b55e5b7f0505f11307120d495cfb197a3de00569b3d39f93c27270df4725243d314a026549692b0e2b4079c60a8053f0f36e83aaa3494307b175fd40643c1bc264eed1c00f8b565f2a3aeeb78bd94970bd9267d21f5a9a1b07df09ed44a3bd4255a139a328235b921833f92904a74ec202b0eac65df1caed05dc84e52b06c1ecf0f7914324ac4d828b7de7189705308959be42401948e3bf4bdd50ce24101c67ae745a73a67d7e366b6b432ce67b05cefd149a17247010f42dae4de1b2ca42a8e71824cd32c5cb2e2055443ec3ff24339c774dc9207744b84e9203fda1f85595f961987d847ed33867f1ddfce0795e3f2d78c5d749a488a4997392b8c9022c810197c93186894faa55cb0b6775b57a7ba2729c617c9430a44098d5081e3c5c4908ddd1a475cf9211408a8ddbe19ee527ddb2596456e1b1481a09b04e091b1c14b7b2e41bb4434a906736e115cb25ada0950ac5d2845b4a9f1e95f4d80f64440e983324c3aa9f3ec8964f9da0d26aa47e86355aa80ad99d0e573fa9932da70bd65cb1a06d8cb77e455fe7cada4561e027ca1608132c2605b6d0489bba6b29f293951883c451f37bd545f6605364ddc75918df097e + +COUNT = 5 +K = 704eb91dd5ba3d85279cf47c01eca2a5 +C = 51c71fd7778aa3648f3e31e1db0c73cb1479372f2e35f65f00188f08f794993a2ee2bb7e91cd1a2b86e92b8ccca7277207fb525ab17600173fa28844ae27f093e0e5ae00585cc714dac90cbe9b6332cbe4cb689b2cd141c102c6881f5b71ec477c5f4a91f7bdcb5871aadd478f1a9ccc6e069b7283f4d70b26e8748eda6d443ab13804c543a44fe2fb366f90de35d83fbf6354a9a9ab4a93ff7d61cbc0bfe05d6102c9c393273e7d3a04d61eba771f05cee29e5dacb7abf34ec9159e121841e2c39848f604c8f743313cbdca828bfa4635a81136e7a37f230c0d3c814d35c2eaabdd94183312909ab3a09b87cce0c719408f837bf24bfb2dad87630aabc9eab35bdb9cc536198389aceec68e8779f9e1eee84392189823a68195b75bbb6d33addf580564e696a362928e2ac506b79480600bc2f9eaa3e96f323390d1d92cf3c6d4bd4147ada5634cff2bf2d97b259904a335eaf11ec3fc84dcd8e27f7538e0fac1dbe7cb4533f4fa58913535d957b90678fac58aa96694a8047ac774afe488ab429c6807e709351f8159dcfbf83b865aeeb26722ef64a537ce932b2cfa6d53ed6cc1ca8ab58748c06a753515fffc56e294f51ab257585b610d261c6fe12def38a1b5dceaa4681569124c679b20984ed2967740419b342e9010eabd291de026f6e829e4dba5300cb668191358ab58e178c29a0194a639233f9c28c50a609bc42f8fa6cd17bc58eccd +FAIL diff --git a/src/test/crypto/KWP_AD_256.txt b/src/test/crypto/KWP_AD_256.txt new file mode 100644 index 0000000000..6d28f39705 --- /dev/null +++ b/src/test/crypto/KWP_AD_256.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AD with AES-256 cipher function' information for test-files +# Seed = bf1ba8f321ce0abadb40026686c9d9e7f0c4a55388f2ea7cefc81aeb0e054d40c94f48093f2739580010d6ad6e6ce734f21e7338100b750ec9c7bb06bf46f7f4 +# Generated on Fri Apr 6 14:47:06 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 08f5c088acec18e6cf1f03a8f85d772e327e7fb07f8c2939eb554e84c42ab93d +C = dff30fd43647d4be54cf2dfd6187e2ddffb55267313f980fb09c833a9c2bfa558a95861711f0acb2a5c7e731ba22f24a9c4dfdd9e9b0216e9088f817a175b9835b0e17615687a20f68c067205626494cd04fbabc0b3eea7c0a4cd6236bc8b3e52e721dfc357fb8a3722bfcc4c690d8f63dbb864bb6e3a15805aea7270f8eb748deebaa2d066fcda11c2e67221f9a91d2c29a6c79ffae76aa80a2590b4f9e35f623fbf2f8ceb2a205493077556a186e25e5bd52dcff7bcc6909b37a66c1d1431be1b363bb40da25386eaaf5fcabc7be6422a04434a21d1d3105328e7c56770b9f59b03395e4138f5f06fc7e6b80dab87b08caa7bfffc45a095c15263efd3f06c651ded6f58074efc20620d704997fc84721a0a8e9e5b9f5cd330bbb156b31d9d1b1c260e4a24535f30404dc5b2dd6b35d916a1391b25a7d8790be09d85483ed1522074a2785812005bda10dd55acb245b3bd3d9bb777dd23f9b02538ba1a114ba53386d7ca4d9524b2f8a18e0ffb21580b560540bb2146f08f04974b90eb324547d56222df95f44bc6e5f183bef283e4816fb1b2933f9c7c6726a245a495e304d8318d0008c51b0be8090f8f668fbc3f31e073be4b9e97468f4dd8c798e9d682868df493db8a85738b58cfd005190f365849072577772672c6f82555c65046eb34e86fe61103327a063bacbbe33cea7eaa3d1de45471b7269e1b6b38608626e323447a3d5fe0599a6 +P = 8b68f66a3d2f59d419851b94d9a6f2f0e667f8125e11d463a6bc2cea46b12dcc40ce8018b204972c735fdd6d2d05b628f4905c6690f5ac5b1b51e12f3af2dc3ae9b9dab616f0a2a66a1ac197592fd5b15900547f32f54110b58d51a0340aa80e9eeb7b2e0eb97e80aa22ba918f2fe1c678c730ed5c3d8d24774f17d8ab6e01a06243d36e764df1dbb8af1faadbc55281f0242abd7a162c984fd0b05ab8b0bcaedffb2962024f009a8d7c9e71281c09f52ec0707ee3bbeb1ecb918be6ae3e9c1fabbcd3512af928db3ba6c109ff9e9839a616b2a53f092160a48222b84d53cd52490515ef93e1ebb33897263492ab8ec6fad2e633276ae367f76d7f926309478c0205d4f22506a451795dc98f5410d8f5d3e049cbedf381620861e7b4ae08f2d8a71abc1f230248cb636a2d7b4e7717ab2b7b5f2dc6e5b5a18e8043254208b50fd6f8929eaf974c48551233661ad67321b64d69245d536d9a8ca2a6a10966dddb9d2ce36641c9281c460ae524b077867258f638e6ac872cb5f5c6fb216b1ae60a9d0c5ea0dbcd060f255da26111175af4e9935df59ddade6a2a70cddff8cae6a98e4f3843c2dd59d09053b07b648a46f5de0eb21ebb192828279a386ea3eedf2cdc355d73d51111e8c1d522e059752bc56226a4225bcab713bfaaaec78167d7cfd33e913b26fda93ca7524aa8a8b17977c88ff9bc23ea810b4de59eac18d1523b + +COUNT = 1 +K = 94c4d5d70f881e58e10e7246cf812d40e2be258adb2b6c13c6603fc7daf7e85a +C = 6c07b5ffd1b9be182413ef8eae4a6eac657108a46008a0d898727f2711e6fa0ca60fd1d51fad683b57d4202fa2b0eb88b856e08b07155439bdb03890cbb7e0f228172bf297a4e0917dadaa5e89a287bb9ba6441c852c5b0cff5084e6c425aaf866815b3fc45f5f7fb5d14b270343e6a30f402e11d62e433a0d84f65684b2df78d4e7758bc0bf81783316905cdc3c1150ec47f225c966f7f339b2538970eb3b8a2c13f95df1310d6e3b2a1f8aed19105846557d8f0018fc0f17146bf836b654dec98e9ad639c7e4b2f922b4396e82c690cdecb65f5e0ea282dd6262f34346ff9adbc8b2f361ddd4356f0feadf7c750fc0580c4e12c00ee049d06eed2242b14727ef4d58386dc8df279a7bf8131c3befaea2f059ab757826e5e381d49a2f11b8cbc2b0021af4da7a779e5df0083edeb54348cc36ce96a19a3d7ff5bd2f19d05fef6b200e76399a02a991111832173353bff4ce1859ff534ae13290dd176ba8e1384ed24d9702dbff127e15e5c66618f94680271732d19f64552ed03df76dc9d46c3cfdd53a1b253992fbcbea6db006f16e8dd92406f0090ad9100856c6b71f7767fcb895136416b374285efe1c6506941911a380e2bf74ffd0f67e853f9ac7b5df6666b177a2908fda9add0eb798f8ccc52801535b2bdf9507f3fb3b46915aa889d62ac5909040a1a28856105dfe2e10d5cfbb569c380551fc8bbe7d83dc87ef7a92faa3fff4b1e2c2 +P = 85693a16ae69d751cfa6799b95a6396de2eabe7e4da74d734691d992cba353a39f3b9615c1325db5b0563ce1a846bb0f0534a86130ce6657736b9a9b35b0f8d89dd1b3a295131d2f3f57f94deef9606dad76a377d0b24e632b3680e4d3338f3e4484609e8063e9ec621297f55802d7c347e8085ba6e514884b8fc1ae109409c5c3a5bddf4daf034d300e31eccba07a9380f5325666c4a3aa12d60b30ca272fe03534aabe78ba0452a7e4648ebfd4645675629676be6f122a54b6b810cf9cc0c68b7c61470a537a5a664ec24dbb3eb4f9fa8355cc7ae8fef27a0146df5ccc585d8c106a1eeb64ad4c701fd5a54ef18295b07e9e47f7f7dd2f67d38ed776a5f0b28843cc4bb5d7fdbea9cb0088dee849ae232e4e016d8cf3681971e8a45d6b25451538212b91f30e17580a8107a7a95587a06d22d615f5475a5f616fdf2fab79152f2643054d96ba88f50888eb0f2f1f154c6fff53dd44c3613269751dca4fa86f45d6b1af9ad0159685223889529609e7003c8f3cab491fd6c1a020305da8f94ec833d721d9fac7e575c2a1bc26eb4fb5010c35ffbd39b98d857f12584f4ab7de92aa6d7e7148a0120cc6b3f7ae47a291ba1cf55a28d38d3a30dfc3917d663458cf840385ca81cf70acce45a5cd509f8387d450bbdd6fa51830cf9a7387887c620b86809c55a3eb322ca784a51693f1054759804314ae86048f0d9c99650a5a12 + +COUNT = 2 +K = d65338fd3771fd58c07b6b689577378939d439628529b92cd5625edd18afac76 +C = 1c429bf25c144a2cc649fbd60de5c26c31a0c352de99b34b86101c551994f082feffe1db8853de59b3e8593785eca100a71c5392f0c71eca9f411cbd87fc77ea1a96376dc13f6ad460a11e9cd5a829875a7b7dcd2ba4eaac08c5bb48ab5d4c338a6f8bc5e760739edcff2db116b5b1802e35f936d473db168edd12532a992bcc418a759cc9fd3f97f561623078af29d7ab489b7ec564ba981c188f11240dd9354c324f8d0cdf1c74252f0fc75e390e837b8be90a670f5803ee53eb75c3ce95b2853b2342e54f86dd9aeb308eb82ceb2bae7b3e0b364d17105eb61b3843f7206bdb6abb818efe0f0d3b1004e370191e8218cca14947aa8070f7c66fd0422b02ab4a1d94fa46197acd24e272c765667353e819588402feb85d7f00243521d0e7a9d9e70753d8b51d374ea9c8355536594bf05a6960ca7176a4b66086b055b099e315a23e042a7e0807316d7a11a657a6dc9043806e248a9af06570f710af65267d436a5fcb001104fe8a7c564afe075d85bc0a2ce3d33d8d93d5ab1e923f51d4ef26cbb6fd4a935a97cb115aed678e75d5d67fbfcd2362cb3d74ed6b9b9fb0cf82569a474a25e5aa39d22fe5cd301045203d9f93cf5c9e9e9451f1bf3566eec75fbd995cf8c640aa68fb04f5419344057fd1c0e655d750a68c523b0fab24cab03d7393ee3a5735039daed52895dfe7937f55d7ae9a8c0256e9d638a8598452f5329353a20c4bd9958c +P = 58b20979cba48a9dc95a8857f5bce433087ff93470fc62546e86e72dfaaf7b233ffe428802390c1db7cba00b1f23678aace4a16a237b41d26bcd83d471030929a34e8467f85eaef070b9b74a57f13e91b4e95a3c0b8dfea87d026196a10168c152c4ac42718989003b7e688ca43207034b674d3cbab6f57db6513f8883d27f2280c742896a62e7d0f3f20377e98a0688652d270887fdacb86daa086ffda17937e6d20e4a82667f80ac7749a889b0d748e906d653f569b86de2b42b5819ade9c92970d4caeeed8cd5759d56fd38205215bd8401b2a5a000990afe6c9bea8d091171e85ed83f45bb5b9a8d74cae897cc36f1eaf0122693990b1fb57d0025ad6d92c90885accb649368fe237c4cf017787609fb93c9ea5b413847a9fcf2d2ccb6283345a278619abf8dc351682928187bf92551a820939ec73928eb9930c48f7088ed0a367882f4a8b20d754c5f06bc82990da02227923eb8d1cb73c23793ea0d19bed4a9986f0d48d7835733d1ed3396ec3cf15e1854473b05535261251f4f0af8a0743b3298888bec2f7656493d05eb2d9b848e6802845fb9f7835b50d6a0f0e6cfdaf9b1afc6caa6573b3350256e6f23cc4681316705e33eb0a5f664b79be556cb1bbdd0208430cdc95a35f61facbe7ca2a9bd329e4a1fa42aab9bb02f6519a5672346a4cfac1b96a969317480dd995e339af888fc0e43692332d583fec6215d + +COUNT = 3 +K = 1726706350c11e6883955f24ea11ab247ce3b2ab54d05e67ad9770b5564483dd +C = b006f26a67d0e1e2cbeb5c23b6b300adc1526d1f17bbe964fe8237ae244878158e6b04cb488786b5258ac973c3a2eafd7fcf3a7ca6c825155659fbc53d112bc78b3a770cf059fdd5e68f2b4bfa36de3721231102e5041c947fba3d906bff39592ec3901a398da23035f1190e99b58659330cc2e856ee87ad4197dcc7d16e1f062275bced1ed5cd82163ae3e58da7368dc2aadac855385bd4fa0b8baadef608d0a5c27172d12b88c70b136eeccf37f36364361a990dc50815743cab1636e661bff04ca8345520c30b935a060b450526b1d6ac09170e5b0a327b88f42327b85c9a621d2ca745963c2815a2bfcf509d50b6058ed6e67f369b5608d2aa885238b67d1b8e0d83f9464aa473bf109350fcc02e360c2619236cbfbf895b607895530d8d3d2e41450750dad05b1c37ef15db7fb4707597ac252e8e58d4c1ab2713b427643d198164c908b5d8ff36e9700157284009c7b283633d8b27b378bb65eff8aa59b5fe5e6437a1d53a99c106c2c4d033d3d23950e313a10eb31d68524ae9f8e4f56437acf66db3e8f77407a15bbff4b393e5559908993146d93c673d2aeb7d4cb8fc8d0169de7ed6e2bbe6ce9958a0f5d201419e7acb17e47da827ba380d6b3ad3b5a8c2101c5fb501110c727169065f23297947f538ab3ec165d61edc1f6a9e1735e9b7fc06d4d3406cf8f9c6a68b196cf262324a986705fbc802cdd2e6b4ebcf68e6bb9e793ae644 +FAIL + +COUNT = 4 +K = 32e57ccfe7563dc0a20c14ee450837a33606c086ce1467fd7ec58467154338ab +C = 977d9c5f6861a69e13cd854299434e348cd0690b4d04e08e0598b47eea621bcd8a22838dc9c35a72c35fb1a6434718d02fd24cb4b3dd90b0430334a938a218467eeb4c373d446a539810bc3ce1e923b7c20d9f58ea931d4f964c79613bce67b268efc44bdb9bb00a68d60037949aec7a399493defb2a466e33d4831efd63ad1cb89e00b530626d2f0165975ddfc4cc5e0f968d3875de0f674b3a517df26480b02b6236ebb377118268cebb30ff1ddf0e280fe1bff61902a017e8decf60753c642f35faf0565303bfe651ec8f0193cf34d4af010c9925b8871f0f8c934a149d874a3b659f78ad148428aacaeab80b1b25dec8b0f7ce54406287bc802ac2c0cca3db4adcaa8400a8636ea339b62f5e94d5e32fd3d1183b374507a2af620ca1346dccc9f83a4fe855b1c0e91db9e7c532828d0944d9a81b553ebdf35e24119ed8164bd0260627ea011e93bc103f208c76498ddb8bca15fd05324da5473157feedd54592aacaee68852968eb54c69eb1ddf607917c57493ea380de0cc6ae304dc49cab80a31b8b456986dc367c70f144e52dd604c8d5edbce5de5efb30d9470bc883445b34fa4414f44bb94a64362a12b546665721fa6db82f0c947f015978412b2ce136c471c98b1f908315a16c83e9318e64508c7e179a4429195a9b1ccc211a1c1d4e4df15c5ebc7ab90926fcf7da03657159e440e93adea31ee35f72f2399f5fe2f8c560c8826e23 +P = 80de48ff805a88f3b359451bb6df61def9cb3551e64fdd3a3a70a3b6d238a69311a85bc5924e395ce92ef394b1e5dc301233e9a212f7fb86272c42ddf5f4857c38d0dd259dc1d663c0d729e033d9b0f7f01ab1f8f1b7192d40921ee0d4696a3e35663c5ffcff5ed167660bf6b4c00e619512a2e827be33c90eecc539e18acc8c76eb332b28b1cc502af571242342f63d155271da3211352128aa0af70c9ce78dfdf084a13049b7bb6f2bd10dd385b412d60bf1ccc9fae1208f39dc53db471a04d0dcc3703b4f7b95e72ea815b64a1499865a7ccc5b740999e76338e1b251c740d75274150a96def8760a08c5a8a6f58273b079c06ee09f79a976eaacc8a04c365bc61a786b496811121c386d274c413a2fbfae9464db6ea775233193395740fc9a5eca1a3820d33f6f7b38a83ccbacaac16479225e108acdf46ca35e573151963721b73b3e1c9a12effff0c3a622eb9f07bef7ae712c96ee3ba245597fb8d511698d6e819a967e0d1868c0c6055333b7c13a98cf63d6a5d87779a95345ca8b7e9e597ec588e96f8fc2a7f0a0b8f1543d9e362a911dfb1f03132a4e6af71b503c41814d6b684a26b8df00cdc657ae129a1f2a18cf4b78a3981de68296b1268609fe3ecb9928b90df4553be37319fc508096fa54b35e4822328569da60a6c660f30c61f02f4c5ab2527cf36cb7da8d7dade4c714ea3fc2da8f65b4199090e114dd + +COUNT = 5 +K = e3982db2032f2b4ce658fc44b76f5964c45cd31bf803708982ae599186fc3765 +C = 975e49a4b9a770957a1bb2be920a4f39b9cfd69ba46983d2473d631c08132b9bf61c44510b8aa8bd48c70a86276aa1149d8fdefad511d15d2e2037d9e920e640cb71a97663d19eb90d0b74d9764d03e17cda87ebec6e35ac2003cb75bf9192920d910188d78e2e664255fdf6c9190319d34adb858162ff0830f37fe1dd44003d3d5a1f9451949e368f46ad1977ce622daadf8483a1f60359992b9b366e8a81ffbe96cee45d3aef2fd0ad8c17cc34927af77a0d6d0c5deef3b4a25c82ec388667a493bb0599ac492b351246cbad6d283bf1820883afa48bd909eb7304b9fc5b7d960344309133aab7a85c49f7de396926f50bc83c95900cd049eac1b387aae7fcba5345496425f9216e1fd15c20da75fbb26da176149b40a701e15a7bacfe899e3ecc534ab8bc5b7bd081fb825b5f40fa57e363d7bce40020e73f638acfa097b89c50cb9edb0bd6d71d429b8003aa5dcb7d61792eb3bcac795954c625a104209b373c28cf02038c3318916edd2b818e6719ec154cfa56afb2f337d333069f915d0d35edd6c278fae23c4440c40be462a1dcab23758e4a7fbe8436493f58e890092ea71cb8bcf1336e9ee16b852ccc488f21682dc9f02bdf6c56fe8ad04d84a3c69d8d06dee3d126c0a75f142d0c90c256139acd4b719573e588b80b4540024a05a35044cf58d89673923a534c3816492e62379797cd6e6a7464da5eaaf11ee7b9c27b9b03d7b53c03 +P = 43d38ca132545b154995cff07082611cc47a6467a980654d2d1f1ccfb3bcd387e9d7ffa281b0e0b00dc8669207e0d8033e9e36613c98978f8644bb7e505fbf491dcefbe19589254c8abf859dd65cb94dfc99e7b9d3d1f0a31f21285963e1f7b45c7490a522ff887786f7940fb6192f5081ce7181944bdca5c5bfcf2589f9173a682b78fcdf971dd9f4e8529033e15cde560984dcf796914206973dcc13f8c9a24b25dd00c11166ec6ecf33c6ad9b487847abd7bd29b4f3b9c8dc93a6a5a31723dc03245884bfadc12b2fddcc82409d7b14660af808d4e8216157bb6ba03a319193ad4dacbd37ac884550962a4de26ae923f8d74f2f694fcd0aa74f2e809da4689aad9f2820684b3b423ec4a7da0ce4a1b599fc21bd2779653283b0ee81d7b0d9fd3f6d1e75bd71af9620630aa87b73f7b12e68ddbdfa02ae86ae06b0b1aee4a997d34f61b466348b92e36f83652763084a215c47dcf689df17e36b64bae3ca1a2cc22c837b5907236833c2c1e5f3ddb74165fb6f0633990122cbe4af8b5920b1bb6961cdb144ea8d7b245d0128ab76f4fc0189ba97385717e89e0f99c962ee8c2b6e55546a18be0ba3dbebf7e4140eed6aa3558c43115b65b6f6e8e8fb4b9cfbe0b6eac006603667b28cefb4dec037f33568a3c94d9e36539e91b3199d728521a9a6b82b96ff1c29dd1d10366d0510f1b9a9494cd104db2390530be3fb6abdb7 diff --git a/src/test/crypto/KWP_AE_128.txt b/src/test/crypto/KWP_AE_128.txt new file mode 100644 index 0000000000..a682d4bb32 --- /dev/null +++ b/src/test/crypto/KWP_AE_128.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AE with AES-128 cipher function' information for test-files +# Seed = 7c0f161159cc9e338308ddc59a655450a030832b3d4576c568d63725d57b020073aee9c77d9fec34eab358b83bf5aae4fb52803343bc03d472283edbebbcf75c +# Generated on Fri Apr 6 14:46:12 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 0e54956a24c7d4a343f90269fb18a17f +P = 817ddabdc5d215eee233adff97e92193c6beec52a71340477f70243a794ce954af51e356c9940e4ab198f0e68c543355f65ad179cb2d60dd369eaeb9ed141fb18c9e4054ac7fdc83506896990a4d20833d2d6e9a34938796ee67c9d7d23058544a4a35f2954103ce443a95a7e785602075ca0a73da37899e4568106bb2dbf1f901377d4d3380c70fa5175ebc550481ac6f15986a4407fde5c23ff317e37544c0a25f87117506597db5bb79850c86247b73a5d0090417d63e4c257ea0220c2c04db07a34f0ab7954e1dfa2007a1466795c4d0c2aa09ca3986c028185b43a466526594afc9c891c263a7c608304bc1957c9873f544dc71e6f847c48d32026ed03b2333825452ee7e12a50e1cd7d678319264c65f78001996d37fae7f9861fbd21cb506c2f8a3b0ee53c7debe17111b6e3f78a5c5677857b082c2c4943dfd1edf6337fea98a44fc25928361156ef38d865948b979cf6f4b46bd2119f12f0891cef7fc9d0638fd105fc05f9968d16948d1cb820751e82e44cb68e99d4f072ffd1577da6c0631b5827bec7e1b9ec72d18b74cf5f233e85013c1668ceb5d7a1f5e0f016b0ff726a0a9d41e2cea8e14a2f56492b14606d3fafd8ac141335f39f90d56863735628e8f17be90e100ef0785f3cd57db8b9d89a6b2189dc2ea00c285d2657983f8bd7883c215477e67a55556401f1d8b27d4e0d541c7fb7ace370c2e428884 +C = 876f3e53ba9cf4f6a521ac198bc813d0ede0f862ab6082e3e0a06ad82b4f279582f7c43bb63574608446bc2a05f401a68f74086cf2776b4b3df6b3679c2edfb91c024db54c6831e0752ae6f86c7596462de905ee0be908c1b9d043ecafe2ad1cbddb904e18ebc9b7a107031be3a87059516a3d1257812d9c801b0b9f21539e70c47150c128d87c5e58fa6e4371aedde69c7b5cd16b73ac422676328131f3ac48c602bb6e0741805aad9d23b33b3523b86cf0588cdf9dc6c4d5f9fa43d88ca17976eaf48fb37a41a598266da04144373df5631cc5126341c200a0c8499b29ae96e6e6e6c2bdf8d8903da62bf8ddae970569b695240e77f8ac5b191da5034008b6ef21936858e69bac372bbafd8794f6b03711503c1875528a9348681844edb199a0664d740f0f0b1f866c4248c80fe8b5700a3c4134cdddb17676e0cd37d6d81831a0f4adfba071bb0935502480eccd48b28be5954ea6c7d873b51b8bd2b709c5b6132ed31296510915073c18f7012f0eff6a9aad5340a19fd5e372d35260b718d9e4807b1954c24e6a4fd48e4dbb8f395474e99ab577367d2ab5ccaa18c947331047dc3986e213a878b41089aa221019dad4191a4feefd095f8606c2700a46d71cbb13efb6957df925ec26071c04d04d5a94e138e5fc5d1f059236aad76208077dcc607b1dd2086f9c04e33f955822b457eecd68bd5f24836ecedbac675e6ed93d8a787cb57ad68e + +COUNT = 1 +K = 6b8ba9cc9b31068ba175abfcc60c1338 +P = 8af887c58dfbc38ee0423eefcc0e032dcc79dd116638ca65ad75dca2a2459f13934dbe61a62cb26d8bbddbabf9bf52bbe137ef1d3e30eacf0fe456ec808d6798dc29fe54fa1f784aa3c11cf39405009581d3f1d596843813a6685e503fac8535e0c06ecca8561b6a1f22c578eefb691912be2e1667946101ae8c3501e6c66eb17e14f2608c9ce6fbab4a1597ed49ccb3930b1060f98c97d8dc4ce81e35279c4d30d1bf86c9b919a3ce4f0109e77929e58c4c3aeb5de1ec5e0afa38ae896df9121c72c255141f2f5c9a51be5072547cf8a3b067404e62f9615a02479cf8c202e7feb2e258314e0ebe62878a5c4ecd4e9df7dab2e1fa9a7b532c2169acedb7998d5cd8a7118848ce7ee9fb2f68e28c2b279ddc064db70ad73c6dbe10c5e1c56a709c1407f93a727cce1075103a4009ae2f7731b7d71756eee119b828ef4ed61eff164935532a94fa8fe62dc2e22cf20f168ae65f4b6785286c253f365f29453a479dc2824b8bdabd962da3b76ae9c8a720155e158fe389c8cc7fa6ad522c951b5c236bf964b5b1bfb098a39835759b95404b72b17f7dbcda936177ae059269f41ecdac81a49f5bbfd2e801392a043ef06873550a67fcbc039f0b5d30ce490baa979dbbaf9e53d45d7e2dff26b2f7e6628ded694217a39f454b288e7906b79faf4a407a7d207646f93096a157f0d1dca05a7f92e318fc1ff62ce2de7f129b187053 +C = aea19443d7f8ad7d4501c1ecadc6b5e3f1c23c29eca608905f9cabdd46e34a55e1f7ac8308e75c903675982bda99173a2ba57d2ccf2e01a02589f89dfd4b3c7fd229ec91c9d0c46ea5dee3c048cd4611bfeadc9bf26daa1e02cb72e222cf3dab120dd1e8c2dd9bd58bbefa5d14526abd1e8d2170a6ba8283c243ec2fd5ef07030b1ef5f69f9620e4b17a3639341005887b9ffc793533594703e5dcae67bd0ce7a3c98ca65815a4d067f27e6e66d6636cebb789732566a52ac3970e14c37310dc2fcee0e739a16291029fd2b4d534e30445474b26711a8b3e1ee3cc88b09e8b1745b6cc0f067624ecb232db750b01fe5457fdea77b251b10fe95d3eeedb083bdf109c41dba26cc9654f787bf95735ff07070b175cea8b62302e6087b91a0415474605691099f1a9e2b626c4b3bb7aeb8ead9922bc3617cb427c669b88be5f98aea7edb8b0063bec80af4c081f89778d7c7242ddae88e8d3aff1f80e575e1aab4a5d115bc27636fd14d19bc59433f697635ecd870d17e7f5b004dee4001cddc34ab6e377eeb3fb08e9476970765105d93e4558fe3d4fc6fe053aab9c6cf032f1116e70c2d65f7c8cdeb6ad63ac4291f93d467ebbb29ead265c05ac684d20a6bef09b71830f717e08bcb4f9d3773bec928f66eeb64dc451e958e357ebbfef5a342df28707ac4b8e3e8c854e8d691cb92e87c0d57558e44cd754424865c229c9e1abb28e003b6819400b + +COUNT = 2 +K = 5f0f2428ad83ea04642d8f90e31f00ad +P = 05f65580a602fc839c5372778bc495219ab159935acf3caed10958ca0e135cbc44a896aa84b1f6b05d65ac45f5930029a223977ca46c7717809e80d8eb91e7ccbc525e69ca0a85d3e45b2f9bcf0c86980b7f9f9eb20e9b558479aa8c814e6113635a9b8b0e264055da68198303ae91683eb8ed3351dea3b8280a504d163e1cf284834e3641a26868e23fcf460d279d3577115d8c4cf0f54374df26ddfbf0c1df5bf71fe509231177ac531514ed878d3ef44a03685dee906ab394bfb69cb56ff973f19fb466ce550ae7b03008feca7046bae160ad47f28c9004da68e594bcd7f9ca8674a645c50768b5ee0ec2088f66602a429f95492e09dc567c11083eb18da97d85968a86f6ad05054f4a9c4bcd57ece2511bdb38afd78084b0a088112b81b993a4d37674d97e8081795ee055d9c37d2907d859be201e73580ebd439620918e7c4b42243f43a59b475095205c40a1d37c0f87a5df19450ccd5f274f186b0f45f7bc153d30ce3c873d7bba5dea5a8bd348b2f0b4f25461927642a2991c3c4bc3ad34d09c104322bf27e8eaa0ca99c0dd47a3c4c618af02599192e362e75de53db336a68b223567bcddcd7feecde0aba91a1eac0ce6327d3c9c5be36698494b2b67ec0efd36b744d794f521bee251b31dd29852a94e6b95f143a53f93f85ec8c2e86e1026ec949aaf97fc9b8d728c13e1deab628e9624ca966992efae61dcd6e5 +C = efa9d549d0e975789a6c84b99991c98cf20db4de6789be6f9f7ae109e322a6f04a1a1c1fe582e4242a5e2f2b390827d7e52f1bd97a4e745920dda69a25fe93fa5de82c09dce8a2593106667820dfa1f55c8a1152ee02ed0c607ab986e5209859cbb36baf90ca63285c3128fdceb18e5145b9d3efe6bc7b8bac7837637ce5553d634b13fa3888201112248bc31e0785f2dbae1a8eaa7f4ebc0b00d21242f03acd4e4b0f4ad5e9bb1e25301a331a4b5535c1253b9ccc7800cb2e80602ec9ef858865a39e0ba28006da93f9c26c1fb9ee8258dfcd7e79f02ab0d9c2df9ade309fa6c604fcbeb38641b2968ff96349bcc34927cc479c30153125f7ad07068e5a4ffaba903d27488c94bfb694d8277251061a7dcc5b5fbf47ea88065ea1df007cbd4c1bdbc2e3609d53bf99bf2c7de8ef457fd7b85882210fb52c5093415dd2b37084b75f83d2a05a669867143fd95705f241fa794526bc65c5f6422410da4783340497bc54852125e3da579bfabc9ba98ec398487444b3da454f40d8b59d384cdd3454ad77f38d4ce657d10b4708dd4ecb0fcc3238aca57258e082e8cb86d802b9722388284f232811ec35b49ebba151d5b0771feeb6120d8bee8d55fa9c4de45d8f950752b3b9c6526fc4fa8fdd02c645fb053cef8293437111c85b07e98b3b38ccd3de93cd6fd08fdfb8a3a94b619f9acd226be3bd3ff906f14b1dda93cccae783c04ed545db685bf0 + +COUNT = 3 +K = 9b5c2e4a9e53d3e5f5670af2440439db +P = c4c726199cc27b81c90ee59f1f44420abad2a4fe0f24647952116ec46f364a3dad1b48ba5bc6c48c056e34764ed4b4a2da04e7e45ff477df8364ee9f116c82dcff7e9ae2db6a7722f2261b6cee1ef7cc66820184988e002886a6b636264e7704383543817e119ba3582d42aa00aa78dcf84d0ac92ff69f09046f89e7b608143699f9fda37b07d3e2949e935bba230e0a42c2d41d285042feaa480bcc1ef8236d615f449107fad97b5be514ae39d586b98ed45cc39784c30a3a256d62336002b1f7f6cedc3e5418d6f444bacd74d7bc89b3ec65c52191b4a2f3b3c7b682f254e6e909c5dcbc11d5def56792f5d2183e5e7119f268b0395e7f4f25b8574bc921da8419af30113bd35b11c21debb979fb698752264595777be7665419052cf3265117bce96b61f27da4072f828bd9315ef25a9ab9eb4f5319907cdbce6104054c8563a997cf477523478e6d23527f91997dd43ebaf629003530de4b8406bdbfae49c0d97920b745c78a665c438ff064d24803382f79fcc2e24265adaf707394b832685def125b69b1787719aaf92e518645ac22dd94d0dcb03f052c87954854d177e2caa6ab5d9cf3dc4ed4ceca90d2fa0b235efd1b7d17ba9758987d127b11204047f22e75158720feec549f29b67a8158e577325e178d6b757770d56a30b85d12ab5fb164a07a5f613b53d075c44cd3c1e3a311515aeed2427f70b3bc018dd796 +C = 09540dbbe97e983a3420ec1ef7d1195f9cd32acc93e9f31d07e68dd9d963f32bb23565a37d97e857e5e40eae27946ef83a4da10ec29ea001fbe892133a4814b04aa3ebabfa295a19429c7e6b263293d07a48f5b2beea2930aabe6268aec00a8fbcf9f4865d4658905413a0f17bb08bbc6466f8d3e1744d9ffe6132f94ce79bec4ffea05785525eb5e071ecf30081281df58f1b5295842967a486ee262a7c366978140bcb8562abf1a417c7b1ffe07b1748978408cfae4beec8290b312f15d3223fa7447f5a89e52ddfdc4f79cf1d9fe4ecad3e674e98136c457e36748da001970cceed0c61d2ea5f9ee5f9660ad561b9fb808be3d4455fe76ae9778a49526c8461f99dc2c271ff9aa389d3e6a546a82d15967a82e8ec9b64cdf25fb73afccd7cc2fde8d07ef4a1345976970997facd5be16566c7761d23f4a31909a5a31bce35e33596a861e426fb56d9ea13e5f6219aceda2450434e04c454ec8569ef9076856cb48f36618e690a710dfe8717526001c3be7eccb671dbb4430289924ae975a30b6fc7079f8c9cbc8a268f5defc03d94d5bae0474987b7ebd4c9e29eec9080e8d85776c570089fcd235372518dc38e481f8a5edc2105bb4191cd81a19e51c60252a18b3058ac1ff9368af59a2caf82a06357764fb3721287bd28204d77dc2eadb0913cabff7cd63c73488e92c818341563476e69a50fbb781f151e02f2df4a9fe05c8419a3393d67 + +COUNT = 4 +K = f2713273ad5b7add10d2b8d8b61d2f68 +P = 1c189d6d34b57c699dd8c79e30457c6735f3697aa6825e14257906e18413580727d676b6e667cd58e0a47b738abfd91144952940de60b161b0dd261d68af12573820f4f172b32d7c032ed7db3715fc6ceb92ec2d614a2005ea1aed14cc5015de143aaf6dc2db406e0fd31976562250e80eb79140bfc8d14fc79bea4159d66e1013704b474b0b1377220741dc55cecde0eed4e0931e58025e7fb672117a8288587cc91b1a41fb4d1e9e4a7ff59fdb814dbd9521dc148abfe16ab62fa45a3dc361cffb422482f4fd93466320a11d9a5438959947ba51d7761b2a3bb9e0b843ea6a71cba54a96bd14de8360ac8f16ece773d7bfb25df63c1aa67a7bcb587f618f2425bf9aa018eba5aebc2911b9fb6276e48d72dc74836a290fad0f5c545934f1231ed57c6944a464905fb08685e815ecddc8dbe4eeec1355f414fd2ce975c0dbe29bf4369e05c1582cfd9a7ddb7659247a0f6db38ba165f1f46a472760bcd2609ffc8bff49c6f54698063fa81305696f7ce25c95b237c6a4fecb3e48167060f556359609b73e738c5c3021d0e5bc62ed31a2a27f43dc417c0041f8a2402fa7bf76c72e2b290c060efd93ef65d0a893314b03c098a6674d0137f12eba3ebb63b3e65fd0aac89847bf9573d9eff0b1283843bd4edd6d459de6d559e48633d964dccd8b9fb5142596430b2373f9448606f9286157e810dd0f6a72843b0cb28ab452d7 +C = dbeedfe9aede047a07402d23b1b349a3e75490d3c404240c872470aa4f6b5a1723b6a0e455a8961fd25f88a9329b4fdc7d8e6bb2b3d7403bd65418f94b933467d882c1aaca702a47e7bf44916ce5eed0647250d572686c8a8c9f9437f2ceb13b361aef929d428c35cd4cf1d8970d98e75c46b4825c871003eceac2412adb6681f0b8a1d6d6bd7eb41c6e58ebe6909ab061d5e832f7580dce89eccb4ea2546204e895ccb54429fe388b0d9173922e08625d43f0a10743ad76230fa458b0971028b18e0f04f3c1049012a8bbc2f93934e0b51b943efd2fa4a14fdb795a1f21302529653583edcfa3c613ada2bf69a93e5a29a20a6f9d9df7891e3b438be4f9f714806cd57afcabc0d5f5753af804d885ca1ae23bb5fd2593fa361e92e7d12eea0805453056022256410369dab0e2e8d2d4c9ffda1d7df8ba5ee19c5bf62133d1ca9a713fa45157d327bc4e1d8e7a105012036939a5d5ae15a5fe8ef989c8323129b8b4e43402bb4f60f35483685893e9ea0b0dc4a8238148d0220cd9bdd7dc42590cc303f706cd81cc5f6ce761ffdc570c62cf734cfffe0ec7be17c7ae3ab34c65f1a53267b03de4da3a5f921cf01b068c4098c7594a3051bb03940cb6c5bac2aa04f95bd30bd15ccc9edb8e29091047de011ee9591cc4c83244750dceb8c2eedda797965c1243521c19228eb2d799775dd83ceab5aae082cfa743d58544b61b7104f3d10f632994a4 + +COUNT = 5 +K = 8967c8058742169ab39d2d528130df80 +P = cbe7ac6c232b327f7d9154acf334891398fa0a1ea64a4f15243f488ecc784749fec41fab7a447e2a2cc75a44fb548a46a46e51f4c245f64901f3acbf18f468a28367c51060c34762c8a88780044761579846557f285c97bcb9c966aa08d4cf352c26d8a2f308803393155437b91776ee74941da9db39ef64d9ac9261f2c7fb6349e52111a7c146d5a4ef6d5d00717ac93338fea5ff0c90cdefe0d66b039045330877d8596de4c35987ce70f660aa8777276eed1b128002dffd90ce03224d505950eb2c73f5e958df92ee1cd8dee667bbb8cc71f87e4ce880585f004160c798ff2cc195b09921c73951332bd437840bbadcd10730272d46b3f23963a7c0396275407bb8041a6e5ceed4d1dc2a65a70d4843660cc039ce94a4ce96573811d2b8be00e91dec7486484ccb050dfd6e319001ba8b8fef48b03d17902cb91792dc420c79e823616cc79af4e738dc077519015c734c1be0150921657f3139e59ba4ad8e75e0f7b51806a96b7249709faac3cdcd53bfed8f0326d19000ebc1ae05a43e96c1fa02f2cfadefaf871250aff6eade8ce8d4b8d5a686d553e8a66d96412f2f189fb724e67c8acdb229501414e45c430a9533d211a217eae0030ff10db12a399567e94c900ce839157ebee1f4f876675794cb3642ace55fbdb261f1369807f284f4e6f8a2c8330eed88e18b8ff564fb754b96f8c85d28dc7a0b18805ecc3ddffc +C = d29eb0bc54c43081030a3e1169f243eac73f1d0fb2b2f983fb25755684df4331a1932b02e237502c1b73a8d744619f521457b72fcd7401973ec48ee915054e5e12577ef1212d0e642c9dc7341cdaffd344dc480eaf9f29019355b07de2f05897cb91b00b4c53714609dea69f18c7c93429b73373311e0c4a45e2f2b3a6bd393b69f90a4125eb635a794563a68e092cc66730d236909cce6585d34f47a10c2968f161ed79c16c18a6c093c1fc455eb502770ee436dd48c091a4cefeaf449c5b27f2fd2f9150a601bccf8a37cda65b25917728ce3f1f12f55a51d9110c16132bde100b6479eb8a1ae670a9ca396f18a2ff19fb5543dbd3b189e51ba5f12516129071c997044df747f4ed70616e784bfb948a63211b224dfdfc5b9116b869fc5313c761d55c759a6e275e9ac53e83a5386e2cfa50f6c1b541cf752da800ff991ff896b79b9e691d67e96fffd166a06f0c300b7d9bb552f5ec7df6eea384751632ed9357df5a5a9fda931568930fedb0536f5491d4067a3951dc72903d1ca2f5f1b713e24cce5347b056e2f8829d34143032f53edb81a765d2ab3e6cbe12536cd85ac3230fea546ee50d991e13c562bea3d694d3990b1783a373a59fe12cfdb74fa0275e367d76bdef698dd33c30afadbbf0fe0dd7ddf288aec7a397e263a5d537df9f606f5c126565c06ed5e0bd6753d3eaf5002b472a1d7f4d7df7d9a8fe8aa2c595ecef8cfb3da3e7 diff --git a/src/test/crypto/KWP_AE_256.txt b/src/test/crypto/KWP_AE_256.txt new file mode 100644 index 0000000000..66d8523987 --- /dev/null +++ b/src/test/crypto/KWP_AE_256.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AE with AES-256 cipher function' information for test-files +# Seed = 6eb04695fd5dc2b0038aff221b225c302fc9cde3d6dbf4195d9d4d2756c8303906335e07679fee9a80e5d0d1580f7cdab8f0fdd78c0fa58f34afccdcba4820a5 +# Generated on Fri Apr 6 14:46:23 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 20f31cded60b8ed8d9d3fd1e1fa6244e76c7cb7628bfd28a5d63ce8aa2c9494d +P = f07225202842c8dede42215301e44b9bb7e625d3812f74f9b6ddbcd024ebd1f33e2cbf280b9004941f3cbf86c880a2357f88f92a6dcf8dad9da7dddcd00f3635efdff0af4382024e93c2af66b991e565eacca6b886f07178c9b4adad6f0d6ada5ff6aa7cd0712519a947a8089cea5e1e3e40ffe1806010b0149f9ffc7c4dd3c31b3d08d5ae1997c52369393d58611dff9bec501c1ab35e6ed3e7f9445a34e211010a8236686f154e0a5ae3433d6a844eb3884961aa6592216d93952b46bb58a4195aa80966ad0ccd4a7e23823912556a90d5ee9c3bb952ecbb9d895dabd3b11ab4f2e3a6c2582de50403289230ef4dc46e7c0d870a3f0cba9d643a0349503c1b162ddb6350e699589eb47bd563999f55a1adb6b78b52f006901b0427ea7d3394bb0adae4637b4f1ad5d5425e2c8ff3083506d7ad7ba4c7405a778b0a3a11760c96900a5256956cc9710091d073a19f46a985d004651fe2b6448ed761bf9bc81619cf273a6783d868d090753bf01318be21afd88d9f3a961a69f93e9d9fb822c80acc7b48cf14a08b5b7ef15c66975721b7cde9761a145b679155472a44dea8fedc0f86ae7ebf6283ecfde5f2444b51569e6723a7a19e28cdf8dec6791ccc14af95abad018f741575b343cb1a20a2a9adf4248f99728069a1e2e78ad8966c41c9918fb7019ef56c153a183a6247d22d9956564bb03075cbfd1b43d96818b28484 +C = a5b63618fc0c4512960f00a1f226d9837a90480baea75265453b9553b12a58c72153080842d7f8710f317f88fbbbf97caf879ab4bf416ba767ee9aeb34357f4a2d0e8b9571054d98e28804a70bc4d74807f2bfd95ee955bfdbb6f4d6969a0c3c3b541a514647d5cd8c9740ac3496095c3f145c50c97ec98b935158fbdf89705d5330015e48ece89188b8c1bcb2ad6825d865b375a9b9056b743dac720feeac033c9f757f6fe73dd7c4a747661b64cf490a0dd43b547cd791a5d78dac97efcd355f7ebac248fa2a33e4fad640dc34e0d40b0d36588aa32f0864c9446739a6b44ff84666d723bd7d646c5172cda932fec34ddaaba342b02a9604087ef042a2be4774194b5d32cb3fb112438fbf2801050b5424635fa2d3d3fb10332965c73e6669e65195310a3a30602640e9809179cdfc50de585aa1c0072423c626815d281a06eac3b6ffa137716318e288e3f9970e415ef0451bdc557968febf9eb6772c1f77cb8e95701246d9c567048142bb25e340351b87d7391822d9ee7fe51378bc0d08135f9f39cf44b348b87937939dc61f430dfe308cada632722e23aed5a0699e039cf0563ab8025163744b136a13ce3c62c748c89f5e17540f105e7c6ec9ba13515b504342f9e6dc7d65b9a633d8c0b5c9fa858dbb9b3a594406d478a81bb9abfa289730408c1e303c663a61d5caca00f615065312580042862397b9aa8c80ca812887664c439c8c68 + +COUNT = 1 +K = 85c8aa6d9c83cbcb11550c95cb9fa991d49dc89fbe2531e10694269f38e9a309 +P = 58ffd9e73b5f64cb951b7df799a505d842ef0cca6d4b21027680793565a67d47db0bc872a7640691216c044fbfc0be1cd33ffe9fc73c5ad8cf38bcfe6d4b7b76639ef4200ced0eff2d361e9c948075f09398fbe5ef24d370a5a16b515c102d415813cbce56f6a8f7b5227db7aab33c4d8162ac3c178e2344f3c7e123f0e60142112b960137c0176c450248167e9714db8566e2d4af731a7ef3f88730a4be4cb99bf46eba39527ff279be67aee1f4af25b3396aef5883fe7086fed7285af5216c7945d9bd61d907a84095d6fad383a7a90ad4061f7241a84358cf9fbc7048c010c7652bf47f9ff58f1c0cfd62a5fef92b02074ecbf4762af71eeda9aeb8497d7b33beeb4fdbd80b72f2c6dc9a12409cf7ee3716fc07515970411ed7e113b7ad9865d822a502976bba1d2ef254dda4acdf6f55a04c4ac5f6a5e011375af4cfd405d1220a10465a4cd01d160d179fc7ce614ef4023d55dad363e3639f1eb8fec716f21d267ca21b0587186d162b4a3cd23f80e33ca330a8c3bd756e70c2d457306cd54be9e0433bd9416efd861ed42fcde10ccc98389652b5c14cff6c73341320c93ee457f9e3f320c5c5564fe50eeecf2d7a62a85639ddac4abdad7495e8926111b18f7866e6b3f95c788d9124ab9788210b1e25524427e86049726200052404d9b3585477bfc11f12c4fa761ed86b7449a5471ce18f964acee913946c0997980f +C = d4112af26f37a7e2e9cdc0fd460cf4cf4f3985fc820348e63a095751ce30b2fa5d6e89a8d37b8ddd3dc2af8fa64e0db729f526bab89562ad7f2bbb9ba02668033ebf6eb742904f37fbf1d87b4d23b1dfc815529a6e3f24bb31ba1d4354820a47e0d7b0acb4e495a50912824715d24cdeab0be3597dc535326ee073eb39b925c7699115dfdffd4162b84afc61048c7802a76ddafec243b91791a26138f202bf365a0b0863555d48c953c50cce5e09fede4a97571d3ebf0582c57c39680714750fd2227a632274adece8a4bd13413825b51969b3e965ad417c96415f4041d8f0d0a3df04cbb24fcfbeb95b401be7ea840da43606cd5bd56bfde26034fe94aadc78bedfaa4eed77e64cdc3d059e3765ff6346e9344f0e0a183085d78711b13d4ddc356a669693d2ad090bb67e8013403c51c81951f8a6241269d3423a739b44215bfa403f8927fc1d100587d602a7efc505f28edfe51ae8df407361fa06c09d06c7e960cb7b7041f3603faa6b4767a8a3c3d1c8ab05f75143a3061396c7289503ec29237eea7a364eedb77f66da308fe8e194c9b1784364072dbcc8aa1145db1a03f3862c38cc1a08b569f8576623915a644700ab43f7eff7e59684eea5595c31b13778824704b4aaa9c1b5d2acc4a6cd2ae21c5b0b32bc1433d536e8bd7a4e6c5e6d1ceb70dcd2ac215bd4619aa1ea8c184033b3279cc9fd094336c5e6f5112bd9f36fbebf304ab04f + +COUNT = 2 +K = 5309c5dff8fb03b007c28bda0a219d329080120b430f4b5af9df60ebbe20890c +P = 22b75fa402aa5b665aa71db0e24e76a6796329fea9128dffbbe03764e23d32443cadfbb6d557e833f841ecf5cc7c9200de81b2c58fc72a1a9451ce51c75abde6504b7868d1864a5824d6ef5dbd57c80e488a011102f7a321639fa57c40a5eb8479cbbc7428bb5497dd6295e270b53b00eaff17c2aac199a637e2baabce9c3f5af8d8c4698eea5390d7bdaa92ddf8190361a2452df2f44220b92519449605069661e0360cfef6841b734c8bdac87feeac243693a4e114995e1985d5c93e512d0f2fa27f12db8803e0aafd974d968cfb82cd9f0d4b09f34a440cff85e09a63462fe9820d5f1ba6bc6a489f30ed570e44c06668b6a4d3b977a6b7fdff3e781f0b2c7a1361108eb95f569c3ac5dfb1352cfe0965d9bea70f923d9f65cccfa83a9272947b3563f81c92f7f5caff203137b034af82847de0653bfee01aea476c9112267f97507b6949dd1d0d7510fa280432159a0b3bd67a8e628ef11599be4459238161914520b556fd5d91e92349169bd2ec175a9e728ca78b56eb2bacd56fd2af2696d1b116f5776e9df459c1023180a18f5d6e3718ac402bb0fdb806cd7b1cda6fc107e635f6a756005df79b720f3f274deffb31552cf238ec93182370d0ebe9f0b819e2e7225bc2b77e702d5df2825fe5d42005f1e39d67414ea539810db552049641ca22bd6539f1efefcb90dd479bf93080cb9df86a9aa99048f950297b8343 +C = c6a64b49464f5db516dd39f3d1dbc444528b5da57016b5f2a5649965a4781962264dee68d4c78e987f9446cdc815aabeff2590d3e2679a2583dbfd3aa6b8bd3bcd86d66d598daa6e766a6ab8fc6fe98a1bc89b5a137a342a59dfb52c7e7b2feb2a33d5f936503121476a9eee8d5c60b6c4c9aa7f8c56ddee5b4a9d7908890d3aa98f45cce20a074896aacefb6d7520e84b6675d02b819f37929746298d2c81407c20328a1ff4ee6da8298fd639fa9bba4ba27da5e19520434e27ae4bef59b73c3660aba0353701573d602d8135442a14cea4dbca3732a3fc8e5ac831929ad48ca30d4ee0f068f54faafb50bc7c6a784e81cf86b29225200f985af914ea6ed166e08c04e6324a372550603c98bb6c010ea7219699e76e4cdd9c4b8e6c9200d10c51e93a57f25b0019ae436dde220868596e96ebea3086089ece51b93a812c8120b8ba60435720ed690a2d1ff7c024cd514cf73bc7cb121a83222e4fba2dca65aaafd8cb51ac096511f83327458c53119f2462caf6cde5ae679fafd4f96756aa496ae6657ce417b2963c09463ab810ef51b4942dd1ecd120d69ae76f8928f3bcf20465f68e526beb7cef276533270f08136c4088182dcd626ffb2c3c4a357d09affb6b935e671bc827779ec6f260acf2c01f1faad57b162fb50579ed45909be1cb2a4a4dd07effcae6a2342022e0bad504261bb8d5d3b73eea932b0135fbb76fd6b8f8d3b6435db6d3 + +COUNT = 3 +K = 8027a28b92435ac111a67ba0f4af5cdd50e6bbf965cb9077189143440db0327d +P = 0d8f70b229ad687ab5f55f0b0ade81a4bcff76dba57aef40abb8074b9745ad74efcc8d8f2202b92a9788f8049c864e5973403279dd02e165b772ed7444795ebfdafd8e13a9a22ac1103d0c15d8d279c617dafff98dbdb9bcbea923a00967287ea9accdfcc90b2b4047021115d5454d057cf3a27a44988a8fa71578ab7751a5d3bbabd1a335248564f5e334c3925119a29c219aff6e71b3963f603b0f6e8213219fa56a36e33066f49409c643dfa1ab7478630d623c7b77efe07b0ad4811a522ed81a1c905831689747efb4edab65062cea3724c3f9decad27290c4ca0ee9c68f1ad9ce3981a0b712acbe2de17eefb7f9bfb3efd66aae691e63e5d64233715eae8957bce7924fe2f259309ddfe050cfca3cf76c52f96f8b7608cc833bb3876f35e37f6a83ac0a21f0c0fe859133c17c5f7422c72741137b8c3b5c3ed3e7737d5d8a29c063f1490558982496b7c852ec95bc452113c48bba93394dc1df0eee1f8dadd38715e93a1d7065c528f08525cd84583e250327320405bcf18d8011e3d864978f948b2839bb6d1d7bcef4b5b6871243a5c93a2afc1f7fc7d06c8307a025ebb522cf0fa1e1e9f8163f20854514504f4f873d757faa7938bb1ef91000df4ea64bb985d903136fcb9c4e9467561cd564f61146ac519c08dfa41f22b773cc821cd1252a3ba237c1e8c1c128ba40e0b75e67adf12cc448e47fb4037850b014f9c3 +C = c1e82d374a1d7c07b1f061656fd58b5df0a04c099f3390e98d2cd460b2e2713eb7274b9f646fc453a5d26c3fc0b9203521de72dc78292cf4d61095febeb0748658dc6764fdee4fa498b93a6bc053b3d81e10564cc9681aff2296ae0864fe3db495298ac4174f6ce06ae6b5bf86985c4603700757ef44c82c0e03beb7f2841df2dc4dd8022d5dc7ea2f711daacc1c8d1805798f646f1041146891e88edc4db9904f122c74656a5ea1694f8e6073920481e6cd30b20fac4cad7e03f58183f6b0beff8dde3ab4f781a1dc36ce0905e41e77be65c83dc9b6ab69d5d4c15a1de3cae6b081ea9568578207d678df77e20196483c2e00d2dba604784495949285a623f33e07e20687aadb86d57adb9935865e6467801a10add2ac7dfb6f8ba98e7635d6766eed38ca8fa8101f93018261b872a2a899065918b672423288cdaa417560701600424da6db15e0ebb7a5d1d0a7e1cc8aca714ac6bde51f779033563b49dcaeb8e0330112902b5ff0ebbdd00aa371c902092566244bd9a1e5e978af39adbe13a7067d056d04b718133575e420058529a42ec032cc5fa62035e872cec7acb3c32eb8b02e0cdd81c59afdba65b673c828b980756e8ff6b0160a1e796e221ed40c342863848b76e4390ef730abd3edc73ba975c037e6ba5e58162d7a60cd55b61643e9a8d4f6e5efa2485da7c45991ec4e8d4c6d9432e630e703ad1fc2f644d7577248ef7c7c974ce7 + +COUNT = 4 +K = 1a5b5f43929215da78d1d8c8fd4ba4508a66273be95c85a892da606eed04d6da +P = bd762d13e2385d333e435ecd792b8b3f13936d3b8372466fa7f65aa537bbd0bca184ec570acdb99ce59633bace041fb1374961be99276ff1d4b82cd49d78b89585aaa6cb1bf18498f4d01f2fc5e41e2b4fa56cedf5fe69e39eb1bed303d0795a715bb7eece6b389b3ae81b23b4f053137d54c03c2bdc68c79184fcca1758652c61a5e3c19d49b3af1a8946403b254da0f9ef227d2be6d0c528e1a7b78a55a2e79d79d6e420e9a3e8b40bb3ee74935be7c8e935414209be01532c991a871b90de0b35e963989475fbcfc37d81cbac674b1b29da1d0f9e78a6c3ef55575a8d3d7637b93895bae528d85130d059415d7c456a5c8d94897e567d6d6748e7c2689c98331ee5a7e1e25909901ace55828737e7464cb670399af3f806b996f416550d256f354dcdd1c332b4ca9eaef70114a7bc171efa08b88f1afd0429b2cf15366105cb812596a05e22b06debe1c69b06d904fe6cce284531ebefbae44b9c9eb156dedd01a4a73a1e631fcabc349e07636741580a916b35560d02e1cbbbea0a335b395487df8f972cf8600b48237e7b63bad5f08c85b85043eac0ca93d34df56cc486857d6eb3de0c332ee1ded06927e8acdaa3fce78636eb06c0f805788189c56023c8a1f5a2427c6f0f8b993f517a9b4a3bc3fa9c3e6318256dc49941c1e92973e3ae4814b87135c9ee767348045ef7877027c8ec441c50b1cab98bf556352333aa +C = bed34ebd1c70d52665103651110f847d83e19a551f5156c9720e2d1b038ed5d5ac0be8cd97ad3d725a8fa3c127939cbe1a6946558015bda4c7fffff7e00662c4b0ae3d5477bf61e175f8e4ca14cf53ccc558b8fb5f371748329b6cfbfb277cd0828a2394ec99f943775b4a2c1c285127a2d4677ba3b918154908badde2a74b58f73ac8f92b440214a3cc990f1517372b2e53272db9bb191cc7711ea31ffba2de2b646e8e312f4eedbe9e51e182603cf075ff69bc4d845410bf66d20215c160df05ab007b49a5396bf69d702d193aa0e2a2f9d62b7f9fb46cdf11864314101c7b7072f5ba8262c32b12b14225edba48b440501163d027e7fcf2292c9b06ce19dd0eabc6b6639769544d71bf068b8e03c4834c4a785a6b2b927c077f839f29da61c6c52c39ef4354839eff993068b67438266e10e9a421a90a103835829320fb1ad9e7e0cf73bdd22aa268436f4df0d973c401f8bb20f55585954079b6de5a2707ab5ce3b9f42af09b7c1996a45a15af12d041f5fe0c055b969a9055f7656cc2806745e84836903804799ffcc15580f8a50362f4dab9694e8ebf6f32cacbc3ba3501959ffbf33db0d231722715705e685df9202f0466c3ac0f1361a71905e23b0aa9fcac76b01d6402598861084477338b5d0463a641e2576f76aab2de95016cb31d37d8f4617e3908a179b43e3d50c516bff1e388bfc0ca519430f85a0de5b0e274cd0bd021f20c34 + +COUNT = 5 +K = 2a919617d24baf991779de6c06e68bbc4894a4e9a702214ba3eabfbd0015a7c0 +P = 8d01e11206e3b9285cfc73c2ad6b238536fefbe824f19588685601bf94e1ee2515c4ac281aa04feef877694840d15f2a6722661159a7216548d768b4754d65f1f1c522daf2baebbf88f638fb324b182e61d315f5f12b04c486022749f6a222d87c7d635c52605c7efc1853cbba466d138d3cda22979e7742e4fc8168f5139a72818109d2124c20ddcecc74fb388f34151e96fd33e8b269a7d7e51b6fc6ebb427219163c3d9f7de4a6831a7f8b596b4b490789f1104fa160115b6e8d191d1db885e42812c01724170e765634fdb8e05555cc7a85c3d8cc7a8e299b4f3708a4a5189871037c0d04216dca8e6127b449ec6004e7f076430d1f870f0d2c0c6812c810b4665f7fc07fbb04b56bd956b81267d2408c31be8dabade28a3c40c609eee3458b7340715272ee26df8d875d1139511649b629df067a8994e1fbcf7ec60f565e8da257db1b7ef3b2b765c77124c8120bf26db8c47f493d2afbbf8b89efe2e33c21c2a2a1141c0aa5b9dc1eeae50714304ff35a7cd34da963f5249bdce8e2c2a3026672bcd2ebf311f446cbfb7171fd35b446f0270320b30b554f96573eac72ea0594306d43c5ed48a6c2d01a363e88646a8759117075596fe17f18c0b36d00b7976098fddeb7abc4e34bf6d6b35238f81b59d169db4578a96c2bcfe78bd583fb31fc2e7fc97c6f3460147a5730bdf31ca197482acc341b7a719c4e1716cde56 +C = eb62ee28d9929093e3c6dbc0592c35dd0ffbe8faa37173195da391b84b1ffb342b9bd9f0df0947c33c7c74a071e7d71312f91c8e1f11d680865563f2d9604dea06892f2adaf2028246b74ecc82686bd7c369426a09f23012b69a89e7e66e7cf008b41dbf04f72cf950fd8f6ed63b3392de53b6feb34e45df3153e4aa6fdd685fcd7e3239f0509f474d8f6a58abfbd36bf14f93fecf913b2f9a0f0763c39cb6828651434ef0520cee88f46daef7932e5ce1549774ec106c2aba54cff64067e0202167f5a3ba1cd6396bda08023d2cfa9100043b44b8d90e2862d7d2a7f6e3f5eb6976dac2d5dbd2adeb69858ca391fdd4a582d3cb79a2d9fb57ad5b2cbd157b1e36ed49fb848e9960b0dd1c715b701f1d379027999fe8e35769e27a9ef60a45aeee56c7a3ebb39f6c50796d1236e721cd0e5d931e6a85f28fd87d652c3e5b706e2e97c12c4b33fc7df4585d60d0326267d8d252ab8970a2528be086d7ad18ee4b1531cc13b9c88cb3a9188a47b8a72cd276eb4d7d726a290398a98be0083e9917f349d2c8137a6fcd2232baedb6edf075d8e938319a12bfc31726b13d0e1a6c18ae33e11ad1f451fb9693f1774aec30fa7b473c98cbaf4dad1b3743d118ce69620503a753b87e4ae25f36564c2538d6650291087ae04d181e4fc44b6804b5bb0c6c8a0aa04a49895ea64d13f7d253e62216736a716258bc98c63349b5b19f5ae9aec3a176f98cb298 diff --git a/src/test/crypto/Makefile b/src/test/crypto/Makefile new file mode 100644 index 0000000000..9b5fce5b30 --- /dev/null +++ b/src/test/crypto/Makefile @@ -0,0 +1,39 @@ +#------------------------------------------------------------------------ +# +# Makefile for test crypto programs +# +# Copyright (c) 1998-2021, PostgreSQL Global Development Group +# +# src/test/crypto/Makefile +# +#------------------------------------------------------------------------ + +PGFILEDESC = "testcrypto - test PG crypto routines" +PGAPPICON=win32 + +subdir = src/test/crypto +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +export with_ssl + +OBJS = \ + $(WIN32RES) \ + testcrypto.o + +PROGS = testcrypto + +all: $(PROGS) + +testcrypto: $(OBJS) | submake-libpgport + $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) + +check: temp-install $(PROGS) + $(prove_check) + +installcheck: $(PROGS) + $(prove_installcheck) + +clean distclean maintainer-clean: + rm -f $(PROGS) $(OBJS) + rm -rf tmp_check log diff --git a/src/test/crypto/README b/src/test/crypto/README new file mode 100644 index 0000000000..3750346a83 --- /dev/null +++ b/src/test/crypto/README @@ -0,0 +1,33 @@ +src/test/crypto/README + +Regression tests for cluster file encryption +============================================ + +This directory contains scripts for testing cluster file encryption. +The first two tests test the encryption/decryption using AES128 and +AES256 in GCM mode and the Key Wrap with Padding (KWP) method. + +The third test tests that the data encryption keys can be encrypted and +decrypted, and tests cluster key rotation via pg_alterckey. The fourth +test checks that the database files are encrypted. + +Running the tests +================= + +Run + make check +or + make installcheck +You can use "make installcheck" if you previously did "make install". +In that case, the code in the installation tree is tested. With +"make check", a temporary installation tree is built from the current +sources and then tested. + +Either way, this test initializes, starts, and stops a test Postgres +cluster. + +Requirements +============ + +OpenSSL must be compiled into the server for these test to run; if not +the tests are skipped. diff --git a/src/test/crypto/gcmDecrypt128.rsp b/src/test/crypto/gcmDecrypt128.rsp new file mode 100644 index 0000000000..53f694d37a --- /dev/null +++ b/src/test/crypto/gcmDecrypt128.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Decrypt with keysize 128 test information +# Generated on Fri Aug 31 11:28:04 2012 + +[Keylen = 128] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = 137f40bc82b01b34047a407f40d5c434 +IV = 1f3de3a272227db8a88ba0cf7b23f7e57f84ce370b289e2f774a0c6184f44f9df9b3e645ae506e3d18cdc819fdc3b31f77d2dae7ee295ba2c776e6f1fefecec740021c4aa1118a6ed3813720411f27ace5c11437a78b3d626b5f421e2f3ffcd0677930e7878110a0cad535c057261c63f0531b538d8bf04f491c16e871f071ac +CT = f6336f2001a5edc519aa64071fabec2fbe4654ff334871e9cb8b12c24ffb7ec9f0d316e43bbdb568b65608f937f24f64eab19b +AAD = +Tag = ea3d8cad +PT = 60881890126afdf881dba4c1a7bd0f9d8c240073517b21c622bfa70c1f05d59544b9dccdd908bd924ca60d23697fe64aab927c + +Count = 1 +Key = 5e49fac6608ee4157a9f0ffde503893c +IV = 53d854bba28eb9d9db71e68b5916f9e73964a0f95c6524411fb18f64977d012d611a15fe8d46ec49bcabf060cfc7989c4a8272567c660c0881f944066b88311992b5d52cd644bf07e9f01d2249e5f9b00ae1af9f2739da627b3a8838e28116b02e7d171d1938edc6dd5d7da3dac63ed148b367961b809bf423510f0a3f44a8d5 +CT = 23bff066410784e0999a3d0fe742ee28ce8095fe822fae0f50b0b17d70c8cdcd705c83e108fcef2d64c5b4db110e9a72d3c4e2 +AAD = +Tag = 01aaf4e6 +FAIL + +Count = 2 +Key = 174b47de82c64196af18214d9d6dce2f +IV = db9653e9ca4ff27f8180f45eaf4861541d30b3b147d2b9b64709983dd08312b5fa709a7bc6afc591fbdd08fa0f21f06a25758ee199a6c6415cffdd9777280e620d7bfaa6d551875c069f5672a730aa68137cf33f86d46cbd548248e68f589c2e699c554ae4680417e9440e5514c1cee03205324f789d953d63518b023f65abe6 +CT = f8b0c1317c5f3550363762dc7fcc847376a7917e1505cddafeb5451ecf4464ef52afd797697f1b0b1e8d3a2376600d6acc1c88 +AAD = +Tag = f6fa70c6 +PT = 523986d433fbcca153af7765d177e2bad72ba52743440250db451e2da643913cb25b33fa364011a8f8d5cb563b461d46952923 + +Count = 3 +Key = 5194e88020a501ddf21a7ba5525bff70 +IV = 444b0037fba6cb9d4f4e33df32289d44f262d8f535564a1921e6a4315611b98a4e1074b37114758b421ca44efda91a48ba8674c695c259649c0bd3c11bcdc8363d3d113a534bd790988a44e72559fa70a5e2deeb9da187e2fa89c7b636a000a38d7bfec2edf2831ddfc9326f0c54956c149e5ffba3f556c8f2a749dc6ed5407a +CT = 31398e58d6e27c50e85c87a4dbfd73134f456ff042c9f869b47a41e3c10db7b24cffc103a2185a696fdc1f2d389cbf41379f04 +AAD = +Tag = 2d28343a +FAIL + +Count = 4 +Key = de250ebe0ea1212480f7193934d1ae84 +IV = 255f658334f1c5258d613ef92da7242079795e9c04c5c4e800aa1c03b6abe77f4dc28cf614278900b52f1de23550fc9c0ace8e8dd91a25d0240281dcd684dcb786839f376496692f3e35a5946bbf7b39cabf7d782f4e77eec53a875d2a5e3106238dae0d048e8b9ec7b05961d9d68489bcbe2805ab0f9cd42b5f8f62b4b28b00 +CT = 6eace8c7c10c713870c676e557c6b49134af26011af157de1e2c6ce0769fab666b9a7844756d6b948ccdf1df77fa98119c83a0 +AAD = +Tag = 87b81d39 +FAIL + +Count = 5 +Key = d0d9c4aa70e5bcad8dd9d72d14c21c54 +IV = da20388ed0709e5af5a52a5293bc4060f02ed6d05d96f9ac3a05c14118a3613ff790b721ff2c888d63c04b9b28a7138ca00d0ebbfcb11b70868f5b90c0dce79c7220fb90d5f3b6e22c6ae3640e7dfaca72ec12813820debdb69f90c8393645d7e96cc36bdf62ecc4c0ec181c64593c65c8f707d6db61b16c6a5c2f05f8851692 +CT = 2f08e5a8027a88f0e223429e6749dbdf91ff9742c7c6130962ae837a434a7247600df8e2004fcf3e81f8ef64e6a23bda002f3e +AAD = +Tag = ef6c0d39 +FAIL + +Count = 6 +Key = 464609f485abe0bf4930102f9637fdee +IV = d0a788fa1494d47bd1cddd55f88515229de5c1a2ca7dca054e08bf35d3c0aba677671d5290c3375fabe40de14e15465138d298edea5efe1ac62452dfee86920fec8e46577bae1c0b43d77cb6b8d1de6ba4557d7410796bede796b8b87b5771e30e05a8bfa55ed92194d506892c33f8ad6038ee8896b98772d1a88a7a40ceb650 +CT = 5e81bd883fbedb3e074ba273751081752c3aeb7a62b08091b768f8f8e6c0913cf633358fc26885150adf65bd1d708d9e45eff2 +AAD = +Tag = 7e715546 +FAIL + +Count = 7 +Key = 018319ad18e5d877dbdd871316fa6610 +IV = 941f8715bbba7b1fdfa98c8c05546a50f86cd02b654862af6c1a70cf4d79618b37abd1f06b7a29bbed74e728a8c78a0ec8ef8312f2552fc4571f48024ec5c6d44c21d69311cb1ba390cda615f246cb175a9e8a522a20b671e6df973b8924fd003c5dab7e59af63f20edf56e9dbc9f98fcac25cc692cd87a3d8fbf8d4b5f150cf +CT = c5a681b2b2e5f68e0cb698150aab9d13e93e900e0e6fb20170a4104839a7e0a5775bb3f20ae814edd109530552ee2b41cf5986 +AAD = +Tag = 75a966e1 +FAIL + +Count = 8 +Key = 3556566a1cdec721469cc86412ade697 +IV = 731a29cc083422da193724a57c8126b587c3817fbc6b76bee106abb3ac48722bfd0cc15deb573ca73f2bc10ad7b60024f297bc52c31a7021bda91c5246c781a95d5d18978c685f096e9310d6f5ed151122262f7d3f5a2b265321683cd72e0693ff608e378199264eea876df44659817997b323e43457e70809c74737d015cf26 +CT = 753174df0aff575fcea79e54e5b8347e69175070a00d58720529b173e9dc64677fdf459ad72385b03244d90810670304129f2e +AAD = +Tag = 41147153 +PT = 7588b757f08b75fc799fc825034ac60585e24c56db94920b6982bbae86030e7a021ddc89a5131e5371e9295e414d3503ec11e3 + +Count = 9 +Key = 2ec96059d1313923fe522bed1e942491 +IV = aafb096a63195cfc959bf93785a48cb7a78b98c60f861c9646129222643d249e50d46f706d670662f60c12a7f0a6ad65b908ede9a56b27c46f07d427754c11549154632e9b5ffa25c9306add4c0b13c7191abd59430a65ccc42c5420ec98093ad6ec6ad1c4d3bb895ca3546236c2c78cd075b4b1f44a6a03d115b91df02b96b9 +CT = 61c6b8bd44c9c6eb075fa9bba7ac0156bc64a6326d32afedb51f35daf232dfd068dbfd0505933519a753bce6de4560b79b31f1 +AAD = +Tag = 26f53f5c +PT = e4b86f8b7684d6a300cdca848ee8185c41fcfe061e61af6f3a563ceccab58751f1884b322019983d99150db217a1e92a8f15bb + +Count = 10 +Key = 48cbb46539dca9faa97b69b32015886f +IV = 1d3152ccf3ff05a81837ac7c55f791c78aba8c95149c80de8724b1126dda847b983a1bfe54ac5206d5915dd0796944e4daf432b14a72c5fcf81822ce1e3e7ade345528641b94a2fdc90afac098b9e4b0b5ad159ba275dcbcc11fec04bba0f12db9b3d1473ea0ed7825d293fd0b5d2d07795b3de922bba5660778adbdb2cf8539 +CT = b2658e897b85d9acf437f9e895189954134c543aaa91fa82c586cb5bb1d96e04b189df64df5bfcb63a567e444f4653aee3fdbf +AAD = +Tag = 48d21620 +PT = ef58f7ad664e36dce523408e497acb871e9a579fe4688be6425adf41c4c41db85e905e9279b93a037648802dfc273b387e7216 + +Count = 11 +Key = c5c2e83c4e5b336e30257c95c8f7f75d +IV = 269c29118eb9cb4ae8fd8ff5b3a5d61f4a63a719416bf2cc5225544cabdae24741239ba7910d1a32364c341169a215264ae62964322b6ac64002d09455f11d9658a9454cc6bafa82dba5b5365b2243e854470dbe74a3043b5f5f82ebefadaca4d468317c83076691c188af6e800dc296cdd941fc1aaed8840874f8cbbf96827e +CT = 891da6e799429c772edfe93d9fc3ad20c25fd5a172d272d0ee6b7063b59b611e4a2805923f182abf99373fbe4a1759339bb13d +AAD = +Tag = 6759d819 +PT = 474a2e7aa4a5422bf17f257b584059fc7abe2d25e7243b434a6f936f5a0eaed692616a43a2aef8b427d83ddf1bece29d96c581 + +Count = 12 +Key = f4d7a0e84b7f024988f0507098415616 +IV = 9d58c03fe45a97d4c556bafa4292766c55bd84d0e467c6c26b0e42162015f3f506c9fa06caac63a3ef6a3f0ea62ca4e122bf453a767b2b00d6277a990329b76073c64ef260421d887eaf89b6deb488a0528608ee8c38faa5f5976b2b71d7e29a08ccbeb2342ec15c46232f3c8867ada7cb8cc8aed0728dc59d706913ba62124a +CT = 11358bd3ea694574c0bde99aafc2621732fcd978626d3a3ef124e98a2c3dcb8261580306b51ffa9557d379952ee31bffd9ade0 +AAD = +Tag = cc46943b +PT = 4a289261f4ced6fbe544fa27d430f04c72ccfdbbe0f93e881fcc1b29a19e706a579a98304cb2b03cfb3aacbe187dc69f923ee5 + +Count = 13 +Key = 31237ffe81db56b9541133156fd0e7d8 +IV = feea389ed83bddb360f42351cfd56321bdd0a8fbbdcc8787725a236365eafe5dbc9c6e9ec8e9ea0f74e623c33a574ea4223a43b1550add0c75d8348315add72ee9561e1750b69defd46214bcc507d4db4c66c6da23612aabc2ebc9b0e6d5aeee270c013a553ddbc7d7d7ac62ecb33a5a6e3db859471ca3d3686203c0c96eed2e +CT = b965d7bb29a8811872c894d535e001d534a7d6e99eea5bd14ebf39c5e79c623a67d549bafbd0dcee1494245c04854f34de677e +AAD = +Tag = e6fac1d4 +FAIL + +Count = 14 +Key = b0e576fd7cc85f7c92d44e4b8c149b3c +IV = 3855af97c5b740baebda1532c196d187b8af0da761489ee9e267fcbeb720bf6f73cd743b69d942f3f44893d68a5c70174a1863dfb831ced0ffbee668a03e1066e8b3dc03dc1471c3a848b3787c0645b20add2fc0cc37e5e0aa57ef08fc69c53897030fcb5579a831ee53c76df2f75d2de5bea93e9ddd8e4e1383bfa1a7c7aa77 +CT = 624aad7fd047a61c09638b8dd5065d7c00960035440303a03ff1b19caac02baa3b6835581f2b66c2381006798c9d5d63d65684 +AAD = +Tag = e6142617 +PT = 0fb24ea93a14e3d76c4d7a991cb55e6bde4083504ac9f3a00d4b0ef53d3929953b53f5ce820c41b9aa75a985e09986d3f4baee diff --git a/src/test/crypto/gcmDecrypt256.rsp b/src/test/crypto/gcmDecrypt256.rsp new file mode 100644 index 0000000000..af72933a60 --- /dev/null +++ b/src/test/crypto/gcmDecrypt256.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Decrypt with keysize 256 test information +# Generated on Fri Aug 31 11:31:25 2012 + +[Keylen = 256] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = 60c9f83fb0ddbdc727e70bf9eb1acc13b1b63e3056e64db7c2ac55c4f2068273 +IV = c33d34a3673b93bb78dd1e00f877c4e6e4cf628438b9effa61cfe81e159155cc9ca7c1418917527ed3f0a51daf2bedbdaca20fad687a7dd086ae086c8ff5094e9b31fd71bd6f8f1f1adbf96bb2690663386c37d7bce891137897aeef70be10a453cef7e31c1b8c0a24ac1baeaf08a46aac445ad5a8103804825fde86dd4720b4 +CT = df6586921250aacd9d25f432977e92b09ddf89a9403c83a80890ff15ce9c4559145ecd85d86f1573bbc1b48992859d22fc13b6 +AAD = +Tag = b0dc70f0 +FAIL + +Count = 1 +Key = 8d7db520bf88c96d46778991a4f0b6de9aa7fd5d35cb6188a6f355499072af5c +IV = 488a50706bfd8ec7fb4c508511bf4c897c8566ef289b5e58a4c59bcbf16b5ae85fbccaee4a1cc0d1ec74156ae911d36d497f5ee71f1fa51649819c9cb88cf65d62d2abb65d621c202bcb33d8d68018a858d04e79deb62b3486658730735a1c87829acb49e73301902c116c9b6ce110f23a6b1a4dd657e47a328e017c19f0ee52 +CT = 497e1a39ba1b38d263bcbf19cc2900ca4070ad37ec12bfdd30139a7068a889825eaac5012cb5c2dcc710a220cc658dcf069f60 +AAD = +Tag = f20b9885 +FAIL + +Count = 2 +Key = 1ae1e42ee656973f5628e3cd11f0494dc8a563cbf5fbc5880cc2dc6d787bc9b9 +IV = 23e3c948cac6eba2ed11d667783557917f066ff6b93ab9409df9c7c84b27d26817dbdebb9fa9d0a64bbc572bfb2c7ef7f0c836528c9bd692505c8c5e522f57aecf6b479723449398e5b1f45cadd81264c5aca8059562d69deba26395034b4b01325d072dce92e540c159dba92d3e41e2d0947d873ad48f9f0b00f4807d420aac +CT = 3839fae7008b88250b602cbdf295e932e3c4e3710d397a2b9a37289104efde75f73302b2820f14664c064e8dec45ae49a74036 +AAD = +Tag = bce334cd +FAIL + +Count = 3 +Key = 8af302bc8684cb91b4d7a6088cf8c94f9f6e027ba046c2b508956ba3c88f2d65 +IV = 3218736e931392e6510b91210a6a6a27680740ba8924062ea176048d6b42f44ed04a46ce31843b735ef4f63dd1d85643f28fb335d21fc2e3c673e97e6b845e363362d32844c9054a165f40658267bb177b74797a8828b1eea723d51b571d93748c758ea5c328103612b109e008f743f9505034ed3c42ab3dc310c20938f8627c +CT = 1f1b133a1a7b58625fc77021f8ad1751bfa2b8addc0a9837dd5c44632cffe5ecc2e9e54b90cafb6cf8b652a8d2da116ecda3f5 +AAD = +Tag = a673129d +FAIL + +Count = 4 +Key = 13a7942b5a5cecb2bdc0e8b0348d4db5a98572544ee31918ea625b0691c10779 +IV = d92b4d05a549b296e18c90a8da55ec5bff3547a679697c489a1d49dc02bfe2dd85c8f050b32c389c4f857eb4b663f53354bcfe9c3a7e30019f2e3994421bcf3a3d1cc093768eed71bad5139f3f3078514d80a4a41d1284b5dc43ce07efac9c475d6ba2acb66dee50cdc62c463a05ca396e72d189f50d44ffb70d2c6112c6ef0e +CT = afe058cf694d64706302b405243db77c7ae2fe4f33c6427416f8992ba92754c69d4e7c1a89e9b6987f2bc0a7b568dca9c9d273 +AAD = +Tag = e0bfbdb0 +PT = d544d114e3d9ab8aa2b9ec588a112f780a6df74d637be3cd34fdefe14506f26281cacd2b98c26fc4adff837a7bd72173b962a2 + +Count = 5 +Key = 7272e6ca6d6d76c483df9a55c6d07bd54fd8fad50b529ed52154959acf01b64a +IV = 5c55bb8f4fe797ce34c0e281c3b04ba0bce8689493451ea569ba8cbacc74ea36ccf319776f77cb4d7f901fd0ff23cd28ff0ca77ad9d4adb0329fb68a60ff004a1c5b12111d2dd705ab1f7734178f14dbb356cfc0c5c208b91c277235f35afe8c2d46ebf43bd5e0a653e67e0c086ebcfca32a56d56dd5f810f562f769cce2794c +CT = 7f378bc30cf2774f21078f42b5d6b66aa355c8c073d3a70f06775f3c7e5948539ec08a2cc50cae6f2ad9680ba47bac190c3068 +AAD = +Tag = e612f4b2 +FAIL + +Count = 6 +Key = 82b1d2ffb53fc79f5ef88742a28eabcf404074836fe28b5b202ce7d5c68f6ebe +IV = 33f834fe23b9639d30de763faf7c1a71568c5dea9d5d253f28723ccb3306a3cac3cac3beca638067a3485ff743b5133577633ec88dae0aec4fec08e894ab5d61c411f0939772df2fa66d5775f74b3ff36ee61695d7cd2726b9be4df80750011477705948b1276db0cafede5d7ac73ccdf01b73a5492a02c43b89632a501f6694 +CT = d9d92b33a10f4252fff828b57ca5f5f118885df0825be80ea5725a874b7e8721af40bd221e7f5c2c8b005d77af6266cd36ddd2 +AAD = +Tag = 86e48fa3 +FAIL + +Count = 7 +Key = 5264831eaebdde1eadf741dbfd585cb0ef6437d1365bd5848d9cb3a22f57d420 +IV = ede74a8f53eac5dac276bc72518255831b616c9fb50a617eacdcdfa50e197d2941004f785f00f8c600e239cda77c8c06088793a674efb8759c98604dc0143e06665dc7e21d5031fd4751a7cd1b947304645e0987ec7e765db80a743122fbcaef9ec83849e8eee8d011dab67fb54317caddcfc472f585e93df91b1edce9695908 +CT = 9ad126b39dc2066542dd30c8fe81cd750b72123d74aa162113c6b0cf10a9cdb217d921e8f03b400f1ff719fc704f44e26ad463 +AAD = +Tag = 5d7bed4e +FAIL + +Count = 8 +Key = 6c2a43ce5610eab9dc40f43f035f7eed6651789dfdd166d4f106c95cef2a67ec +IV = 60e3a8ddb899108c11550a461720bdbf9adef26c300f098c73c3767621b06eac4f5619b9855d96e4d972ddd38f4538f8e25b7524b46c6341e8780e22c3b42ccf43f41fddfc5680432b64fb4025b378204045bb2d7ea56f4340a4018a4c99eb8b91012b28024d1b2bdb603fa10a28130e84bce38384fbb7c43548c0072c5c657d +CT = e073e948ddfc414948b12b4540d43dfeb9cbfa525b3cacccd21da89ecfb254c840722b9179057cb3ee69358f05e4ad0e41a543 +AAD = +Tag = ac0497a5 +FAIL + +Count = 9 +Key = 7c36a9bb3033bc6f7395155eadf0e07c8e5b3441d0ad66b21625d4950760386c +IV = c8393fb1d80ce92801a4fd906a568f7f404a82b02096e859e70e46d1ca5e231a073c5acbaa4cb4c33581e6887c402753bd55f95c76e68bfcbb1cb21bd37ab7a226e03d03e9dca6589c3020f5f916c50676e8c387f9b1710579a728ba7e7b60955ee5e383bb75d2b9d0f2abc72c02edd925bb32dc5a994f032e9a856931eb1ca3 +CT = 559b2ff3f5fd147b9889146f9fdea6758e5e0c716395cf1caf577dc2707764833099bda0910626c62bbb1ca010b66c54114982 +AAD = +Tag = cd2ed4fc +PT = 178745b297a23a897ec5cfe3a9e373befbdb840d9eb657885ad0423628c4a18f934e6fb57974a52436c517f4463cc5f9370c54 + +Count = 10 +Key = 981afbf7e7b74f08d186616d1f71b682bccc3cff6c5560696d267ad455d111f6 +IV = a2d07ac3ef29978c44ebbd83e1ad330a8fcfad8213fe2e924390015bc966a944a0a76831189a011094ec4ef98535efeb56b871e7e1aa36748e639dd5f9d1bf3286a1b2965bfc029faa0f855622c30cad67331bd11dbcea51e397185cbc4f0f0341fd8e744d2f09b2e3c2bd03af15850dbe2a701855ed4247f97acf9754f5e4f9 +CT = cbb1f9a5bd84c4b1b8df2714f87db878f7d2658cc7c37f75d784e2157687398a391ecdfd1119e087bd12f6af79db50ae7711bc +AAD = +Tag = ad794c97 +PT = 06ea96ad8e6044978ea676056df8c647b7bdfce3923750983cdca875089841612737e6fe078496d77906b9606532b309851cdd + +Count = 11 +Key = 03183678896e28b84e16ac41ebb14f4f436efe386ee6df4e8ad2a7aaf11f17d6 +IV = 718ec99fa1b9b1d29a06ca3973d9c0323b14a2cc34cfa2816481aa2da97b435b0a075a2ed6412bb482bea23df9deddd16944492b1756c65138c3d189b8d2d695150667f46edce88755e868a2d90bf13f170d9b6bb29d9210f3c9f507663756866ede0b362aa5c859e15cd96da4f8c7f7852b3924bdf35ff3a515ba5150e1b017 +CT = 6ecd41a492ae5d6295e9c18290c9a36999c79c87f8b69ff20cb42ccb7c6678baaf159c75ecfb15cb87db99a3236734001545d2 +AAD = +Tag = cad99689 +FAIL + +Count = 12 +Key = 2ca8b01d1cbb8d392bae40bd8a51205a9020be27a23533da51dfe1ad0c4c1d41 +IV = 5b5edcc2f17942afb9577c3d2ed7d5ecaf009ac3ebac985fcf1e0fac0dfdfe747fdfe3d05795337baf41cea3b26e4f35caee1c13fc52d1192da145f376b4ea810ce7dc94845a9ca9184203c3b8e803e7a9bfcbf4a310c85b28b04a007e8e9bd14ff0ae28a1966918a6e22ae8415334e7df0d530b0507a24f755f70f117581820 +CT = c420656fb66e89d5b10fbe3ec0929286683ddc4a34cbaca638493f5c09673609814127709b6b1bb765902f6857761a8d57d98f +AAD = +Tag = 23f1dc74 +FAIL + +Count = 13 +Key = 25cb38a4b7ff73bb632ecae5f75d46e45a108ffaf3ec2d6ad39d3af4b3c64ca5 +IV = 372510e18f877e0f74c1cc54b19d265b27a452cbe91339bb720aa1bdfaf9bbe5365c571ce8f01d2e96aef8bf089c3f4402f186213be72b46b200337c9ebf943bf3d2db1f68c8e655534d9198825737e623745c26f6b0a82585660a7cc3985a271dea9b20f93653701a8d383bbf3155864809decc03ffefc9ce018379d12d8bfc +CT = ea01ae67abee8f8552ad260ca9d08ea5b35b53667a3455718545e007e5ac0c62c1ff0c5b06f8c031079fce5f2367889a6a068e +AAD = +Tag = c6a365f1 +PT = 3bc70116886ed9b4ef795e45c6ec8ea65f6285b3449174f89ceb1294ea73dae9b2f037107f57355be7242abb7da818c98d2755 + +Count = 14 +Key = 19bb98022f5d140cdbb5b1c02aae8eeec1e96dc6eb489d70967588b6f414330a +IV = 6d7b41c7f949f8ff3e9e18ff7af3d67eff5ddaa62eefdbc0b0a49dfb6fb07582998250d1c8e609d57510c859333a268f7e89bca06adf1646cdeb2e592bc86769aba402410cbd71f572dbe065beb37d8766ac61c12e7ac322d213407e073d4bb3c28848c42959cab21f9e39d7f4ff8debd50f40bfff96cbf81af07fbffb6bb2b0 +CT = 60a77e3d27fcea5e505221382d82e9ee39c2bfaa01d7d6ce0d293e7fc7bd0d7f900afa9a7f080c33c04cedd76573a914409e39 +AAD = +Tag = b561ee30 +FAIL diff --git a/src/test/crypto/gcmEncryptExtIV128.rsp b/src/test/crypto/gcmEncryptExtIV128.rsp new file mode 100644 index 0000000000..69c74ca6c5 --- /dev/null +++ b/src/test/crypto/gcmEncryptExtIV128.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Encrypt with keysize 128 test information +# Generated on Fri Aug 31 11:23:06 2012 + +[Keylen = 128] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = f8fe56171fa546a34b1b28e0b1d31cfb +IV = 960d57f1336271e069c12f11044dd5a5bea996fc0290d37b5b2f47c8df3ae3ee37214a6871d963b830aec266026364984cfe31eb88c2a6229f5594ca9d3b6d26c7fadb91a0282cdd0a321714b745dd5e161e7cd192420cf2eacd552c4df5cee8fb5f0e06b7c353017b4b9523ce56899db770c344da720327817ba823a8f71382 +PT = fd229158f18f5b8c2a96c86fa3d8084014660eb2314bbab4ca09fa72c3a98b6faa2ebb83a1809de9ccbc8973d23af34014fb27 +AAD = +CT = 4dc0fd07c86ec84c264f0544456bfee14f688af2109455d73aa1e58a3354727e05387c94568edf352f8342a6156c64d87c44d7 +Tag = ac350afd + +Count = 1 +Key = 803007b69146363999afad4433c0f3b5 +IV = 2fc7688faf0b3783094294526ac07c38a73ff961ab41f20deb66edcabfe68e094aa0f1fc52318706a0e2f9b3901a768346494e846cf17b662d05a3788d77c1468408a49ffe5c0cf68b3b8b26193dfd84c63c4631eebf0c7974283e05e39494d9aaad038018a6e999912b1f92681375214e634f5937cb32ccc4face42d013980c +PT = bb311134866deba57fe506445c5a312ea1ba1ba16469731c1647c6a482bc84fa8349d82bbd01d3edb6cfe8f25b37ff8ec9d621 +AAD = +CT = aa1d6ae4151aaec1030a6f0297c48e67b84d1c397e9eaf0c5c8c3d252bf8638bf8591342c5c20c7f88f41140b0334f55cb7b04 +Tag = 30076c5b + +Count = 2 +Key = bfb96f58535443205ce281e03bdc5c38 +IV = 3961f47e9e800f1c72526d73fe372cd6e69a7eb1d2692c58bf4297bb05503edf95e074a7cf2644981f72421f229d93866a6e1fcf5c13953b39bf36b56fcbd4c09b505e550b5ba0e9bc26efbe0b9621a47b81842caa8c945fa5bb606f0dab824a4bb5a2625668a916e47f0a0d8a995bcae6940e120724f6d53629545da5456008 +PT = 2c07d76ceac2a77809906bebd3452e2f898ea5467d47be1f17573b3f7fc7d11c9b868d1d1a24010b63dabe9c6c6b4e123df559 +AAD = +CT = ca443acf08d121e1ae4221013ec40dca2237d035e89ae67040602132972417e07a7c770d75d96fbb5b8a38e048abb15bb978d2 +Tag = f61f75ee + +Count = 3 +Key = f3a44f15d104f81b4bb263eecf806737 +IV = bbf1b0646991c2b9735066dad5860fcc08a6b92944a5e90dfb120acf2a75403d3175f5e61a1d84b89a0c1bdd3b3414450faf6ffb8820ab1ea01a2b3cc05f1cd1de9bd48ee1308ddc7d87e6db33d3a171e7f63fce6b8e0417359afa833f6b5f293195bedf444ac56103ee0c8706a69c08fe59a95c8474f28a4a12661905f8781c +PT = 62d05ea0b0ddff1c0418b01a21267230ebbf23b63a6c14caa769c9148150c2454c055cbe4a72a08e7cd8dcb456e1ec17bc3a63 +AAD = +CT = df08c4d223a168d4dea9445f5b88d40ef0a796de3e77a6a116bfed840ccbe7b988345d070640f62f5757878420b5c50dd9f567 +Tag = 1518c2d1 + +Count = 4 +Key = ccd58a6017ac344ce5f8ebebdbe03593 +IV = b2c53f013021f494f6637876f62ce5b5dec6d548ffe58d0952aa8fd8fd5c8d2b835165e1b0ca01e72c19f962e38cd4458229a3415d7b4f9afddf5bb63215999b750c07a080677ba4e40f6c5e42038882503c9923a6eb2cf0d3b82f9f94e624f9938830fc22430f16f6c93c362cb3c11cb05d63becdb4c572f03431e7108369c2 +PT = 0b89a455a2470b3b2b7e04afac15c45a0742061494b78f88c57f2505e1f5804f35c0a829f8b6443e427fb6ecd374642217b6f7 +AAD = +CT = a8b6203b914391c61dbc123efc6902b892107ea9724341a22ecdd0deb16c48d885d606f68724cb43a956c07ef9ef654c042906 +Tag = 02aeaa63 + +Count = 5 +Key = 50bcf114df40a431c0e1e88033154a25 +IV = 0ffcd9b14c6bf5f630c86b41cb3cf96004f3fa4fd48ee87b7235d34be0be1fdddeaa79abb3c0c9198a1eea0ccf04e8cfc8d24e4badbf438c59a70b435fbf07d44f55b75e5e48fa0f3d7714aaf9e34b430640614646d0008014ad432a464a252c66584c922b29a90cb2c2e4237c8545f913cd2ed3910a4f075062a55b71228411 +PT = 9f5206b2afe824ac4303d58a97255dea6f026b8531651105db9695f09acb16afd9488928060219307fd41edc16e49f01b7d646 +AAD = +CT = 0143c0a035ce29d1418acb2561dfe88c74f24a9808a8672427797cc0ba01693c2d66c0e365961cfd58fd039bf08fb4c2b1be29 +Tag = 552e27c7 + +Count = 6 +Key = 8dbf11f923374b8e8be93788de939806 +IV = 2fcb130962ba3e3eb8e31a7a26e2082a643f39d67cce11d8b2ba8a782f63d4df375b21b1fedaf67bd8d73b2208937ae941afc99420ebbe328214fe6a456bf00979d5ebfb22b79fd3cdea81056747bf4e4ded33f2f26f2d228965128a3d0a32696db44e4aff6ca5467d3c749830a5d2e9a41b4ffa3a422e5bb870cee84f4b64af +PT = 1c57186740c4901e022e63b2e7b085f9cb60c83763e357a591c1968277920ade1987334e88ed9c3c96665e37fe492a975153d0 +AAD = +CT = ce4c7e5fb87ed02424a13cd2cfcfaacf67f1072198dedc594db26d6453991821861ba6cc843c6d2750e846e162415123b7e01e +Tag = ab10b040 + +Count = 7 +Key = 6e88bed99ebe380c1f8297f46019d8a0 +IV = 80fb82c0016d054491f396a7722217b0a07bfd0fc954a7141bc1e2e7958cb24541a21492ec85d3c744489f93ae3abb9af101a78f2366226080389d29eef564d5205f377ab0902043bbf7ba64c30c9d2c945cd6f29654738106dd282194fa02344ec177b5547531061b31cebfac4a2b0f46b68e44c8c89f6942f9c9c13e50c58e +PT = 6c90ea3dad3e172ae6784cf1b7c08fc04d0f9b4372f1103d11393a8f02e2f495b53c57cf46f1df6f55b2fdb184fd2b7d590402 +AAD = +CT = f55d67b524b8e633019e9b1736f3db1a254e53cd71fecb48f2dceffa62256a0bfc3775f6506db52db6c6eb91971f5bb5688325 +Tag = b4585815 + +Count = 8 +Key = 99de57ce03d1db62a751c3b2c7d38f3c +IV = 2e74c283e216ab5ac9a1b214dd9280431d7ff942a17715fadc3a22978fa1ccc0743f969358baea3d79abe93388a7db82d82ad6a917bf67795fb4360543d7f22f7ee49f41029ebb87573ac03ddb7e279f1846f4a88b85bea63e2c9b9ecf6f91777434b3def0d4a42d3e025eb43a666a28d5f6c834c7e7991897bf051915e646c6 +PT = 72dc7e5533e862efe0d23c62095506b11c9b256d8d18d11511aa1ed4eae67b0017ec74e322f3a7a18e7d199e7093cadf26680a +AAD = +CT = af31d880d1820a35c9248ee0b1aa0da31339f90182e60451493bec8d4dec3baf922268741c2717831b8365bebf072aa7931ab4 +Tag = b1700396 + +Count = 9 +Key = acdc98a8baa0f003c130ce196135334d +IV = b7c9885d302842c41a2880b9382584806e8cb55b49183b80bc50403c82cfaf0d28a00fef813ba5b7e35b80dc1f0b7c3b0669a3bc739f499d77ed9cc47a467ca62fc34c5bf4c374ff396c01a472c3dfdab394e6926545a1c20363960c72dfad3eebf9a970e6579e3eff7a38f6bcf0373a8494d450d12445f9ff62c233dc1d2379 +PT = 30f1c7fb5fc152dad6911623ca4af1eb495e108ed94b6e6cc19eabeaa7b85262ea3cc4dc5297aa6f7cf504ac6e07db5db550e6 +AAD = +CT = b2cbae4df8898ca223213824a5c08e16eda81f063916e813bf2d0d8c7e8a75b2d0a9f6de91e08d5422970534331cf1dd53fa8e +Tag = 8ef08260 + +Count = 10 +Key = 75d651a377ccdc0e743d73b8205fb38c +IV = c7296796ef031d372284f7b1481a13862aec243792ca73f40e11bfe39da28984f11d591d3294c833babf05a1f19b603f4f4a9ca1102f201c1405b6cb45facc8ef408541963abcafa907b2eb8e5c1c404b0e4884a48bbb2d43add4dc1c44c526295bcfbd8f2b7041ed49189e835cdf4ed00dcbb450eef4070482f5e8b52360966 +PT = 13ab743d9db511857f0f79a84742c225b3692a9aee8a63697d42fe50d74fe028ccf95e18a2dd2d9b778392ee7a5d2f23b399eb +AAD = +CT = 09c3f153ef712fcb3b5fca2b9bf7f25740fb748bec64bc35576ed01682030ae2728d4282140264819e8c4dedd48e29199a6236 +Tag = ae374dd4 + +Count = 11 +Key = 15414fda594aa87a3e7af69df769adbf +IV = 7f021a78293e7cb4dd0af221efda3149b0cab87241b597865267cbab5aad530ea4aa4b10815ba9318a45fcb22fd0e6d692d7beecc2042fa2791f6cca5f9916b0bbae79e9d91133aa54d15a1397f8b063695a3d36b8e573a866fc94964f39016e9490c37189cbb0638db09548a91688d73e2c0d4542f5bd08e03ac0d75e36f519 +PT = bc8263ccf50d0224d088546bc16e2577925567ca52d98ec45cb43b190159bbd5f0326d4498a8a88c0ea0b0a79420b906cd5115 +AAD = +CT = 34b07942793b066b74b7fb8a4ce71a04f9b29e40dca351f5b6e0939bc2819cd95e69bd58163a4df9729e3d6220a6df60a4ba9e +Tag = bcf4aed9 + +Count = 12 +Key = 4d73cae96ded98e1688104a63f462c76 +IV = 331b51fd88cd4731e0eca051717b642f86ea6d6941f9a7331ff361e0edc3f9ea4c013d585f3eff70004a696c4b51d7c589ca97e5fd30f4b2f99c0f3ac83769c2397e10669b7b83aef714a2388638b8941efde8e631098dd78742772f484edd568fefe26b9d981b437e4e4f3ad25445e1aa8c8608c655a5090d2cac158ee67e98 +PT = bf55d48ae5015c39bb167782cab391510d7b7d698e9d3faacd7d409fe4d86fea0a6f61128b6aa305ac0fe4cee4d582dd30e717 +AAD = +CT = 3cf29f4d50feac5cda4c7cef91d563b2573096b7f32c723355d8d59b8e1a0e229dc2f6114f1db6bdc26212043d153709597c1d +Tag = 76fcb529 + +Count = 13 +Key = 7045a9d7caae2c0a39f58998720974ec +IV = a9028008f708ae19dd28e75a02c6e84a05096779781ce0a908047152208468cc4b2a57d25608767f936cf70fbe82dd8a61497a180e1fc967caf4e6310ed850082e6919c922e021ec070dc64b040ed9edbbf5883676630d69e48953068b2bd006bd6d5417038604ee5aed04980c9ed2316c531ce6a3a73dfad90c04c58596d5e9 +PT = 635b946047c38533bb2cb4c9799b44f6eae0e63626901b0741f6dcf3c2bb02270343c7708a72dfe303b20f7805cb732386b341 +AAD = +CT = aebb8c1e914ea9bc1ada9b01f84cc8dbbc611f2cd386d5fc89497d37e5a469b28fe2fdf0ab0f1c882dcce50620b1b18a2d8343 +Tag = 27bcaf06 + +Count = 14 +Key = 79d2a35efbf03f57b66e875c232e10d5 +IV = 7cc96b48afff401adf4bab2ebedd021377b18a819f3c3af39fda42e24c5d62e67ec30f8bcab00263dc5a9bb06cbfe1750c98555901d34d775fdcc86841bd08fbeb44ba68ee794dc351a29a1a9de576d83c17a730d50db79cab88d538a441bb9ff6aa073a2a976de820ab5cc61f834753220d4e472a275dcd13db3e51a23a84a5 +PT = d91e1b2811b3e3894b46c563e6ea0b4a33990ba4fce8a354c941e1effc5691671de5d97c4c1a35e3730b43584944695f00544a +AAD = +CT = 6a45beb05c0dba6c38c997f8c37ef07c7cf78eacff6ff4dd6fa000e745e8053d2d270a746994f29c8628f41fc7fecdac158655 +Tag = 36caee75 diff --git a/src/test/crypto/gcmEncryptExtIV256.rsp b/src/test/crypto/gcmEncryptExtIV256.rsp new file mode 100644 index 0000000000..3faa8956bb --- /dev/null +++ b/src/test/crypto/gcmEncryptExtIV256.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Encrypt with keysize 256 test information +# Generated on Fri Aug 31 11:26:26 2012 + +[Keylen = 256] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = 0df25c2bc9444b4a01e26d357a3ac0635fb6ff2e65ce1e759aae491a17772243 +IV = 54652573ba189cffed3bcfa60efbfb417eb4b0e8de80c7e53765d018cbddfae74617269eb35f29faf628d28c40737f9d9e1eb0b8757c984d94340ecc5ddd108f0a5a0e96335ea805950d378fea7569b98693e88bc1cfd9f6eb8d25de177122fc774a5b1957bcb80e92230c12fe401a8e00d0c04897e234644bda36ae761ea619 +PT = b29af460c6a5dbe56f1d67751346d7182c93413a6c328c6d85176cd8dea8ecce1cad3063c8708c0be9ae73d42bbb10421e73f1 +AAD = +CT = 3c786a3d0c8945bf320a21ca63f3b8bf5c6bf56a8412f836d7894e42c9e0695a8e41bf59b23fc52d17f8b341183f1cdce02e22 +Tag = 0b31bffe + +Count = 1 +Key = 1b6b7b8d00e543f0a17bd0bf595319a4a1f8a55ff41ce2380381d4e83c83243b +IV = 791d9546f180b838b50bb7b66b68f7f1aa09a2cef411a1dbf2e64c5ed026613200ac8f0e5b961925853621a1d4339322ea4b7bbc4adcbc008efeaaee0dc948a916b22ff8693a7d441620d0ff67680b23567f9582e22eec529408c6d0a00de1bd1ee5a11ed7fa29b7567f990e412ce90ee12d5b1d8ba8c2b528d0d9963fdc49a5 +PT = 0a76090de676d71d9f5ee8511864f0c9440a0fa12b5155a5bffc36127bc957b293d4fb624af3b956385783124f28f6f3c0f0e8 +AAD = +CT = fdaa9ef2c65d7666b61367f843863a3b273249192551b5633cfaf84a5ab9ecba42916395177f9a16c1ad385e77393cd93d71b3 +Tag = fd854d63 + +Count = 2 +Key = 261a0382c739093634502303d60dd0d2a568b5155147d661e7789f7bd70de82c +IV = e22330a2981b9b60354a740c49dcd17c9016cff50423977f7fc8500fc36a81610c979eb37a1f9c4d54d11b790906492205263178dca6d269a230a595ee95edeb60e45d92a2d6169877dfe5514b23db143dbcf3ca44e13cf5b7402e4f95a9e6760451be2d57d1d5fefb015ffe4b69456d87338865ee8775d1cce238f75c345dad +PT = 42c8c79988c52bcf38865a6f341a5165294af0947bc4fa597c6f648a1172070851c4ee154604d9a21c8d53f7eba77fbb72cf2b +AAD = +CT = 0fb368563c6ad0c097207239a791bc685274d434c4a54cb0ea97a98b4cee9eb100c4cd626ba82c338a3f51cb29391b1da3a44f +Tag = a9607e42 + +Count = 3 +Key = 5f3d6b5c76cb7ad0be3d771f1f7d1bafd88c8c7c7c84922f89f45f3b7453eb44 +IV = 7072d302e4acb642090c5d48e4549a8823725be1389316df0152e68c41f77937eadc4ee1d164101717cdcdd3bc9be3c2669c0b1394953090ac787fb117500b6275122f608e70a3f5063b21fae42ca4b00724b21c50e27c37a77d4befb118018ada7999888442e1271410f6b804c36a27a41e95a438792de4f06fd7122b177cc3 +PT = 00d82cc00c07d3cf0fd9b31ddc091d351aab9b58af07d2c59f3e28c7202ef6e3ec35726cbdd14f8f1985a770d092470f115be7 +AAD = +CT = 0aab0dbb31c113b33b116659294e0af5497c987af870ae8ce55288d48e8a4f1b3e8377cb34d06460daf3cc99fbefb90724a39f +Tag = 26e56b13 + +Count = 4 +Key = 1d4b6b6ba43269d46ca5c0fab38b61f1d2bf3d24dae46181fac73420a8e6194f +IV = 01c283a90c588dafc585f437dc111e94dd8ea98a622a5d2e554a87086ec10e8dac9b205fae70be3c024b3f9fbdc26208a6e44d082ad92a51a7fa1f861d0e93e7e74d4e41426e70b2e7ef7fbcbc5a302aed4bc8a42963ca53334259f1924e74225c3bf9aa2ffdf97a2be6474c72f6dedba7c454e36fbc3537596a63b61a4cf3ce +PT = 4eb036d945c14de26c4e0c83d446dacb57e91b5bdec2ef612bbed8135e57ce62b28843fbac555580ef23c74959eb869a017872 +AAD = +CT = 24d67f2e94294255b9fd90c16b67c7a8b3a47a42782954aa15e80ffe732d64342b1ec65aa66fc4d6b8fe9dfcdbf4d4c1a979e7 +Tag = 37358e86 + +Count = 5 +Key = cbfbb15a4fae4af3de55ffcaa4b8cdbd515ecc72fd50060b22acc8fad57a5f4b +IV = 86e47ba98d8ed360262f94afc364df373d1e44c788b4ee5ed7542822858aaeca2b3c07b5464d6f7edefca9759107bbb64c086d526cc4c9a6b9a7cd6dbb50acec2297597612fd436d8c8ddbc83664305b214cf2c3b6fab3e545c499c7f1cbba1897041475f94952d8a4fc10110986b5ccc7a10c1d9e6427dabba8d942ada9290a +PT = 5470a60f2818ffef374be3cc591084096a4a8caa0dac0024c12d304382c301264526b02efb674dfbcc3be5c818a1d88a7c19f1 +AAD = +CT = a6a8ebadfe5c5ac754e4a2a9cdc25fdf3588c69a287b0f9aba2347bfcdc110a1134426a45c62d577e52a8d61fae39ea38bad51 +Tag = 97406b4b + +Count = 6 +Key = 3ced5c63078a36f6b02d3bec0debacd19021c8dbc501a9d86f556c4ad2cdef8a +IV = b9629801cba22f7493b4f6394620d49a76ec686e524f5ebcb3a76b5a189473484060cbb01bcbf10048427ef21527626085c8a75aa5264b6338abbc26171c2c3a44f6b3b5c3fd05a892c8290a8f99be962deea48d7ae4e626c45a45ffda5efaad6e54e98ba876b039a5dde3d6061f217e57da4774acfaf1f5da9495083aad4dad +PT = c29788107799077ac6dbccad29a346727f263676a8510fbcadb9b5bf53df978b3382fdfc1b5c3312eaa0f7621b6efdaeebe1c0 +AAD = +CT = 5811d4a96169b39cd8f8ff3f931efdda68550f17558b48de8bb5adf455af5179c8c5ff4a73f363f8819daf846427132daf6a17 +Tag = 06e8e7fa + +Count = 7 +Key = 39ae4fdfe56b74a71325a1a685a6593b44936892890cd05d2719a420c97a7c64 +IV = 8f110d0fd20ece7f35c35b2eed32316fd742ae9346fc6907bd749361a4436427f80185b376b16a36fb95429cfaf2e22e46c210442fe5efc14985a9d9d847c3ceb5db02e0d999acbc3ba0afdfecbfaf65024258cc7f6fc8e3f568cbedec1c7eabe3ff3ab3c7331722b6400429d46b54820bd1f88ac03cfba5cbd0812d91342c4b +PT = 8b086d11be7ff55312addad86b49585ea38ea1ee7c4200964cd269a4bb5cfe0f518e6f9b733efb4ba3ce35ce2e803b0ad47d24 +AAD = +CT = be46d6e61dc11b2fc8ac4c9c5f49ceca0fb6fe1fc7221c7cc5d8ba254a92282500b1b31528314035cd125578de960b3bafb69d +Tag = 7c00aa14 + +Count = 8 +Key = c244e9afd90ec810acb7e586cf7a06386e6892e01c7d111d5c1455fc95250d1b +IV = 6ff9103f9751ff4743d856c5cd54491e1a537384260fbfe076f772ad0d66ff6ddd0aaff57023885b0a4d60d2b25c80b1cdc54802607770a61a2503c23cf26f1fe529573c8d4745b19dedaff5769a6a796c01540776d4fa99be9057ad87cdc973e7938640f9497753e88c4cdc358c1cfa06f1ddd826c33f44c55e2d183927baaa +PT = cd1296133cefe6f5cf6f8ec68b37172bfb793a8582d92a539f24f3582569bfcfebb706fe9f276716b185558fbfd6fe8ea99a1b +AAD = +CT = dd1694b178f15612c454885b3100c576c7b68206c57898161d4fdc51e75a428840c5cfee104c3d85fcbd92edbd1d80f22d8e64 +Tag = 741953f5 + +Count = 9 +Key = 785ea9cd2403ecaeec3e4940dce7c41ec012203a2610c780bca5d15af64748c5 +IV = fbc177ec9d47f69e2fcaba9fef7de30735e46a5d20dc66bf66c8a76a382051d780f58dccae8e2054bbe437a5bc0814381bac2b0efa99202aa1f1bf7f51b842907dc9b60f83987c31eae086e26e2018243bdd47a291b523ac6905b40dfb442ba239c876cdfaa581b2ef0683456ac944829a97b663ecd48c116d06f1f054ddfcdb +PT = a3465ad9a4009e26c39acb1d424b7ba6556a74dcc2b78a5ab65b5d07c2a97f382aadb7415395fa6fc90bd137f6894a75f70907 +AAD = +CT = 2b3eb5ab251282da1795a25f9b43ccaa5a27643d042fc315ad662947b0c5cddeb5848e6c69869ae5e81f5c76729bbaceda9889 +Tag = 64173526 + +Count = 10 +Key = 0becf9c7a069db5b9a25f2871fb0594e452126262ec1c48bca3d3024d85d0c51 +IV = e05643a72ab0b1fc42a6f17302d6446fc507bbb4f0ad59178f5084530f02534df2e673e92e67802629f93221bba545a13fb9143eb2a2ee4bce047be621a9d96a450a19951d93c527eb698ac7e132e4f00985dff91e079ae791549d37da3105c77ef5d8cb7649f1aa761b5a5fbc9e0d7dfb5aade98f3df4a641cca02f33eed55c +PT = 70ba9822801760e5ec647a41e27cf6069978be6a28be0b1da0f3661e124847823dec7ec6af737d6dea597ce5bd5baa5d6f2651 +AAD = +CT = 0d45b5ad77226524374a3bca3680528787cefd9978b3f1407688886ce28fea86e7d9005d7d6acde2a7c2fb158479918c06bba4 +Tag = 43dfd859 + +Count = 11 +Key = d1f374702b4481b83e7c7783853c1850e887b0c80cd28c0686b4cab6adc744c2 +IV = 5e15c44cef36c1c89e02b2979f4e7d275fe5973a80580753f6fd51ea477ba84346a030b90b22a9adfff02096bc0b1691c37ee21cdc1b5f4862696d195859cd2bf0b423f5d19da5e1475bcc99f96b8c7c51fe85930aea0e97304d4c025b52bd386666d0537036360e939a6fbdfd6ce3b012e551d333fa74f3cbb9d33a59477364 +PT = 996428a232b1b9ac81ff5260eb77938f0d531a4a0ed3bd774c72c18128af72e964f8d05fb7ec4e0ac8e37056a48c85713a3a01 +AAD = +CT = aa9c103ae51bf556f7c57f9948d28c859f458a74e22f039fea183e9e262b023bdac9bb5a2f167b34cde4a694c555eb0c905f34 +Tag = d1ed9cf0 + +Count = 12 +Key = a6a69524d66f015b4afb746ad0942410baf06d1aba18ef1a8b40e35633f04ce1 +IV = 5e1df758c63728b1269a1d3f610b6e3724bc7ff797dcb4a7aca9dfe1c8e1717ce4281d1c5b4b33d0aafcee342a4f4eb30089eeab2b0470d3f9b709622bde4654d41b3bbc6bf59c11edea28f26b099d83d4fefdc63ab7218221238fe1c230b1290235465b445f60eaa9822eaa8da2c08f6f2fdaaef87dacf74c7e5d6cea191c74 +PT = 0ffc6c3090817fc1a0e6f3802269e263e40d17772fabf5cbee905962878d77c8bc4223e5671bd9f310d8db56ceaebae41fc79e +AAD = +CT = f24537db3fe27eb07f1d2b9b7059dfae97df86d4e609930491c4ec3154462df308ba1c85204dd754521d8de9619603f90f8c44 +Tag = 360adc36 + +Count = 13 +Key = ff0007317f9ecc43c1601708614eb8d443a407318cbf3c085c0a9a7c67faafa8 +IV = 8186f92bd394ecfbf4bcbb9e7e442bd6faa1e59f8785d9aec82551fee38aa10212a8477be3055927379a906902c153598d50c63e316c4e4f3956ef04ce40e5b6f1901fb5d08a23b913e5ad53512b02bb75ce96f8c50a2cebdfa058ded996fe1dc5370ee50c6d90e948129d544dc89c2f28dda8429bb338d6aed0d9557f96889e +PT = e7f4063319fd31e07192a5fc9e486fc0a2a3470671a46356ab0d32989803259c0dae103a4033c533fbde585866fc2af5eba151 +AAD = +CT = 8f86644614e4f6483f28eec02fab33e7cbbd56381a8f878522c2015b91d48652472e356f608c62361939dc983e1cd364a28665 +Tag = 83335b3a + +Count = 14 +Key = 43646d9ddebec23447febe71596e6f9b2387965db1faf3f06feec8bc5c808342 +IV = 0d3774607261b07427dc77f0dcf57b026226dbd2c1df1b9d74598582da44c677af36a6bd80b6bc1f000632c84c5dfa701f8b51b4da228d340d8b4ccc4d2f5d7b5fad00809133eae9250ecd18d7a8741bb57c394396feb81c20bec23de8520f883b8e22b362dde6e6bd9dad73e8919695384a04c09a28bdb6de2fa356b0d259b1 +PT = 2aab9a484b817b759bd9d876967c90160a18208cfa753e7bccd4f73a715aaa6acc6ce666e97bc22fbcf11f263dfed332418707 +AAD = +CT = dd80974e31ad15ba26ec8a8dacfa5f57f0bc69f7113cc6c39fb038ddce37ffeee3f789a02cd86b0418fe16ca104b5fbdb26432 +Tag = 75c35f4c diff --git a/src/test/crypto/t/001_testcrypto.pl b/src/test/crypto/t/001_testcrypto.pl new file mode 100644 index 0000000000..586a37c5bd --- /dev/null +++ b/src/test/crypto/t/001_testcrypto.pl @@ -0,0 +1,137 @@ +# Reads and parses .rsp files and runs them through testcrypto +# +# Test vectors downloaded from: +# +# https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/cavp-testing-block-cipher-modes +# +# Specifically GCM Test Vectors (SP 800-38D): +# +# https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip +# +# Note that the AADlen > 0 cases were removed from our set, since we don't support that and +# the test code will just skip them currently, and we also don't bother testing 192-bit, +# but one could download the full set and run all of the files through if they wished and +# all should pass (they did when this test suite was written originally). + +use strict; +use warnings; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 56; +} +else +{ + plan skip_all => 'SSL not supported by this build'; +} + + +# XXX add CTR tests here + +my $algo = "AES-GCM"; +my @rspfiles = ( + "gcmDecrypt128.rsp", "gcmDecrypt256.rsp", + "gcmEncryptExtIV128.rsp", "gcmEncryptExtIV256.rsp"); + +note "running tests"; + +foreach my $rspfile (@rspfiles) +{ + open(my $in_rspfile, '<', $rspfile) || die; + my %testrun; + my %lengths; + + while (my $line = <$in_rspfile>) + { + + chomp($line); + + # Remove CR, if it's there. + $line =~ s/\r$//; + + # Skip comments + if ($line =~ /^[[:space:]]*#/) { next; } + + # If we hit a blank, time to run a test + if ($line =~ /^[[:space:]]*$/) + { + if (%testrun) + { + my @testargs; + + # Set up the command to run + push(@testargs, ("$ENV{TESTDIR}/testcrypto", '-a', $algo)); + + if ($testrun{'Key'}) + { + push(@testargs, ('-k', $testrun{'Key'})); + } + + if ($testrun{'IV'}) + { + push(@testargs, ('-i', $testrun{'IV'})); + } + + if ($testrun{'CT'}) + { + push(@testargs, ('-c', $testrun{'CT'})); + } + + if ($testrun{'AAD'}) + { + # Don't currently support AAD + undef(%testrun); + next; + } + + if ($testrun{'Tag'}) + { + push(@testargs, ('-t', $testrun{'Tag'})); + } + + if ($testrun{'PT'}) + { + push(@testargs, ('-p', $testrun{'PT'})); + } + + if ($testrun{fail}) + { + command_exit_is(\@testargs, 1, + "Run $testrun{Count} of Keylen: $lengths{Keylen}, IVlen: $lengths{IVlen}, PTlen: $lengths{PTlen}, AADlen: $lengths{AADlen}, Taglen: $lengths{Taglen}" + ); + } + else + { + command_ok(\@testargs, + "Run $testrun{Count} of Keylen: $lengths{Keylen}, IVlen: $lengths{IVlen}, PTlen: $lengths{PTlen}, AADlen: $lengths{AADlen}, Taglen: $lengths{Taglen}" + ); + } + undef(%testrun); + undef(%lengths); + } + else + { + next; + } + } + + # Grab length information, just to have. + if ($line =~ /^\[([A-Za-z]*) = ([0-9]*)]$/) + { + $lengths{$1} = $2; + next; + } + + if ($line =~ /^([A-Za-z]*) = ([a-f0-9]*)$/) + { + $testrun{$1} = $2; + } + + if ($line =~ /^FAIL$/) + { + $testrun{fail} = 1; + } + } +} diff --git a/src/test/crypto/t/002_testkwp.pl b/src/test/crypto/t/002_testkwp.pl new file mode 100644 index 0000000000..25911cfd9f --- /dev/null +++ b/src/test/crypto/t/002_testkwp.pl @@ -0,0 +1,126 @@ +# Reads and parses .rsp files and runs them through testcrypto +# +# (Partial) Test vectors downloaded from: +# +# https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/cavp-testing-block-cipher-modes +# +# Specifically Key Wrap Test Vectors (SP 800-38F): +# +# https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/kwtestvectors.zip +# +# We don't include the 192-bit tests, though they all worked when this test suite was written. +# We also don't include the _inv tests as those aren't supported in OpenSSL yet. + +use strict; +use warnings; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 20; +} +else +{ + plan skip_all => 'SSL not supported by this build'; +} + + +my $algo; +# my @txtfiles = ("KWP_AD_128.txt", "KWP_AD_192.txt", "KWP_AD_256.txt", "KWP_AE_128.txt", "KWP_AE_192.txt", "KWP_AE_256.txt"); +my @txtfiles = + ("KWP_AD_128.txt", "KWP_AD_256.txt", "KWP_AE_128.txt", "KWP_AE_256.txt"); + +note "running tests"; + +foreach my $txtfile (@txtfiles) +{ + open(my $in_txtfile, '<', $txtfile) || die; + my %testrun; + my %lengths; + + if ($txtfile =~ /^KWP_/) + { + $algo = 'AES-KWP'; + } + + while (my $line = <$in_txtfile>) + { + + chomp($line); + + # Remove CR, if it's there. + $line =~ s/\r$//; + + # Skip comments + if ($line =~ /^[[:space:]]*#/) { next; } + + # If we hit a blank, time to run a test + if ($line =~ /^[[:space:]]*$/) + { + if (%testrun) + { + my @testargs; + + # Set up the command to run + push(@testargs, ("$ENV{TESTDIR}/testcrypto", '-a', $algo)); + + if ($testrun{'K'}) + { + push(@testargs, ('-k', $testrun{'K'})); + } + + if ($testrun{'C'}) + { + push(@testargs, ('-c', $testrun{'C'})); + } + + if ($testrun{'P'}) + { + push(@testargs, ('-p', $testrun{'P'})); + } + + if ($testrun{fail}) + { + command_exit_is(\@testargs, 1, + "Run $testrun{COUNT} of Plaintext Length: $lengths{'PLAINTEXT LENGTH'}" + ); + } + else + { + command_ok(\@testargs, + "Run $testrun{COUNT} of Plaintext Length: $lengths{'PLAINTEXT LENGTH'}" + ); + } + undef(%testrun); + undef(%lengths); + } + else + { + next; + } + } + + # Grab length information, just to have. + if ($line =~ /^\[([A-Za-z ]*) = ([0-9]*)]$/) + { + $lengths{$1} = $2; + next; + } + + if ($line =~ /^([A-Z]) = ([a-f0-9]*)$/) + { + $testrun{$1} = $2; + } + + if ($line =~ /^COUNT = ([0-9]*)$/) + { + $testrun{COUNT} = $1; + } + + if ($line =~ /^FAIL$/) + { + $testrun{fail} = 1; + } + } +} diff --git a/src/test/crypto/t/003_clusterkey.pl b/src/test/crypto/t/003_clusterkey.pl new file mode 100644 index 0000000000..5b8a255ace --- /dev/null +++ b/src/test/crypto/t/003_clusterkey.pl @@ -0,0 +1,93 @@ +# Test cluster file encryption key managment +# + +use strict; +use warnings; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 6; +} +else +{ + plan skip_all => "tests cannot run without OpenSSL"; +} + +# generate two cluster file encryption keys of random hex digits +my ($rand_hex, $rand_hex2); +$rand_hex .= sprintf("%x", rand 16) for 1 .. 64; +$rand_hex2 .= sprintf("%x", rand 16) for 1 .. 64; + +# initialize cluster using the first cluster key +my $node = get_new_node('node'); +$node->init( + extra => [ + '--file-encryption-method', 'AES256', + '--cluster-key-command', "echo $rand_hex" + ]); + +# Set wal_level to 'replica'; encryption can't use 'minimal' +$node->append_conf('postgresql.conf', 'wal_level=replica'); + +$node->start; + +# check encryption method +my $file_encryption_method = + $node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq 'AES256', 'file_encryption_method is valid'); + +# record pg_proc count +my $old_pg_proc_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM pg_proc;'); +ok($old_pg_proc_count > 0, 'pg_proc count is valid'); + +# create permanent table +$node->safe_psql('postgres', + 'CREATE TABLE perm_table (x) AS SELECT * FROM generate_series(1, 100);'); + +# create unlogged table +# Non-permanent tables like unlogged tables use a special nonce bit, so test those here. +$node->safe_psql('postgres', + 'CREATE UNLOGGED TABLE unlog_table (x) AS SELECT * FROM generate_series(1, 200);' +); + +# We can run pg_alterckey and change the cluster_key_command here +# without affecting the running server. +system_or_bail( + 'pg_alterckey', + "echo $rand_hex", + "echo $rand_hex2", + $node->data_dir); + +$node->safe_psql('postgres', + "ALTER SYSTEM SET cluster_key_command TO 'echo $rand_hex2'"); + +$node->stop; + +# start/stop with new cluster key +$node->start; + +# check encryption method +$file_encryption_method = + $node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq 'AES256', 'file_encryption_method is valid'); + +# check pg_proc count +my $new_pg_proc_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM pg_proc;'); +ok($new_pg_proc_count == $old_pg_proc_count, 'old/new pg_proc counts match'); + +# check permanent table count +my $perm_table_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM perm_table;'); +ok($perm_table_count == 100, 'perm_table_count count matches'); + +# check unlogged table count +my $unlog_table_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM unlog_table;'); +ok($unlog_table_count == 200, 'unlog_table_count count matches'); + +$node->stop; diff --git a/src/test/crypto/t/004_buffers.pl b/src/test/crypto/t/004_buffers.pl new file mode 100644 index 0000000000..a1379b4772 --- /dev/null +++ b/src/test/crypto/t/004_buffers.pl @@ -0,0 +1,157 @@ +# Test cluster file encryption buffer management + +# This tests that an encrypted server actually encrypts the database +# files. It does this by checking for strings in the database files in +# both non-encrypted and encrypted clusters. We test a system table, a +# permanent relation, and a unlogged/non-permanent table. +# (Non-permanent relations use a special nonce bit, which is why we test +# it here.) + +use strict; +use warnings; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 17; +} +else +{ + plan skip_all => "tests cannot run without OpenSSL"; +} + +my %file_match_count; + +sub get_cluster_file_contents +{ + my $node = shift(); + my %relnode; + + # get postgres database oid + my $postgres_db_oid = $node->safe_psql('postgres', + "SELECT oid FROM pg_database WHERE datname = 'postgres';"); + ok($postgres_db_oid != 0, 'retrieving postgres database oid'); + + # get pg_proc relfilenode + $relnode{pg_proc} = + $node->safe_psql('postgres', "SELECT pg_relation_filenode('pg_proc');"); + ok($relnode{pg_proc} != 0, 'retrieving pg_proc relfilenode'); + + # create permanent table + $node->safe_psql('postgres', + "CREATE TABLE perm_table (x) AS SELECT 'aaaaaaaa' FROM generate_series(1, 100);" + ); + + # get permanent table relfilenode + $relnode{perm} = $node->safe_psql('postgres', + "SELECT pg_relation_filenode('perm_table');"); + ok($relnode{perm} != 0, 'retrieving permanent table relfilenode'); + + # create unlogged table + $node->safe_psql('postgres', + "CREATE UNLOGGED TABLE unlog_table (x) AS SELECT 'bbbbbbbb' FROM generate_series(1, 200);" + ); + + # get unlogged table relfilenode + $relnode{unlog} = $node->safe_psql('postgres', + "SELECT pg_relation_filenode('unlog_table');"); + ok($relnode{unlog} != 0, 'retrieving unlogged table relfilenode'); + + my $file_contents = + slurp_file($node->basedir . + '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{pg_proc}); + # () converts to list context + $file_match_count{pg_proc} = () = $file_contents =~ m/pg_[a-z]{3,}/g; + + $file_contents = + slurp_file($node->basedir . + '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{perm}); + $file_match_count{perm} = () = $file_contents =~ m/a{8,}/g; + + $file_contents = + slurp_file($node->basedir . + '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{unlog}); + $file_match_count{unlog} = () = $file_contents =~ m/b{8,}/g; +} + +# +# Test with disabled encryption +# + +# initialize cluster +my $non_encrypted_node = get_new_node('non_encrypted_node'); +$non_encrypted_node->init(); + +$non_encrypted_node->start; + +# check encryption is disabled +my $file_encryption_method = + $non_encrypted_node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq '', 'file_encryption_method is valid'); + +get_cluster_file_contents($non_encrypted_node); + +# record pg_proc count +my $query_pg_proc_count = $non_encrypted_node->safe_psql('postgres', + "SELECT COUNT(*) FROM pg_proc WHERE proname ~ '^pg_[a-z]{3,}';"); +ok($query_pg_proc_count > 0, 'pg_proc count is valid'); + +# check pg_proc count +ok($file_match_count{pg_proc} != $query_pg_proc_count, + 'SQL/file pg_proc counts match'); + +# check permanent table count +ok($file_match_count{perm} != 100, 'perm_table count matches'); + +# check unlogged table count +ok($file_match_count{unlog} != 200, 'unlog_table count matches'); + +$non_encrypted_node->stop; + + +#--------------------------------------------------------------------------- + +# +# Test with enabled encryption +# + +my $rand_hex; +$rand_hex .= sprintf("%x", rand 16) for 1 .. 64; + +# initialize cluster using a cluster key +my $encrypted_node = get_new_node('encrypted_node'); +$encrypted_node->init( + extra => [ + # We tested AES256 in 003, so use AES128 + '--file-encryption-method', 'AES128', + '--cluster-key-command', "echo $rand_hex" + ]); + +# Set wal_level to 'replica'; encryption can't use 'minimal' +$encrypted_node->append_conf('postgresql.conf', 'wal_level=replica'); + +$encrypted_node->start; + +# check encryption method +$file_encryption_method = + $encrypted_node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq 'AES128', 'file_encryption_method is valid'); + +get_cluster_file_contents($encrypted_node); + +# Because the files are encrypted, we should get zero matches for all +# comparisons below. However, technically the encrypted data might +# match the desired string, so we allow one such match. + +# check pg_proc +ok($file_match_count{pg_proc} <= 1, 'pg_proc is encrypted'); + +# check permanent table +ok($file_match_count{perm} <= 1, 'perm_table is encrypted'); + +# check unlogged table +ok($file_match_count{unlog} <= 1, 'unlog_table is encrypted'); + +$encrypted_node->stop; diff --git a/src/test/crypto/testcrypto.c b/src/test/crypto/testcrypto.c new file mode 100644 index 0000000000..85bf921c09 --- /dev/null +++ b/src/test/crypto/testcrypto.c @@ -0,0 +1,458 @@ +/*------------------------------------------------------------------------- + * + * testcrypto.c + * A utility to test our encryption / decryption routines. + * + * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/test/crypto/testcrypto.c + * + *------------------------------------------------------------------------- + */ + +#define FRONTEND 1 + +#define EXITSUCCESS 0 +#define EXITDECRYPTFAIL 1 +#define EXITFAILURE 2 + +#include "postgres_fe.h" + +#include <signal.h> +#include <string.h> +#include <unistd.h> +#include <sys/stat.h> + +#include "common/hex.h" +#include "common/cipher.h" +#include "common/logging.h" +#include "getopt_long.h" +#include "pg_getopt.h" + +static const char *progname; + +static void +usage(const char *progname) +{ + printf(_("%s tests encryption/decryption routines in PG common library.\n\n"), progname); + printf(_("Usage:\n")); + printf(_(" %s [OPTION]\n"), progname); + printf("\n"); + printf(_(" Performs one encryption and one decryption run.\n")); + printf("\n"); + printf(_(" Encrypts the provided plaintext (or the empty string if none given) and generates a tag, if using AES-GCM, using the key and IV given.\n")); + printf(_(" After encryption, compares provided ciphertext to resulting ciphertext.\n")); + printf(_(" Compares provided tag, if any, to resulting tag.\n")); + printf(_(" If no tag is provided, then the tag created during encryption is used during decryption.\n")); + printf("\n"); + printf(_(" Decrypts the provided ciphertext (or the empty string if none given) using the key, and IV + tag given if using AES-GCM.\n")); + printf(_(" After successful decryption (requires tag to match for AES-GCM), compares provided plaintext to resulting plaintext.\n")); + printf(_(" Exits with '1' if decryption fails.\n")); + printf("\n"); + printf(_(" Exits with '2' for any other failure.\n")); + printf("\n"); + printf(_(" Key is always required, IV is required for AES-GCM mode.\n")); + printf("\n"); + printf(_(" Algorithms supported are AES-GCM and AES-KWP.\n")); + printf("\n"); + printf(_("\nOptions:\n")); + printf(_(" -a, --algorithm=ALG Crypto algorithm to use\n")); + printf(_(" -i, --init-vector=IV Initialization vector to use\n")); + printf(_(" -k, --key=KEY Key to use, in hex\n")); + printf(_(" -p, --plain-text=PT Plain text to encrypt\n")); + printf(_(" -c, --cipher-text=CT Cipher text to decrypt\n")); + printf(_(" -t, --tag=TAG Tag to use for decryption\n")); + printf(_(" -T, --tag-length=LEN Length of tag to use for encryption\n")); + printf(_(" -v, --verbose verbose output\n")); + printf(_(" -V, --version output version information, then exit\n")); + printf(_(" -?, --help show this help, then exit\n")); + printf("\n"); + printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT); +} + + +int +main(int argc, char *argv[]) +{ + char *algorithm = NULL, + *iv_hex = NULL, + *key_hex = NULL, + *plaintext_hex = NULL, + *ciphertext_hex = NULL, + *tag_hex = NULL; + + unsigned char *plaintext = NULL, + *ciphertext = NULL, + *key = NULL, + *iv = NULL, + *tag = NULL, + *tag_result = NULL, + *result = NULL; + + int verbose = 0, + plaintext_len = 0, + ciphertext_len = 0, + key_len = 0, + iv_len = 0, + tag_len = 16, + result_len = 0, + blocksize = 0, + cipher = PG_CIPHER_AES_GCM; + + PgCipherCtx *ctx = NULL; + + static struct option long_options[] = { + {"algorithm", required_argument, NULL, 'a'}, + {"init-vector", required_argument, NULL, 'i'}, + {"key", required_argument, NULL, 'k'}, + {"plain-text", required_argument, NULL, 'p'}, + {"cipher-text", required_argument, NULL, 'c'}, + {"tag", required_argument, NULL, 't'}, + {"tag-length", required_argument, NULL, 'T'}, + {"verbose", required_argument, NULL, 'v'}, + {NULL, 0, NULL, 0} + }; + + int c; + + pg_logging_init(argv[0]); + set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("testcrypto")); + progname = get_progname(argv[0]); + + if (argc > 1) + { + if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0) + { + usage(progname); + exit(EXITSUCCESS); + } + if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0) + { + puts("testcrypto (PostgreSQL) " PG_VERSION); + exit(EXITSUCCESS); + } + } + + /* Process command-line argument */ + + while ((c = getopt_long(argc, argv, "a:i:k:p:c:t:T:v", long_options, NULL)) != -1) + { + switch (c) + { + case 'a': + algorithm = pg_strdup(optarg); + break; + + case 'i': + iv_hex = pg_strdup(optarg); + break; + + case 'k': + key_hex = pg_strdup(optarg); + break; + + case 'p': + plaintext_hex = pg_strdup(optarg); + break; + + case 'c': + ciphertext_hex = pg_strdup(optarg); + break; + + case 't': + tag_hex = pg_strdup(optarg); + break; + + case 'T': + tag_len = atoi(optarg); + break; + + case 'v': + verbose = 1; + break; + + default: + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(EXITFAILURE); + } + } + + /* Complain if any arguments remain */ + if (optind < argc) + { + pg_log_error("too many command-line arguments (first is \"%s\")", + argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + + /* Check options passed in */ + if (algorithm) + { + if (strcmp(algorithm, "AES-GCM") == 0) + cipher = PG_CIPHER_AES_GCM; + else if (strcmp(algorithm, "AES-KWP") == 0) + cipher = PG_CIPHER_AES_KWP; + else + { + pg_log_error("Unsupported algorithm selected."); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + } + + if (key_hex == NULL) + { + pg_log_error("Key must be provided"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + else + { + size_t key_hex_len = strlen(key_hex); + + key_len = pg_hex_dec_len(key_hex_len); + + key = pg_malloc0(key_len); + pg_hex_decode(key_hex, key_hex_len, (char *) key, key_len); + } + + if (cipher == PG_CIPHER_AES_GCM && iv_hex == NULL) + { + pg_log_error("Initialization vector must be provided"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + else if (cipher == PG_CIPHER_AES_GCM) + { + size_t iv_hex_len = strlen(iv_hex); + + iv_len = pg_hex_dec_len(iv_hex_len); + + iv = pg_malloc0(iv_len); + pg_hex_decode(iv_hex, iv_hex_len, (char *) iv, iv_len); + } + + if (plaintext_hex) + { + size_t plaintext_hex_len = strlen(plaintext_hex); + + plaintext_len = pg_hex_dec_len(plaintext_hex_len); + + plaintext = pg_malloc0(plaintext_len); + pg_hex_decode(plaintext_hex, plaintext_hex_len, (char *) plaintext, + plaintext_len); + } + + /* + * OpenSSL 1.1.1d and earlier crashes on some zero-length plaintext and + * ciphertext strings. It crashes on an encryption call to + * EVP_EncryptFinal_ex(() in GCM mode of zero-length strings if plaintext + * is NULL, even though plaintext_len is zero. Setting plaintext to + * non-NULL allows it to work. In KWP mode, zero-length strings fail if + * plaintext_len = 0 and plaintext is non-NULL (the opposite). OpenSSL + * 1.1.1e+ is fine with all options. + */ + else if (cipher == PG_CIPHER_AES_GCM) + { + plaintext_len = 0; + plaintext = pg_malloc0(1); + } + + if (ciphertext_hex) + { + size_t ciphertext_hex_len = strlen(ciphertext_hex); + + ciphertext_len = pg_hex_dec_len(ciphertext_hex_len); + + ciphertext = pg_malloc0(ciphertext_len); + pg_hex_decode(ciphertext_hex, ciphertext_hex_len, + (char *) ciphertext, ciphertext_len); + } + /* see OpenSSL 1.1.1d item above, though crash only happens in GCM mode */ + else if (cipher == PG_CIPHER_AES_GCM) + { + ciphertext_len = 0; + ciphertext = pg_malloc0(1); + } + + if (cipher == PG_CIPHER_AES_GCM) + tag_result = pg_malloc0(tag_len); + + if (tag_hex) + { + size_t tag_hex_len = strlen(tag_hex); + + tag_len = pg_hex_dec_len(tag_hex_len); + + tag = pg_malloc0(tag_len); + pg_hex_decode(tag_hex, tag_hex_len, (char *) tag, tag_len); + } + else + tag = tag_result; + + if (verbose) + { + printf("Alrogithm: %d\n", cipher); + printf("Key length: %d (%d bits)\n", key_len, key_len * 8); + printf("IV length: %d (%d bits)\n", iv_len, iv_len * 8); + printf("Tag length: %d (%d bits)\n", tag_len, tag_len * 8); + printf("Plaintext length: %d\n", plaintext_len); + printf("Ciphertext length: %d\n", ciphertext_len); + } + + /* + * Encryption + * + * We run through the encryption even if there wasn't a plaintext + * provided- in that case we just encrypt the empty string. + */ + ctx = pg_cipher_ctx_create(cipher, key, key_len, true); + if (!ctx) + { + pg_log_error("Error creating encryption context, be sure key is of supported length"); + exit(EXITFAILURE); + } + + blocksize = pg_cipher_blocksize(ctx); + + /* If we were provided with a plaintext input */ + if (plaintext_len != 0) + { + /* Encryption might result in as much as input length + blocksize */ + result_len = plaintext_len + blocksize; + result = palloc0(result_len); + + if (ciphertext_hex == NULL) + { + ciphertext = result; + ciphertext_len = plaintext_len; + } + } + + if (cipher == PG_CIPHER_AES_GCM) + { + if (!pg_cipher_encrypt(ctx, cipher, + plaintext, plaintext_len, + result, &result_len, + iv, iv_len, + tag_result, tag_len)) + { + pg_log_error("Error during encryption."); + exit(EXITFAILURE); + } + } + else if (cipher == PG_CIPHER_AES_KWP) + { + if (!pg_cipher_keywrap(ctx, + plaintext, plaintext_len, + result, &result_len)) + { + pg_log_error("Error during encryption."); + exit(EXITFAILURE); + } + } + + if (verbose || ciphertext == NULL) + { + uint64 result_hex_len = pg_hex_enc_len(result_len); + char *result_hex = palloc0(result_hex_len + 1); + + pg_hex_encode((char *) result, result_len, result_hex, result_hex_len); + result_hex[result_hex_len] = '\0'; + + printf("ciphertext: %s\n", result_hex); + + if (cipher == PG_CIPHER_AES_GCM) + { + result_hex_len = pg_hex_enc_len(tag_len); + result_hex = palloc0(result_hex_len + 1); + + pg_hex_encode((char *) tag_result, tag_len, result_hex, result_hex_len); + result_hex[result_hex_len] = '\0'; + + printf("tag: %s\n", result_hex); + } + } + + /* + * Report on non-matching results, but still go through the decryption + * routine to make sure that we get the correct result, and then error + * out. + */ + if (plaintext_len != 0 && ciphertext != NULL && memcmp(ciphertext, result, plaintext_len) != 0) + pg_log_error("Provided ciphertext does not match"); + + if (cipher == PG_CIPHER_AES_GCM && tag != tag_result && memcmp(tag, tag_result, tag_len) != 0) + pg_log_error("Provided tag does not match"); + + /* + * If a ciphertext was provided then use that as the max size of our + * plaintext result. We shouldn't ever get a result larger. + */ + if (ciphertext_len != 0) + { + result_len = ciphertext_len; + result = palloc0(result_len); + } + + /* + * Decryption + * + * We run through the decryption even if there wasn't a ciphertext + * provided- in that case we just decrypt the empty string. + */ + ctx = pg_cipher_ctx_create(cipher, key, key_len, false); + if (!ctx) + { + pg_log_error("Error creating decryption context, be sure key is of supported length"); + exit(EXITFAILURE); + } + + if (cipher == PG_CIPHER_AES_GCM) + { + if (!pg_cipher_decrypt(ctx, cipher, + ciphertext, ciphertext_len, + result, &result_len, + iv, iv_len, + tag, tag_len)) + { + pg_log_error("Error during decryption."); + exit(EXITDECRYPTFAIL); + } + } + else if (cipher == PG_CIPHER_AES_KWP) + { + if (!pg_cipher_keyunwrap(ctx, + ciphertext, ciphertext_len, + result, &result_len)) + { + pg_log_error("Error during decryption."); + exit(EXITDECRYPTFAIL); + } + } + + if (verbose || plaintext == NULL) + { + uint64 result_hex_len = pg_hex_enc_len(result_len); + char *result_hex = palloc0(result_hex_len + 1); + + pg_hex_encode((char *) result, result_len, result_hex, result_hex_len); + result_hex[result_hex_len] = '\0'; + + printf("plaintext: %s\n", result_hex); + } + + if (ciphertext_len != 0 && plaintext != NULL && memcmp(plaintext, result, plaintext_len) != 0) + { + pg_log_error("Provided plaintext does not match"); + exit(EXITFAILURE); + } + + exit(EXITSUCCESS); +} -- 2.37.0 (Apple Git-136) [application/octet-stream] 0010-cfe-10-hint_over_cfe-09-test-squash-commit.patch (17.7K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/11-0010-cfe-10-hint_over_cfe-09-test-squash-commit.patch) download | inline diff: From ec1d0780fdf5cce01235528701a4dc353d902995 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:03 -0400 Subject: [PATCH 10/12] cfe-10-hint_over_cfe-09-test squash commit --- doc/src/sgml/config.sgml | 7 +- src/backend/access/rmgrdesc/xlogdesc.c | 6 +- src/backend/access/transam/xlog.c | 4 + src/backend/access/transam/xloginsert.c | 24 +++++ src/backend/replication/logical/decode.c | 1 + src/backend/storage/buffer/bufmgr.c | 131 ++++++++++++++++------- src/bin/pg_alterckey/pg_alterckey.c | 76 +++++++++++-- src/include/access/xlog.h | 14 +-- src/include/access/xloginsert.h | 2 + src/include/catalog/pg_control.h | 1 + 10 files changed, 214 insertions(+), 52 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 69ee55a568..3da17c76f2 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3164,9 +3164,10 @@ include_dir 'conf.d' </para> <para> - If data checksums are enabled, hint bit updates are always WAL-logged - and this setting is ignored. You can use this setting to test how much - extra WAL-logging would occur if your database had data checksums + If data checksums or cluster file encryption is enabled, + hint bit updates are always WAL-logged and this setting is + ignored. You can use this setting to test how much extra + WAL-logging would occur if your database had data checksums enabled. </para> diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c index 3fd7185f21..e08c364d03 100644 --- a/src/backend/access/rmgrdesc/xlogdesc.c +++ b/src/backend/access/rmgrdesc/xlogdesc.c @@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record) appendStringInfoString(buf, xlrec->rp_name); } - else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT) + else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT || + info == XLOG_ENCRYPTION_LSN) { /* no further information to print */ } @@ -196,6 +197,9 @@ xlog_identify(uint8 info) case XLOG_FPI_FOR_HINT: id = "FPI_FOR_HINT"; break; + case XLOG_ENCRYPTION_LSN: + id = "ENCRYPTION_LSN"; + break; } return id; diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index de92e88d81..43429647b5 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -8012,6 +8012,10 @@ xlog_redo(XLogReaderState *record) UnlockReleaseBuffer(buffer); } } + else if (info == XLOG_ENCRYPTION_LSN) + { + /* nothing to do here */ + } else if (info == XLOG_BACKUP_END) { /* nothing to do here, handled in xlogrecovery_redo() */ diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c index f811523448..b4bbacae26 100644 --- a/src/backend/access/transam/xloginsert.c +++ b/src/backend/access/transam/xloginsert.c @@ -1081,6 +1081,30 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std) return recptr; } +/* + * This function returns either a WAL or fake LSN, for use by encryption. + */ +XLogRecPtr +LSNForEncryption(bool use_wal_lsn) +{ + if (use_wal_lsn) + { + int dummy = 0; + + Assert(FileEncryptionEnabled); + /* + * Records other than SWITCH_WAL must have content. We use an integer 0 to + * follow the restriction. + */ + XLogBeginInsert(); + XLogSetRecordFlags(XLOG_MARK_UNIMPORTANT); + XLogRegisterData((char *) &dummy, sizeof(dummy)); + return XLogInsert(RM_XLOG_ID, XLOG_ENCRYPTION_LSN); + } + else + return GetFakeLSNForUnloggedRel(); +} + /* * Write a WAL record containing a full image of a page. Caller is responsible * for writing the page to disk after calling this routine. diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index 2cc0ac9eb0..6922f6c0e2 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -162,6 +162,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf) case XLOG_FPI_FOR_HINT: case XLOG_FPI: case XLOG_OVERWRITE_CONTRECORD: + case XLOG_ENCRYPTION_LSN: break; default: elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info); diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 6b95381481..5ec3e405ab 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -3991,11 +3991,12 @@ IncrBufferRefCount(Buffer buffer) * This is essentially the same as MarkBufferDirty, except: * * 1. The caller does not write WAL; so if checksums are enabled, we may need - * to write an XLOG_FPI_FOR_HINT WAL record to protect against torn pages. + * to write an XLOG_FPI_FOR_HINT record to protect against torn pages, or + * XLOG_ENCRYPTION_LSN to generate a new LSN for the page. * 2. The caller might have only share-lock instead of exclusive-lock on the - * buffer's content lock. + * buffer's content lock. * 3. This function does not guarantee that the buffer is always marked dirty - * (due to a race condition), so it cannot be used for important changes. + * (due to a race condition), so it cannot be used for important changes. */ void MarkBufferDirtyHint(Buffer buffer, bool buffer_std) @@ -4041,53 +4042,111 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * If we need to protect hint bit updates from torn writes, WAL-log a * full page image of the page. This full page image is only necessary * if the hint bit update is the first change to the page since the - * last checkpoint. + * last checkpoint. If cluster file encryption is enabled, we also + * need to generate new page LSNs for all other cases of page writes. * * We don't check full_page_writes here because that logic is included * when we call XLogInsert() since the value changes dynamically. */ - if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + if (XLogHintBitIsNeeded()) { /* - * If we must not write WAL, due to a relfilelocator-specific - * condition or being in recovery, don't dirty the page. We can - * set the hint, just not dirty the page as a result so the hint - * is lost when we evict the page or shutdown. + * If we must not write WAL during recovery so don't dirty the page. + * We can set the hint, just not dirty the page as a result so the + * hint is lost when we evict the page or shutdown. * * See src/backend/storage/page/README for longer discussion. */ if (RecoveryInProgress() || - RelFileLocatorSkippingWAL(BufTagGetRelFileLocator(&bufHdr->tag))) + (RelFileLocatorSkippingWAL(BufTagGetRelFileLocator(&bufHdr->tag)) && + !FileEncryptionEnabled)) return; /* - * If the block is already dirty because we either made a change - * or set a hint already, then we don't need to write a full page - * image. Note that aggressive cleaning of blocks dirtied by hint - * bit setting would increase the call rate. Bulk setting of hint - * bits would reduce the call rate... - * - * We must issue the WAL record before we mark the buffer dirty. - * Otherwise we might write the page before we write the WAL. That - * causes a race condition, since a checkpoint might occur between - * writing the WAL record and marking the buffer dirty. We solve - * that with a kluge, but one that is already in use during - * transaction commit to prevent race conditions. Basically, we - * simply prevent the checkpoint WAL record from being written - * until we have marked the buffer dirty. We don't start the - * checkpoint flush until we have marked dirty, so our checkpoint - * must flush the change to disk successfully or the checkpoint - * never gets written, so crash recovery will fix. - * - * It's possible we may enter here without an xid, so it is - * essential that CreateCheckPoint waits for virtual transactions - * rather than full transactionids. + * Non-BM_PERMANENT objects don't need full page images because + * they are not restored. WAL-skipped relfilenodes should never + * have full page images generated. */ - Assert((MyProc->delayChkptFlags & DELAY_CHKPT_START) == 0); - MyProc->delayChkptFlags |= DELAY_CHKPT_START; - delayChkptFlags = true; - lsn = XLogSaveBufferForHint(buffer, buffer_std); + if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT && + !RelFileLocatorSkippingWAL(BufTagGetRelFileLocator(&bufHdr->tag))) + { + /* + * If the block is already dirty because we either made a change + * or set a hint already, then we don't need to write a full + * page image. Note that aggressive cleaning of blocks dirtied + * by hint bit setting would increase the call rate. Bulk + * setting of hint bits would reduce the call rate... + * + * We must issue the WAL record before we mark the buffer + * dirty. Otherwise we might write the page before we write + * the WAL. That causes a race condition, since a checkpoint + * might occur between writing the WAL record and marking the + * buffer dirty. We solve that with a kluge, but one that is + * already in use during transaction commit to prevent race + * conditions. Basically, we simply prevent the checkpoint WAL + * record from being written until we have marked the buffer + * dirty. We don't start the checkpoint flush until we have + * marked dirty, so our checkpoint must flush the change to disk + * successfully or the checkpoint never gets written, so crash + * recovery will fix. + * + * It's possible we may enter here without an xid, so it + * is essential that CreateCheckpoint waits for virtual + * transactions rather than full transactionids. + */ + MyProc->delayChkptFlags |= DELAY_CHKPT_START; + delayChkptFlags = true; + lsn = XLogSaveBufferForHint(buffer, buffer_std); + } + + /* + * Above, for hint bit changes, we might have generated a new page + * LSN and a full-page WAL record for a page's first-clean-to-dirty + * during a checkpoint for permanent, non-WAL-skipped relfilenodes. + * If we didn't (the lsn variable is invalid), and we are + * doing cluster file encryption, we must generate a new + * page LSN here for either non-permanent relations or page + * non-first-clean-to-dirty during a checkpoint. (Cluster file + * encryption does not support WAL-skip relfilenodes.) We must + * update the page LSN even if the page with the hint bit change is + * later overwritten in the file system with an earlier version of + * the page during crash recovery. + * + * XXX Can we rely on the full page write above with no lock being + * held to avoid torn pages? Above, the LSN and page image are + * tied together, but here is just the page LSN update. + */ + if (XLogRecPtrIsInvalid(lsn) && FileEncryptionEnabled) + { + /* + * For cluster file encryption we need a new page LSN because + * the LSN is used, with the page number and permanent flag, as + * part of the nonce, and the nonce must be unique for every + * page write. If we reencrypt a page with hint bit changes + * using the same nonce as previous writes, it would expose the + * hint bit change locations. To avoid this, we write a simple + * WAL record to advance the lsn, which can then be assigned to + * the page below. + * + * Above we are relying on the full page writes to revert + * any partial pages writes caused by this LSN change for + * permanent, non-WAL-skip relfilenodes. For non-permanent + * relations, they crash recover as empty. For WAL-skip + * relfilenodes, they recover with their original contents, so + * that works too. + */ + /* XXX Do we need the checkpoint delay here? */ + MyProc->delayChkptFlags |= DELAY_CHKPT_START; + delayChkptFlags = true; + /* + * XXX We probably don't need to replay this WAL on the primary + * since the full page image is restored, but do we have + * to replay this on the repicas (for relations that are + * replicated)? + */ + lsn = LSNForEncryption( + pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT); + } } buf_state = LockBufHdr(bufHdr); diff --git a/src/bin/pg_alterckey/pg_alterckey.c b/src/bin/pg_alterckey/pg_alterckey.c index 7e3d4ad915..c699c59761 100644 --- a/src/bin/pg_alterckey/pg_alterckey.c +++ b/src/bin/pg_alterckey/pg_alterckey.c @@ -38,10 +38,9 @@ #include "common/file_perm.h" #include "common/file_utils.h" -#include "common/hex.h" #include "common/restricted_token.h" -#include "crypto/kmgr.h" #include "common/logging.h" +#include "crypto/kmgr.h" #include "getopt_long.h" #include "pg_getopt.h" @@ -84,6 +83,9 @@ static void bzero_keys_and_exit(exit_action action); static void reencrypt_data_keys(void); static void install_new_keys(void); +static uint64 hex_decode(const char *src, size_t len, char *dst); + + static void usage(const char *progname) { @@ -511,8 +513,8 @@ retrieve_cluster_keys(void) (char *) cluster_key_hex, ALLOC_KMGR_CLUSTER_KEY_LEN, live_path, terminal_fd); - if (pg_hex_decode(cluster_key_hex, cluster_key_len, - (char *) old_cluster_key, KMGR_CLUSTER_KEY_LEN) != + if (hex_decode(cluster_key_hex, cluster_key_len, + (char *) old_cluster_key) != KMGR_CLUSTER_KEY_LEN) { pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); @@ -534,8 +536,8 @@ retrieve_cluster_keys(void) (char *) cluster_key_hex, ALLOC_KMGR_CLUSTER_KEY_LEN, new_path, terminal_fd); - if (pg_hex_decode(cluster_key_hex, cluster_key_len, - (char *) new_cluster_key, KMGR_CLUSTER_KEY_LEN) != + if (hex_decode(cluster_key_hex, cluster_key_len, + (char *) new_cluster_key) != KMGR_CLUSTER_KEY_LEN) { pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); @@ -722,3 +724,65 @@ bzero_keys_and_exit(exit_action action) /* return 0 or 1 */ exit(action != SUCCESS_EXIT); } + +/* + * HEX + */ + +static const int8 hexlookup[128] = { + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, +}; + +static inline char +get_hex(const char *cp) +{ + unsigned char c = (unsigned char) *cp; + int res = -1; + + if (c < 127) + res = hexlookup[c]; + + if (res < 0) + pg_fatal("invalid hexadecimal digit: \"%s\"",cp); + + return (char) res; +} + +static uint64 +hex_decode(const char *src, size_t len, char *dst) +{ + const char *s, + *srcend; + char v1, + v2, + *p; + + srcend = src + len; + s = src; + p = dst; + while (s < srcend) + { + if (*s == ' ' || *s == '\n' || *s == '\t' || *s == '\r') + { + s++; + continue; + } + v1 = get_hex(s) << 4; + s++; + if (s >= srcend) + pg_fatal("invalid hexadecimal data: odd number of digits"); + + v2 = get_hex(s); + s++; + *p++ = v1 | v2; + } + + return p - dst; +} diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 8f219fbacc..78def5753a 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -107,13 +107,15 @@ extern PGDLLIMPORT int wal_level; /* * Is a full-page image needed for hint bit updates? * - * Normally, we don't WAL-log hint bit updates, but if checksums are enabled, - * we have to protect them against torn page writes. When you only set - * individual bits on a page, it's still consistent no matter what combination - * of the bits make it to disk, but the checksum wouldn't match. Also WAL-log - * them if forced by wal_log_hints=on. + * Normally, we don't WAL-log hint bit updates, but if checksums or encryption + * is enabled, we have to protect them against torn page writes. When you + * only set individual bits on a page, it's still consistent no matter what + * combination of the bits make it to disk, but the checksum wouldn't match. + * Cluster file encryption requires a new LSN for hint bit changes, and can't + * tolerate torn pages. Also WAL-log them if forced by wal_log_hints=on. */ -#define XLogHintBitIsNeeded() (DataChecksumsEnabled() || wal_log_hints) +#define XLogHintBitIsNeeded() \ + (DataChecksumsEnabled() || FileEncryptionEnabled || wal_log_hints) /* Do we need to WAL-log information required only for Hot Standby and logical replication? */ #define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA) diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h index 001ff2f521..dfb501a7c3 100644 --- a/src/include/access/xloginsert.h +++ b/src/include/access/xloginsert.h @@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forknum, BlockNumber startblk, BlockNumber endblk, bool page_std); extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std); +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + extern void InitXLogInsert(void); #endif /* XLOGINSERT_H */ diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index db7de77287..1c97ffd2cd 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -78,6 +78,7 @@ typedef struct CheckPoint #define XLOG_FPI 0xB0 /* 0xC0 is used in Postgres 9.5-11 */ #define XLOG_OVERWRITE_CONTRECORD 0xD0 +#define XLOG_ENCRYPTION_LSN 0xE0 /* -- 2.37.0 (Apple Git-136) [application/octet-stream] 0011-cfe-11-gist_over_cfe-10-hint-squash-commit.patch (7.1K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/12-0011-cfe-11-gist_over_cfe-10-hint-squash-commit.patch) download | inline diff: From 385d85b87e4f0a7c3a3e551a1575466e8d348d93 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:05 -0400 Subject: [PATCH 11/12] cfe-11-gist_over_cfe-10-hint squash commit --- src/backend/access/gist/README | 10 ++++++++++ src/backend/access/gist/gist.c | 21 +++++++++++++++------ src/backend/access/gist/gistbuild.c | 12 +++++++++--- src/backend/access/gist/gistvacuum.c | 20 ++++++++++++++------ 4 files changed, 48 insertions(+), 15 deletions(-) diff --git a/src/backend/access/gist/README b/src/backend/access/gist/README index efb2730e18..c4d908fcb2 100644 --- a/src/backend/access/gist/README +++ b/src/backend/access/gist/README @@ -477,6 +477,16 @@ value. The page is not recycled, until that XID is no longer visible to anyone. That's much more conservative than necessary, but let's keep it simple. +GiST and Encryption +------------------- + +GiST uses LSNs and NSNs for concurrency. This means that unlogged and +temporary relations also need LSNs. GiST has a mechanism to assign LSNs +to such relations, but sometimes uses fixed LSNs and or calls +gistGetFakeLSN(), which can cause duplicate LSNs to be used. Therefore, +when encryption is enabled and fake LSNs are needed, the GiST code calls +LSNForEncryption(). See src/backend/crypto/README for more details. + Authors: Teodor Sigaev <[email protected]> diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 30069f139c..74e7e485e6 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -501,13 +501,16 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, * we don't need to be able to detect concurrent splits yet.) */ if (is_build) - recptr = GistBuildLSN; + recptr = !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(rel)); else { if (RelationNeedsWAL(rel)) recptr = gistXLogSplit(is_leaf, dist, oldrlink, oldnsn, leftchildbuf, markfollowright); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else recptr = gistGetFakeLSN(rel); } @@ -568,7 +571,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, MarkBufferDirty(leftchildbuf); if (is_build) - recptr = GistBuildLSN; + recptr = !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(rel)); else { if (RelationNeedsWAL(rel)) @@ -586,6 +590,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, deloffs, ndeloffs, itup, ntup, leftchildbuf); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else recptr = gistGetFakeLSN(rel); } @@ -1665,6 +1671,8 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel) if (ndeletable > 0) { + XLogRecPtr recptr; + TransactionId latestRemovedXid = InvalidTransactionId; if (XLogStandbyInfoActive() && RelationNeedsWAL(rel)) @@ -1690,16 +1698,17 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel) /* XLOG stuff */ if (RelationNeedsWAL(rel)) { - XLogRecPtr recptr; - recptr = gistXLogDelete(buffer, deletable, ndeletable, latestRemovedXid); - PageSetLSN(page, recptr); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - PageSetLSN(page, gistGetFakeLSN(rel)); + recptr = gistGetFakeLSN(rel); + + PageSetLSN(page, recptr); END_CRIT_SECTION(); } diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index fb0f466708..bf5162b410 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -48,6 +48,8 @@ #include "utils/rel.h" #include "utils/tuplesort.h" +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + /* Step of index tuples for check whether to switch to buffering build mode */ #define BUFFERING_MODE_SWITCH_CHECK_STEP 256 @@ -307,7 +309,8 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) GISTInitBuffer(buffer, F_LEAF); MarkBufferDirty(buffer); - PageSetLSN(page, GistBuildLSN); + PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(index))); UnlockReleaseBuffer(buffer); @@ -458,7 +461,9 @@ gist_indexsortbuild(GISTBuildState *state) gist_indexsortbuild_flush_ready_pages(state); /* Write out the root */ - PageSetLSN(levelstate->pages[0], GistBuildLSN); + RelationGetSmgr(state->indexrel); + PageSetLSN(levelstate->pages[0], !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(state->indexrel))); PageSetChecksumInplace(levelstate->pages[0], GIST_ROOT_BLKNO); smgrwrite(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, GIST_ROOT_BLKNO, levelstate->pages[0], true); @@ -655,7 +660,8 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state) if (blkno != state->pages_written) elog(ERROR, "unexpected block number to flush GiST sorting build"); - PageSetLSN(page, GistBuildLSN); + PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(state->indexrel))); PageSetChecksumInplace(page, blkno); smgrextend(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, blkno, page, true); diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c index 0aa6e58a62..35265ae0e7 100644 --- a/src/backend/access/gist/gistvacuum.c +++ b/src/backend/access/gist/gistvacuum.c @@ -24,6 +24,8 @@ #include "storage/lmgr.h" #include "utils/memutils.h" +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + /* Working state needed by gistbulkdelete */ typedef struct { @@ -180,6 +182,8 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, vstate.callback_state = callback_state; if (RelationNeedsWAL(rel)) vstate.startNSN = GetInsertRecPtr(); + else if (FileEncryptionEnabled) + vstate.startNSN = LSNForEncryption(RelationIsPermanent(rel)); else vstate.startNSN = gistGetFakeLSN(rel); @@ -359,6 +363,8 @@ restart: */ if (ntodelete > 0) { + XLogRecPtr recptr; + START_CRIT_SECTION(); MarkBufferDirty(buffer); @@ -367,16 +373,15 @@ restart: GistMarkTuplesDeleted(page); if (RelationNeedsWAL(rel)) - { - XLogRecPtr recptr; - recptr = gistXLogUpdate(buffer, todelete, ntodelete, NULL, 0, InvalidBuffer); - PageSetLSN(page, recptr); - } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - PageSetLSN(page, gistGetFakeLSN(rel)); + recptr = gistGetFakeLSN(rel); + + PageSetLSN(page, recptr); END_CRIT_SECTION(); @@ -663,8 +668,11 @@ gistdeletepage(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (RelationNeedsWAL(info->index)) recptr = gistXLogPageDelete(leafBuffer, txid, parentBuffer, downlink); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(info->index)); else recptr = gistGetFakeLSN(info->index); + PageSetLSN(parentPage, recptr); PageSetLSN(leafPage, recptr); -- 2.37.0 (Apple Git-136) [application/octet-stream] 0012-cfe-12-rel_over_cfe-11-gist-squash-commit.patch (33.0K, ../../CAOxo6XJh95xPOpvTxuP_kiGRs8eHcaNrmy3kkzWrzWxvyVkKkQ@mail.gmail.com/13-0012-cfe-12-rel_over_cfe-11-gist-squash-commit.patch) download | inline diff: From 042d9e3292cf1970d15cd98a84440552d10412e2 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:08 -0400 Subject: [PATCH 12/12] cfe-12-rel_over_cfe-11-gist squash commit --- contrib/basic_archive/basic_archive.c | 2 +- contrib/bloom/blinsert.c | 3 + src/backend/access/gist/gist.c | 51 ++++---- src/backend/access/gist/gistbuild.c | 12 ++ src/backend/access/hash/hashpage.c | 1 + src/backend/access/heap/rewriteheap.c | 7 + src/backend/access/nbtree/nbtree.c | 3 + src/backend/access/nbtree/nbtsort.c | 4 +- src/backend/access/spgist/spginsert.c | 6 + src/backend/access/transam/xlog.c | 2 + src/backend/bootstrap/bootstrap.c | 1 + src/backend/catalog/storage.c | 7 +- src/backend/crypto/Makefile | 1 + src/backend/crypto/bufenc.c | 176 ++++++++++++++++++++++++++ src/backend/postmaster/postmaster.c | 2 + src/backend/storage/buffer/bufmgr.c | 22 +++- src/backend/storage/buffer/localbuf.c | 8 +- src/backend/storage/file/copydir.c | 28 +++- src/backend/storage/file/reinit.c | 2 +- src/backend/storage/page/bufpage.c | 51 +++++++- src/backend/tcop/postgres.c | 2 + src/include/access/gist.h | 5 +- src/include/crypto/bufenc.h | 28 ++++ src/include/storage/bufpage.h | 18 ++- src/include/storage/copydir.h | 2 +- 25 files changed, 396 insertions(+), 48 deletions(-) create mode 100644 src/backend/crypto/bufenc.c create mode 100644 src/include/crypto/bufenc.h diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c index 9f221816bb..7a59e34a42 100644 --- a/contrib/basic_archive/basic_archive.c +++ b/contrib/basic_archive/basic_archive.c @@ -275,7 +275,7 @@ basic_archive_file_internal(const char *file, const char *path) * Copy the file to its temporary destination. Note that this will fail * if temp already exists. */ - copy_file(unconstify(char *, path), temp); + copy_file(unconstify(char *, path), temp, false); /* * Sync the temporary file to disk and move it to its final destination. diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index dd26d6ac29..3204950817 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -176,7 +176,10 @@ blbuildempty(Relation index) * XLOG_DBASE_CREATE* or XLOG_TBLSPC_CREATE record. Therefore, we need * this even when wal_level=minimal. */ + PageEncryptInplace(metapage, INIT_FORKNUM, RelationIsPermanent(index), + BLOOM_METAPAGE_BLKNO); PageSetChecksumInplace(metapage, BLOOM_METAPAGE_BLKNO); + smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, BLOOM_METAPAGE_BLKNO, (char *) metapage, true); log_newpage(&(RelationGetSmgr(index))->smgr_rlocator.locator, INIT_FORKNUM, diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 74e7e485e6..3472afe363 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -503,17 +503,14 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, if (is_build) recptr = !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(rel)); + else if (RelationNeedsWAL(rel)) + recptr = gistXLogSplit(is_leaf, + dist, oldrlink, oldnsn, leftchildbuf, + markfollowright); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - { - if (RelationNeedsWAL(rel)) - recptr = gistXLogSplit(is_leaf, - dist, oldrlink, oldnsn, leftchildbuf, - markfollowright); - else if (FileEncryptionEnabled) - recptr = LSNForEncryption(RelationIsPermanent(rel)); - else - recptr = gistGetFakeLSN(rel); - } + recptr = gistGetFakeLSN(rel); for (ptr = dist; ptr; ptr = ptr->next) PageSetLSN(ptr->page, recptr); @@ -573,28 +570,26 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, if (is_build) recptr = !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(rel)); - else + else if (RelationNeedsWAL(rel)) { - if (RelationNeedsWAL(rel)) - { - OffsetNumber ndeloffs = 0, - deloffs[1]; + OffsetNumber ndeloffs = 0, + deloffs[1]; - if (OffsetNumberIsValid(oldoffnum)) - { - deloffs[0] = oldoffnum; - ndeloffs = 1; - } - - recptr = gistXLogUpdate(buffer, - deloffs, ndeloffs, itup, ntup, - leftchildbuf); + if (OffsetNumberIsValid(oldoffnum)) + { + deloffs[0] = oldoffnum; + ndeloffs = 1; } - else if (FileEncryptionEnabled) - recptr = LSNForEncryption(RelationIsPermanent(rel)); - else - recptr = gistGetFakeLSN(rel); + + recptr = gistXLogUpdate(buffer, + deloffs, ndeloffs, itup, ntup, + leftchildbuf); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); + else + recptr = gistGetFakeLSN(rel); + PageSetLSN(page, recptr); if (newblkno) diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index bf5162b410..b54d50c440 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -464,6 +464,12 @@ gist_indexsortbuild(GISTBuildState *state) RelationGetSmgr(state->indexrel); PageSetLSN(levelstate->pages[0], !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(state->indexrel))); + /* Make sure LSNs are vaild, and if encryption, are not constant. */ + Assert(!XLogRecPtrIsInvalid(PageGetLSN(levelstate->pages[0])) && + (!FileEncryptionEnabled || + PageGetLSN(levelstate->pages[0]) != GistBuildLSN)); + PageEncryptInplace(levelstate->pages[0], MAIN_FORKNUM, RelationIsPermanent(state->indexrel), + GIST_ROOT_BLKNO); PageSetChecksumInplace(levelstate->pages[0], GIST_ROOT_BLKNO); smgrwrite(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, GIST_ROOT_BLKNO, levelstate->pages[0], true); @@ -662,6 +668,12 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state) PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(state->indexrel))); + /* Make sure LSNs are vaild, and if encryption, are not constant. */ + Assert(!XLogRecPtrIsInvalid(PageGetLSN(page)) && + (!FileEncryptionEnabled || + PageGetLSN(page) != GistBuildLSN)); + PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(state->indexrel), + blkno); PageSetChecksumInplace(page, blkno); smgrextend(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, blkno, page, true); diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c index d2edcd4617..4ba41ce288 100644 --- a/src/backend/access/hash/hashpage.c +++ b/src/backend/access/hash/hashpage.c @@ -1025,6 +1025,7 @@ _hash_alloc_buckets(Relation rel, BlockNumber firstblock, uint32 nblocks) zerobuf.data, true); + PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(rel), lastblock); PageSetChecksumInplace(page, lastblock); smgrextend(RelationGetSmgr(rel), MAIN_FORKNUM, lastblock, zerobuf.data, false); diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c index b01b39b008..f4936893f9 100644 --- a/src/backend/access/heap/rewriteheap.c +++ b/src/backend/access/heap/rewriteheap.c @@ -325,6 +325,9 @@ end_heap_rewrite(RewriteState state) state->rs_buffer, true); + PageEncryptInplace(state->rs_buffer, MAIN_FORKNUM, + RelationIsPermanent(state->rs_new_rel), + state->rs_blockno); PageSetChecksumInplace(state->rs_buffer, state->rs_blockno); smgrextend(RelationGetSmgr(state->rs_new_rel), MAIN_FORKNUM, @@ -691,8 +694,12 @@ raw_heap_insert(RewriteState state, HeapTuple tup) * need for smgr to schedule an fsync for this write; we'll do it * ourselves in end_heap_rewrite. */ + PageEncryptInplace(page, MAIN_FORKNUM, + RelationIsPermanent(state->rs_new_rel), + state->rs_blockno); PageSetChecksumInplace(page, state->rs_blockno); + /* XXX - is this still needed? */ smgrextend(RelationGetSmgr(state->rs_new_rel), MAIN_FORKNUM, state->rs_blockno, (char *) page, true); diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index b52eca8f38..e60582e7e3 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -163,7 +163,10 @@ btbuildempty(Relation index) * XLOG_DBASE_CREATE* or XLOG_TBLSPC_CREATE record. Therefore, we need * this even when wal_level=minimal. */ + PageEncryptInplace(metapage, INIT_FORKNUM, RelationIsPermanent(index), + BTREE_METAPAGE); PageSetChecksumInplace(metapage, BTREE_METAPAGE); + smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, BTREE_METAPAGE, (char *) metapage, true); log_newpage(&RelationGetSmgr(index)->smgr_rlocator.locator, INIT_FORKNUM, diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index bd1685c441..0188202bff 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -661,13 +661,15 @@ _bt_blwritepage(BTWriteState *wstate, Page page, BlockNumber blkno) { if (!wstate->btws_zeropage) wstate->btws_zeropage = (Page) palloc0(BLCKSZ); - /* don't set checksum for all-zero page */ + /* don't set checksum or encryption for all-zero page */ smgrextend(RelationGetSmgr(wstate->index), MAIN_FORKNUM, wstate->btws_pages_written++, (char *) wstate->btws_zeropage, true); } + PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(wstate->index), + blkno); PageSetChecksumInplace(page, blkno); /* diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index c6821b5952..b36bc41854 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -168,6 +168,8 @@ spgbuildempty(Relation index) * of their existing content when the corresponding create records are * replayed. */ + PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index), + SPGIST_METAPAGE_BLKNO); PageSetChecksumInplace(page, SPGIST_METAPAGE_BLKNO); smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, SPGIST_METAPAGE_BLKNO, (char *) page, true); @@ -177,6 +179,8 @@ spgbuildempty(Relation index) /* Likewise for the root page. */ SpGistInitPage(page, SPGIST_LEAF); + PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index), + SPGIST_ROOT_BLKNO); PageSetChecksumInplace(page, SPGIST_ROOT_BLKNO); smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, SPGIST_ROOT_BLKNO, (char *) page, true); @@ -186,6 +190,8 @@ spgbuildempty(Relation index) /* Likewise for the null-tuples root page. */ SpGistInitPage(page, SPGIST_LEAF | SPGIST_NULLS); + PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index), + SPGIST_NULL_BLKNO); PageSetChecksumInplace(page, SPGIST_NULL_BLKNO); smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, SPGIST_NULL_BLKNO, (char *) page, true); diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 43429647b5..9347e63205 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -71,6 +71,7 @@ #include "common/file_utils.h" #include "crypto/kmgr.h" #include "executor/instrument.h" +#include "crypto/bufenc.h" #include "miscadmin.h" #include "pg_trace.h" #include "pgstat.h" @@ -4832,6 +4833,7 @@ BootStrapXLOG(void) WriteControlFile(); BootStrapKmgr(); + InitializeBufferEncryption(); if (terminal_fd != -1) { diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 9fc09b34de..7bdd37b96d 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -29,6 +29,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/link-canary.h" +#include "crypto/bufenc.h" #include "libpq/pqsignal.h" #include "miscadmin.h" #include "nodes/makefuncs.h" diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c index d708af19ed..925195d5a3 100644 --- a/src/backend/catalog/storage.c +++ b/src/backend/catalog/storage.c @@ -486,8 +486,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst, smgrread(src, forkNum, blkno, buf.data); - if (!PageIsVerifiedExtended(page, blkno, - PIV_LOG_WARNING | PIV_REPORT_STAT)) + if (!PageIsVerifiedExtended(page, forkNum, + relpersistence == RELPERSISTENCE_PERMANENT, + blkno, PIV_LOG_WARNING | PIV_REPORT_STAT)) { /* * For paranoia's sake, capture the file path before invoking the @@ -514,6 +515,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst, if (use_wal) log_newpage(&dst->smgr_rlocator.locator, forkNum, blkno, page, false); + PageEncryptInplace(page, forkNum, + relpersistence == RELPERSISTENCE_PERMANENT, blkno); PageSetChecksumInplace(page, blkno); /* diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile index c27362029d..8985a66875 100644 --- a/src/backend/crypto/Makefile +++ b/src/backend/crypto/Makefile @@ -13,6 +13,7 @@ top_builddir = ../../.. include $(top_builddir)/src/Makefile.global OBJS = \ + bufenc.o \ kmgr.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/crypto/bufenc.c b/src/backend/crypto/bufenc.c new file mode 100644 index 0000000000..0213fb23f5 --- /dev/null +++ b/src/backend/crypto/bufenc.c @@ -0,0 +1,176 @@ +/*------------------------------------------------------------------------- + * + * bufenc.c + * + * Copyright (c) 2020, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/crypto/bufenc.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "miscadmin.h" +#include "lib/stringinfo.h" + +#include "access/gist.h" +#include "access/xlog.h" +#include "crypto/bufenc.h" +#include "storage/bufpage.h" +#include "storage/fd.h" + +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + +/* + * We use the page LSN, page number, and permanent-bit to indicate if a fake + * LSN was used to create a nonce for each page. + */ +#define BUFENC_IV_SIZE 16 + +static unsigned char buf_encryption_iv[BUFENC_IV_SIZE]; + +PgCipherCtx *BufEncCtx = NULL; +PgCipherCtx *BufDecCtx = NULL; + +static void set_buffer_encryption_iv(Page page, BlockNumber blkno, + bool relation_is_permanent); + +void +InitializeBufferEncryption(void) +{ + const CryptoKey *key; + + if (!FileEncryptionEnabled) + return; + + key = KmgrGetKey(KMGR_KEY_ID_REL); + + BufEncCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR, + (unsigned char *) key->key, + (key->klen), true); + if (!BufEncCtx) + elog(ERROR, "cannot intialize encryption context"); + + BufDecCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR, + (unsigned char *) key->key, + (key->klen), false); + if (!BufDecCtx) + elog(ERROR, "cannot intialize decryption context"); +} + +/* Encrypt the given page with the relation key */ +void +EncryptPage(Page page, bool relation_is_permanent, BlockNumber blkno) +{ + unsigned char *ptr = (unsigned char *) page + PageEncryptOffset; + bool is_gist_page_or_similar; + + int enclen; + + Assert(BufEncCtx != NULL); + + /* + * Permanent pages have valid LSNs, and non-permanent pages usually have + * invalid (not set) LSNs. (One exception are GiST fake LSNs, see below.) + * However, we need valid ones on all pages for encryption. There are too + * many places that set the page LSN for permanent pages to do the same + * for non-permanent pages, so we just set it here. + * + * Also, while permanent relations get new LSNs every time the page is + * modified, for non-permanent relations do not, so we just update the LSN + * here before it is encrypted. + * + * GiST indexes uses LSNs, which are also stored in NSN fields, to detect + * page splits. Therefore, we allow the GiST code to assign LSNs and we + * don't change them here. + */ + + /* Permanent relations should already have valid LSNs. */ + Assert(!XLogRecPtrIsInvalid(PageGetLSN(page)) || !relation_is_permanent); + + /* + * Check if the page has a special size == GISTPageOpaqueData, a valid + * GIST_PAGE_ID, no invalid GiST flag bits are set, and a valid LSN. This + * is true for all GiST pages, and perhaps a few pages that are not. The + * only downside of guessing wrong is that we might not update the LSN for + * some non-permanent relation page changes, and therefore reuse the IV, + * which seems acceptable. + */ + is_gist_page_or_similar = + (PageGetSpecialSize(page) == MAXALIGN(sizeof(GISTPageOpaqueData)) && + GistPageGetOpaque(page)->gist_page_id == GIST_PAGE_ID && + (GistPageGetOpaque(page)->flags & ~GIST_FLAG_BITMASK) == 0 && + !XLogRecPtrIsInvalid(PageGetLSN(page))); + + if (!relation_is_permanent && !is_gist_page_or_similar) + PageSetLSN(page, LSNForEncryption(relation_is_permanent)); + + set_buffer_encryption_iv(page, blkno, relation_is_permanent); + if (unlikely(!pg_cipher_encrypt(BufEncCtx, PG_CIPHER_AES_CTR, + (const unsigned char *) ptr, /* input */ + SizeOfPageEncryption, + ptr, /* length */ + &enclen, /* resulting length */ + buf_encryption_iv, /* iv */ + BUFENC_IV_SIZE, + NULL, 0))) + elog(ERROR, "cannot encrypt page %u", blkno); + + Assert(enclen == SizeOfPageEncryption); +} + +/* Decrypt the given page with the relation key */ +void +DecryptPage(Page page, bool relation_is_permanent, BlockNumber blkno) +{ + unsigned char *ptr = (unsigned char *) page + PageEncryptOffset; + int enclen; + + Assert(BufDecCtx != NULL); + + set_buffer_encryption_iv(page, blkno, relation_is_permanent); + if (unlikely(!pg_cipher_decrypt(BufDecCtx, PG_CIPHER_AES_CTR, + (const unsigned char *) ptr, /* input */ + SizeOfPageEncryption, + ptr, /* output */ + &enclen, /* resulting length */ + buf_encryption_iv, /* iv */ + BUFENC_IV_SIZE, + NULL, 0))) + elog(ERROR, "cannot decrypt page %u", blkno); + + Assert(enclen == SizeOfPageEncryption); +} + +/* Construct iv for the given page */ +static void +set_buffer_encryption_iv(Page page, BlockNumber blkno, + bool relation_is_permanent) +{ + unsigned char *p = buf_encryption_iv; + + MemSet(buf_encryption_iv, 0, BUFENC_IV_SIZE); + + /* page lsn (8 byte) */ + memcpy(p, &((PageHeader) page)->pd_lsn, sizeof(PageXLogRecPtr)); + p += sizeof(PageXLogRecPtr); + + /* block number (4 byte) */ + memcpy(p, &blkno, sizeof(BlockNumber)); + p += sizeof(BlockNumber); + + /* + * Mark use of fake LSNs in IV so if the real and fake LSN counters + * overlap, the IV will remain unique. XXX Is there a better value? + */ + if (!relation_is_permanent) + *p++ = 0x80; + + /* + * The maximum required counter for AES-CTR is 2048, which fits in the + * last three bytes. + */ +} diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index aec9b094bd..491258a7ee 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -98,6 +98,7 @@ #include "common/ip.h" #include "common/pg_prng.h" #include "common/string.h" +#include "crypto/bufenc.h" #include "crypto/kmgr.h" #include "lib/ilist.h" #include "libpq/auth.h" @@ -1354,6 +1355,7 @@ PostmasterMain(int argc, char *argv[]) } InitializeKmgr(); + InitializeBufferEncryption(); if (terminal_fd != -1) close(terminal_fd); diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 5ec3e405ab..36e74b2813 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -39,6 +39,7 @@ #include "catalog/catalog.h" #include "catalog/storage.h" #include "catalog/storage_xlog.h" +#include "crypto/bufenc.h" #include "executor/instrument.h" #include "lib/binaryheap.h" #include "miscadmin.h" @@ -1029,7 +1030,9 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, } /* check for garbage data */ - if (!PageIsVerifiedExtended((Page) bufBlock, blockNum, + if (!PageIsVerifiedExtended((Page) bufBlock, forkNum, + relpersistence == RELPERSISTENCE_PERMANENT, + blockNum, PIV_LOG_WARNING | PIV_REPORT_STAT)) { if (mode == RBM_ZERO_ON_ERROR || zero_damaged_pages) @@ -2893,12 +2896,24 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln) */ bufBlock = BufHdrGetBlock(buf); + if (FileEncryptionEnabled) + { + /* + * Technically BM_PERMANENT could indicate an init fork, but that's + * okay since forkNum would also tell us not to encrypt init forks. + */ + bufToWrite = PageEncryptCopy((Page) bufBlock, buf->tag.forkNum, + buf_state & BM_PERMANENT, buf->tag.blockNum); + bufToWrite = PageSetChecksumCopy((Page) bufToWrite, buf->tag.blockNum); + } + else + bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum); + /* * Update page checksum if desired. Since we have only shared lock on the * buffer, other processes might be updating hint bits in it, so we must * copy the page to private storage if we do checksumming. */ - bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum); if (track_io_timing) INSTR_TIME_SET_CURRENT(io_start); @@ -3543,6 +3558,9 @@ FlushRelationBuffers(Relation rel) errcallback.previous = error_context_stack; error_context_stack = &errcallback; + /* XXX should we be writing a copy of the page here? */ + PageEncryptInplace(localpage, bufHdr->tag.forkNum, + RelationIsPermanent(rel), bufHdr->tag.blockNum); PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); smgrwrite(RelationGetSmgr(rel), diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 30d67d1c40..09b3ec71d9 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -159,7 +159,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } return bufHdr; } - + #ifdef LBDEBUG fprintf(stderr, "LB ALLOC (%u,%d,%d) %d\n", smgr->smgr_rlocator.locator.relNumber, forkNum, blockNum, @@ -217,6 +217,12 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, /* Find smgr relation for buffer */ oreln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag), MyBackendId); + /* + * Technically BM_PERMANENT could indicate an init fork, but that's + * okay since forkNum would also tell us not to encrypt init forks. + */ + PageEncryptInplace(localpage, bufHdr->tag.forkNum, + buf_state & BM_PERMANENT, bufHdr->tag.blockNum); PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); /* And write... */ diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c index 8a866191e1..d2f4d13ec0 100644 --- a/src/backend/storage/file/copydir.c +++ b/src/backend/storage/file/copydir.c @@ -22,12 +22,15 @@ #include <unistd.h> #include <sys/stat.h> +#include "crypto/bufenc.h" #include "common/file_utils.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/copydir.h" #include "storage/fd.h" +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + /* * copydir: copy a directory * @@ -72,7 +75,7 @@ copydir(char *fromdir, char *todir, bool recurse) copydir(fromfile, tofile, true); } else if (xlde_type == PGFILETYPE_REG) - copy_file(fromfile, tofile); + copy_file(fromfile, tofile, false); } FreeDir(xldir); @@ -115,7 +118,7 @@ copydir(char *fromdir, char *todir, bool recurse) * copy one file */ void -copy_file(char *fromfile, char *tofile) +copy_file(char *fromfile, char *tofile, bool encrypt_init_file) { char *buffer; int srcfd; @@ -123,9 +126,8 @@ copy_file(char *fromfile, char *tofile) int nbytes; off_t offset; off_t flush_offset; - /* Size of copy buffer (read and write requests) */ -#define COPY_BUF_SIZE (8 * BLCKSZ) + int copy_buf_size = (encrypt_init_file) ? BLCKSZ : 8 * BLCKSZ; /* * Size of data flush requests. It seems beneficial on most platforms to @@ -140,7 +142,7 @@ copy_file(char *fromfile, char *tofile) #endif /* Use palloc to ensure we get a maxaligned buffer */ - buffer = palloc(COPY_BUF_SIZE); + buffer = palloc(copy_buf_size); /* * Open the files @@ -178,7 +180,7 @@ copy_file(char *fromfile, char *tofile) } pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ); - nbytes = read(srcfd, buffer, COPY_BUF_SIZE); + nbytes = read(srcfd, buffer, copy_buf_size); pgstat_report_wait_end(); if (nbytes < 0) ereport(ERROR, @@ -186,6 +188,20 @@ copy_file(char *fromfile, char *tofile) errmsg("could not read file \"%s\": %m", fromfile))); if (nbytes == 0) break; + /* + * When we copy an init fork page to be part of an empty unlogged + * relation, its real LSN must be replaced with a fake one, and the + * page encrypted. + */ + if (encrypt_init_file) + { + Page page = (Page) buffer; + + Assert(nbytes == BLCKSZ); + PageSetLSN(page, LSNForEncryption(false)); + PageEncryptInplace(page, MAIN_FORKNUM, false, offset / BLCKSZ); + } + errno = 0; pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_WRITE); if ((int) write(dstfd, buffer, nbytes) != nbytes) diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c index 647c458b52..2e57943768 100644 --- a/src/backend/storage/file/reinit.c +++ b/src/backend/storage/file/reinit.c @@ -312,7 +312,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) /* OK, we're ready to perform the actual copy. */ elog(DEBUG2, "copying %s to %s", srcpath, dstpath); - copy_file(srcpath, dstpath); + copy_file(srcpath, dstpath, true); } FreeDir(dbspace_dir); diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c index 8b617c7e79..1255566f57 100644 --- a/src/backend/storage/page/bufpage.c +++ b/src/backend/storage/page/bufpage.c @@ -17,6 +17,7 @@ #include "access/htup_details.h" #include "access/itup.h" #include "access/xlog.h" +#include "crypto/bufenc.h" #include "pgstat.h" #include "storage/checksum.h" #include "utils/memdebug.h" @@ -85,7 +86,8 @@ PageInit(Page page, Size pageSize, Size specialSize) * to pgstat. */ bool -PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags) +PageIsVerifiedExtended(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno, int flags) { PageHeader p = (PageHeader) page; size_t *pagebytes; @@ -108,6 +110,8 @@ PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags) checksum_failure = true; } + PageDecryptInplace(page, forknum, relation_is_permanent, blkno); + /* * The following checks don't prove the header is correct, only that * it looks sane enough to allow into the buffer pool. Later usage of @@ -1544,3 +1548,48 @@ PageSetChecksumInplace(Page page, BlockNumber blkno) ((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blkno); } + +char * +PageEncryptCopy(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno) +{ + static char *pageCopy = NULL; + + /* If we don't need a checksum, just return the passed-in data */ + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum)) + return (char *) page; + + /* + * We allocate the copy space once and use it over on each subsequent + * call. The point of palloc'ing here, rather than having a static char + * array, is first to ensure adequate alignment for the checksumming code + * and second to avoid wasting space in processes that never call this. + */ + if (pageCopy == NULL) + pageCopy = MemoryContextAlloc(TopMemoryContext, BLCKSZ); + + memcpy(pageCopy, (char *) page, BLCKSZ); + EncryptPage(pageCopy, relation_is_permanent, blkno); + return pageCopy; +} + +void +PageEncryptInplace(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno) +{ + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum)) + return; + + EncryptPage(page, relation_is_permanent, blkno); +} + + +void +PageDecryptInplace(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno) +{ + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum)) + return; + + DecryptPage(page, relation_is_permanent, blkno); +} diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 5970ec97a5..9ac4a737fa 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -35,6 +35,7 @@ #include "commands/async.h" #include "commands/prepare.h" #include "common/pg_prng.h" +#include "crypto/bufenc.h" #include "crypto/kmgr.h" #include "jit/jit.h" #include "libpq/libpq.h" @@ -4076,6 +4077,7 @@ PostgresMain(const char *dbname, const char *username) if (!IsUnderPostmaster) { InitializeKmgr(); + InitializeBufferEncryption(); if (terminal_fd != -1) close(terminal_fd); diff --git a/src/include/access/gist.h b/src/include/access/gist.h index a3337627b8..3e116a66d8 100644 --- a/src/include/access/gist.h +++ b/src/include/access/gist.h @@ -41,7 +41,7 @@ #define GISTNProcs 11 /* - * Page opaque data in a GiST index page. + * Page opaque data flags in a GiST index page. */ #define F_LEAF (1 << 0) /* leaf page */ #define F_DELETED (1 << 1) /* the page has been deleted */ @@ -51,6 +51,9 @@ #define F_HAS_GARBAGE (1 << 4) /* some tuples on the page are dead, * but not deleted yet */ +/* Specifies the bits that can be set in the GiST flags field */ +#define GIST_FLAG_BITMASK 0x1F + /* * NSN (node sequence number) is a special-purpose LSN which is stored on each * index page in GISTPageOpaqueData and updated only during page splits. By diff --git a/src/include/crypto/bufenc.h b/src/include/crypto/bufenc.h new file mode 100644 index 0000000000..cc86ceb821 --- /dev/null +++ b/src/include/crypto/bufenc.h @@ -0,0 +1,28 @@ +/*------------------------------------------------------------------------- + * + * bufenc.h + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/crypto/bufenc.h + * + *------------------------------------------------------------------------- + */ +#ifndef BUFENC_H +#define BUFENC_H + +#include "storage/bufmgr.h" +#include "crypto/kmgr.h" + +/* Cluster encryption encrypts only main forks */ +#define PageNeedsToBeEncrypted(forknum) \ + (FileEncryptionEnabled && (forknum) == MAIN_FORKNUM) + + +extern void InitializeBufferEncryption(void); +extern void EncryptPage(Page page, bool relation_is_permanent, + BlockNumber blkno); +extern void DecryptPage(Page page, bool relation_is_permanent, + BlockNumber blkno); + +#endif /* BUFENC_H */ diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h index 2708c4b683..d06c88d98c 100644 --- a/src/include/storage/bufpage.h +++ b/src/include/storage/bufpage.h @@ -15,6 +15,7 @@ #define BUFPAGE_H #include "access/xlogdefs.h" +#include "common/relpath.h" #include "storage/block.h" #include "storage/item.h" #include "storage/off.h" @@ -168,6 +169,8 @@ typedef struct PageHeaderData } PageHeaderData; typedef PageHeaderData *PageHeader; +#define PageEncryptOffset offsetof(PageHeaderData, pd_special) +#define SizeOfPageEncryption (BLCKSZ - PageEncryptOffset) /* * pd_flags contains the following flag bits. Undefined bits are initialized @@ -471,8 +474,8 @@ do { \ ((overwrite) ? PAI_OVERWRITE : 0) | \ ((is_heap) ? PAI_IS_HEAP : 0)) -#define PageIsVerified(page, blkno) \ - PageIsVerifiedExtended(page, blkno, \ +#define PageIsVerified(page, relation_is_permanent, blkno) \ + PageIsVerifiedExtended(page, MAIN_FORKNUM, relation_is_permanent, blkno, \ PIV_LOG_WARNING | PIV_REPORT_STAT) /* @@ -486,7 +489,10 @@ StaticAssertDecl(BLCKSZ == ((BLCKSZ / sizeof(size_t)) * sizeof(size_t)), "BLCKSZ has to be a multiple of sizeof(size_t)"); extern void PageInit(Page page, Size pageSize, Size specialSize); -extern bool PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags); +extern bool PageIsVerifiedExtended(Page page, ForkNumber forknum, + bool relation_is_permanent, + BlockNumber blkno, + int flags); extern OffsetNumber PageAddItemExtended(Page page, Item item, Size size, OffsetNumber offsetNumber, int flags); extern Page PageGetTempPage(Page page); @@ -506,5 +512,11 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum, Item newtup, Size newsize); extern char *PageSetChecksumCopy(Page page, BlockNumber blkno); extern void PageSetChecksumInplace(Page page, BlockNumber blkno); +extern char *PageEncryptCopy(Page page, ForkNumber forknum, + bool relation_is_permanent, BlockNumber blkno); +extern void PageEncryptInplace(Page page, ForkNumber forknum, + bool relation_is_permanent, BlockNumber blkno); +extern void PageDecryptInplace(Page page, ForkNumber forknum, + bool relation_is_permanent, BlockNumber blkno); #endif /* BUFPAGE_H */ diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h index 50a26edeb0..7fdfb71067 100644 --- a/src/include/storage/copydir.h +++ b/src/include/storage/copydir.h @@ -14,6 +14,6 @@ #define COPYDIR_H extern void copydir(char *fromdir, char *todir, bool recurse); -extern void copy_file(char *fromfile, char *tofile); +extern void copy_file(char *fromfile, char *tofile, bool encrypt_init_file); #endif /* COPYDIR_H */ -- 2.37.0 (Apple Git-136) ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> @ 2022-11-03 14:09 ` Aleksander Alekseev <[email protected]> 2022-11-03 22:06 ` Re: Moving forward with TDE David Christensen <[email protected]> 1 sibling, 1 reply; 9+ messages in thread From: Aleksander Alekseev @ 2022-11-03 14:09 UTC (permalink / raw) To: David Christensen <[email protected]>; +Cc: pgsql-hackers; Stephen Frost <[email protected]> Hi David, > Working with Stephen, I am attempting to pick up some of the work that > was left off with TDE and the key management infrastructure. I have > rebased Bruce's KMS/TDE patches as they existed on the > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption wiki > page, which are enclosed in this email. I'm happy to see that the TDE effort was picked up. > I would love to open a discussion about how to move forward and get > some of these features built out. The historical threads here are > quite long and complicated; is there a "current state" other than the > wiki that reflects the general thinking on this feature? Any major > developments in direction that would not be reflected in the code from > May 2021? The patches seem to be well documented and decomposed in small pieces. That's good. Unless somebody in the community remembers open questions/issues with TDE that were never addressed I suggest simply iterating with our usual testing/reviewing process. For now I'm going to change the status of the CF entry [1] to "Waiting for Author" since the patchset doesn't pass the CI [2]. One limitation of the design described on the wiki I see is that it seems to heavily rely on AES: > We will use Advanced Encryption Standard (AES) [4]. We will offer three key length options (128, 192, and 256-bits) selected at initdb time with --file-encryption-method (there doesn't seem to be any mention of the hash/MAC algorithms, that's odd). In the future we should be able to add the support of alternative algorithms. The reason is that the algorithms can become weak every 20 years or so, and the preferred algorithms may also depend on the region. This should NOT be implemented in this particular patchset, but the design shouldn't prevent from implementing this in the future. [1]: https://commitfest.postgresql.org/40/3985/ [2]: http://cfbot.cputube.org/ -- Best regards, Aleksander Alekseev ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-03 14:09 ` Re: Moving forward with TDE Aleksander Alekseev <[email protected]> @ 2022-11-03 22:06 ` David Christensen <[email protected]> 2022-11-04 08:42 ` Re: Moving forward with TDE Dilip Kumar <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: David Christensen @ 2022-11-03 22:06 UTC (permalink / raw) To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Stephen Frost <[email protected]> > Unless somebody in the community remembers open questions/issues with > TDE that were never addressed I suggest simply iterating with our > usual testing/reviewing process. For now I'm going to change the > status of the CF entry [1] to "Waiting for Author" since the patchset > doesn't pass the CI [2]. Thanks, enclosed is a new version that is rebased on HEAD and fixes a bug that the new pg_control_init() test picked up. Known issues (just discovered by me in testing the latest revision) is that databases created from `template0` are not decrypting properly, but `template1` works fine, so going to dig in on that soon. > One limitation of the design described on the wiki I see is that it > seems to heavily rely on AES: > > > We will use Advanced Encryption Standard (AES) [4]. We will offer three key length options (128, 192, and 256-bits) selected at initdb time with --file-encryption-method > > (there doesn't seem to be any mention of the hash/MAC algorithms, > that's odd). In the future we should be able to add the support of > alternative algorithms. The reason is that the algorithms can become > weak every 20 years or so, and the preferred algorithms may also > depend on the region. This should NOT be implemented in this > particular patchset, but the design shouldn't prevent from > implementing this in the future. Yes, we definitely are considering multiple algorithms support as part of this effort. Best, David Attachments: [application/octet-stream] v2-0001-cfe-01-doc_over_master-squash-commit.patch (10.1K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/2-v2-0001-cfe-01-doc_over_master-squash-commit.patch) download | inline diff: From 6f4cb27162b8b719e1f00d9d4bec77c72b629d3e Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:43 -0400 Subject: [PATCH v2 01/12] cfe-01-doc_over_master squash commit --- doc/src/sgml/database-encryption.sgml | 149 ++++++++++++++++++++++++++ doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/installation.sgml | 2 +- doc/src/sgml/monitoring.sgml | 13 +++ doc/src/sgml/postgres.sgml | 1 + 5 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 doc/src/sgml/database-encryption.sgml diff --git a/doc/src/sgml/database-encryption.sgml b/doc/src/sgml/database-encryption.sgml new file mode 100644 index 0000000000..c4377ada64 --- /dev/null +++ b/doc/src/sgml/database-encryption.sgml @@ -0,0 +1,149 @@ +<!-- doc/src/sgml/database-encryption.sgml --> + +<chapter id="cluster-file-encryption"> + <title>Cluster File Encryption</title> + + <indexterm zone="cluster-file-encryption"> + <primary>Cluster File Encryption</primary> + </indexterm> + + <para> + The purpose of cluster file encryption is to prevent users with read + access on the directories used to store database files and write-ahead + log files from being able to access the data stored in those files. + For example, when using cluster file encryption, users who have read + access to the cluster directories for backup purposes will not be able + to decrypt the data stored in these files. Read-only access for a group + of users can be enabled using the <application>initdb</application> + <option>--allow-group-access</option> option. Cluster file encryption + also provides data-at-rest security, protecting users from data loss + should the physical storage media be stolen or improperly erased before + disposal. + </para> + + <para> + Cluster file encryption does not protect against unauthorized file + system writes. Such writes can allow data decryption if used to weaken + the system's security and the weakened system is later supplied with + the externally-stored cluster encryption key. This also does not always + detect if users with write access remove or modify database files. + </para> + + <para> + This also does not protect against users who have read access to database + process memory because all in-memory data pages and data encryption keys + are stored unencrypted in memory. Therefore, an attacker who is able + to read memory can read the data encryption keys and decrypt the entire + cluster. The Postgres operating system user and the operating system + administrator, e.g., the <literal>root</literal> user, have such access. + </para> + + <sect1 id="cluster-encryption-keys"> + <title>Keys</title> + + <para> + Cluster file encryption uses two levels of encryption — an upper + key which encrypts lower-level keys. The upper-level key is often + referred to as a Key Encryption Key (<acronym>KEK</acronym>). This key + is <emphasis>not</emphasis> stored in the file system, but provided at + <command>initdb</command> time and each time the server is started. This + key can be easily changed via <command>pg_alterckey</command> without + requiring any changes to the the data files or <command>WAL</command> + files. + </para> + + <para> + The lower level keys are data encryption keys, specifically for relations + and <acronym>WAL</acronym>. The relation key is used to encrypt database + heap and index files. The WAL key is used to encrypt write-ahead log + (WAL) files. Two different keys are used so that primary and standby + servers can use different relation keys, but the same WAL key, so that + these keys can (in a future release) be rotated by switching the + primary to the standby and then changing the WAL key. Eventually, + encryption will be able to added to non-encrypted clusters by creating + encrypted replicas and switching over to them. + </para> + + <para> + Postgres stores the data encryption (lower-level) keys in the data + directory encrypted (wrapped) by key encryption (upper-level) key. + Though the data encryption keys technically exist in the file system, + the key encryption key does not, so the data encryption keys are + securely stored. Data encryption keys are used to security encrypt + other database files. + </para> + </sect1> + + <sect1 id="cluster-encryption-initialization"> + <title>Initialization</title> + + <para> + Cluster file encryption is enabled when + <productname>PostgreSQL</productname> is built + with <literal>--with-openssl</literal> and <xref + linkend="app-initdb-cluster-key-command"/> is specified + during <command>initdb</command>. The cluster key + provided by the <option>--cluster-key-command</option> + option during <command>initdb</command> and the one generated + by <xref linkend="guc-cluster-key-command"/> in the + <filename>postgresql.conf</filename> must match for the database + cluster to start. Note that the cluster key command + passed to <command>initdb</command> must return a key of + 64 hexadecimal characters. For example: +<programlisting> +initdb -D dbname --cluster-key-command='ckey_passphrase.sh' +</programlisting> + Cluster file encryption does not support a <varname>wal_level</varname> + of <literal>minimal</literal>. + </para> + </sect1> + + <sect1 id="cluster-encryption-operation"> + <title>Operation</title> + + <para> + During the <command>initdb</command> process, if + <option>--cluster-key-command</option> is specified, two data-level + encryption keys are created. These two keys are then encrypted with + the key encryption key (KEK) supplied by the cluster key command before + being stored in the database directory. The key or passphrase that + derives the key must be supplied from the terminal or stored in a + trusted key store, such as key vault software or a hardware security + module. + </para> + + <para> + If the <productname>PostgreSQL</productname> server has + been initialized to require a cluster key, each time the + server starts the <filename>postgresql.conf</filename> + <varname>cluster_key_command</varname> command will be executed + and the cluster key retrieved. The data encryption keys in the + <filename>pg_cryptokeys</filename> directory will then be decrypted + using the supplied key and integrity-checked to ensure it matches the + initdb-supplied key. (If this check fails, the server will refuse + to start.) The cluster encryption key will then be removed from + system memory. The decrypted data encryption keys will remain in + shared memory until the server is stopped. + </para> + + <para> + The data encryption keys are randomly generated and can be 128, 192, + or 256-bits in length, depending on whether <literal>AES128</literal>, + <literal>AES192</literal>, or <literal>AES256</literal> is specified. + They are encrypted by the key encryption key (KEK) using Advanced + Encryption Standard (<acronym>AES256</acronym>) encryption in Key + Wrap Padded Mode, which also provides KEK authentication; see <ulink + url="https://tools.ietf.org/html/rfc5649">RFC 5649</ulink>. While + 128-bit encryption is sufficient for most sites, 256-bit encryption + is thought to be more immune to future quantum cryptographic attacks. + </para> + + <para>. + If you prefer to create the random keys on your own, you can create + a empty directory with a <filename>pg_cryptokeys/live</filename> + subdirectory, generate the keys there using your tools. and use the + <command>initdb</command> <option>--copy-encryption-keys</option> + to copy those keys into the newly-created cluster. + </para> + </sect1> +</chapter> diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index de450cd661..efeb542cf7 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -49,6 +49,7 @@ <!ENTITY wal SYSTEM "wal.sgml"> <!ENTITY logical-replication SYSTEM "logical-replication.sgml"> <!ENTITY jit SYSTEM "jit.sgml"> +<!ENTITY database-encryption SYSTEM "database-encryption.sgml"> <!-- programmer's guide --> <!ENTITY bgworker SYSTEM "bgworker.sgml"> diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml index 319c7e6966..b67d3b7ffb 100644 --- a/doc/src/sgml/installation.sgml +++ b/doc/src/sgml/installation.sgml @@ -1011,7 +1011,7 @@ build-postgresql: <listitem> <para> Build with support for <acronym>SSL</acronym> (encrypted) - connections. The only <replaceable>LIBRARY</replaceable> + connections and cluster file encryption. The only <replaceable>LIBRARY</replaceable> supported is <option>openssl</option>. This requires the <productname>OpenSSL</productname> package to be installed. <filename>configure</filename> will check for the required diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index e5d622d514..3e3e1adea6 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1340,6 +1340,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser <entry><literal>DataFileWrite</literal></entry> <entry>Waiting for a write to a relation data file.</entry> </row> + <row> + <entry><literal>KeyFileRead</literal></entry> + <entry>Waiting for a read of the wrapped data encryption keys.</entry> + </row> + <row> + <entry><literal>KeyFileWrite</literal></entry> + <entry>Waiting for a write of the wrapped data encryption keys.</entry> + </row> + <row> + <entry><literal>KeyFileSync</literal></entry> + <entry>Waiting for changes to the wrapped data encryption keys to reach + durable storage.</entry> + </row> <row> <entry><literal>LockFileAddToDataDirRead</literal></entry> <entry>Waiting for a read while adding a line to the data directory lock diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 73439c049e..d99b24605f 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -171,6 +171,7 @@ break is not needed in a wider output rendering. &wal; &logical-replication; &jit; + &database-encryption; ®ress; </part> -- 2.31.1 [application/octet-stream] v2-0005-cfe-05-crypto_over_cfe-04-common-squash-commit.patch (23.7K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/3-v2-0005-cfe-05-crypto_over_cfe-04-common-squash-commit.patch) download | inline diff: From 1a69fe5bd62a573427426e2305679562d72090a9 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:51 -0400 Subject: [PATCH v2 05/12] cfe-05-crypto_over_cfe-04-common squash commit --- doc/src/sgml/config.sgml | 120 ++++++++-- src/backend/crypto/Makefile | 18 ++ src/backend/crypto/kmgr.c | 438 ++++++++++++++++++++++++++++++++++++ src/common/kmgr_utils.c | 3 +- src/include/crypto/kmgr.h | 25 ++ 5 files changed, 588 insertions(+), 16 deletions(-) create mode 100644 src/backend/crypto/Makefile create mode 100644 src/backend/crypto/kmgr.c create mode 100644 src/include/crypto/kmgr.h diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 559eb898a9..4e95f7e22a 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1558,19 +1558,31 @@ include_dir 'conf.d' mechanism is used. </para> <para> - The command must print the passphrase to the standard output and exit - with code 0. In the parameter value, <literal>%p</literal> is - replaced by a prompt string. (Write <literal>%%</literal> for a - literal <literal>%</literal>.) Note that the prompt string will - probably contain whitespace, so be sure to quote adequately. A single - newline is stripped from the end of the output if present. + The command must print the passphrase to the standard output + and exit with code 0. It can prompt from the terminal if + <option>--authprompt</option> is used. In the parameter + value, <literal>%R</literal> is replaced by a file descriptor + number opened to the terminal that started the server. A file + descriptor is only available if enabled at server start via + <option>-R</option>. If <literal>%R</literal> is specified and + no file descriptor is available, the server will not start. Value + <literal>%p</literal> is replaced by a pre-defined prompt string. + (Write <literal>%%</literal> for a literal <literal>%</literal>.) + Note that the prompt string will probably contain whitespace, + so be sure to quote its use adequately. Newlines are stripped + from the end of the output if present. </para> + <para> - The command does not actually have to prompt the user for a - passphrase. It can read it from a file, obtain it from a keychain - facility, or similar. It is up to the user to make sure the chosen - mechanism is adequately secure. + Sample scripts can be found in + <filename>$SHAREDIR/auth_commands</filename>, + where <literal>$SHAREDIR</literal> means the + <productname>PostgreSQL</productname> installation's shared-data + directory, often <filename>/usr/local/share/postgresql</filename> + (use <command>pg_config --sharedir</command> to determine it if + you're not sure). </para> + <para> This parameter can only be set in the <filename>postgresql.conf</filename> file or on the server command line. @@ -1592,10 +1604,12 @@ include_dir 'conf.d' parameter is off (the default), then <varname>ssl_passphrase_command</varname> will be ignored during a reload and the SSL configuration will not be reloaded if a passphrase - is needed. That setting is appropriate for a command that requires a - TTY for prompting, which might not be available when the server is - running. Setting this parameter to on might be appropriate if the - passphrase is obtained from a file, for example. + is needed. This setting is appropriate for a command that requires a + terminal for prompting, which will likely not be available when the server is + running. (<option>--authprompt</option> closes the terminal file + descriptor soon after server start.) Setting this parameter on + might be appropriate, for example, if the passphrase is obtained + from a file. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> @@ -2794,7 +2808,9 @@ include_dir 'conf.d' <varname>max_wal_senders</varname> is non-zero. Note that changing <varname>wal_level</varname> to <literal>minimal</literal> makes previous base backups unusable - for point-in-time recovery and standby servers. + for point-in-time recovery and standby servers, which may + lead to data loss. Cluster file encryption also does not support + <varname>wal_level</varname> <literal>minimal</literal>. </para> <para> In <literal>logical</literal> level, the same information is logged as @@ -8483,6 +8499,64 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; </variablelist> </sect1> + <sect1 id="runtime-config-encryption"> + <title>Cluster File Encryption</title> + + <variablelist> + <varlistentry id="guc-cluster-key-command" xreflabel="cluster_key_command"> + <term><varname>cluster_key_command</varname> (<type>string</type>) + <indexterm> + <primary><varname>cluster_key_command</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + This option specifies an external command to obtain the cluster-level + key for cluster file encryption during server initialization and + server start. + </para> + <para> + The command must print the cluster key to the standard + output as 64 hexadecimal characters, and exit with code 0. + The command can prompt for the passphrase or PIN from the + terminal if <option>--authprompt</option> is used. In the + parameter value, <literal>%R</literal> is replaced by a file + descriptor number opened to the terminal that started the server. + A file descriptor is only available if enabled at server start + via <option>-R</option>. If <literal>%R</literal> is specified + and no file descriptor is available, the server will not start. + Value <literal>%p</literal> is replaced by a pre-defined + prompt string. Value <literal>%d</literal> is replaced by the + directory containing the keys; this is useful if the command + must create files with the keys, e.g., to store a cluster-level + key encrypted by a key stored in a hardware security module. + (Write <literal>%%</literal> for a literal <literal>%</literal>.) + Note that the prompt string will probably contain whitespace, + so be sure to quote its use adequately. Newlines are stripped + from the end of the output if present. + </para> + + <para> + Sample script can be found in + <filename>$SHAREDIR/auth_commands</filename>, + where <literal>$SHAREDIR</literal> means the + <productname>PostgreSQL</productname> installation's shared-data + directory, often <filename>/usr/local/share/postgresql</filename> + (use <command>pg_config --sharedir</command> to determine it if + you're not sure). + </para> + + <para> + This parameter can only be set by + <application>initdb</application>, in the + <filename>postgresql.conf</filename> file, or on the server + command line. + </para> + </listitem> + </varlistentry> + </variablelist> + </sect1> + <sect1 id="runtime-config-client"> <title>Client Connection Defaults</title> @@ -10508,6 +10582,22 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' </listitem> </varlistentry> + <varlistentry id="guc-file-encryption-method" xreflabel="file_encryption_method"> + <term><varname>file_encryption_method</varname> (<type>boolean</type>) + <indexterm> + <primary>Cluster file encryption method</primary> + </indexterm> + </term> + <listitem> + <para> + Reports the cluster file + encryption method. See <xref + linkend="app-initdb-cluster-key-command"/> for more + information. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-data-directory-mode" xreflabel="data_directory_mode"> <term><varname>data_directory_mode</varname> (<type>integer</type>) <indexterm> diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile new file mode 100644 index 0000000000..c27362029d --- /dev/null +++ b/src/backend/crypto/Makefile @@ -0,0 +1,18 @@ +#------------------------------------------------------------------------- +# +# Makefile +# Makefile for src/backend/crypto +# +# IDENTIFICATION +# src/backend/crypto/Makefile +# +#------------------------------------------------------------------------- + +subdir = src/backend/crypto +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = \ + kmgr.o + +include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c new file mode 100644 index 0000000000..4b74961323 --- /dev/null +++ b/src/backend/crypto/kmgr.c @@ -0,0 +1,438 @@ +/*------------------------------------------------------------------------- + * + * kmgr.c + * Cluster file encryption routines + * + * Cluster file encryption is enabled if user requests it during initdb. + * During bootstrap, we generate data encryption keys, wrap them with the + * cluster-level key, and store them into each file located at KMGR_DIR. + * During startup, we decrypt all internal keys and load them to the shared + * memory. Internal keys in the shared memory are read-only. The wrapping + * and unwrapping key routines require the OpenSSL library. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/crypto/kmgr.c + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include <sys/stat.h> +#include <unistd.h> + +#include "funcapi.h" +#include "miscadmin.h" +#include "pgstat.h" + +#include "access/xlog.h" +#include "common/file_perm.h" +#include "common/kmgr_utils.h" +#include "common/sha2.h" +#include "access/xlog.h" +#include "common/controldata_utils.h" +#include "crypto/kmgr.h" +#include "postmaster/postmaster.h" +#include "storage/copydir.h" +#include "storage/fd.h" +#include "storage/ipc.h" +#include "storage/shmem.h" +#include "utils/builtins.h" +#include "utils/memutils.h" +/* Struct stores file encryption keys in plaintext format */ +typedef struct KmgrShmemData +{ + CryptoKey intlKeys[KMGR_NUM_DATA_KEYS]; +} KmgrShmemData; +static KmgrShmemData *KmgrShmem; + +/* GUC variables */ +char *cluster_key_command = NULL; + +CryptoKey bootstrap_keys[KMGR_NUM_DATA_KEYS]; + +extern char *bootstrap_old_key_datadir; +extern int bootstrap_file_encryption_method; + +static void bzeroKmgrKeys(int status, Datum arg); +static void KmgrWriteCryptoKeys(const char *dir, unsigned char **keys, int *key_lens); +static CryptoKey *generate_crypto_key(int len); + +/* + * This function must be called ONCE during initdb. It creates the DEK + * files wrapped with the KEK supplied by kmgr_run_cluster_key_command(). + * There is also an option for the keys to be copied from another cluster. + */ +void +BootStrapKmgr(void) +{ + char live_path[MAXPGPATH]; + unsigned char *keys_wrap[KMGR_NUM_DATA_KEYS]; + int key_lens[KMGR_NUM_DATA_KEYS]; + char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN]; + int cluster_key_hex_len; + unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN]; + + if (!FileEncryptionEnabled) + return; + +#ifndef USE_OPENSSL + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"), + errhint("Compile with --with-openssl to use this feature.")))); +#endif + + /* + * There are too many optimizations for wal_level=minimal that don't set + * LSNs for permanent tables, so just disallow it. + */ + if (!XLogIsNeeded()) + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported with a wal_level of \"minimal\"")))); + + snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR); + + /* + * Copy cluster file encryption keys from an old cluster? This is useful + * for pg_upgrade upgrades where the copied database files are already + * encrypted using the old cluster's DEK keys. + */ + if (bootstrap_old_key_datadir != NULL) + { + char old_key_dir[MAXPGPATH]; + + snprintf(old_key_dir, sizeof(old_key_dir), "%s/%s", + bootstrap_old_key_datadir, LIVE_KMGR_DIR); + copydir(old_key_dir, LIVE_KMGR_DIR, true); + } + /* create an empty directory */ + else + { + if (mkdir(LIVE_KMGR_DIR, pg_dir_create_mode) < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not create cluster file encryption directory \"%s\": %m", + LIVE_KMGR_DIR))); + } + + /* + * Get key encryption key (KEK) from the cluster_key command. The cluster + * key command might need to check for the existence of files in the live + * directory, e.g., PIV, so run this _after_ copying the directory in + * place. + */ + cluster_key_hex_len = kmgr_run_cluster_key_command(cluster_key_command, + cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + live_path, terminal_fd); + + /* decode supplied hex */ + if (hex_decode(cluster_key_hex, cluster_key_hex_len, + (char *) cluster_key) != + KMGR_CLUSTER_KEY_LEN) + ereport(ERROR, + (errmsg("cluster key must be %d hexadecimal characters", + KMGR_CLUSTER_KEY_LEN * 2))); + + /* We are not in copy mode? Generate new cluster file encryption keys. */ + if (bootstrap_old_key_datadir == NULL) + { + unsigned char *bootstrap_keys_wrap[KMGR_NUM_DATA_KEYS]; + int key_lens[KMGR_NUM_DATA_KEYS]; + PgCipherCtx *cluster_key_ctx; + + /* Create KEK encryption context */ + cluster_key_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, cluster_key, + KMGR_CLUSTER_KEY_LEN, true); + if (!cluster_key_ctx) + elog(ERROR, "could not initialize encryption context"); + + /* Wrap data encryption keys (DEK) using the key encryption key (KEK) */ + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + { + CryptoKey *key; + + /* generate a DEK */ + key = generate_crypto_key( + encryption_methods[bootstrap_file_encryption_method].bit_length / 8); + + /* output generated random string as hex, for testing */ + { + char str[MAXPGPATH]; + int out_len; + + out_len = hex_encode((char *) (key->key), key->klen, + str); + str[out_len] = '\0'; + } + + bootstrap_keys_wrap[id] = palloc0(KMGR_MAX_KEY_LEN_BYTES + + pg_cipher_blocksize(cluster_key_ctx)); + + /* wrap DEK with KEK */ + if (!kmgr_wrap_data_key(cluster_key_ctx, key, bootstrap_keys_wrap[id], &(key_lens[id]))) + { + pg_cipher_ctx_free(cluster_key_ctx); + elog(ERROR, "failed to wrap data encryption key"); + } + + /* remove DEK from memory */ + explicit_bzero(key, sizeof(CryptoKey)); + } + + /* Write data encryption keys to the disk */ + KmgrWriteCryptoKeys(LIVE_KMGR_DIR, bootstrap_keys_wrap, key_lens); + + pg_cipher_ctx_free(cluster_key_ctx); + } + + /* + * We are either decrypting keys we copied from an old cluster, or + * decrypting keys we just wrote above --- either way, we decrypt them + * here and store them in a file-scoped variable for use in later + * encrypting during bootstrap mode. + */ + + /* Get the crypto keys from the live directory */ + kmgr_read_wrapped_data_keys(LIVE_KMGR_DIR, keys_wrap, key_lens); + + if (!kmgr_verify_cluster_key(cluster_key, keys_wrap, key_lens, bootstrap_keys)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("supplied cluster key does not match expected cluster_key"))); + + /* bzero DEK on exit */ + on_proc_exit(bzeroKmgrKeys, 0); + + /* bzero KEK */ + explicit_bzero(cluster_key_hex, cluster_key_hex_len); + explicit_bzero(cluster_key, KMGR_CLUSTER_KEY_LEN); +} + +/* Report shared-memory space needed by KmgrShmem */ +Size +KmgrShmemSize(void) +{ + if (!FileEncryptionEnabled) + return 0; + + return MAXALIGN(sizeof(KmgrShmemData)); +} + +/* Allocate and initialize key manager memory */ +void +KmgrShmemInit(void) +{ + bool found; + + if (!FileEncryptionEnabled) + return; + + KmgrShmem = (KmgrShmemData *) ShmemInitStruct("File encryption key manager", + KmgrShmemSize(), &found); + + /* bzero DEK on exit */ + on_shmem_exit(bzeroKmgrKeys, 0); +} + +/* + * Get cluster key and verify it, then get the data encryption keys. + * This function is called by postmaster at startup time. + */ +void +InitializeKmgr(void) +{ + unsigned char *keys_wrap[KMGR_NUM_DATA_KEYS]; + int key_lens[KMGR_NUM_DATA_KEYS]; + char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN]; + int cluster_key_hex_len; + struct stat buffer; + char live_path[MAXPGPATH]; + unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN]; + +#ifndef USE_OPENSSL + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"), + errhint("Compile with --with-openssl to use this feature.")))); +#endif + + /* + * There are too many optimizations for wal_level=minimal that don't set + * LSNs for permanent tables, so just disallow it. + */ + if (!XLogIsNeeded()) + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported with a wal_level of \"minimal\"")))); + + elog(DEBUG1, "starting up cluster file encryption manager"); + + if (stat(KMGR_DIR, &buffer) != 0 || !S_ISDIR(buffer.st_mode)) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster file encryption directory %s is missing", KMGR_DIR)))); + + if (stat(KMGR_DIR_PID, &buffer) == 0) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster had a pg_alterckey failure that needs repair or pg_alterckey is running"), + errhint("Run pg_alterckey --repair or wait for it to complete.")))); + + /* + * We want OLD deleted since it allows access to the data encryption keys + * using the old cluster key. If NEW exists, it means either the new + * directory is partly written, or NEW wasn't renamed to LIVE --- in + * either case, it needs to be repaired. See src/bin/pg_alterckey/README + * for more details. + */ + if (stat(OLD_KMGR_DIR, &buffer) == 0 || stat(NEW_KMGR_DIR, &buffer) == 0) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster had a pg_alterckey failure that needs repair"), + errhint("Run pg_alterckey --repair.")))); + + /* If OLD, NEW, and LIVE do not exist, there is a serious problem. */ + if (stat(LIVE_KMGR_DIR, &buffer) != 0 || !S_ISDIR(buffer.st_mode)) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + (errmsg("cluster has no data encryption keys")))); + + /* Get the cluster key (KEK) */ + snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR); + cluster_key_hex_len = kmgr_run_cluster_key_command(cluster_key_command, + cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + live_path, terminal_fd); + + /* decode supplied hex */ + if (hex_decode(cluster_key_hex, cluster_key_hex_len, + (char *) cluster_key) != + KMGR_CLUSTER_KEY_LEN) + ereport(ERROR, + (errmsg("cluster key must be %d hexadecimal characters", + KMGR_CLUSTER_KEY_LEN * 2))); + + /* Load wrapped DEKs from their files into an array */ + kmgr_read_wrapped_data_keys(LIVE_KMGR_DIR, keys_wrap, key_lens); + + /* + * Verify cluster key and store the unwrapped data encryption keys in + * shared memory. + */ + if (!kmgr_verify_cluster_key(cluster_key, keys_wrap, key_lens, KmgrShmem->intlKeys)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("supplied cluster key does not match expected cluster key"))); + + /* Check that retrieved key lengths match controldata length. */ + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + if (KmgrShmem->intlKeys[id].klen * 8 != + encryption_methods[GetFileEncryptionMethod()].bit_length) + { + char path[MAXPGPATH]; + + CryptoKeyFilePath(path, DataDir, id); + + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("data encryption key %s of length %d does not match controldata key length %d", + path, KmgrShmem->intlKeys[id].klen * 8, + encryption_methods[GetFileEncryptionMethod()].bit_length))); + } + + /* bzero KEK */ + explicit_bzero(cluster_key_hex, cluster_key_hex_len); + explicit_bzero(cluster_key, KMGR_CLUSTER_KEY_LEN); +} + +static void +bzeroKmgrKeys(int status, Datum arg) +{ + if (IsBootstrapProcessingMode()) + explicit_bzero(bootstrap_keys, sizeof(bootstrap_keys)); + else + explicit_bzero(KmgrShmem->intlKeys, sizeof(KmgrShmem->intlKeys)); +} + +/* return requested DEK */ +const CryptoKey * +KmgrGetKey(int id) +{ + Assert(id < KMGR_NUM_DATA_KEYS); + + return (const CryptoKey *) (IsBootstrapProcessingMode() ? + &(bootstrap_keys[id]) : &(KmgrShmem->intlKeys[id])); +} + +/* Generate a DEK inside a CryptoKey */ +static CryptoKey * +generate_crypto_key(int len) +{ + CryptoKey *newkey; + + Assert(len <= KMGR_MAX_KEY_LEN); + newkey = (CryptoKey *) palloc0(sizeof(CryptoKey)); + + newkey->klen = len; + + if (!pg_strong_random(newkey->key, len)) + elog(ERROR, "failed to generate new file encryption key"); + + return newkey; +} + +/* + * Write the DEKs to the disk. + */ +static void +KmgrWriteCryptoKeys(const char *dir, unsigned char **keys, int *key_lens) +{ + elog(DEBUG2, "writing data encryption keys wrapped using the cluster key"); + + for (int i = 0; i < KMGR_NUM_DATA_KEYS; i++) + { + int fd; + char path[MAXPGPATH]; + + CryptoKeyFilePath(path, dir, i); + + if ((fd = BasicOpenFile(path, O_RDWR | O_CREAT | O_EXCL | PG_BINARY)) < 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\": %m", + path))); + + errno = 0; + pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_WRITE); + if (write(fd, keys[i], key_lens[i]) != key_lens[i]) + { + /* if write didn't set errno, assume problem is no disk space */ + if (errno == 0) + errno = ENOSPC; + + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not write file \"%s\": %m", + path))); + } + pgstat_report_wait_end(); + + pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_SYNC); + if (pg_fsync(fd) != 0) + ereport(PANIC, + (errcode_for_file_access(), + errmsg("could not fsync file \"%s\": %m", + path))); + pgstat_report_wait_end(); + + if (close(fd) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", + path))); + } +} diff --git a/src/common/kmgr_utils.c b/src/common/kmgr_utils.c index 6c50cb093b..92b6441186 100644 --- a/src/common/kmgr_utils.c +++ b/src/common/kmgr_utils.c @@ -30,11 +30,12 @@ #endif #include "common/cryptohash.h" #include "common/file_perm.h" -#include "common/hex.h" #include "common/string.h" #include "crypto/kmgr.h" #include "lib/stringinfo.h" #include "storage/fd.h" +#include "postgres.h" +#include "utils/builtins.h" #ifndef FRONTEND #include "pgstat.h" diff --git a/src/include/crypto/kmgr.h b/src/include/crypto/kmgr.h new file mode 100644 index 0000000000..ec341c70b9 --- /dev/null +++ b/src/include/crypto/kmgr.h @@ -0,0 +1,25 @@ +/*------------------------------------------------------------------------- + * + * kmgr.h + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/crypto/kmgr.h + * + *------------------------------------------------------------------------- + */ +#ifndef KMGR_H +#define KMGR_H + +#include "common/kmgr_utils.h" + +/* GUC parameters */ +extern char *cluster_key_command; + +extern Size KmgrShmemSize(void); +extern void KmgrShmemInit(void); +extern void BootStrapKmgr(void); +extern void InitializeKmgr(void); +extern const CryptoKey *KmgrGetKey(int id); + +#endif /* KMGR_H */ -- 2.31.1 [application/octet-stream] v2-0002-cfe-02-internaldoc_over_cfe-01-doc-squash-commit.patch (11.4K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/4-v2-0002-cfe-02-internaldoc_over_cfe-01-doc-squash-commit.patch) download | inline diff: From bbaaac474026fb8ab624f0396da5222bb5c52909 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:45 -0400 Subject: [PATCH v2 02/12] cfe-02-internaldoc_over_cfe-01-doc squash commit --- src/backend/crypto/README | 231 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 src/backend/crypto/README diff --git a/src/backend/crypto/README b/src/backend/crypto/README new file mode 100644 index 0000000000..be5e5557ba --- /dev/null +++ b/src/backend/crypto/README @@ -0,0 +1,231 @@ +Cluster File Encryption +======================= + +This directory contains support functions and sample scripts to be used +for cluster file encryption. + +Architecture +------------ + +Fundamentally, cluster file encryption must store data in a file system +in such a way that the keys required to decrypt the file system data can +only be accessed using somewhere outside of the file system itself. The +external requirement can be someone typing in a passphrase, getting a +key from a key management server (KMS), or decrypting a key stored in +the file system using a hardware security module (HSM). The current +architecture supports all of these methods, and includes sample scripts +for them. + +The simplest method for accessing data keys using some external +requirement would be to retrieve all data encryption keys from a KMS. +However, retrieved keys would still need to be verified as valid. This +method also introduces unacceptable complexity for simpler use-cases, +like user-supplied passphrases or HSM usage. External key rotation +would also be very hard since it would require re-encrypting all the +file system data with the new externally-stored keys. + +For these reason, a two-tiered architecture is used, which uses two +types of encryption keys: a key encryption key (KEK) and data encryption +keys (DEK). The KEK should not be present unencrypted in the file system +--- it should be supplied the user, stored externally (e.g., in a KMS) +or stored in the file system encrypted with a HSM (e.g., PIV device). +The DEK is used to encrypt database files and is stored in the same file +system as the database but is encrypted using the KEK. Because the DEK +is encrypted, its storage in the file system is no more of a security +weakness and the storage of the encrypted database files in the same +file system. + +Implementation +-------------- + +To enable cluster file encryption, the initdb option +--cluster-key-command must be used, which specifies a command to +retrieve the KEK. initdb records the cluster_key_command in +postgresql.conf. Every time the KEK is needed, the command is run and +must return 64 hex characters which are decoded into the KEK. The +command is called twice during initdb, and every time the server starts. +initdb also sets the encryption method in controldata during server +bootstrap. + +initdb runs "postgres --boot", which calls function +kmgr.c::BootStrapKmgr(), which calls the cluster key command. The +cluster key command returns a KEK which is used to encrypt random bytes +for each DEK and writes them to the file system by +kmgr.c::KmgrWriteCryptoKeys() (unless --copy-encryption-keys is used). +Currently the DEK files are 0 and 1 and are stored in +$PGDATA/pg_cryptokeys/live. The wrapped DEK files use Key Wrapping with +Padding which verifies the validity of the KEK. + +initdb also does a non-boot backend start which calls +kmgr.c::InitializeKmgr(), which calls the cluster key command a second +time. This decrypts/unwraps the DEK keys and stores them in the shared +memory structure KmgrShmem. This step also happens every time the server +starts. Later patches will use the keys stored in KmgrShmem to +encrypt/decrypt database files. KmgrShmem is erased via +explicit_bzero() on server shutdown. + +Limitations +----------- + +There doesn't seem to be a reasonable way to detect all malicious data +modification or key extraction if a user has write permission on the +files in PGDATA. It might be possible to limit the key extraction risk +if postgresql.auto.conf were able to be moved to a directory outside of +PGDATA, and if postmaster.opts could be moved or ignored when cluster +file encryption is used. (This file is used by pg_ctl restart.) + +It doesn't appear possible to detect all malicious writes --- even if +you add message authentication code (MAC) checks to encrypted files, +modifying non-encrypted files could still affect encrypted ones, e.g., +modifying files in pg_xact could affect how heap rows are interpreted. +Basically you would need to encrypt all files, and at that point you +might as well just use an encrypted file system. There also doesn't seem +to be a way to prevent key extraction if someone has read permission on +postgres process memory. + +Initialization Vector +--------------------- + +Nonce means "number used once". An Initialization Vector (IV) is a +specific type of nonce. That is, unique but not necessarily random or +secret, as specified by the NIST +(https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf). +To generate unique IVs, the NIST recommends two methods: + + The first method is to apply the forward cipher function, under + the same key that is used for the encryption of the plaintext, + to a nonce. The nonce must be a data block that is unique to + each execution of the encryption operation. For example, the + nonce may be a counter, as described in Appendix B, or a message + number. The second method is to generate a random data block + using a FIPS-approved random number generator. + +We will use the first method to generate IVs. That is, select nonce +carefully and use a cipher with the key to make it unique enough to use +as an IV. The nonce selection for buffer encryption and WAL encryption +are described below. + +If the IV was used more than once with the same key (and we only use one +data encryption key), changes in the unencrypted data would be visible +in the encrypted data. + +IV for Heap/Index Encryption +- - - - - - - - - - - - - - + +To create the 16-byte IV needed by AES for each page version, we will +use the page LSN (8 bytes) and page number (4 bytes). In the remaining +four bytes, one bit will be used to indicate if the LSN is WAL (real) or +fake (see below). The LSN is ideal for use in the IV because it is +always increasing, and is changed every time a page is updated. The +same LSN is never used for two relations with different page contents. + +However, the same LSN can be used in multiple pages in the same relation +--- this can happen when a heap update expires an old tuple and adds a +new tuple to another page. By adding the page number to the IV, we keep +the IV unique. + +By not using the database id in the IV, CREATE DATABASE can copy the +heap/index files from the old database to a new one without +decryption/encryption. Both page copies are valid. Once a database +changes its pages, it gets new LSNs, and hence new IV. Using only the +LSN and page number also avoids requiring pg_upgrade to preserve +database oids, tablespace oids, and relfilenodes. + +As part of WAL logging, every change of a WAL-logged page gets a new +LSN, and therefore a new IV automatically. + +However, the LSN must then be visible on encrypted pages, so we will not +encrypt the LSN on the page. We will also not encrypt the CRC so +pg_checksums can still check pages offline without access to the keys. + +Non-Permanent Relations +- - - - - - - - - - - - + +To avoid the overhead of generating WAL for non-permanent (unlogged and +temporary) relations, we assign fake LSNs that are derived from a +counter via xlog.c::GetFakeLSNForUnloggedRel(). (GiST also uses this +counter for LSNs.) We also set a bit in the IV so the use of the same +value for WAL (real) and fake LSNs will still generate unique IVs. Only +main forks are encrypted, not init, vm, or fsm files. + +In the code, we need to identify if a page uses WAL or fake LSNs in +four places, when: + +1. Reading a page from the file system and decrypting +2. Setting the WAL or fake LSN on a page +3. Hint bits changes requiring new LSNs for the encryption IV +4. Encrypting and writing a page to the file system + +For all these case, we have access to the fork number and either the +relation's persistence state or the buffer state. If it is a "main" +fork and the relation persistence state is RELPERSISTENCE_PERMANENT, or +if it is an "init" fork, we use a real LSN. If it is a main fork and +RELPERSISTENCE_PERMANENT is false, we use a fake LSN. The buffer state +BM_PERMANENT is true if the relation is PERMANENT or is an init fork. + +Init Forks +- - - - - + +Init forks for unlogged relations get permanent LSNs because unlogged +relation creation is WAL logged/crash safe, even though the relation's +contents are not. When the init fork is copied to represent an empty +relation during crash recovery, it becomes a non-permanent page and must +be successfully decrypted as such. Therefore, when it is copied, its +LSN is changed to e fake LSN and then encrypted. This prevents a real +LSN from being encrypted with the fake nonce bit. + +LSN Assignment, GiST, & Non-Permanent Relations +- - - - - - - - - - - - - - - - - - - - - - - - + +LSN assignment has to be slightly modified for encryption. In normal, +non-encryption mode, LSNs are assigned to pages following these rules: + +1. During GiST builds, some pages are assigned fixed LSNs (GistBuildLSN) + +2. During GiST builds, non-permanent pages not assigned fixed LSNs in +#1 are assigned fake LSNs, via gistutil.c::gistGetFakeLSN(). + +3. All other permanent pages are assigned WAL-based LSNs based on the +WAL position of their WAL records. + +4. All other non-permanent pages have LSNs of zero. + +When encryption is enabled: + +1. During GiST builds, permanent pages are assigned WAL-based LSNs +generated by xloginsert.c::LSNForEncryption(). + +2. During GiST builds, non-permanent pages are assigned fake LSNs. +(No constant LSNs are used in #1 or #2.) + +3. same as #3 above + +4. All other non-permanent pages are assigned fake LSNs before page +encryption. + +When switching to an encrypted replica from a non-encrypted primary, +GiST indexes will be using fixed LSNs for permanent tables, so it is +recommended to rebuild GiST indexes. Non-permanent relations are not +replicated, so they are not an issue. + +Hint Bits +- - - - - + +For hint bit changes, the LSN normally doesn't change, which is a +problem. By enabling wal_log_hints, you get full page writes to the WAL +after the first hint bit change of the checkpoint. This is useful for +two reasons. First, it generates a new LSN, which is needed for the IV +to be secure. Second, full page images protect against torn pages, +which is an even bigger requirement for encryption because the new LSN +is re-encrypting the entire page, not just the hint bit changes. You +can safely lose the hint bit changes, but you need to use the same LSN +to decrypt the entire page, so a torn page with an LSN change cannot be +decrypted. To prevent this, wal_log_hints guarantees that the +pre-hint-bit version (and previous LSN version) of the page is restored. + +However, if a hint-bit-modified page is written to the file system +during a checkpoint, and there is a later hint bit change switching the +same page from clean to dirty during the same checkpoint, we need a new +LSN, and wal_log_hints doesn't give us a new LSN here. The fix for this +is to update the page LSN by writing a dummy WAL record via +xloginsert.c::LSNForEncryption() in such cases. -- 2.31.1 [application/octet-stream] v2-0003-cfe-03-scripts_over_cfe-02-internaldoc-squash-com.patch (13.8K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/5-v2-0003-cfe-03-scripts_over_cfe-02-internaldoc-squash-com.patch) download | inline diff: From aa2b8fe000bbcf98731d7dbef13170ebde1d33d7 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:47 -0400 Subject: [PATCH v2 03/12] cfe-03-scripts_over_cfe-02-internaldoc squash commit --- src/backend/Makefile | 15 +++- src/backend/crypto/ckey_aws.sh.sample | 53 +++++++++++++ src/backend/crypto/ckey_direct.sh.sample | 39 ++++++++++ src/backend/crypto/ckey_passphrase.sh.sample | 35 +++++++++ src/backend/crypto/ckey_piv_nopin.sh.sample | 68 ++++++++++++++++ src/backend/crypto/ckey_piv_pin.sh.sample | 81 ++++++++++++++++++++ src/backend/crypto/ssl_passphrase.sh.sample | 35 +++++++++ 7 files changed, 325 insertions(+), 1 deletion(-) create mode 100644 src/backend/crypto/ckey_aws.sh.sample create mode 100644 src/backend/crypto/ckey_direct.sh.sample create mode 100644 src/backend/crypto/ckey_passphrase.sh.sample create mode 100644 src/backend/crypto/ckey_piv_nopin.sh.sample create mode 100644 src/backend/crypto/ckey_piv_pin.sh.sample create mode 100644 src/backend/crypto/ssl_passphrase.sh.sample diff --git a/src/backend/Makefile b/src/backend/Makefile index 181c217fae..1a5239b822 100644 --- a/src/backend/Makefile +++ b/src/backend/Makefile @@ -198,6 +198,12 @@ endif $(INSTALL_DATA) $(srcdir)/libpq/pg_hba.conf.sample '$(DESTDIR)$(datadir)/pg_hba.conf.sample' $(INSTALL_DATA) $(srcdir)/libpq/pg_ident.conf.sample '$(DESTDIR)$(datadir)/pg_ident.conf.sample' $(INSTALL_DATA) $(srcdir)/utils/misc/postgresql.conf.sample '$(DESTDIR)$(datadir)/postgresql.conf.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_aws.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_aws.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_direct.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_direct.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_passphrase.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_passphrase.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_piv_nopin.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_nopin.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ckey_piv_pin.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_pin.sh.sample' + $(INSTALL_DATA) $(srcdir)/crypto/ssl_passphrase.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ssl_passphrase.sh.sample' ifeq ($(with_llvm), yes) install-bin: install-postgres-bitcode @@ -223,6 +229,7 @@ endif installdirs: $(MKDIR_P) '$(DESTDIR)$(bindir)' '$(DESTDIR)$(datadir)' + $(MKDIR_P) '$(DESTDIR)$(datadir)' '$(DESTDIR)$(datadir)/auth_commands' ifeq ($(PORTNAME), cygwin) ifeq ($(MAKE_DLL), true) $(MKDIR_P) '$(DESTDIR)$(libdir)' @@ -262,7 +269,13 @@ endif $(MAKE) -C utils uninstall-data rm -f '$(DESTDIR)$(datadir)/pg_hba.conf.sample' \ '$(DESTDIR)$(datadir)/pg_ident.conf.sample' \ - '$(DESTDIR)$(datadir)/postgresql.conf.sample' + '$(DESTDIR)$(datadir)/postgresql.conf.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_aws.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_direct.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_passphrase.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_nopin.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_pin.sh.sample' \ + '$(DESTDIR)$(datadir)/auth_commands/ssl_passphrase.sh.sample' ifeq ($(with_llvm), yes) $(call uninstall_llvm_module,postgres) endif diff --git a/src/backend/crypto/ckey_aws.sh.sample b/src/backend/crypto/ckey_aws.sh.sample new file mode 100644 index 0000000000..d9bee53132 --- /dev/null +++ b/src/backend/crypto/ckey_aws.sh.sample @@ -0,0 +1,53 @@ +#!/bin/sh + +# This uses the AWS Secrets Manager using the AWS CLI and OpenSSL. +# This stores the AWS secret Id in $DIR. +# Do not create any file with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -ne 1 ] && echo "cluster_key_command usage: $0 \"%d\"" 1>&2 && exit 1 +# No need for %R or -R since we are not prompting + +DIR="$1" +[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1 +[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1 + +# File containing the id of the AWS secret +AWS_ID_FILE="$DIR/aws-secret.id" + + +# ---------------------------------------------------------------------- + + +# Create an AWS Secrets Manager secret? +if [ ! -e "$AWS_ID_FILE" ] +then # The 'postgres' operating system user must have permission to + # access the AWS CLI + + # The epoch-time/directory/hostname combination is unique + HASH=$(echo -n "$(date '+%s')$DIR$(hostname)" | sha1sum | cut -d' ' -f1) + AWS_SECRET_ID="Postgres-cluster-key-$HASH" + + # Use stdin to avoid passing the secret on the command line + openssl rand -hex 32 | + aws secretsmanager create-secret \ + --name "$AWS_SECRET_ID" \ + --description "Postgres cluster file encryption on $(hostname)" \ + --secret-string 'file:///dev/stdin' \ + --output text > /dev/null + if [ "$?" -ne 0 ] + then echo 'cluster key generation failed' 1>&2 + exit 1 + fi + + echo "$AWS_SECRET_ID" > "$AWS_ID_FILE" +fi + +if ! aws secretsmanager get-secret-value \ + --secret-id "$(cat "$AWS_ID_FILE")" \ + --output text +then echo 'cluster key retrieval failed' 1>&2 + exit 1 +fi | awk -F'\t' 'NR == 1 {print $4}' + +exit 0 diff --git a/src/backend/crypto/ckey_direct.sh.sample b/src/backend/crypto/ckey_direct.sh.sample new file mode 100644 index 0000000000..492defcffe --- /dev/null +++ b/src/backend/crypto/ckey_direct.sh.sample @@ -0,0 +1,39 @@ +#!/bin/sh + +# This uses a 64-character hex key supplied by the user. +# If OpenSSL is installed, you can generate a pseudo-random key by running: +# openssl rand -hex 32 +# To get a true random key, run: +# wget -q -O - 'https://www.random.org/cgi-bin/randbyte?nbytes=32&format=h' | tr -d ' \n'; echo +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 1 ] && echo "cluster_key_command usage: $0 %R [%p]" 1>&2 && exit 1 +# Supports environment variable PROMPT + +FD="$1" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 + +[ "$2" ] && PROMPT="$2" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter cluster key as 64 hexadecimal characters: ' + +stty -echo <&"$FD" + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" +read KEY <&"$FD" + +stty echo <&"$FD" + +if [ "$(expr "$KEY" : '[0-9a-fA-F]*$')" -ne 64 ] +then echo 'invalid; must be 64 hexadecimal characters' 1>&2 + exit 1 +fi + +echo "$KEY" + +exit 0 diff --git a/src/backend/crypto/ckey_passphrase.sh.sample b/src/backend/crypto/ckey_passphrase.sh.sample new file mode 100644 index 0000000000..a5d837b45e --- /dev/null +++ b/src/backend/crypto/ckey_passphrase.sh.sample @@ -0,0 +1,35 @@ +#!/bin/sh + +# This uses a passphrase supplied by the user. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 1 ] && echo "cluster_key_command usage: $0 %R [\"%p\"]" 1>&2 && exit 1 + +FD="$1" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 +# Supports environment variable PROMPT + +[ "$2" ] && PROMPT="$2" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter cluster passphrase: ' + +stty -echo <&"$FD" + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" +read PASS <&"$FD" + +stty echo <&"$FD" + +if [ ! "$PASS" ] +then echo 'invalid: empty passphrase' 1>&2 + exit 1 +fi + +echo "$PASS" | sha256sum | cut -d' ' -f1 + +exit 0 diff --git a/src/backend/crypto/ckey_piv_nopin.sh.sample b/src/backend/crypto/ckey_piv_nopin.sh.sample new file mode 100644 index 0000000000..e90a579dea --- /dev/null +++ b/src/backend/crypto/ckey_piv_nopin.sh.sample @@ -0,0 +1,68 @@ +#!/bin/sh + +# This uses the public/private keys on a PIV device, like a CAC or Yubikey. +# It uses a PIN stored in a file. +# It uses OpenSSL with PKCS11 enabled via OpenSC. +# This stores the cluster encryption key encrypted with the PIV public +# key in $DIR. This is technically a three-level encryption +# architecture, with the third level requiring the PIV and PIN. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -ne 1 ] && echo "cluster_key_command usage: $0 \"%d\"" 1>&2 && exit 1 +# Supports environment variable PIV_PIN_FILE +# No need for %R or -R since we are not prompting for a PIN + +DIR="$1" +[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1 +[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1 + +# Set these here or pass in as environment variables. +# File that stores the PIN to unlock the PIV +#PIV_PIN_FILE='' +# PIV slot 3 is the "Key Management" slot, so we use '0:3' +PIV_SLOT='0:3' + +# File containing the cluster key encrypted with the PIV_SLOT's public key +KEY_FILE="$DIR/pivpass.key" + + +# ---------------------------------------------------------------------- + +[ ! "$PIV_PIN_FILE" ] && echo 'PIV_PIN_FILE undefined' 1>&2 && exit 1 +[ ! -e "$PIV_PIN_FILE" ] && echo "$PIV_PIN_FILE does not exist" 1>&2 && exit 1 +[ -d "$PIV_PIN_FILE" ] && echo "$PIV_PIN_FILE is a directory" 1>&2 && exit 1 + +[ ! "$KEY_FILE" ] && echo 'KEY_FILE undefined' 1>&2 && exit 1 +[ -d "$KEY_FILE" ] && echo "$KEY_FILE is a directory" 1>&2 && exit 1 + +# Create a cluster key encrypted with the PIV_SLOT's public key? +if [ ! -e "$KEY_FILE" ] +then # The 'postgres' operating system user must have permission to + # access the PIV device. + + openssl rand -hex 32 | + if ! openssl rsautl -engine pkcs11 -keyform engine -encrypt \ + -inkey "$PIV_SLOT" -passin file:"$PIV_PIN_FILE" -out "$KEY_FILE" + then echo 'cluster key generation failed' 1>&2 + exit 1 + fi + + # Warn the user to save the cluster key in a safe place + cat 1>&2 <<END + +WARNING: The PIV device can be locked and require a reset if too many PIN +attempts fail. It is recommended to run this command manually and save +the cluster key in a secure location for possible recovery. +END + +fi + +# Decrypt the cluster key encrypted with the PIV_SLOT's public key +if ! openssl rsautl -engine pkcs11 -keyform engine -decrypt \ + -inkey "$PIV_SLOT" -passin file:"$PIV_PIN_FILE" -in "$KEY_FILE" +then echo 'cluster key decryption failed' 1>&2 + exit 1 +fi + +exit 0 diff --git a/src/backend/crypto/ckey_piv_pin.sh.sample b/src/backend/crypto/ckey_piv_pin.sh.sample new file mode 100644 index 0000000000..e693ac31ba --- /dev/null +++ b/src/backend/crypto/ckey_piv_pin.sh.sample @@ -0,0 +1,81 @@ +#!/bin/sh + +# This uses the public/private keys on a PIV device, like a CAC or Yubikey. +# It requires a user-entered PIN. +# It uses OpenSSL with PKCS11 enabled via OpenSC. +# This stores the cluster encryption key encrypted with the PIV public +# key in $DIR. This is technically a three-level encryption +# architecture, with the third level requiring the PIV and PIN. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 2 ] && echo "cluster_key_command usage: $0 \"%d\" %R [\"%p\"]" 1>&2 && exit 1 +# Supports environment variable PROMPT + +DIR="$1" +[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1 +[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1 + +FD="$2" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 + +[ "$3" ] && PROMPT="$3" + +# PIV slot 3 is the "Key Management" slot, so we use '0:3' +PIV_SLOT='0:3' + +# File containing the cluster key encrypted with the PIV_SLOT's public key +KEY_FILE="$DIR/pivpass.key" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter PIV PIN: ' + +stty -echo <&"$FD" + +# Create a cluster key encrypted with the PIV_SLOT's public key? +if [ ! -e "$KEY_FILE" ] +then echo 1>&"$FD" + echo -n "$PROMPT" 1>&"$FD" + + # The 'postgres' operating system user must have permission to + # access the PIV device. + + openssl rand -hex 32 | + # 'engine "pkcs11" set.' message confuses prompting + if ! openssl rsautl -engine pkcs11 -keyform engine -encrypt \ + -inkey "$PIV_SLOT" -passin fd:"$FD" -out "$KEY_FILE" 2>&1 + then stty echo <&"$FD" + echo 'cluster key generation failed' 1>&2 + exit 1 + fi | grep -v 'engine "pkcs11" set\.' + + echo 1>&"$FD" + + # Warn the user to save the cluster key in a safe place + cat 1>&"$FD" <<END + +WARNING: The PIV can be locked and require a reset if too many PIN +attempts fail. It is recommended to run this command manually and save +the cluster key in a secure location for possible recovery. +END + +fi + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" + +# Decrypt the cluster key encrypted with the PIV_SLOT's public key +if ! openssl rsautl -engine pkcs11 -keyform engine -decrypt \ + -inkey "$PIV_SLOT" -passin fd:"$FD" -in "$KEY_FILE" 2>&1 +then stty echo <&"$FD" + echo 'cluster key retrieval failed' 1>&2 + exit 1 +fi | grep -v 'engine "pkcs11" set\.' + +echo 1>&"$FD" + +stty echo <&"$FD" + +exit 0 diff --git a/src/backend/crypto/ssl_passphrase.sh.sample b/src/backend/crypto/ssl_passphrase.sh.sample new file mode 100644 index 0000000000..efbf5c0720 --- /dev/null +++ b/src/backend/crypto/ssl_passphrase.sh.sample @@ -0,0 +1,35 @@ +#!/bin/sh + +# This uses a passphrase supplied by the user. +# Do not create any fie with extension "wkey" in $DIR; these are +# reserved for wrapped data key files. + +[ "$#" -lt 1 ] && echo "ssl_passphrase_command usage: $0 %R [\"%p\"]" 1>&2 && exit 1 + +FD="$1" +[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1 +# Supports environment variable PROMPT + +[ "$2" ] && PROMPT="$2" + + +# ---------------------------------------------------------------------- + +[ ! "$PROMPT" ] && PROMPT='Enter cluster passphrase: ' + +stty -echo <&"$FD" + +echo 1>&"$FD" +echo -n "$PROMPT" 1>&"$FD" +read PASS <&"$FD" + +stty echo <&"$FD" + +if [ ! "$PASS" ] +then echo 'invalid: empty passphrase' 1>&2 + exit 1 +fi + +echo "$PASS" + +exit 0 -- 2.31.1 [application/octet-stream] v2-0004-cfe-04-common_over_cfe-03-scripts-squash-commit.patch (32.8K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/6-v2-0004-cfe-04-common_over_cfe-03-scripts-squash-commit.patch) download | inline diff: From 00a0b4698c469ef448710b8fd62fe76c38540d22 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:49 -0400 Subject: [PATCH v2 04/12] cfe-04-common_over_cfe-03-scripts squash commit --- src/common/Makefile | 3 + src/common/cipher.c | 98 +++++++ src/common/cipher_openssl.c | 419 ++++++++++++++++++++++++++++ src/common/kmgr_utils.c | 465 ++++++++++++++++++++++++++++++++ src/include/common/cipher.h | 74 +++++ src/include/common/kmgr_utils.h | 94 +++++++ src/include/utils/wait_event.h | 3 + src/tools/msvc/Mkvcbuild.pm | 4 +- 8 files changed, 1159 insertions(+), 1 deletion(-) create mode 100644 src/common/cipher.c create mode 100644 src/common/cipher_openssl.c create mode 100644 src/common/kmgr_utils.c create mode 100644 src/include/common/cipher.h create mode 100644 src/include/common/kmgr_utils.h diff --git a/src/common/Makefile b/src/common/Makefile index e9af7346c9..9f63b01609 100644 --- a/src/common/Makefile +++ b/src/common/Makefile @@ -62,6 +62,7 @@ OBJS_COMMON = \ ip.o \ jsonapi.o \ keywords.o \ + kmgr_utils.o \ kwlookup.o \ link-canary.o \ md5_common.o \ @@ -83,11 +84,13 @@ OBJS_COMMON = \ ifeq ($(with_ssl),openssl) OBJS_COMMON += \ + cipher_openssl.o \ protocol_openssl.o \ cryptohash_openssl.o \ hmac_openssl.o else OBJS_COMMON += \ + cipher.o \ cryptohash.o \ hmac.o \ md5.o \ diff --git a/src/common/cipher.c b/src/common/cipher.c new file mode 100644 index 0000000000..7aa3ea6138 --- /dev/null +++ b/src/common/cipher.c @@ -0,0 +1,98 @@ +/*------------------------------------------------------------------------- + * + * cipher.c + * Shared frontend/backend for cryptographic functions + * + * This is the set of in-core functions used when there are no other + * alternative options like OpenSSL. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/common/cipher.c + * + *------------------------------------------------------------------------- + */ + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/cipher.h" + +static void cipher_failure(void) pg_attribute_noreturn(); + + +PgCipherCtx * +pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc) +{ + cipher_failure(); + return NULL; /* keep compiler quiet */ +} + +void +pg_cipher_ctx_free(PgCipherCtx *ctx) +{ + cipher_failure(); +} + +int +pg_cipher_blocksize(PgCipherCtx *ctx) +{ + cipher_failure(); + return -1; /* keep compiler quiet */ +} + +bool +pg_cipher_encrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *plaintext, + const int inlen, unsigned char *ciphertext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *outtag, const int taglen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +bool +pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *ciphertext, + const int inlen, unsigned char *plaintext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *intag, const int taglen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +bool +pg_cipher_keywrap(PgCipherCtx *ctx, const unsigned char *plaintext, + const int inlen, unsigned char *ciphertext, int *outlen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +bool +pg_cipher_keyunwrap(PgCipherCtx *ctx, const unsigned char *ciphertext, + const int inlen, unsigned char *plaintext, int *outlen) +{ + cipher_failure(); + return false; /* keep compiler quiet */ +} + +static void +cipher_failure(void) +{ +#ifndef FRONTEND + ereport(ERROR, + (errcode(ERRCODE_CONFIG_FILE_ERROR), + (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"), + errhint("Compile with --with-openssl to use this feature.")))); +#else + fprintf(stderr, _("cluster file encryption is not supported because OpenSSL is not supported by this build")); + exit(1); +#endif +} diff --git a/src/common/cipher_openssl.c b/src/common/cipher_openssl.c new file mode 100644 index 0000000000..a03fbe435f --- /dev/null +++ b/src/common/cipher_openssl.c @@ -0,0 +1,419 @@ +/*------------------------------------------------------------------------- + * cipher_openssl.c + * Cryptographic function using OpenSSL + * + * This contains the common low-level functions needed in both frontend and + * backend, for implement the database encryption. + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/common/cipher_openssl.c + * + *------------------------------------------------------------------------- + */ +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include "common/cipher.h" +#include <openssl/conf.h> +#include <openssl/err.h> +#include <openssl/evp.h> +#include <openssl/ssl.h> + +/* + * prototype for the EVP functions that return an algorithm, e.g. + * EVP_aes_128_gcm(). + */ +typedef const EVP_CIPHER *(*ossl_EVP_cipher_func) (void); + +static ossl_EVP_cipher_func get_evp_aes_gcm(int klen); +static EVP_CIPHER_CTX *ossl_cipher_ctx_create(int cipher, unsigned char *key, + int klen, bool enc); + +/* + * Return a newly created cipher context. 'cipher' specifies cipher algorithm + * by identifier like PG_CIPHER_XXX. + */ +PgCipherCtx * +pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc) +{ + PgCipherCtx *ctx = NULL; + + if (cipher > PG_MAX_CIPHER_ID) + return NULL; + + ctx = ossl_cipher_ctx_create(cipher, key, klen, enc); + + return ctx; +} + +void +pg_cipher_ctx_free(PgCipherCtx *ctx) +{ + EVP_CIPHER_CTX_free(ctx); +} + +int +pg_cipher_blocksize(PgCipherCtx *ctx) +{ + Assert(ctx); + + return EVP_CIPHER_CTX_block_size(ctx); +} + +/* + * Encryption routine to encrypt data provided. + * + * ctx is the encryption context which must have been created previously. + * + * plaintext is the data we are going to encrypt + * inlen is the length of the data to encrypt + * + * ciphertext is the encrypted result + * outlen is the encrypted length + * + * iv is the IV to use. + * ivlen is the IV length to use. + * + * outtag is the resulting tag. + * taglen is the length of the tag. + */ +bool +pg_cipher_encrypt(PgCipherCtx *ctx, int cipher, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *outtag, const int taglen) +{ + int len; + int enclen; + + Assert(ctx != NULL); + + /* + * Here we are setting the IV for the context which was passed in. Note + * that we signal to OpenSSL that we are configuring a new value for the + * context by passing in 'NULL' for the 2nd ('type') parameter. + */ + + /* + * We don't use GCM mode, but it has a MAC, so we support it and test it + * in case we need it later. XXX is this correct for GCM and CTR? + */ + /* Set the GCM IV length first */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, ivlen, NULL)) + return false; + + /* Set the IV for this encryption. */ + if (!EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv)) + return false; + + /* + * This is the function which is actually performing the encryption for + * us. + */ + if (!EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, inlen)) + return false; + + enclen = len; + + /* Finalize the encryption, which could add more to output. */ + if (!EVP_EncryptFinal_ex(ctx, ciphertext + enclen, &len)) + return false; + + *outlen = enclen + len; + + /* + * Once all of the encryption has been completed we grab the tag. + */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, taglen, outtag)) + return false; + + return true; +} + +/* + * Decryption routine + * + * ctx is the encryption context which must have been created previously. + * + * ciphertext is the data we are going to decrypt + * inlen is the length of the data to decrypt + * + * plaintext is the decrypted result + * outlen is the decrypted length + * + * iv is the IV to use. + * ivlen is the length of the IV. + * + * intag is the tag to use to verify. + * taglen is the length of the tag. + */ +bool +pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *intag, const int taglen) +{ + int declen; + int len; + + /* + * Here we are setting the IV for the context which was passed in. Note + * that we signal to OpenSSL that we are configuring a new value for the + * context by passing in 'NULL' for the 2nd ('type') parameter. + */ + + /* XXX is this correct for GCM and CTR? */ + /* Set the GCM IV length first */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, ivlen, NULL)) + return false; + + /* Set the IV for this decryption. */ + if (!EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv)) + return false; + + /* + * This is the function which is actually performing the decryption for + * us. + */ + if (!EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, inlen)) + return false; + + declen = len; + + /* Set the expected tag value. */ + if (cipher == PG_CIPHER_AES_GCM && + !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, taglen, intag)) + return false; + + /* + * Finalize the decryption, which could add more to output, this is also + * the step which checks the tag and we MUST fail if this indicates an + * invalid tag! + */ + if (!EVP_DecryptFinal_ex(ctx, plaintext + declen, &len)) + return false; + + *outlen = declen + len; + + return true; +} + +/* + * Routine to perform key wrapping for data provided. + * + * ctx is the encryption context which must have been created previously. + * + * plaintext is the data/key we are going to encrypt/wrap + * inlen is the length of the data + * + * ciphertext is the wrapped result + * outlen is the encrypted length (will be larger than input!) + */ +bool +pg_cipher_keywrap(PgCipherCtx *ctx, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen) +{ + int len; + int enclen; + + Assert(ctx != NULL); + + /* + * This is the function which is actually performing the encryption for + * us. + */ + if (!EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, inlen)) + return false; + + enclen = len; + + /* Finalize the encryption, which could add more to output. */ + if (!EVP_EncryptFinal_ex(ctx, ciphertext + enclen, &len)) + return false; + + *outlen = enclen + len; + + return true; +} + +/* + * Routine to perform key unwrapping of the data provided. + * + * ctx is the encryption context which must have been created previously. + * + * ciphertext is the wrapped key we are going to unwrap + * inlen is the length of the data to decrypt/unwrap + * + * plaintext is the decrypted result + * outlen is the decrypted length (will be smaller than input!) + */ +bool +pg_cipher_keyunwrap(PgCipherCtx *ctx, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen) +{ + int declen; + int len; + + /* + * This is the function which is actually performing the decryption for + * us. + */ + if (!EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, inlen)) + return false; + + declen = len; + + /* + * Finalize the decryption, which could add more to output, this is also + * the step which checks the tag and we MUST fail if this indicates an + * invalid result! + */ + if (!EVP_DecryptFinal_ex(ctx, plaintext + declen, &len)) + return false; + + *outlen = declen + len; + + return true; +} + +/* + * Returns the correct GCM cipher functions for OpenSSL based + * on the key length requested. + */ +static ossl_EVP_cipher_func +get_evp_aes_gcm(int klen) +{ + switch (klen) + { + case PG_AES128_KEY_LEN: + return EVP_aes_128_gcm; + case PG_AES192_KEY_LEN: + return EVP_aes_192_gcm; + case PG_AES256_KEY_LEN: + return EVP_aes_256_gcm; + default: + return NULL; + } +} + +/* + * Returns the correct KWP cipher functions for OpenSSL based + * on the key length requested. + */ +static ossl_EVP_cipher_func +get_evp_aes_kwp(int klen) +{ + switch (klen) + { + case PG_AES128_KEY_LEN: + return EVP_aes_128_wrap_pad; + case PG_AES192_KEY_LEN: + return EVP_aes_192_wrap_pad; + case PG_AES256_KEY_LEN: + return EVP_aes_256_wrap_pad; + default: + return NULL; + } +} + +/* + * Returns the correct CTR cipher functions for OpenSSL based + * on the key length requested. + */ +static ossl_EVP_cipher_func +get_evp_aes_ctr(int klen) +{ + switch (klen) + { + case PG_AES128_KEY_LEN: + return EVP_aes_128_ctr; + case PG_AES192_KEY_LEN: + return EVP_aes_192_ctr; + case PG_AES256_KEY_LEN: + return EVP_aes_256_ctr; + default: + return NULL; + } +} + +/* + * Initialize and return an EVP_CIPHER_CTX. Returns NULL if the given + * cipher algorithm is not supported or on failure. + */ +static EVP_CIPHER_CTX * +ossl_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc) +{ + EVP_CIPHER_CTX *ctx; + ossl_EVP_cipher_func func; + int ret; + + ctx = EVP_CIPHER_CTX_new(); + + /* + * We currently only support AES GCM but others could be added in the + * future. + */ + switch (cipher) + { + case PG_CIPHER_AES_GCM: + func = get_evp_aes_gcm(klen); + if (!func) + goto failed; + break; + case PG_CIPHER_AES_KWP: + func = get_evp_aes_kwp(klen); + + /* + * Since wrapping will produce more output then input, and we have + * to be ready for that, OpenSSL requires that we explicitly + * enable wrapping for the context. + */ + EVP_CIPHER_CTX_set_flags(ctx, EVP_CIPHER_CTX_FLAG_WRAP_ALLOW); + if (!func) + goto failed; + break; + case PG_CIPHER_AES_CTR: + func = get_evp_aes_ctr(klen); + if (!func) + goto failed; + break; + default: + goto failed; + } + + /* + * We create the context here based on the cipher requested and the + * provided key. Note that the IV will be provided in the actual + * encryption call through another EVP_EncryptInit_ex call- this is fine + * as long as 'type' is passed in as NULL! + */ + if (enc) + ret = EVP_EncryptInit_ex(ctx, (const EVP_CIPHER *) func(), NULL, key, NULL); + else + ret = EVP_DecryptInit_ex(ctx, (const EVP_CIPHER *) func(), NULL, key, NULL); + + if (!ret) + goto failed; + + /* Set the key length based on the key length requested. */ + if (!EVP_CIPHER_CTX_set_key_length(ctx, klen)) + goto failed; + + return ctx; + +failed: + EVP_CIPHER_CTX_free(ctx); + return NULL; +} diff --git a/src/common/kmgr_utils.c b/src/common/kmgr_utils.c new file mode 100644 index 0000000000..6c50cb093b --- /dev/null +++ b/src/common/kmgr_utils.c @@ -0,0 +1,465 @@ +/*------------------------------------------------------------------------- + * + * kmgr_utils.c + * Shared frontend/backend for cluster file encryption + * + * These functions handle reading the wrapped DEK files from the + * file system, and wrapping and unwrapping them. It also handles + * running the cluster_key_command. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/common/kmgr_utils.c + * + *------------------------------------------------------------------------- + */ + +#ifndef FRONTEND +#include "postgres.h" +#else +#include "postgres_fe.h" +#endif + +#include <unistd.h> +#include <sys/stat.h> +#include <fcntl.h> + +#ifdef FRONTEND +#include "common/logging.h" +#endif +#include "common/cryptohash.h" +#include "common/file_perm.h" +#include "common/hex.h" +#include "common/string.h" +#include "crypto/kmgr.h" +#include "lib/stringinfo.h" +#include "storage/fd.h" + +#ifndef FRONTEND +#include "pgstat.h" +#include "storage/fd.h" +#endif + +#define KMGR_PROMPT_MSG "Enter authentication needed to generate the cluster key: " + +#ifdef FRONTEND +static FILE *open_pipe_stream(const char *command); +static int close_pipe_stream(FILE *file); +#endif + +static void read_wrapped_data_key(const char *cryptoKeyDir, uint32 id, unsigned char **key_p, int *key_len); + +/* + * We need this array in frontend and backend code, so define it here. + * You can only add to the end of this array since the array index is + * stored in pg_control. + */ +encryption_method encryption_methods[NUM_ENCRYPTION_METHODS] = { + {"", 0}, + {"AES128", 128}, + {"AES192", 192}, + {"AES256", 256} +}; + +/* This maps wrapped key filesnames to their array slots */ +char *wkey_filenames[KMGR_NUM_DATA_KEYS] = { + "relation", + "wal" +}; + + +/* + * Wrap the given CryptoKey. + * + * Returns true and writes encrypted/wrapped/padded data to 'out', and the length + * of the result to outlen, if successful. + * + * Otherwise returns false. The caller must allocate sufficient space + * for cipher data calculated by using KmgrSizeOfCipherText(). Please note that + * this function modifies 'out' data even on failure. + */ +bool +kmgr_wrap_data_key(PgCipherCtx *ctx, CryptoKey *in, unsigned char *out, int *outlen) +{ + Assert(ctx && in && out); + + if (!pg_cipher_keywrap(ctx, (unsigned char *) in, sizeof(CryptoKey), out, outlen)) + return false; + + return true; +} + +/* + * Decrypt the given data. Return true and set plain text data to `out` if + * successful. Otherwise return false. The caller must allocate sufficient + * space for cipher data calculated by using KmgrSizeOfPlainText(). Please + * note that this function modifies 'out' data even on failure. + */ +bool +kmgr_unwrap_data_key(PgCipherCtx *ctx, unsigned char *in, int inlen, CryptoKey *out) +{ + int outlen; + + Assert(ctx && in && out); + + if (!pg_cipher_keyunwrap(ctx, in, inlen, (unsigned char *) out, &outlen)) + return false; + + Assert(outlen == sizeof(CryptoKey)); + + return true; +} + +/* + * Verify the correctness of the given cluster key by unwrapping the given keys. + * If the given cluster key is correct we set unwrapped keys to out_keys and return + * true. Otherwise return false. Please note that this function changes the + * contents of out_keys even on failure. Both in_keys and out_keys must be the + * same length. + */ +bool +kmgr_verify_cluster_key(unsigned char *cluster_key, + unsigned char **in_keys, int *key_lens, CryptoKey *out_keys) +{ + PgCipherCtx *ctx; + + /* Create decryption context with the KEK. */ + ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, cluster_key, + KMGR_CLUSTER_KEY_LEN, false); + + /* unwrap each DEK */ + for (int i = 0; i < KMGR_NUM_DATA_KEYS; i++) + { + if (!kmgr_unwrap_data_key(ctx, in_keys[i], key_lens[i], &(out_keys[i]))) + { + /* The cluster key is not correct */ + pg_cipher_ctx_free(ctx); + return false; + } + explicit_bzero(in_keys[i], key_lens[i]); + } + + /* The cluster key is correct, free the cipher context */ + pg_cipher_ctx_free(ctx); + + return true; +} + +/* + * Run cluster key command. + * + * Substitute %d for directory, %p for prompt, %R for file descriptor. + * + * The result will be put in buffer buf, which is of size "size". + * The return value is the length of the actual result. + */ +int +kmgr_run_cluster_key_command(char *cluster_key_command, char *buf, + int size, char *dir, int terminal_fd) +{ + StringInfoData command; + const char *sp; + FILE *fh; + int pclose_rc; + size_t len = 0; + + buf[0] = '\0'; + + Assert(size > 0); + + /* + * Build the command to be executed. + */ + initStringInfo(&command); + + for (sp = cluster_key_command; *sp; sp++) + { + if (*sp == '%') + { + switch (sp[1]) + { + /* directory */ + case 'd': + { + char *nativePath; + + sp++; + + /* + * This needs to use a placeholder to not modify the + * input with the conversion done via + * make_native_path(). + */ + nativePath = pstrdup(dir); + make_native_path(nativePath); + appendStringInfoString(&command, nativePath); + pfree(nativePath); + break; + } + /* prompt string */ + case 'p': + sp++; + appendStringInfoString(&command, KMGR_PROMPT_MSG); + break; + /* file descriptor number */ + case 'R': + { + char fd_str[20]; + + if (terminal_fd == -1) + { +#ifdef FRONTEND + pg_fatal("cluster key command referenced %%R, but --authprompt not specified"); +#else + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("cluster key command referenced %%R, but --authprompt not specified"))); +#endif + } + + sp++; + snprintf(fd_str, sizeof(fd_str), "%d", terminal_fd); + appendStringInfoString(&command, fd_str); + break; + } + /* literal "%" */ + case '%': + /* convert %% to a single % */ + sp++; + appendStringInfoChar(&command, *sp); + break; + default: + /* otherwise treat the % as not special */ + appendStringInfoChar(&command, *sp); + break; + } + } + else + { + appendStringInfoChar(&command, *sp); + } + } + +#ifdef FRONTEND + fh = open_pipe_stream(command.data); + if (fh == NULL) + { + pg_fatal("could not execute command \"%s\": %m", + command.data); + } +#else + fh = OpenPipeStream(command.data, "r"); + if (fh == NULL) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not execute command \"%s\": %m", + command.data))); +#endif + + if (!fgets(buf, size, fh)) + { + if (ferror(fh)) + { +#ifdef FRONTEND + pg_fatal("could not read from command \"%s\": %m", + command.data); +#else + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read from command \"%s\": %m", + command.data))); +#endif + } + } + +#ifdef FRONTEND + pclose_rc = close_pipe_stream(fh); +#else + pclose_rc = ClosePipeStream(fh); +#endif + + if (pclose_rc == -1) + { +#ifdef FRONTEND + pg_fatal("could not close pipe to external command: %m"); +#else + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close pipe to external command: %m"))); +#endif + } + else if (pclose_rc != 0) + { +#ifdef FRONTEND + pg_fatal("command \"%s\" failed", command.data); +#else + ereport(ERROR, + (errcode_for_file_access(), + errmsg("command \"%s\" failed", + command.data), + errdetail_internal("%s", wait_result_to_str(pclose_rc)))); +#endif + } + + /* strip trailing newline and carriage returns */ + len = pg_strip_crlf(buf); + + pfree(command.data); + + return len; +} + +#ifdef FRONTEND +static FILE * +open_pipe_stream(const char *command) +{ + FILE *res; + +#ifdef WIN32 + size_t cmdlen = strlen(command); + char *buf; + int save_errno; + + buf = malloc(cmdlen + 2 + 1); + if (buf == NULL) + { + errno = ENOMEM; + return NULL; + } + buf[0] = '"'; + memcpy(&buf[1], command, cmdlen); + buf[cmdlen + 1] = '"'; + buf[cmdlen + 2] = '\0'; + + res = _popen(buf, "r"); + + save_errno = errno; + free(buf); + errno = save_errno; +#else + res = popen(command, "r"); +#endif /* WIN32 */ + return res; +} + +static int +close_pipe_stream(FILE *file) +{ +#ifdef WIN32 + return _pclose(file); +#else + return pclose(file); +#endif /* WIN32 */ +} +#endif /* FRONTEND */ + +/* + * Reads the keys at path. + * + * This routine simply reads in the raw encrypted/wrapped keys; + * it does not handle any decryption, see kmgr_key_unwrap(). + * + * For each key returned, the key and key length are returned + * in the keys and key_lens arrays respectfully. + * + * Note that keys and key_lens must be allocated before calling + * this function as arrays of at least KMGR_NUM_DATA_KEYS length. + */ +void +kmgr_read_wrapped_data_keys(const char *path, unsigned char **keys, int *key_lens) +{ +/* StaticAssertStmt(lengthof(wkey_filenames) == KMGR_NUM_DATA_KEYS, + "wkey_filenames[] must match KMGR_NUM_DATA_KEYS"); +*/ + StaticAssertStmt(1 == 1, + "wkey_filenames[] must match KMGR_NUM_DATA_KEYS"); + + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + read_wrapped_data_key(path, id, &(keys[id]), &(key_lens[id])); + + return; +} + +/* Read a wrapped DEK file */ +static void +read_wrapped_data_key(const char *cryptoKeyDir, uint32 id, unsigned char **key_p, int *key_len) +{ + char path[MAXPGPATH]; + int fd; + int r; + struct stat st; + + CryptoKeyFilePath(path, cryptoKeyDir, id); + +#ifndef FRONTEND + if ((fd = OpenTransientFile(path, O_RDONLY | PG_BINARY)) == -1) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not open file \"%s\" for reading: %m", + path))); + else if (fstat(fd, &st)) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + path))); +#else + if ((fd = open(path, O_RDONLY | PG_BINARY, 0)) == -1) + pg_fatal("could not open file \"%s\" for reading: %m", + path); + else if (fstat(fd, &st)) + pg_fatal("could not stat file \"%s\": %m", + path); +#endif + + *key_len = st.st_size; + +#ifndef FRONTEND + pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_READ); +#endif + + *key_p = (unsigned char *) palloc0(*key_len); + + /* Get key bytes */ + r = read(fd, *key_p, *key_len); + if (r != *key_len) + { + if (r < 0) + { +#ifndef FRONTEND + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", path))); +#else + pg_fatal("could not read file \"%s\": %m", path); +#endif + } + else + { +#ifndef FRONTEND + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg("could not read file \"%s\": read %d of %u", + path, r, *key_len))); +#else + pg_fatal("could not read file \"%s\": read %d of %u", + path, r, *key_len); +#endif + } + } + +#ifndef FRONTEND + pgstat_report_wait_end(); +#endif + +#ifndef FRONTEND + if (CloseTransientFile(fd) != 0) + ereport(ERROR, + (errcode_for_file_access(), + errmsg("could not close file \"%s\": %m", + path))); +#else + if (close(fd) != 0) + pg_fatal("could not close file \"%s\": %m", path); +#endif +} diff --git a/src/include/common/cipher.h b/src/include/common/cipher.h new file mode 100644 index 0000000000..da97ca3776 --- /dev/null +++ b/src/include/common/cipher.h @@ -0,0 +1,74 @@ +/*------------------------------------------------------------------------- + * + * cipher.h + * Declarations for cryptographic functions + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/common/cipher.h + * + *------------------------------------------------------------------------- + */ +#ifndef PG_CIPHER_H +#define PG_CIPHER_H + +#ifdef USE_OPENSSL +#include <openssl/evp.h> +#include <openssl/conf.h> +#include <openssl/err.h> +#endif + +/* + * Supported symmetric encryption algorithm. These identifiers are passed + * to pg_cipher_ctx_create() function, and then actual encryption + * implementations need to initialize their context of the given encryption + * algorithm. + */ +#define PG_CIPHER_AES_GCM 0 +#define PG_CIPHER_AES_CTR 1 +#define PG_CIPHER_AES_KWP 2 +#define PG_MAX_CIPHER_ID 2 + +/* AES128/192/256 various length definitions */ +#define PG_AES128_KEY_LEN (128 / 8) +#define PG_AES192_KEY_LEN (192 / 8) +#define PG_AES256_KEY_LEN (256 / 8) + +/* + * The encrypted data is a series of blocks of size. Initialization + * vector(IV) is the same size of cipher block. + */ +#define PG_AES_BLOCK_SIZE 16 +#define PG_AES_IV_SIZE (PG_AES_BLOCK_SIZE) + +#ifdef USE_OPENSSL +typedef EVP_CIPHER_CTX PgCipherCtx; +#else +typedef void PgCipherCtx; +#endif + +extern PgCipherCtx *pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, + bool enc); + +extern void pg_cipher_ctx_free(PgCipherCtx *ctx); +extern bool pg_cipher_encrypt(PgCipherCtx *ctx, int cipher, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *tag, const int taglen); +extern bool pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen, + const unsigned char *iv, const int ivlen, + unsigned char *intag, const int taglen); + +extern bool pg_cipher_keywrap(PgCipherCtx *ctx, + const unsigned char *plaintext, const int inlen, + unsigned char *ciphertext, int *outlen); +extern bool pg_cipher_keyunwrap(PgCipherCtx *ctx, + const unsigned char *ciphertext, const int inlen, + unsigned char *plaintext, int *outlen); + +extern int pg_cipher_blocksize(PgCipherCtx *ctx); + +#endif /* PG_CIPHER_H */ diff --git a/src/include/common/kmgr_utils.h b/src/include/common/kmgr_utils.h new file mode 100644 index 0000000000..ca264db728 --- /dev/null +++ b/src/include/common/kmgr_utils.h @@ -0,0 +1,94 @@ +/*------------------------------------------------------------------------- + * + * kmgr_utils.h + * Declarations for utility function for file encryption key + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/common/kmgr_utils.h + * + *------------------------------------------------------------------------- + */ +#ifndef KMGR_UTILS_H +#define KMGR_UTILS_H + +#include "common/cipher.h" + +/* Current version number */ +#define KMGR_VERSION 1 + +/* + * Directories where cluster file encryption keys reside within PGDATA. + */ +#define KMGR_DIR "pg_cryptokeys" +#define KMGR_DIR_PID KMGR_DIR"/pg_alterckey.pid" +#define LIVE_KMGR_DIR KMGR_DIR"/live" +/* used during cluster key rotation */ +#define NEW_KMGR_DIR KMGR_DIR"/new" +#define OLD_KMGR_DIR KMGR_DIR"/old" + +/* CryptoKey file name is keys id */ +#define CryptoKeyFilePath(path, dir, id) \ + snprintf((path), MAXPGPATH, "%s/%s.wkey", (dir), (wkey_filenames[id])) + +/* + * Identifiers of internal keys. + */ +#define KMGR_KEY_ID_REL 0 +#define KMGR_KEY_ID_WAL 1 +#define KMGR_NUM_DATA_KEYS 2 + +/* We always, today, use a 256-bit AES key. */ +#define KMGR_CLUSTER_KEY_LEN PG_AES256_KEY_LEN + +/* double for hex format, plus some for spaces, \r,\n, and null byte */ +#define ALLOC_KMGR_CLUSTER_KEY_LEN (KMGR_CLUSTER_KEY_LEN * 2 + 10 + 2 + 1) + +/* Maximum length of key the key manager can store */ +#define KMGR_MAX_KEY_LEN 256 +#define KMGR_MAX_KEY_LEN_BYTES KMGR_MAX_KEY_LEN / 8 + + +/* + * Cryptographic key data structure. + * + * This is the structure we use to write out the encrypted keys and + * which we use to store the keys in shared memory. + * + * Note that wrapping this structure results in an encrypted byte + * string which is what we actually write and then read back in. + * + * klen is the key length in bytes + * key is the encryption key of klen length + */ +typedef struct CryptoKey +{ + int klen; /* key length in bytes */ + unsigned char key[KMGR_MAX_KEY_LEN_BYTES]; +} CryptoKey; + +/* Encryption method array */ +typedef struct encryption_method +{ + const char *name; + const int bit_length; +} encryption_method; + +#define NUM_ENCRYPTION_METHODS 4 +#define DISABLED_ENCRYPTION_METHOD 0 +#define DEFAULT_ENABLED_ENCRYPTION_METHOD 1 + +extern encryption_method encryption_methods[NUM_ENCRYPTION_METHODS]; +extern char *wkey_filenames[KMGR_NUM_DATA_KEYS]; + +extern bool kmgr_wrap_data_key(PgCipherCtx *ctx, CryptoKey *in, unsigned char *out, int *outlen); +extern bool kmgr_unwrap_data_key(PgCipherCtx *ctx, unsigned char *in, int inlen, CryptoKey *out); +extern bool kmgr_verify_cluster_key(unsigned char *cluster_key, + unsigned char **in_keys, int *klens, CryptoKey *out_keys); +extern int kmgr_run_cluster_key_command(char *cluster_key_command, + char *buf, int size, char *dir, + int terminal_fd); +extern void kmgr_read_wrapped_data_keys(const char *path, unsigned char **keys, + int *key_lens); + +#endif /* KMGR_UTILS_H */ diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h index 6f2d5612e0..4b049f2ab4 100644 --- a/src/include/utils/wait_event.h +++ b/src/include/utils/wait_event.h @@ -180,6 +180,9 @@ typedef enum WAIT_EVENT_DATA_FILE_WRITE, WAIT_EVENT_DSM_ALLOCATE, WAIT_EVENT_DSM_FILL_ZERO_WRITE, + WAIT_EVENT_KEY_FILE_READ, + WAIT_EVENT_KEY_FILE_WRITE, + WAIT_EVENT_KEY_FILE_SYNC, WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ, WAIT_EVENT_LOCK_FILE_ADDTODATADIR_SYNC, WAIT_EVENT_LOCK_FILE_ADDTODATADIR_WRITE, diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm index 83a3e40425..7c448183f4 100644 --- a/src/tools/msvc/Mkvcbuild.pm +++ b/src/tools/msvc/Mkvcbuild.pm @@ -136,19 +136,21 @@ sub mkvcbuild archive.c base64.c checksum_helper.c compression.c config_info.c controldata_utils.c d2s.c encnames.c exec.c f2s.c file_perm.c file_utils.c hashfn.c ip.c jsonapi.c - keywords.c kwlookup.c link-canary.c md5_common.c + keywords.c kmgr_utils.c kwlookup.c link-canary.c md5_common.c pg_get_line.c pg_lzcompress.c pg_prng.c pgfnames.c psprintf.c relpath.c rmtree.c saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c username.c wait_error.c wchar.c); if ($solution->{options}->{openssl}) { + push(@pgcommonallfiles, 'cipher_openssl.c'); push(@pgcommonallfiles, 'cryptohash_openssl.c'); push(@pgcommonallfiles, 'hmac_openssl.c'); push(@pgcommonallfiles, 'protocol_openssl.c'); } else { + push(@pgcommonallfiles, 'cipher.c'); push(@pgcommonallfiles, 'cryptohash.c'); push(@pgcommonallfiles, 'hmac.c'); push(@pgcommonallfiles, 'md5.c'); -- 2.31.1 [application/octet-stream] v2-0007-cfe-07-bin_over_cfe-06-backend-squash-commit.patch (33.3K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/7-v2-0007-cfe-07-bin_over_cfe-06-backend-squash-commit.patch) download | inline diff: From 00884d21abdf24cc344515521660fc4360e1bbde Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:56 -0400 Subject: [PATCH v2 07/12] cfe-07-bin_over_cfe-06-backend squash commit --- doc/src/sgml/ref/initdb.sgml | 47 +++++++++ doc/src/sgml/ref/pg_ctl-ref.sgml | 14 +++ doc/src/sgml/ref/pgupgrade.sgml | 20 +++- doc/src/sgml/ref/postgres-ref.sgml | 13 +++ src/bin/initdb/initdb.c | 125 ++++++++++++++++++++++-- src/bin/pg_controldata/pg_controldata.c | 3 + src/bin/pg_ctl/pg_ctl.c | 59 +++++++++-- src/bin/pg_resetwal/pg_resetwal.c | 3 + src/bin/pg_rewind/filemap.c | 8 ++ src/bin/pg_upgrade/check.c | 34 +++++++ src/bin/pg_upgrade/controldata.c | 41 +++++++- src/bin/pg_upgrade/file.c | 2 + src/bin/pg_upgrade/option.c | 7 +- src/bin/pg_upgrade/pg_upgrade.h | 4 + src/bin/pg_upgrade/server.c | 2 +- 15 files changed, 361 insertions(+), 21 deletions(-) diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml index 8158896298..06b10fbaaf 100644 --- a/doc/src/sgml/ref/initdb.sgml +++ b/doc/src/sgml/ref/initdb.sgml @@ -186,6 +186,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry id="app-initdb-cluster-key-command" xreflabel="cluster key command"> + <term><option>-c <replaceable class="parameter">command</replaceable></option></term> + <term><option>--cluster-key-command=<replaceable class="parameter">command</replaceable></option></term> + <listitem> + <para> + This option specifies an external command to obtain the cluster-level + key for cluster file encryption during server initialization and + server start; see <xref linkend="guc-cluster-key-command"/> for details. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-D <replaceable class="parameter">directory</replaceable></option></term> <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term> @@ -257,6 +269,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry id="app-initdb-file-encryption-method" + xreflabel="file encryption"> + <term><option>-K <replaceable class="parameter">encryption_method</replaceable></option></term> + <term><option>--file-encryption-method=<replaceable class="parameter">encryption_method</replaceable></option></term> + <listitem> + <para> + Specifies the cluster file encryption method. The + value values are <literal>AES128</literal> (the default), <literal>AES192</literal>, and <literal>AES256</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>--locale=<replaceable>locale</replaceable></option></term> <listitem> @@ -344,6 +368,17 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem> + <para> + Allows the <option>--cluster-key-command</option> command + to prompt for a passphrase or PIN. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-S</option></term> <term><option>--sync-only</option></term> @@ -369,6 +404,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-u <replaceable>datadir</replaceable></option></term> + <term><option>--copy-encryption-keys=<replaceable>datadir</replaceable></option></term> + <listitem> + <para> + Copies cluster file encryption keys from another cluster; required + when using <application>pg_upgrade</application> on a cluster + with cluster file encryption enabled. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-U <replaceable class="parameter">username</replaceable></option></term> <term><option>--username=<replaceable class="parameter">username</replaceable></option></term> diff --git a/doc/src/sgml/ref/pg_ctl-ref.sgml b/doc/src/sgml/ref/pg_ctl-ref.sgml index 46906966eb..e3705c724e 100644 --- a/doc/src/sgml/ref/pg_ctl-ref.sgml +++ b/doc/src/sgml/ref/pg_ctl-ref.sgml @@ -38,6 +38,7 @@ PostgreSQL documentation <arg choice="opt"><option>-s</option></arg> <arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg> <arg choice="opt"><option>-p</option> <replaceable>path</replaceable></arg> + <arg choice="opt"><option>-R</option></arg> <arg choice="opt"><option>-c</option></arg> </cmdsynopsis> @@ -72,6 +73,7 @@ PostgreSQL documentation <arg choice="opt"><option>-t</option> <replaceable>seconds</replaceable></arg> <arg choice="opt"><option>-s</option></arg> <arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg> + <arg choice="opt"><option>-R</option></arg> <arg choice="opt"><option>-c</option></arg> </cmdsynopsis> @@ -373,6 +375,18 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem> + <para> + Allows <option>ssl_passphrase_command</option> or + <option>cluster_key_command</option> to prompt for a passphrase + or PIN. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-s</option></term> <term><option>--silent</option></term> diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index 8f7a3025c3..4362e551c9 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -182,6 +182,15 @@ PostgreSQL documentation </para></listitem> </varlistentry> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem><para>allows <option>ssl_passphrase_command</option> or + <option>cluster_key_command</option> to prompt for a passphrase + or PIN. + </para></listitem> + </varlistentry> + <varlistentry> <term><option>-s</option> <replaceable>dir</replaceable></term> <term><option>--socketdir=</option><replaceable>dir</replaceable></term> @@ -309,7 +318,9 @@ make prefix=/usr/local/pgsql.new install Again, use compatible <command>initdb</command> flags that match the old cluster. Many prebuilt installers do this step automatically. There is no need to - start the new cluster. + start the new cluster. If upgrading a cluster that uses + cluster file encryption, the <command>initdb</command> option + <option>--copy-encryption-keys</option> must be specified. </para> </step> @@ -842,6 +853,13 @@ psql --username=postgres --file=script.sql postgres is down. </para> + <para> + If the old cluster uses file encryption, the new cluster must use + the same keys, so <command>pg_upgrade</command> copies them to the + new cluster. It is necessary to initialize the new cluster with + the same <varname>cluster_key_command</varname> and the same + file encryption method. + </para> </refsect1> <refsect1> diff --git a/doc/src/sgml/ref/postgres-ref.sgml b/doc/src/sgml/ref/postgres-ref.sgml index 55a3f6c69d..f0fa52ebdc 100644 --- a/doc/src/sgml/ref/postgres-ref.sgml +++ b/doc/src/sgml/ref/postgres-ref.sgml @@ -305,6 +305,19 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-R <replaceable class="parameter">file-descriptor</replaceable></option></term> + <listitem> + <para> + Makes <command>postgres</command> prompt for a passphrase or PIN + from the specified open numeric file descriptor. The descriptor + is closed after the key is read. The file descriptor number + <literal>-1</literal> duplicates standard error for the terminal; + this is useful for single-user mode. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-s</option></term> <listitem> diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index f61a043055..8c7e1e39c5 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -68,6 +68,7 @@ #include "catalog/pg_database_d.h" /* pgrminclude ignore */ #include "common/file_perm.h" #include "common/file_utils.h" +#include "common/kmgr_utils.h" #include "common/logging.h" #include "common/pg_prng.h" #include "common/restricted_token.h" @@ -147,11 +148,16 @@ static bool noclean = false; static bool noinstructions = false; static bool do_sync = true; static bool sync_only = false; +static bool pass_terminal_fd = false; +static char *term_fd_opt = NULL; +static int file_encryption_method = DISABLED_ENCRYPTION_METHOD; static bool show_setting = false; static bool data_checksums = false; static char *xlog_dir = NULL; static char *str_wal_segment_size_mb = NULL; static int wal_segment_size_mb; +static char *cluster_key_cmd = NULL; +static char *old_key_datadir = NULL; /* internal vars */ @@ -214,6 +220,7 @@ static const char *const subdirs[] = { "global", "pg_wal/archive_status", "pg_commit_ts", + "pg_cryptokeys", "pg_dynshmem", "pg_notify", "pg_serial", @@ -904,12 +911,13 @@ test_config_settings(void) test_buffs = MIN_BUFS_FOR_CONNS(test_conns); snprintf(cmd, sizeof(cmd), - "\"%s\" --check %s %s " + "\"%s\" --check -x0 %s %s %s " "-c max_connections=%d " "-c shared_buffers=%d " "-c dynamic_shared_memory_type=%s " "< \"%s\" > \"%s\" 2>&1", backend_exec, boot_options, extra_options, + term_fd_opt ? term_fd_opt : "", test_conns, test_buffs, dynamic_shared_memory_type, DEVNULL, DEVNULL); @@ -941,12 +949,13 @@ test_config_settings(void) } snprintf(cmd, sizeof(cmd), - "\"%s\" --check %s %s " + "\"%s\" --check %s %s %s " "-c max_connections=%d " "-c shared_buffers=%d " "-c dynamic_shared_memory_type=%s " "< \"%s\" > \"%s\" 2>&1", backend_exec, boot_options, extra_options, + term_fd_opt ? term_fd_opt : "", n_connections, test_buffs, dynamic_shared_memory_type, DEVNULL, DEVNULL); @@ -1133,6 +1142,13 @@ setup_config(void) "password_encryption = md5"); } + if (cluster_key_cmd) + { + snprintf(repltok, sizeof(repltok), "cluster_key_command = '%s'", + escape_quotes(cluster_key_cmd)); + conflines = replace_token(conflines, "#cluster_key_command = ''", repltok); + } + /* * If group access has been enabled for the cluster then it makes sense to * ensure that the log files also allow group access. Otherwise a backup @@ -1321,11 +1337,17 @@ bootstrap_template1(void) unsetenv("PGCLIENTENCODING"); snprintf(cmd, sizeof(cmd), - "\"%s\" --boot -X %d %s %s %s %s", + "\"%s\" --boot -X %d %s %s %s %s %s %s %s %s %s", backend_exec, wal_segment_size_mb * (1024 * 1024), data_checksums ? "-k" : "", - boot_options, extra_options, + cluster_key_cmd ? "-K" : "", + cluster_key_cmd ? encryption_methods[file_encryption_method].name : "", + old_key_datadir ? "-u" : "", + old_key_datadir ? old_key_datadir : "", + boot_options, + extra_options, + term_fd_opt ? term_fd_opt : "", debug ? "-d 5" : ""); @@ -2159,18 +2181,26 @@ usage(const char *progname) printf(_(" -T, --text-search-config=CFG\n" " default text search configuration\n")); printf(_(" -U, --username=NAME database superuser name\n")); - printf(_(" -W, --pwprompt prompt for a password for the new superuser\n")); + printf(_(" -W, --pwprompt prompt for the new superuser password\n")); printf(_(" -X, --waldir=WALDIR location for the write-ahead log directory\n")); printf(_(" --wal-segsize=SIZE size of WAL segments, in megabytes\n")); printf(_("\nLess commonly used options:\n")); + printf(_(" -c, --cluster-key-command=COMMAND\n" + " enable cluster file encryption and set command\n" + " to obtain the cluster key\n")); printf(_(" -d, --debug generate lots of debugging output\n")); printf(_(" --discard-caches set debug_discard_caches=1\n")); + printf(_(" -K, --file-encryption-method=METHOD\n" + " cluster file encryption method\n")); printf(_(" -L DIRECTORY where to find the input files\n")); printf(_(" -n, --no-clean do not clean up after errors\n")); printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n")); + printf(_(" -R, --authprompt prompt for a passphrase or PIN\n")); printf(_(" --no-instructions do not print instructions for next steps\n")); printf(_(" -s, --show show internal settings\n")); printf(_(" -S, --sync-only only sync database files to disk, then exit\n")); + printf(_(" -u, --copy-encryption-keys=DATADIR\n" + " copy the file encryption key from another cluster\n")); printf(_("\nOther options:\n")); printf(_(" -V, --version output version information, then exit\n")); printf(_(" -?, --help show this help, then exit\n")); @@ -2700,6 +2730,23 @@ initialize_data_directory(void) /* Top level PG_VERSION is checked by bootstrapper, so make it first */ write_version_file(NULL); + if (pass_terminal_fd) + { +#ifndef WIN32 + int terminal_fd = open("/dev/tty", O_RDWR, 0); +#else + int terminal_fd = open("CONOUT$", O_RDWR, 0); +#endif + + if (terminal_fd < 0) + { + pg_log_error(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + term_fd_opt = psprintf("-R %d", terminal_fd); + } + /* Select suitable configuration settings */ set_null_conf(); test_config_settings(); @@ -2723,8 +2770,9 @@ initialize_data_directory(void) fflush(stdout); snprintf(cmd, sizeof(cmd), - "\"%s\" %s %s template1 >%s", + "\"%s\" %s %s %s template1 >%s", backend_exec, backend_options, extra_options, + term_fd_opt ? term_fd_opt : "", DEVNULL); PG_CMD_OPEN; @@ -2802,10 +2850,14 @@ main(int argc, char *argv[]) {"waldir", required_argument, NULL, 'X'}, {"wal-segsize", required_argument, NULL, 12}, {"data-checksums", no_argument, NULL, 'k'}, + {"authprompt", no_argument, NULL, 'R'}, + {"file-encryption-method", required_argument, NULL, 'K'}, {"allow-group-access", no_argument, NULL, 'g'}, {"discard-caches", no_argument, NULL, 14}, {"locale-provider", required_argument, NULL, 15}, {"icu-locale", required_argument, NULL, 16}, + {"cluster-key-command", required_argument, NULL, 'c'}, + {"copy-encryption-keys", required_argument, NULL, 'u'}, {NULL, 0, NULL, 0} }; @@ -2847,7 +2899,7 @@ main(int argc, char *argv[]) /* process command-line options */ - while ((c = getopt_long(argc, argv, "A:dD:E:gkL:nNsST:U:WX:", long_options, &option_index)) != -1) + while ((c = getopt_long(argc, argv, "A:c:dD:E:gkK:L:nNRsST:u:U:WX:", long_options, &option_index)) != -1) { switch (c) { @@ -2893,6 +2945,28 @@ main(int argc, char *argv[]) case 'N': do_sync = false; break; + case 'R': + pass_terminal_fd = true; + break; + case 'K': + { + int i; + + /* method 0/disabled cannot be specified */ + for (i = DISABLED_ENCRYPTION_METHOD + 1; + i < NUM_ENCRYPTION_METHODS; i++) + if (pg_strcasecmp(optarg, encryption_methods[i].name) == 0) + { + file_encryption_method = i; + break; + } + if (i == NUM_ENCRYPTION_METHODS) + { + fprintf(stderr, _("invalid cluster encryption method\n")); + exit(1); + } + } + break; case 'S': sync_only = true; break; @@ -2929,6 +3003,12 @@ main(int argc, char *argv[]) case 9: pwfilename = pg_strdup(optarg); break; + case 'c': + cluster_key_cmd = pg_strdup(optarg); + break; + case 'u': + old_key_datadir = pg_strdup(optarg); + break; case 's': show_setting = true; break; @@ -3014,6 +3094,32 @@ main(int argc, char *argv[]) if (pwprompt && pwfilename) pg_fatal("password prompt and password file cannot be specified together"); +#ifndef USE_OPENSSL + if (cluster_key_cmd) + { + pg_log_error("cluster file encryption is not supported because OpenSSL is not supported by this build"); + exit(1); + } +#endif + + if (old_key_datadir != NULL && cluster_key_cmd == NULL) + { + pg_log_error("copying encryption keys requires the cluster key command to be specified"); + exit(1); + } + + if (file_encryption_method != DISABLED_ENCRYPTION_METHOD && + cluster_key_cmd == NULL) + { + pg_log_error("a file encryption method requires the cluster key command to be specified"); + exit(1); + } + + /* set the default */ + if (file_encryption_method == DISABLED_ENCRYPTION_METHOD && + cluster_key_cmd != NULL) + file_encryption_method = DEFAULT_ENABLED_ENCRYPTION_METHOD; + check_authmethod_unspecified(&authmethodlocal); check_authmethod_unspecified(&authmethodhost); @@ -3072,6 +3178,11 @@ main(int argc, char *argv[]) else printf(_("Data page checksums are disabled.\n")); + if (cluster_key_cmd) + printf(_("Cluster file encryption is enabled.\n")); + else + printf(_("Cluster file encryption is disabled.\n")); + if (pwprompt || pwfilename) get_su_pwd(); diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index c390ec51ce..d25d13c992 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -25,6 +25,7 @@ #include "access/xlog_internal.h" #include "catalog/pg_control.h" #include "common/controldata_utils.h" +#include "common/kmgr_utils.h" #include "common/logging.h" #include "getopt_long.h" #include "pg_getopt.h" @@ -328,5 +329,7 @@ main(int argc, char *argv[]) ControlFile->data_checksum_version); printf(_("Mock authentication nonce: %s\n"), mock_auth_nonce_str); + printf(_("File encryption method: %s\n"), + encryption_methods[ControlFile->file_encryption_method].name); return 0; } diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c index a509fc9109..d90c07db10 100644 --- a/src/bin/pg_ctl/pg_ctl.c +++ b/src/bin/pg_ctl/pg_ctl.c @@ -74,6 +74,7 @@ typedef enum static bool do_wait = true; static int wait_seconds = DEFAULT_WAIT; static bool wait_seconds_arg = false; +static bool pass_terminal_fd = false; static bool silent_mode = false; static ShutdownMode shutdown_mode = FAST_MODE; static int sig = SIGINT; /* default */ @@ -439,7 +440,7 @@ free_readfile(char **optlines) static pid_t start_postmaster(void) { - char *cmd; + char *cmd, *term_fd_opt = NULL; #ifndef WIN32 pid_t pm_pid; @@ -467,6 +468,19 @@ start_postmaster(void) /* fork succeeded, in child */ + if (pass_terminal_fd) + { + int terminal_fd = open("/dev/tty", O_RDWR, 0); + + if (terminal_fd < 0) + { + write_stderr(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + term_fd_opt = psprintf(" -R %d", terminal_fd); + } + /* * If possible, detach the postmaster process from the launching process * group and make it a group leader, so that it doesn't get signaled along @@ -487,12 +501,14 @@ start_postmaster(void) * has the same PID as the current child process. */ if (log_file != NULL) - cmd = psprintf("exec \"%s\" %s%s < \"%s\" >> \"%s\" 2>&1", + cmd = psprintf("exec \"%s\" %s%s%s < \"%s\" >> \"%s\" 2>&1", exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL, log_file); else - cmd = psprintf("exec \"%s\" %s%s < \"%s\" 2>&1", - exec_path, pgdata_opt, post_opts, DEVNULL); + cmd = psprintf("exec \"%s\" %s%s%s < \"%s\" 2>&1", + exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL); (void) execl("/bin/sh", "/bin/sh", "-c", cmd, (char *) NULL); @@ -513,6 +529,21 @@ start_postmaster(void) PROCESS_INFORMATION pi; const char *comspec; + if (pass_terminal_fd) + { + /* Hopefully we can read and write CONOUT, see simple_prompt() XXX */ + /* Do CreateRestrictedProcess() children even inherit open file descriptors? XXX */ + int terminal_fd = open("CONOUT$", O_RDWR, 0); + + if (terminal_fd < 0) + { + write_stderr(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + term_fd_opt = psprintf(" -R %d", terminal_fd); + } + /* Find CMD.EXE location using COMSPEC, if it's set */ comspec = getenv("COMSPEC"); if (comspec == NULL) @@ -553,12 +584,14 @@ start_postmaster(void) else close(fd); - cmd = psprintf("\"%s\" /C \"\"%s\" %s%s < \"%s\" >> \"%s\" 2>&1\"", - comspec, exec_path, pgdata_opt, post_opts, DEVNULL, log_file); + cmd = psprintf("\"%s\" /C \"\"%s\" %s%s%s < \"%s\" >> \"%s\" 2>&1\"", + comspec, exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL, log_file); } else - cmd = psprintf("\"%s\" /C \"\"%s\" %s%s < \"%s\" 2>&1\"", - comspec, exec_path, pgdata_opt, post_opts, DEVNULL); + cmd = psprintf("\"%s\" /C \"\"%s\" %s%s%s < \"%s\" 2>&1\"", + comspec, exec_path, pgdata_opt, post_opts, + term_fd_opt ? term_fd_opt : "", DEVNULL); if (!CreateRestrictedProcess(cmd, &pi, false)) { @@ -689,7 +722,8 @@ wait_for_postmaster_start(pid_t pm_pid, bool do_checkpoint) } else #endif - print_msg("."); + if (!pass_terminal_fd) + print_msg("."); } pg_usleep(USEC_PER_SEC / WAITS_PER_SEC); @@ -2005,6 +2039,7 @@ do_help(void) printf(_(" -o, --options=OPTIONS command line options to pass to postgres\n" " (PostgreSQL server executable) or initdb\n")); printf(_(" -p PATH-TO-POSTGRES normally not necessary\n")); + printf(_(" -R, --authprompt prompt for a passphrase or PIN\n")); printf(_("\nOptions for stop or restart:\n")); printf(_(" -m, --mode=MODE MODE can be \"smart\", \"fast\", or \"immediate\"\n")); @@ -2200,6 +2235,7 @@ main(int argc, char **argv) {"mode", required_argument, NULL, 'm'}, {"pgdata", required_argument, NULL, 'D'}, {"options", required_argument, NULL, 'o'}, + {"authprompt", no_argument, NULL, 'R'}, {"silent", no_argument, NULL, 's'}, {"timeout", required_argument, NULL, 't'}, {"core-files", no_argument, NULL, 'c'}, @@ -2272,7 +2308,7 @@ main(int argc, char **argv) /* process command-line options */ while (optind < argc) { - while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:sS:t:U:wW", + while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:RsS:t:U:wW", long_options, &option_index)) != -1) { switch (c) @@ -2324,6 +2360,9 @@ main(int argc, char **argv) case 'P': register_password = pg_strdup(optarg); break; + case 'R': + pass_terminal_fd = true; + break; case 's': silent_mode = true; break; diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 089063f471..73cb048b3e 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -52,6 +52,7 @@ #include "common/controldata_utils.h" #include "common/fe_memutils.h" #include "common/file_perm.h" +#include "common/kmgr_utils.h" #include "common/logging.h" #include "common/restricted_token.h" #include "common/string.h" @@ -772,6 +773,8 @@ PrintControlValues(bool guessed) (ControlFile.float8ByVal ? _("by value") : _("by reference"))); printf(_("Data page checksum version: %u\n"), ControlFile.data_checksum_version); + printf(_("File encryption method: %s\n"), + encryption_methods[ControlFile.file_encryption_method].name); } diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index 269ed6446e..d0b71f972a 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -28,6 +28,7 @@ #include "catalog/pg_tablespace_d.h" #include "common/hashfn.h" +#include "common/kmgr_utils.h" #include "common/string.h" #include "datapagemap.h" #include "filemap.h" @@ -106,6 +107,13 @@ static const char *excludeDirContents[] = /* Contents removed on startup, see AsyncShmemInit(). */ "pg_notify", + /* + * Skip cryptographic keys. It's generally not a good idea to copy the + * cryptographic keys from source database because these might use + * different cluster key. + */ + KMGR_DIR, + /* * Old contents are loaded for possible debugging but are not required for * normal operation, see SerialInit(). diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c index f1bc1e6886..ae7aa601a7 100644 --- a/src/bin/pg_upgrade/check.c +++ b/src/bin/pg_upgrade/check.c @@ -11,6 +11,7 @@ #include "catalog/pg_authid_d.h" #include "catalog/pg_collation.h" +#include "common/kmgr_utils.h" #include "fe_utils/string_utils.h" #include "mb/pg_wchar.h" #include "pg_upgrade.h" @@ -30,6 +31,7 @@ static void check_for_composite_data_type_usage(ClusterInfo *cluster); static void check_for_reg_data_type_usage(ClusterInfo *cluster); static void check_for_jsonb_9_4_usage(ClusterInfo *cluster); static void check_for_pg_role_prefix(ClusterInfo *cluster); +static void check_for_cluster_key_failure(ClusterInfo *cluster); static void check_for_new_tablespace_dir(ClusterInfo *new_cluster); static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster); static char *get_canonical_locale_name(int category, const char *locale); @@ -160,6 +162,9 @@ check_and_dump_old_cluster(bool live_check) if (GET_MAJOR_VERSION(old_cluster.major_version) <= 905) check_for_pg_role_prefix(&old_cluster); + if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1400) + check_for_cluster_key_failure(&old_cluster); + if (GET_MAJOR_VERSION(old_cluster.major_version) == 904 && old_cluster.controldata.cat_ver < JSONB_FORMAT_CHANGE_CAT_VER) check_for_jsonb_9_4_usage(&old_cluster); @@ -190,6 +195,9 @@ check_new_cluster(void) check_loadable_libraries(); + if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1400) + check_for_cluster_key_failure(&new_cluster); + switch (user_opts.transfer_mode) { case TRANSFER_MODE_CLONE: @@ -1466,6 +1474,32 @@ check_for_user_defined_encoding_conversions(ClusterInfo *cluster) } +/* + * check_for_cluster_key_failure() + * + * Make sure there was no unrepaired pg_alterckey failure + */ +static void +check_for_cluster_key_failure(ClusterInfo *cluster) +{ + struct stat buffer; + + if (stat (KMGR_DIR_PID, &buffer) == 0) + { + if (cluster == &old_cluster) + pg_fatal("The source cluster had a pg_alterckey failure that needs repair or\n" + "pg_alterckey is running. Run pg_alterckey --repair or wait for it\n" + "to complete.\n"); + else + pg_fatal("The target cluster had a pg_alterckey failure that needs repair or\n" + "pg_alterckey is running. Run pg_alterckey --repair or wait for it\n" + "to complete.\n"); + } + + check_ok(); +} + + /* * get_canonical_locale_name * diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c index 018cd310f7..ee6a48438b 100644 --- a/src/bin/pg_upgrade/controldata.c +++ b/src/bin/pg_upgrade/controldata.c @@ -9,12 +9,18 @@ #include "postgres_fe.h" +#include <dirent.h> #include <ctype.h> #include "pg_upgrade.h" #include "common/string.h" +#include "access/xlog_internal.h" +#include "common/controldata_utils.h" +#include "common/file_utils.h" +#include "common/kmgr_utils.h" + /* * get_control_data() * @@ -62,6 +68,7 @@ get_control_data(ClusterInfo *cluster, bool live_check) bool got_date_is_int = false; bool got_data_checksum_version = false; bool got_cluster_state = false; + int got_file_encryption_method = false; char *lc_collate = NULL; char *lc_ctype = NULL; char *lc_monetary = NULL; @@ -203,6 +210,13 @@ get_control_data(ClusterInfo *cluster, bool live_check) got_data_checksum_version = true; } + /* Only in <= 14 */ + if (GET_MAJOR_VERSION(cluster->major_version) <= 1400) + { + cluster->controldata.file_encryption_method = DISABLED_ENCRYPTION_METHOD; + got_file_encryption_method = true; + } + /* we have the result of cmd in "output". so parse it line by line now */ while (fgets(bufin, sizeof(bufin), output)) { @@ -498,6 +512,18 @@ get_control_data(ClusterInfo *cluster, bool live_check) cluster->controldata.data_checksum_version = str2uint(p); got_data_checksum_version = true; } + else if ((p = strstr(bufin, "Cluster file encryption method:")) != NULL) + { + p = strchr(p, ':'); + + if (p == NULL || strlen(p) <= 1) + pg_fatal("%d: controldata retrieval problem\n", __LINE__); + + p++; /* remove ':' char */ + /* used later for contrib check */ + cluster->controldata.file_encryption_method = atoi(p); + got_file_encryption_method = true; + } } pclose(output); @@ -566,7 +592,8 @@ get_control_data(ClusterInfo *cluster, bool live_check) !got_index || !got_toast || (!got_large_object && cluster->controldata.ctrl_ver >= LARGE_OBJECT_SIZE_PG_CONTROL_VER) || - !got_date_is_int || !got_data_checksum_version) + !got_date_is_int || !got_data_checksum_version || + !got_file_encryption_method) { if (cluster == &old_cluster) pg_log(PG_REPORT, @@ -635,6 +662,10 @@ get_control_data(ClusterInfo *cluster, bool live_check) if (!got_data_checksum_version) pg_log(PG_REPORT, " data checksum version"); + /* value added in Postgres 14 */ + if (!got_file_encryption_method) + pg_log(PG_REPORT, " file encryption method\n"); + pg_fatal("Cannot continue without required control information, terminating"); } } @@ -699,6 +730,14 @@ check_control_data(ControlData *oldctrl, pg_fatal("old cluster uses data checksums but the new one does not"); else if (oldctrl->data_checksum_version != newctrl->data_checksum_version) pg_fatal("old and new cluster pg_controldata checksum versions do not match"); + + /* + * We cannot upgrade if the old cluster file encryption method + * doesn't match the new one. + */ + if (oldctrl->file_encryption_method != newctrl->file_encryption_method) + pg_fatal("old and new clusters use different file encryption methods or\n" + "one cluster uses encryption and the other does not"); } diff --git a/src/bin/pg_upgrade/file.c b/src/bin/pg_upgrade/file.c index 079fbda838..29fd003318 100644 --- a/src/bin/pg_upgrade/file.c +++ b/src/bin/pg_upgrade/file.c @@ -11,6 +11,7 @@ #include <sys/stat.h> #include <fcntl.h> +#include <dirent.h> #ifdef HAVE_COPYFILE_H #include <copyfile.h> #endif @@ -21,6 +22,7 @@ #include "access/visibilitymapdefs.h" #include "common/file_perm.h" +#include "common/file_utils.h" #include "pg_upgrade.h" #include "storage/bufpage.h" #include "storage/checksum.h" diff --git a/src/bin/pg_upgrade/option.c b/src/bin/pg_upgrade/option.c index 3f719f1762..b6285d68ed 100644 --- a/src/bin/pg_upgrade/option.c +++ b/src/bin/pg_upgrade/option.c @@ -52,6 +52,7 @@ parseCommandLine(int argc, char *argv[]) {"check", no_argument, NULL, 'c'}, {"link", no_argument, NULL, 'k'}, {"retain", no_argument, NULL, 'r'}, + {"authprompt", no_argument, NULL, 'R'}, {"jobs", required_argument, NULL, 'j'}, {"socketdir", required_argument, NULL, 's'}, {"verbose", no_argument, NULL, 'v'}, @@ -99,7 +100,7 @@ parseCommandLine(int argc, char *argv[]) if (os_user_effective_id == 0) pg_fatal("%s: cannot be run as root", os_info.progname); - while ((option = getopt_long(argc, argv, "d:D:b:B:cj:kNo:O:p:P:rs:U:v", + while ((option = getopt_long(argc, argv, "d:D:b:B:cj:kNo:O:p:P:rRs:U:v", long_options, &optindex)) != -1) { switch (option) @@ -176,6 +177,10 @@ parseCommandLine(int argc, char *argv[]) log_opts.retain = true; break; + case 'R': + user_opts.pass_terminal_fd = true; + break; + case 's': user_opts.socketdir = pg_strdup(optarg); break; diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index 31589b0fdc..878f8e8032 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -12,6 +12,7 @@ #include "common/relpath.h" #include "libpq-fe.h" +#include "common/kmgr_utils.h" /* For now, pg_upgrade does not use common/logging.c; use our own pg_fatal */ #undef pg_fatal @@ -218,6 +219,7 @@ typedef struct bool date_is_int; bool float8_pass_by_value; uint32 data_checksum_version; + int file_encryption_method; } ControlData; /* @@ -296,6 +298,8 @@ typedef struct transferMode transfer_mode; /* copy files or link them? */ int jobs; /* number of processes/threads to use */ char *socketdir; /* directory to use for Unix sockets */ + bool ind_coll_unknown; /* mark unknown index collation versions */ + bool pass_terminal_fd; /* pass -R to pg_ctl? */ } UserOpts; typedef struct diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c index 9a1a20fb27..0775010379 100644 --- a/src/bin/pg_upgrade/server.c +++ b/src/bin/pg_upgrade/server.c @@ -239,7 +239,7 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error) * vacuumdb --freeze actually freezes the tuples. */ snprintf(cmd, sizeof(cmd), - "\"%s/pg_ctl\" -w -l \"%s/%s\" -D \"%s\" -o \"-p %d -b%s %s%s\" start", + "\"%s/pg_ctl\" -w -l \"%s/%s\" -D \"%s\" -o \"-p %d%s -b %s%s\" start", cluster->bindir, log_opts.logdir, SERVER_LOG_FILE, cluster->pgconfig, cluster->port, -- 2.31.1 [application/octet-stream] v2-0006-cfe-06-backend_over_cfe-05-crypto-squash-commit.patch (23.0K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/8-v2-0006-cfe-06-backend_over_cfe-05-crypto-squash-commit.patch) download | inline diff: From 85cbe762e5c7a74d0e3e02aa40ff9142e79b668e Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:53 -0400 Subject: [PATCH v2 06/12] cfe-06-backend_over_cfe-05-crypto squash commit --- doc/src/sgml/storage.sgml | 5 +++ src/backend/Makefile | 2 +- src/backend/access/transam/xlog.c | 33 +++++++++++++++++++ src/backend/backup/basebackup.c | 5 +++ src/backend/bootstrap/bootstrap.c | 29 +++++++++++++++- src/backend/crypto/kmgr.c | 3 ++ src/backend/libpq/be-secure-common.c | 14 ++++++++ src/backend/main/main.c | 3 ++ src/backend/postmaster/postmaster.c | 13 +++++++- src/backend/storage/ipc/ipci.c | 3 ++ src/backend/storage/lmgr/lwlocknames.txt | 1 + src/backend/tcop/postgres.c | 28 +++++++++++++++- src/backend/utils/activity/wait_event.c | 9 +++++ src/backend/utils/misc/guc_tables.c | 25 ++++++++++++++ src/backend/utils/misc/pg_controldata.c | 11 +++++-- src/backend/utils/misc/postgresql.conf.sample | 5 +++ src/include/access/xlog.h | 5 +++ src/include/catalog/pg_control.h | 5 ++- src/include/catalog/pg_proc.dat | 6 ++-- src/include/postmaster/postmaster.h | 2 ++ src/include/utils/guc_tables.h | 1 + 21 files changed, 197 insertions(+), 11 deletions(-) diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index e5b9f3f1ff..f9dbf04735 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -77,6 +77,11 @@ Item <entry>Subdirectory containing transaction commit timestamp data</entry> </row> +<row> + <entry><filename>pg_cryptokeys</filename></entry> + <entry>Subdirectory containing file encryption keys</entry> +</row> + <row> <entry><filename>pg_dynshmem</filename></entry> <entry>Subdirectory containing files used by the dynamic shared memory diff --git a/src/backend/Makefile b/src/backend/Makefile index 1a5239b822..4b8cdd1b44 100644 --- a/src/backend/Makefile +++ b/src/backend/Makefile @@ -22,7 +22,7 @@ SUBDIRS = access backup bootstrap catalog parser commands executor \ main nodes optimizer partitioning port postmaster \ regex replication rewrite \ statistics storage tcop tsearch utils $(top_builddir)/src/timezone \ - jit + jit crypto include $(srcdir)/common.mk diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index be54c23187..e247376ead 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -69,6 +69,7 @@ #include "catalog/pg_database.h" #include "common/controldata_utils.h" #include "common/file_utils.h" +#include "crypto/kmgr.h" #include "executor/instrument.h" #include "miscadmin.h" #include "pg_trace.h" @@ -76,6 +77,7 @@ #include "port/atomics.h" #include "port/pg_iovec.h" #include "postmaster/bgwriter.h" +#include "postmaster/postmaster.h" #include "postmaster/startup.h" #include "postmaster/walwriter.h" #include "replication/logical.h" @@ -109,6 +111,7 @@ #include "utils/varlena.h" extern uint32 bootstrap_data_checksum_version; +extern int bootstrap_file_encryption_method; /* timeline ID to be used when bootstrapping */ #define BootstrapTimeLineID 1 @@ -3898,6 +3901,7 @@ InitControlFile(uint64 sysidentifier) ControlFile->wal_log_hints = wal_log_hints; ControlFile->track_commit_timestamp = track_commit_timestamp; ControlFile->data_checksum_version = bootstrap_data_checksum_version; + ControlFile->file_encryption_method = bootstrap_file_encryption_method; } static void @@ -4185,6 +4189,12 @@ ReadControlFile(void) /* Make the initdb settings visible as GUC variables, too */ SetConfigOption("data_checksums", DataChecksumsEnabled() ? "yes" : "no", PGC_INTERNAL, PGC_S_DYNAMIC_DEFAULT); + + StaticAssertStmt(lengthof(encryption_methods) == NUM_ENCRYPTION_METHODS, + "encryption_methods[] must match NUM_ENCRYPTION_METHODS"); + SetConfigOption("file_encryption_method", + encryption_methods[ControlFile->file_encryption_method].name, + PGC_INTERNAL, PGC_S_OVERRIDE); } /* @@ -4227,6 +4237,21 @@ DataChecksumsEnabled(void) return (ControlFile->data_checksum_version > 0); } +/* + * Is cluster file encryption enabled? + */ +int +GetFileEncryptionMethod(void) +{ + if (IsBootstrapProcessingMode()) + return bootstrap_file_encryption_method; + else + { + Assert(ControlFile != NULL); + return ControlFile->file_encryption_method; + } +} + /* * Returns a fake LSN for unlogged relations. * @@ -4806,6 +4831,14 @@ BootStrapXLOG(void) /* some additional ControlFile fields are set in WriteControlFile() */ WriteControlFile(); + BootStrapKmgr(); + + if (terminal_fd != -1) + { + close(terminal_fd); + terminal_fd = -1; + } + /* Bootstrap the commit log, too */ BootStrapCLOG(); BootStrapCommitTs(); diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 74fb529380..35e955e036 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -24,6 +24,7 @@ #include "backup/basebackup_target.h" #include "commands/defrem.h" #include "common/compression.h" +#include "common/kmgr_utils.h" #include "common/file_perm.h" #include "lib/stringinfo.h" #include "miscadmin.h" @@ -129,6 +130,10 @@ struct exclude_list_item */ static const char *const excludeDirContents[] = { + /* Skip temporary crypto key directories */ + NEW_KMGR_DIR, + OLD_KMGR_DIR, + /* * Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped * because extensions like pg_stat_statements store data there. diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 661ebacb0c..87918a4d28 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -33,6 +33,7 @@ #include "miscadmin.h" #include "nodes/makefuncs.h" #include "pg_getopt.h" +#include "postmaster/postmaster.h" /* TODO: verify we need this still */ #include "storage/bufmgr.h" #include "storage/bufpage.h" #include "storage/condition_variable.h" @@ -46,6 +47,8 @@ #include "utils/relmapper.h" uint32 bootstrap_data_checksum_version = 0; /* No checksum */ +int bootstrap_file_encryption_method = DISABLED_ENCRYPTION_METHOD; +char *bootstrap_old_key_datadir = NULL; /* disabled */ static void CheckerModeMain(void); @@ -221,7 +224,7 @@ BootstrapModeMain(int argc, char *argv[], bool check_only) argv++; argc--; - while ((flag = getopt(argc, argv, "B:c:d:D:Fkr:X:-:")) != -1) + while ((flag = getopt(argc, argv, "B:c:d:D:FkK:r:R:u:X:-:")) != -1) { switch (flag) { @@ -250,9 +253,33 @@ BootstrapModeMain(int argc, char *argv[], bool check_only) case 'k': bootstrap_data_checksum_version = PG_DATA_CHECKSUM_VERSION; break; + case 'K': + { + int i; + + /* method 0/disabled cannot be specified */ + for (i = DISABLED_ENCRYPTION_METHOD + 1; + i < NUM_ENCRYPTION_METHODS; i++) + if (pg_strcasecmp(optarg, encryption_methods[i].name) == 0) + { + bootstrap_file_encryption_method = i; + break; + } + if (i == NUM_ENCRYPTION_METHODS) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid encryption method specified"))); + } + break; case 'r': strlcpy(OutputFileName, optarg, MAXPGPATH); break; + case 'R': + terminal_fd = atoi(optarg); + break; + case 'u': + bootstrap_old_key_datadir = pstrdup(optarg); + break; case 'X': { int WalSegSz = strtoul(optarg, NULL, 0); diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c index 4b74961323..a83effbca8 100644 --- a/src/backend/crypto/kmgr.c +++ b/src/backend/crypto/kmgr.c @@ -253,6 +253,9 @@ InitializeKmgr(void) char live_path[MAXPGPATH]; unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN]; + if (!FileEncryptionEnabled) + return; + #ifndef USE_OPENSSL ereport(ERROR, (errcode(ERRCODE_CONFIG_FILE_ERROR), diff --git a/src/backend/libpq/be-secure-common.c b/src/backend/libpq/be-secure-common.c index 2719ce1403..9357c80ea6 100644 --- a/src/backend/libpq/be-secure-common.c +++ b/src/backend/libpq/be-secure-common.c @@ -22,6 +22,7 @@ #include <sys/stat.h> #include <unistd.h> +#include "postmaster/postmaster.h" #include "common/string.h" #include "libpq/libpq.h" #include "storage/fd.h" @@ -61,6 +62,19 @@ run_ssl_passphrase_command(const char *prompt, bool is_server_start, char *buf, appendStringInfoString(&command, prompt); p++; break; + case 'R': + { + char fd_str[20]; + + if (terminal_fd == -1) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("ssl_passphrase_command referenced %%R, but -R not specified"))); + p++; + snprintf(fd_str, sizeof(fd_str), "%d", terminal_fd); + appendStringInfoString(&command, fd_str); + break; + } case '%': appendStringInfoChar(&command, '%'); p++; diff --git a/src/backend/main/main.c b/src/backend/main/main.c index f5da4260a1..ddf3fcb6fb 100644 --- a/src/backend/main/main.c +++ b/src/backend/main/main.c @@ -343,6 +343,7 @@ help(const char *progname) #endif printf(_(" -N MAX-CONNECT maximum number of allowed connections\n")); printf(_(" -p PORT port number to listen on\n")); + printf(_(" -R fd prompt for the cluster key\n")); printf(_(" -s show statistics after each query\n")); printf(_(" -S WORK-MEM set amount of memory for sorts (in kB)\n")); printf(_(" -V, --version output version information, then exit\n")); @@ -371,7 +372,9 @@ help(const char *progname) printf(_(" --boot selects bootstrapping mode (must be first argument)\n")); printf(_(" --check selects check mode (must be first argument)\n")); printf(_(" DBNAME database name (mandatory argument in bootstrapping mode)\n")); + printf(_(" -K LEN enable cluster file encryption with specified key bit length\n")); printf(_(" -r FILENAME send stdout and stderr to given file\n")); + printf(_(" -u DATADIR copy encryption keys from datadir\n")); printf(_("\nPlease read the documentation for the complete list of run-time\n" "configuration settings and how to set them on the command line or in\n" diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 0b637ba6a2..1673717da9 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -98,6 +98,7 @@ #include "common/ip.h" #include "common/pg_prng.h" #include "common/string.h" +#include "crypto/kmgr.h" #include "lib/ilist.h" #include "libpq/auth.h" #include "libpq/libpq.h" @@ -231,6 +232,7 @@ static int SendStop = false; /* still more option variables */ bool EnableSSL = false; +int terminal_fd = -1; int PreAuthDelay = 0; int AuthenticationTimeout = 60; @@ -697,7 +699,7 @@ PostmasterMain(int argc, char *argv[]) * tcop/postgres.c (the option sets should not conflict) and with the * common help() function in main/main.c. */ - while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:W:-:")) != -1) + while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:R:S:sTt:W:-:")) != -1) { switch (opt) { @@ -788,6 +790,10 @@ PostmasterMain(int argc, char *argv[]) /* only used by single-user backend */ break; + case 'R': + terminal_fd = atoi(optarg); + break; + case 'S': SetConfigOption("work_mem", optarg, PGC_POSTMASTER, PGC_S_ARGV); break; @@ -1347,6 +1353,11 @@ PostmasterMain(int argc, char *argv[]) pfree(rawstring); } + InitializeKmgr(); + + if (terminal_fd != -1) + close(terminal_fd); + /* * check that we have some socket to listen on */ diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index b204ecdbc3..061da11a58 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -25,6 +25,7 @@ #include "access/xlogprefetcher.h" #include "access/xlogrecovery.h" #include "commands/async.h" +#include "crypto/kmgr.h" #include "miscadmin.h" #include "pgstat.h" #include "postmaster/autovacuum.h" @@ -142,6 +143,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, SyncScanShmemSize()); size = add_size(size, AsyncShmemSize()); size = add_size(size, StatsShmemSize()); + size = add_size(size, KmgrShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -294,6 +296,7 @@ CreateSharedMemoryAndSemaphores(void) SyncScanShmemInit(); AsyncShmemInit(); StatsShmemInit(); + KmgrShmemInit(); #ifdef EXEC_BACKEND diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt index 6c7cf6c295..56ac32407f 100644 --- a/src/backend/storage/lmgr/lwlocknames.txt +++ b/src/backend/storage/lmgr/lwlocknames.txt @@ -53,3 +53,4 @@ XactTruncationLock 44 # 45 was XactTruncationLock until removal of BackendRandomLock WrapLimitsVacuumLock 46 NotifyQueueTailLock 47 +KmgrFileLock 48 diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 3082093d1e..5ee6c43394 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -35,6 +35,7 @@ #include "commands/async.h" #include "commands/prepare.h" #include "common/pg_prng.h" +#include "crypto/kmgr.h" #include "jit/jit.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" @@ -3718,7 +3719,7 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx, * postmaster/postmaster.c (the option sets should not conflict) and with * the common help() function in main/main.c. */ - while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:v:W:-:")) != -1) + while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:R:S:sTt:v:W:-:")) != -1) { switch (flag) { @@ -3810,6 +3811,19 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx, strlcpy(OutputFileName, optarg, MAXPGPATH); break; + case 'R': + terminal_fd = atoi(optarg); + if (terminal_fd == -1) + { + /* + * Allow file descriptor closing to be bypassed via -1. + * We just duplicate sterr. This is useful for + * single-user mode. + */ + terminal_fd = dup(2); + } + break; + case 'S': SetConfigOption("work_mem", optarg, ctx, gucsource); break; @@ -4055,6 +4069,18 @@ PostgresMain(const char *dbname, const char *username) SetProcessingMode(InitProcessing); + /* + * Initialize kmgr for cluster encryption. Since kmgr needs to attach to + * shared memory the initialization must be called after BaseInit(). + */ + if (!IsUnderPostmaster) + { + InitializeKmgr(); + + if (terminal_fd != -1) + close(terminal_fd); + } + /* * Set up signal handlers. (InitPostmasterChild or InitStandaloneProcess * has already set up BlockSig and made that the active signal mask.) diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c index 92f24a6c9b..6f7953accb 100644 --- a/src/backend/utils/activity/wait_event.c +++ b/src/backend/utils/activity/wait_event.c @@ -591,6 +591,15 @@ pgstat_get_wait_io(WaitEventIO w) case WAIT_EVENT_DSM_FILL_ZERO_WRITE: event_name = "DSMFillZeroWrite"; break; + case WAIT_EVENT_KEY_FILE_READ: + event_name = "KeyFileRead"; + break; + case WAIT_EVENT_KEY_FILE_WRITE: + event_name = "KeyFileWrite"; + break; + case WAIT_EVENT_KEY_FILE_SYNC: + event_name = "KeyFileSync"; + break; case WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ: event_name = "LockFileAddToDataDirRead"; break; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 836b49484a..e7d7cad2b2 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -40,6 +40,7 @@ #include "commands/trigger.h" #include "commands/user.h" #include "commands/vacuum.h" +#include "crypto/kmgr.h" #include "jit/jit.h" #include "libpq/auth.h" #include "libpq/libpq.h" @@ -579,6 +580,7 @@ static char *recovery_target_string; static char *recovery_target_xid_string; static char *recovery_target_name_string; static char *recovery_target_lsn_string; +static char *file_encryption_method_str; /* should be static, but commands/variable.c needs to get at this */ char *role_string; @@ -701,6 +703,8 @@ const char *const config_group_names[] = gettext_noop("Statistics / Monitoring"), /* STATS_CUMULATIVE */ gettext_noop("Statistics / Cumulative Query and Index Statistics"), + /* ENCRYPTION */ + gettext_noop("Encryption"), /* AUTOVACUUM */ gettext_noop("Autovacuum"), /* CLIENT_CONN_STATEMENT */ @@ -4403,6 +4407,27 @@ struct config_string ConfigureNamesString[] = NULL, NULL, NULL }, + { + {"cluster_key_command", PGC_SIGHUP, ENCRYPTION, + gettext_noop("Command to obtain cluster key for cluster file encryption."), + NULL + }, + &cluster_key_command, + "", + NULL, NULL, NULL + }, + + { + {"file_encryption_method", PGC_INTERNAL, PRESET_OPTIONS, + gettext_noop("Shows the cluster file encryption method."), + NULL, + GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE + }, + &file_encryption_method_str, + "", + NULL, NULL, NULL + }, + { {"application_name", PGC_USERSET, LOGGING_WHAT, gettext_noop("Sets the application name to be reported in statistics and logs."), diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 781f8b8758..d5fbf8f7df 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -263,8 +263,8 @@ pg_control_recovery(PG_FUNCTION_ARGS) Datum pg_control_init(PG_FUNCTION_ARGS) { - Datum values[11]; - bool nulls[11]; + Datum values[12]; + bool nulls[12]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -274,7 +274,7 @@ pg_control_init(PG_FUNCTION_ARGS) * Construct a tuple descriptor for the result row. This must match this * function's pg_proc entry! */ - tupdesc = CreateTemplateTupleDesc(11); + tupdesc = CreateTemplateTupleDesc(12); TupleDescInitEntry(tupdesc, (AttrNumber) 1, "max_data_alignment", INT4OID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 2, "database_block_size", @@ -297,6 +297,8 @@ pg_control_init(PG_FUNCTION_ARGS) BOOLOID, -1, 0); TupleDescInitEntry(tupdesc, (AttrNumber) 11, "data_page_checksum_version", INT4OID, -1, 0); + TupleDescInitEntry(tupdesc, (AttrNumber) 12, "file_encryption_method", + INT4OID, -1, 0); tupdesc = BlessTupleDesc(tupdesc); /* read the control file */ @@ -338,6 +340,9 @@ pg_control_init(PG_FUNCTION_ARGS) values[10] = Int32GetDatum(ControlFile->data_checksum_version); nulls[10] = false; + values[11] = Int32GetDatum(ControlFile->file_encryption_method); + nulls[11] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 868d21c351..96b5c869e1 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -659,6 +659,11 @@ # autovacuum, -1 means use # vacuum_cost_limit +#------------------------------------------------------------------------------ +# ENCRYPTION +#------------------------------------------------------------------------------ + +#cluster_key_command = '' #------------------------------------------------------------------------------ # CLIENT CONNECTION DEFAULTS diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 1fbd48fbda..8f219fbacc 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -13,6 +13,7 @@ #include "access/xlogbackup.h" #include "access/xlogdefs.h" +#include "common/kmgr_utils.h" #include "datatype/timestamp.h" #include "lib/stringinfo.h" #include "nodes/pg_list.h" @@ -224,6 +225,10 @@ extern XLogRecPtr GetXLogWriteRecPtr(void); extern uint64 GetSystemIdentifier(void); extern char *GetMockAuthenticationNonce(void); extern bool DataChecksumsEnabled(void); + +#define FileEncryptionEnabled (GetFileEncryptionMethod() != DISABLED_ENCRYPTION_METHOD) +extern int GetFileEncryptionMethod(void); + extern XLogRecPtr GetFakeLSNForUnloggedRel(void); extern Size XLOGShmemSize(void); extern void XLOGShmemInit(void); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 06368e2366..db7de77287 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -22,7 +22,7 @@ /* Version identifier for this pg_control format */ -#define PG_CONTROL_VERSION 1300 +#define PG_CONTROL_VERSION 1400 /* Nonce key length, see below */ #define MOCK_AUTH_NONCE_LEN 32 @@ -226,6 +226,9 @@ typedef struct ControlFileData */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; + /* File encryption method; index into encryption_methods[]. */ + int file_encryption_method; + /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 20f5aa56ea..f5e7a85519 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11674,9 +11674,9 @@ descr => 'pg_controldata init state information as a function', proname => 'pg_control_init', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,int4,int4,int4,int4,int4,int4,int4,bool,int4}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{max_data_alignment,database_block_size,blocks_per_segment,wal_block_size,bytes_per_wal_segment,max_identifier_length,max_index_columns,max_toast_chunk_size,large_object_chunk_size,float8_pass_by_value,data_page_checksum_version}', + proallargtypes => '{int4,int4,int4,int4,int4,int4,int4,int4,int4,bool,int4,int4}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{max_data_alignment,database_block_size,blocks_per_segment,wal_block_size,bytes_per_wal_segment,max_identifier_length,max_index_columns,max_toast_chunk_size,large_object_chunk_size,float8_pass_by_value,data_page_checksum_version,file_encryption_method}', prosrc => 'pg_control_init' }, # subscripting support for built-in types diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h index 90e333ccd2..deac25e115 100644 --- a/src/include/postmaster/postmaster.h +++ b/src/include/postmaster/postmaster.h @@ -31,6 +31,8 @@ extern PGDLLIMPORT char *bonjour_name; extern PGDLLIMPORT bool restart_after_crash; extern PGDLLIMPORT bool remove_temp_files_after_crash; +extern int terminal_fd; + #ifdef WIN32 extern PGDLLIMPORT HANDLE PostmasterHandle; #else diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h index 51f7f09fde..1446b334b1 100644 --- a/src/include/utils/guc_tables.h +++ b/src/include/utils/guc_tables.h @@ -86,6 +86,7 @@ enum config_group PROCESS_TITLE, STATS_MONITORING, STATS_CUMULATIVE, + ENCRYPTION, AUTOVACUUM, CLIENT_CONN_STATEMENT, CLIENT_CONN_LOCALE, -- 2.31.1 [application/octet-stream] v2-0008-cfe-08-pg_alterckey_over_cfe-07-bin-squash-commit.patch (30.8K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/9-v2-0008-cfe-08-pg_alterckey_over_cfe-07-bin-squash-commit.patch) download | inline diff: From 61583c5d1af66c91a603d41b5c59e1f025542788 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:56:58 -0400 Subject: [PATCH v2 08/12] cfe-08-pg_alterckey_over_cfe-07-bin squash commit --- doc/src/sgml/ref/allfiles.sgml | 1 + doc/src/sgml/ref/pg_alterckey.sgml | 210 ++++++++ doc/src/sgml/reference.sgml | 1 + src/bin/Makefile | 1 + src/bin/pg_alterckey/.gitignore | 1 + src/bin/pg_alterckey/Makefile | 38 ++ src/bin/pg_alterckey/README | 24 + src/bin/pg_alterckey/pg_alterckey.c | 724 ++++++++++++++++++++++++++++ 8 files changed, 1000 insertions(+) create mode 100644 doc/src/sgml/ref/pg_alterckey.sgml create mode 100644 src/bin/pg_alterckey/.gitignore create mode 100644 src/bin/pg_alterckey/Makefile create mode 100644 src/bin/pg_alterckey/README create mode 100644 src/bin/pg_alterckey/pg_alterckey.c diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index e90a0e1f83..a2b96129ba 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -197,6 +197,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY dropuser SYSTEM "dropuser.sgml"> <!ENTITY ecpgRef SYSTEM "ecpg-ref.sgml"> <!ENTITY initdb SYSTEM "initdb.sgml"> +<!ENTITY pgalterckey SYSTEM "pg_alterckey.sgml"> <!ENTITY pgamcheck SYSTEM "pg_amcheck.sgml"> <!ENTITY pgarchivecleanup SYSTEM "pgarchivecleanup.sgml"> <!ENTITY pgBasebackup SYSTEM "pg_basebackup.sgml"> diff --git a/doc/src/sgml/ref/pg_alterckey.sgml b/doc/src/sgml/ref/pg_alterckey.sgml new file mode 100644 index 0000000000..867c439b8e --- /dev/null +++ b/doc/src/sgml/ref/pg_alterckey.sgml @@ -0,0 +1,210 @@ +<!-- +doc/src/sgml/ref/pg_alterckey.sgml +PostgreSQL documentation +--> + +<refentry id="app-pg_alterckey"> + <indexterm zone="app-pg_alterckey"> + <primary>pg_alterckey</primary> + </indexterm> + + <refmeta> + <refentrytitle><application>pg_alterckey</application></refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo>Application</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>pg_alterckey</refname> + <refpurpose>alter the <productname>PostgreSQL</productname> cluster key</refpurpose> + </refnamediv> + + <refsynopsisdiv> + <cmdsynopsis> + <command>pg_alterckey</command> + + <group choice="plain"> + <arg choice="plain"><option>-R</option></arg> + <arg choice="plain"><option>--authprompt</option></arg> + </group> + + <arg choice="plain"> + <replaceable class="parameter">old_cluster_key_command</replaceable> + <replaceable class="parameter">new_cluster_key_command</replaceable> + </arg> + + <arg choice="opt"> + <group choice="plain"> + <arg choice="plain"><option>-D</option></arg> + <arg choice="plain"><option>--pgdata</option></arg> + </group> + <replaceable class="parameter">datadir</replaceable> + </arg> + + </cmdsynopsis> + + <cmdsynopsis> + <command>pg_alterckey</command> + + <group choice="opt"> + <arg choice="plain"><option>-R</option></arg> + <arg choice="plain"><option>--authprompt</option></arg> + </group> + + <group choice="plain"> + <arg choice="plain"><option>-r</option></arg> + <arg choice="plain"><option>--repair</option></arg> + </group> + + <arg choice="opt"> + <group choice="opt"> + <arg choice="plain"><option>-D</option></arg> + <arg choice="plain"><option>--pgdata</option></arg> + </group> + <replaceable class="parameter">datadir</replaceable> + </arg> + + </cmdsynopsis> + </refsynopsisdiv> + + <refsect1 id="r1-app-pg_alterckey-1"> + <title>Description</title> + <para> + <command>pg_alterckey</command> alters the cluster key used + for cluster file encryption. The cluster key is initially set + during <xref linkend="app-initdb"/>. The command can be run while the + server is running or stopped. The new password must be used the next + time the server is started. + </para> + + <para> + <command>pg_alterckey</command> changes the key encryption key + (<acronym>KEK</acronym>) which encrypts the data encryption keys; + it does not change the data encryption keys. It does this by + decrypting each data encryption key using the <replaceable + class="parameter">old_cluster_key_command</replaceable>, + re-encrypting it using the <replaceable + class="parameter">new_cluster_key_command</replaceable>, and then + writes the result back to the cluster directory. + </para> + + <para> + See the <xref linkend="app-initdb"/> documentation for how to define + the old and new passphrase commands. You can use different executables + for these commands, or you can use the same executable with different + arguments to specify retrieval of the old or new key. + </para> + + <para> + <command>pg_alterckey</command> manages data encryption keys, + which are critical to allowing Postgres to access its decrypted + data. For this reason, it is very careful to preserve these + keys in most possible failure conditions, e.g., operating system + failure during cluster encryption key rotation. + </para> + + <para> + When started, <command>pg_alterckey</command> repairs any files that + remain from previous failures before altering the cluster encryption + key. During this repair phase, <command>pg_alterckey</command> will + either roll back the cluster key or roll forward the changes that + were previously requested. The server will not start if repair is + needed, though a running server is unaffected by an unrepaired cluster + key configuration. Therefore, if <command>pg_alterckey</command> + fails for any reason, it is recommended you run the command with + <option>--repair</option> to simply roll back or forward any previous + changes. This will report if it rolled the cluster key back or forward, + and then run the command again to change the cluster key if needed. + </para> + + <para> + You can specify the data directory on the command line, or use + the environment variable <envar>PGDATA</envar>. + </para> + </refsect1> + + <refsect1> + <title>Options</title> + + <para> + <variablelist> + <varlistentry> + <term><option>-R</option></term> + <term><option>--authprompt</option></term> + <listitem> + <para> + Allows the <option>old_cluster_key_command</option> and + <option>new_cluster_key_command</option> commands + to prompt for a passphrase or PIN. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + Other options: + + <variablelist> + <varlistentry> + <term><option>-V</option></term> + <term><option>--version</option></term> + <listitem> + <para> + Print the <application>pg_alterckey</application> version and exit. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><option>-?</option></term> + <term><option>--help</option></term> + <listitem> + <para> + Show help about <application>pg_alterckey</application> command line + arguments, and exit. + </para> + </listitem> + </varlistentry> + + </variablelist> + </para> + + </refsect1> + + <refsect1> + <title>Environment</title> + + <variablelist> + <varlistentry> + <term><envar>PGDATA</envar></term> + + <listitem> + <para> + Default data directory location + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><envar>PG_COLOR</envar></term> + <listitem> + <para> + Specifies whether to use color in diagnostic messages. Possible values + are <literal>always</literal>, <literal>auto</literal> and + <literal>never</literal>. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="app-initdb"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index a3b743e8c1..46f7834f72 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -241,6 +241,7 @@ </para> </partintro> + &pgalterckey; &clusterdb; &createdb; &createuser; diff --git a/src/bin/Makefile b/src/bin/Makefile index 7f9dde924e..018ee6592f 100644 --- a/src/bin/Makefile +++ b/src/bin/Makefile @@ -17,6 +17,7 @@ SUBDIRS = \ initdb \ pg_amcheck \ pg_archivecleanup \ + pg_alterckey \ pg_basebackup \ pg_checksums \ pg_config \ diff --git a/src/bin/pg_alterckey/.gitignore b/src/bin/pg_alterckey/.gitignore new file mode 100644 index 0000000000..4c4f39f2cc --- /dev/null +++ b/src/bin/pg_alterckey/.gitignore @@ -0,0 +1 @@ +/pg_alterckey diff --git a/src/bin/pg_alterckey/Makefile b/src/bin/pg_alterckey/Makefile new file mode 100644 index 0000000000..2133a4ba06 --- /dev/null +++ b/src/bin/pg_alterckey/Makefile @@ -0,0 +1,38 @@ +#------------------------------------------------------------------------- +# +# Makefile for src/bin/pg_alterckey +# +# Copyright (c) 1998-2021, PostgreSQL Global Development Group +# +# src/bin/pg_alterckey/Makefile +# +#------------------------------------------------------------------------- + +PGFILEDESC = "pg_alterckey - alter the cluster key" +PGAPPICON=win32 + +subdir = src/bin/pg_alterckey +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +OBJS = \ + $(WIN32RES) \ + pg_alterckey.o + +all: pg_alterckey + +pg_alterckey: $(OBJS) | submake-libpgport + $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) + +install: all installdirs + $(INSTALL_PROGRAM) pg_alterckey$(X) '$(DESTDIR)$(bindir)/pg_alterckey$(X)' + +installdirs: + $(MKDIR_P) '$(DESTDIR)$(bindir)' + +uninstall: + rm -f '$(DESTDIR)$(bindir)/pg_alterckey$(X)' + +clean distclean maintainer-clean: + rm -f pg_alterckey$(X) $(OBJS) + rm -rf tmp_check diff --git a/src/bin/pg_alterckey/README b/src/bin/pg_alterckey/README new file mode 100644 index 0000000000..6db3739271 --- /dev/null +++ b/src/bin/pg_alterckey/README @@ -0,0 +1,24 @@ +pg_alterckey +============ + +This directory contains the code to generate the pg_alterckey binary. + +Architecture +------------ + +pg_alterckey allows altering of the cluster encryption key (key +encryption key or KEK) which is stored outside of the file system; see +src/backend/crypto/README for more details. This must be done in a +crash-safe manner since the keys are critical to reading an encrypted +cluster. The active data encryption keys (DEK) are encrypted/wrapped by +the KEK and stored in PGDATA/pg_cryptokeys/live as separate files, +currently files 0 and 1. + +This process can be interrupted at anytime; the new execution of +pg_alterckey will repair any previously interrupted execution of +pg_alterckey. + +pg_alterckey should never be run concurrently. A lock file prevents +almost all concurrent execution. pg_alterckey can be run if the +database server is running or stopped, so it can't use database locking +that is only available when the server is running. diff --git a/src/bin/pg_alterckey/pg_alterckey.c b/src/bin/pg_alterckey/pg_alterckey.c new file mode 100644 index 0000000000..7e3d4ad915 --- /dev/null +++ b/src/bin/pg_alterckey/pg_alterckey.c @@ -0,0 +1,724 @@ +/*------------------------------------------------------------------------- + * + * pg_alterckey.c + * A utility to change the cluster key (key encryption key, KEK) + * used for cluster file encryption. The KEK wrap data encryption + * keys (DEK). + * + * The theory of operation is fairly simple: + * 1. Create lock file + * 2. Retrieve current and new cluster key using the supplied + * commands. + * 3. Revert any failed alter operation. + * 4. Create a "new" directory + * 5. Unwrap each DEK in "live" using the old KEK + * 6. Wrap each DEK using the new KEK and write it to "new" + * 7. Rename "live" to "old" + * 8. Rename "new" to "live" + * 9. Remove "old" + * 10. Remove lock file + * + * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/bin/pg_alterckey/pg_alterckey.c + * + *------------------------------------------------------------------------- + */ + + +#define FRONTEND 1 + +#include "postgres_fe.h" + +#include <signal.h> +#include <unistd.h> +#include <sys/stat.h> +#include <fcntl.h> + +#include "common/file_perm.h" +#include "common/file_utils.h" +#include "common/hex.h" +#include "common/restricted_token.h" +#include "crypto/kmgr.h" +#include "common/logging.h" +#include "getopt_long.h" +#include "pg_getopt.h" + +typedef enum +{ + SUCCESS_EXIT = 0, + ERROR_EXIT, + RMDIR_EXIT, + REPAIR_EXIT +} exit_action; + +#define MAX_WRAPPED_KEY_LENGTH 64 + +static int lock_fd = -1; +static bool pass_terminal_fd = false; +int terminal_fd = -1; +static bool repair_mode = false; +static char *old_cluster_key_cmd = NULL, + *new_cluster_key_cmd = NULL; +static char old_cluster_key[KMGR_CLUSTER_KEY_LEN], + new_cluster_key[KMGR_CLUSTER_KEY_LEN]; +static CryptoKey data_key; +unsigned char in_key[MAX_WRAPPED_KEY_LENGTH], + out_key[MAX_WRAPPED_KEY_LENGTH]; +int in_klen, + out_klen; +static char top_path[MAXPGPATH], + pid_path[MAXPGPATH], + live_path[MAXPGPATH], + new_path[MAXPGPATH], + old_path[MAXPGPATH]; + +static char *DataDir = NULL; +static const char *progname; + +static void create_lockfile(void); +static void recover_failure(void); +static void retrieve_cluster_keys(void); +static void bzero_keys_and_exit(exit_action action); +static void reencrypt_data_keys(void); +static void install_new_keys(void); + +static void +usage(const char *progname) +{ + printf(_("%s changes the cluster key of a PostgreSQL database cluster.\n\n"), progname); + printf(_("Usage:\n")); + printf(_(" %s [OPTION] old_cluster_key_command new_cluster_key_command [DATADIR]\n"), progname); + printf(_(" %s [repair_option] [DATADIR]\n"), progname); + printf(_("\nOptions:\n")); + printf(_(" -R, --authprompt prompt for a passphrase or PIN\n")); + printf(_(" [-D, --pgdata=]DATADIR data directory\n")); + printf(_(" -V, --version output version information, then exit\n")); + printf(_(" -?, --help show this help, then exit\n")); + printf(_("\nRepair options:\n")); + printf(_(" -r, --repair repair previous failure\n")); + printf(_("\nIf no data directory (DATADIR) is specified, " + "the environment variable PGDATA\nis used.\n\n")); + printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT); + printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL); +} + + +int +main(int argc, char *argv[]) +{ + static struct option long_options[] = { + {"authprompt", required_argument, NULL, 'R'}, + {"repair", required_argument, NULL, 'r'}, + {"pgdata", required_argument, NULL, 'D'}, + {NULL, 0, NULL, 0} + }; + + int c; + + pg_logging_init(argv[0]); + set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_alterckey")); + progname = get_progname(argv[0]); + + if (argc > 1) + { + if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0) + { + usage(progname); + exit(0); + } + if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0) + { + puts("pg_alterckey (PostgreSQL) " PG_VERSION); + exit(0); + } + } + + /* check for -r/-R */ + while ((c = getopt_long(argc, argv, "D:rR", long_options, NULL)) != -1) + { + switch (c) + { + case 'D': + DataDir = optarg; + break; + case 'r': + repair_mode = true; + break; + case 'R': + pass_terminal_fd = true; + break; + default: + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(1); + } + } + + if (!repair_mode) + { + /* get cluster key commands */ + if (optind < argc) + old_cluster_key_cmd = argv[optind++]; + else + { + pg_log_error("missing old_cluster_key_command"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + if (optind < argc) + new_cluster_key_cmd = argv[optind++]; + else + { + pg_log_error("missing new_cluster_key_command"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + } + + if (DataDir == NULL) + { + if (optind < argc) + DataDir = argv[optind++]; + else + DataDir = getenv("PGDATA"); + + /* If no DataDir was specified, and none could be found, error out */ + if (DataDir == NULL) + { + pg_log_error("no data directory specified"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(1); + } + } + + /* Complain if any arguments remain */ + if (optind < argc) + { + pg_log_error("too many command-line arguments (first is \"%s\")", + argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + /* + * Disallow running as root because we create directories in PGDATA + */ +#ifndef WIN32 + if (geteuid() == 0) + { + pg_log_error("%s: cannot be run as root\n" + "Please log in (using, e.g., \"su\") as the " + "(unprivileged) user that will\n" + "own the server process.\n", + progname); + exit(1); + } +#endif + + get_restricted_token(); + + /* Set mask based on PGDATA permissions */ + if (!GetDataDirectoryCreatePerm(DataDir)) + { + pg_log_error("could not read permissions of directory \"%s\": %m", + DataDir); + exit(1); + } + + umask(pg_mode_mask); + + snprintf(top_path, sizeof(top_path), "%s/%s", DataDir, KMGR_DIR); + snprintf(pid_path, sizeof(pid_path), "%s/%s", DataDir, KMGR_DIR_PID); + snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR); + snprintf(new_path, sizeof(new_path), "%s/%s", DataDir, NEW_KMGR_DIR); + snprintf(old_path, sizeof(old_path), "%s/%s", DataDir, OLD_KMGR_DIR); + + /* Complain if any arguments remain */ + if (optind < argc) + { + pg_log_error("too many command-line arguments (first is \"%s\")", + argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(1); + } + + if (DataDir == NULL) + { + pg_log_error("no data directory specified"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(1); + } + + create_lockfile(); + + recover_failure(); + + if (!repair_mode) + { + retrieve_cluster_keys(); + reencrypt_data_keys(); + install_new_keys(); + } + +#ifndef WIN32 + /* remove file system reference to file */ + if (unlink(pid_path) < 0) + { + pg_log_error("could not delete lock file \"%s\": %m", KMGR_DIR_PID); + exit(1); + } +#endif + + close(lock_fd); + + bzero_keys_and_exit(SUCCESS_EXIT); +} + +/* Create a lock file; this prevents almost all cases of concurrent access */ +void +create_lockfile(void) +{ + struct stat buffer; + char lock_pid_str[20]; + + if (stat(top_path, &buffer) != 0 || !S_ISDIR(buffer.st_mode)) + { + pg_log_error("cluster file encryption directory \"%s\" is missing; is it enabled?", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + /* Does a lockfile exist? */ + if ((lock_fd = open(pid_path, O_RDONLY, 0)) != -1) + { + int lock_pid; + int len; + + /* read the PID */ + if ((len = read(lock_fd, lock_pid_str, sizeof(lock_pid_str) - 1)) == 0) + { + pg_log_error("cannot read pid from lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + lock_pid_str[len] = '\0'; + + if ((lock_pid = atoi(lock_pid_str)) == 0) + { + pg_log_error("invalid pid in lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + /* Is the PID running? */ + if (kill(lock_pid, 0) == 0) + { + pg_log_error("active process %d currently holds a lock on this operation, recorded in \"%s\"", + lock_pid, KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + close(lock_fd); + + if (repair_mode) + printf("old lock file removed\n"); + + /* ---------- + * pid is no longer running, so remove the lock file. + * This is not 100% safe from concurrent access, e.g.: + * + * process 1 exits and leaves stale lock file + * process 2 checks stale lock file of process 1 + * process 3 checks stale lock file of process 1 + * process 2 remove the lock file of process 1 + * process 4 creates a lock file + * process 3 remove the lock file of process 4 + * process 5 creates a lock file + * + * The sleep(2) helps with this since it reduces the likelihood + * a process that did an unlock will interfere with another unlock + * process. We could ask users to remove the lock, but that seems + * even more error-prone, especially since this might happen + * on server start. Many PG tools seem to have problems with + * concurrent access. + * ---------- + */ + unlink(pid_path); + + /* Sleep to reduce the likelihood of concurrent unlink */ + pg_usleep(2000000L); /* 2 seconds */ + } + + /* Create our own lockfile? */ +#ifndef WIN32 + lock_fd = open(pid_path, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode); +#else + /* delete on close */ + lock_fd = open(pid_path, O_RDWR | O_CREAT | O_EXCL | O_TEMPORARY, + pg_file_create_mode); +#endif + + if (lock_fd == -1) + { + if (errno == EEXIST) + pg_log_error("an active process currently holds a lock on this operation, recorded in \"%s\"", + KMGR_DIR_PID); + else + pg_log_error("unable to create lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + + snprintf(lock_pid_str, sizeof(lock_pid_str), "%d\n", getpid()); + if (write(lock_fd, lock_pid_str, strlen(lock_pid_str)) != strlen(lock_pid_str)) + { + pg_log_error("could not write pid to lock file \"%s\": %m", KMGR_DIR_PID); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } +} + +/* + * ---------- + * recover_failure + * + * A previous pg_alterckey might have failed, so it might need recovery. + * The normal operation is: + * 1. reencrypt LIVE_KMGR_DIR -> NEW_KMGR_DIR + * 2. rename KMGR_DIR -> OLD_KMGR_DIR + * 3. rename NEW_KMGR_DIR -> LIVE_KMGR_DIR + * remove OLD_KMGR_DIR + * + * There are eight possible directory configurations: + * + * LIVE_KMGR_DIR NEW_KMGR_DIR OLD_KMGR_DIR + * + * Normal: + * 0. normal X + * 1. remove new X X + * 2. install new X X + * 3. remove old X X + * + * Abnormal: + * fatal + * restore old X + * install new X + * remove old and new X X X + * + * We don't handle the abnormal cases, just report an error. + * ---------- + */ +static void +recover_failure(void) +{ + struct stat buffer; + bool is_live, + is_new, + is_old; + + is_live = !stat(live_path, &buffer); + is_new = !stat(new_path, &buffer); + is_old = !stat(old_path, &buffer); + + /* normal #0 */ + if (is_live && !is_new && !is_old) + { + if (repair_mode) + printf("repair unnecessary\n"); + return; + } + /* remove new #1 */ + else if (is_live && is_new && !is_old) + { + if (!rmtree(new_path, true)) + { + pg_log_error("unable to remove new directory \"%s\": %m", NEW_KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + printf(_("removed files created during previously aborted alter operation\n")); + return; + } + /* install new #2 */ + else if (!is_live && is_new && is_old) + { + if (rename(new_path, live_path) != 0) + { + pg_log_error("unable to rename directory \"%s\" to \"%s\": %m", + NEW_KMGR_DIR, LIVE_KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + printf(_("Installed new cluster password supplied in previous alter operation\n")); + return; + } + /* remove old #3 */ + else if (is_live && !is_new && is_old) + { + if (!rmtree(old_path, true)) + { + pg_log_error("unable to remove old directory \"%s\": %m", OLD_KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } + printf(_("Removed old files invalidated during previous alter operation\n")); + return; + } + else + { + pg_log_error("cluster file encryption directory \"%s\" is in an abnormal state and cannot be processed", + KMGR_DIR); + fprintf(stderr, _("Exiting with no changes made.\n")); + exit(1); + } +} + +/* Retrieve old and new cluster keys */ +void +retrieve_cluster_keys(void) +{ + int cluster_key_len; + char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN]; + + /* + * If we have been asked to pass an open file descriptor to the user + * terminal to the commands, set one up. + */ + if (pass_terminal_fd) + { +#ifndef WIN32 + terminal_fd = open("/dev/tty", O_RDWR, 0); +#else + terminal_fd = open("CONOUT$", O_RDWR, 0); +#endif + if (terminal_fd < 0) + { + pg_log_error(_("%s: could not open terminal: %s\n"), + progname, strerror(errno)); + exit(1); + } + } + + /* Get old key encryption key from the cluster key command */ + cluster_key_len = kmgr_run_cluster_key_command(old_cluster_key_cmd, + (char *) cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + live_path, terminal_fd); + if (pg_hex_decode(cluster_key_hex, cluster_key_len, + (char *) old_cluster_key, KMGR_CLUSTER_KEY_LEN) != + KMGR_CLUSTER_KEY_LEN) + { + pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); + bzero_keys_and_exit(ERROR_EXIT); + } + + /* + * Create new key directory here in case the new cluster key command needs + * it to exist. + */ + if (mkdir(new_path, pg_dir_create_mode) != 0) + { + pg_log_error("unable to create new cluster key directory \"%s\": %m", NEW_KMGR_DIR); + bzero_keys_and_exit(ERROR_EXIT); + } + + /* Get new key */ + cluster_key_len = kmgr_run_cluster_key_command(new_cluster_key_cmd, + (char *) cluster_key_hex, + ALLOC_KMGR_CLUSTER_KEY_LEN, + new_path, terminal_fd); + if (pg_hex_decode(cluster_key_hex, cluster_key_len, + (char *) new_cluster_key, KMGR_CLUSTER_KEY_LEN) != + KMGR_CLUSTER_KEY_LEN) + { + pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); + bzero_keys_and_exit(ERROR_EXIT); + } + + if (pass_terminal_fd) + close(terminal_fd); + + /* output newline */ + puts(""); + + if (memcmp(old_cluster_key, new_cluster_key, KMGR_CLUSTER_KEY_LEN) == 0) + { + pg_log_error("cluster keys are identical, exiting\n"); + bzero_keys_and_exit(RMDIR_EXIT); + } +} + +/* Decrypt old keys encrypted with old pass phrase and reencrypt with new one */ +void +reencrypt_data_keys(void) +{ + PgCipherCtx *old_ctx, + *new_ctx; + + old_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, + (unsigned char *) old_cluster_key, + KMGR_CLUSTER_KEY_LEN, false); + if (!old_ctx) + pg_log_error("could not initialize encryption context"); + + new_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, + (unsigned char *) new_cluster_key, + KMGR_CLUSTER_KEY_LEN, true); + if (!new_ctx) + pg_log_error("could not initialize encryption context"); + + for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++) + { + char src_path[MAXPGPATH], + dst_path[MAXPGPATH]; + int src_fd, + dst_fd; + int len; + struct stat st; + + CryptoKeyFilePath(src_path, live_path, id); + CryptoKeyFilePath(dst_path, new_path, id); + + if ((src_fd = open(src_path, O_RDONLY | PG_BINARY, 0)) < 0) + { + pg_log_error("could not open file \"%s\": %m", src_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + if ((dst_fd = open(dst_path, O_RDWR | O_CREAT | O_TRUNC | PG_BINARY, + pg_file_create_mode)) < 0) + { + pg_log_error("could not open file \"%s\": %m", dst_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + if (fstat(src_fd, &st)) + { + pg_log_error("could not stat file \"%s\": %m", src_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + in_klen = st.st_size; + + if (in_klen > MAX_WRAPPED_KEY_LENGTH) + { + pg_log_error("invalid wrapped key length (%d) for file \"%s\"", in_klen, src_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* Read the source key */ + len = read(src_fd, in_key, in_klen); + if (len != in_klen) + { + if (len < 0) + pg_log_error("could read file \"%s\": %m", src_path); + else + pg_log_error("could read file \"%s\": read %d of %u", + src_path, len, in_klen); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* decrypt with old key */ + if (!kmgr_unwrap_data_key(old_ctx, in_key, in_klen, &data_key)) + { + pg_log_error("incorrect old key specified"); + bzero_keys_and_exit(RMDIR_EXIT); + } + + if (KMGR_MAX_KEY_LEN_BYTES + pg_cipher_blocksize(new_ctx) > MAX_WRAPPED_KEY_LENGTH) + { + pg_log_error("invalid max wrapped key length"); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* encrypt with new key */ + if (!kmgr_wrap_data_key(new_ctx, &data_key, out_key, &out_klen)) + { + pg_log_error("could not encrypt new key"); + bzero_keys_and_exit(RMDIR_EXIT); + } + + /* Write to the dest key */ + len = write(dst_fd, out_key, out_klen); + if (len != out_klen) + { + pg_log_error("could not write fie \"%s\"", dst_path); + bzero_keys_and_exit(RMDIR_EXIT); + } + + close(src_fd); + close(dst_fd); + } + + /* The cluster key is correct, free the cipher context */ + pg_cipher_ctx_free(old_ctx); + pg_cipher_ctx_free(new_ctx); +} + +/* Install new keys */ +void +install_new_keys(void) +{ + /* + * Issue fsync's so key rotation is less likely to be left in an + * inconsistent state in case of a crash during this operation. + */ + + if (rename(live_path, old_path) != 0) + { + pg_log_error("unable to rename directory \"%s\" to \"%s\": %m", + LIVE_KMGR_DIR, OLD_KMGR_DIR); + bzero_keys_and_exit(RMDIR_EXIT); + } + fsync_dir_recurse(top_path); + + if (rename(new_path, live_path) != 0) + { + pg_log_error("unable to rename directory \"%s\" to \"%s\": %m", + NEW_KMGR_DIR, LIVE_KMGR_DIR); + bzero_keys_and_exit(REPAIR_EXIT); + } + fsync_dir_recurse(top_path); + + if (!rmtree(old_path, true)) + { + pg_log_error("unable to remove old directory \"%s\": %m", OLD_KMGR_DIR); + bzero_keys_and_exit(REPAIR_EXIT); + } + fsync_dir_recurse(top_path); +} + +/* Erase memory and exit */ +void +bzero_keys_and_exit(exit_action action) +{ + explicit_bzero(old_cluster_key, sizeof(old_cluster_key)); + explicit_bzero(new_cluster_key, sizeof(new_cluster_key)); + + explicit_bzero(in_key, sizeof(in_key)); + explicit_bzero(&data_key, sizeof(data_key)); + explicit_bzero(out_key, sizeof(out_key)); + + if (action == RMDIR_EXIT) + { + if (!rmtree(new_path, true)) + pg_log_error("unable to remove new directory \"%s\": %m", NEW_KMGR_DIR); + printf("Re-running pg_alterckey to repair might be needed before the next server start\n"); + exit(1); + } + else if (action == REPAIR_EXIT) + { + unlink(pid_path); + printf("Re-running pg_alterckey to repair might be needed before the next server start\n"); + } + + /* return 0 or 1 */ + exit(action != SUCCESS_EXIT); +} -- 2.31.1 [application/octet-stream] v2-0009-cfe-09-test_over_cfe-08-pg_alterckey-squash-commi.patch (114.0K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/10-v2-0009-cfe-09-test_over_cfe-08-pg_alterckey-squash-commi.patch) download | inline diff: From f2419b9f3b5fcc4c8efd3f9f8a0ac8955e029684 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:00 -0400 Subject: [PATCH v2 09/12] cfe-09-test_over_cfe-08-pg_alterckey squash commit --- src/test/Makefile | 3 + src/test/README | 3 + src/test/crypto/.gitignore | 4 + src/test/crypto/KWP_AD_128.txt | 35 ++ src/test/crypto/KWP_AD_256.txt | 35 ++ src/test/crypto/KWP_AE_128.txt | 35 ++ src/test/crypto/KWP_AE_256.txt | 35 ++ src/test/crypto/Makefile | 39 +++ src/test/crypto/README | 33 ++ src/test/crypto/gcmDecrypt128.rsp | 129 +++++++ src/test/crypto/gcmDecrypt256.rsp | 129 +++++++ src/test/crypto/gcmEncryptExtIV128.rsp | 129 +++++++ src/test/crypto/gcmEncryptExtIV256.rsp | 129 +++++++ src/test/crypto/t/001_testcrypto.pl | 137 ++++++++ src/test/crypto/t/002_testkwp.pl | 126 +++++++ src/test/crypto/t/003_clusterkey.pl | 93 +++++ src/test/crypto/t/004_buffers.pl | 157 +++++++++ src/test/crypto/testcrypto.c | 458 +++++++++++++++++++++++++ 18 files changed, 1709 insertions(+) create mode 100644 src/test/crypto/.gitignore create mode 100644 src/test/crypto/KWP_AD_128.txt create mode 100644 src/test/crypto/KWP_AD_256.txt create mode 100644 src/test/crypto/KWP_AE_128.txt create mode 100644 src/test/crypto/KWP_AE_256.txt create mode 100644 src/test/crypto/Makefile create mode 100644 src/test/crypto/README create mode 100644 src/test/crypto/gcmDecrypt128.rsp create mode 100644 src/test/crypto/gcmDecrypt256.rsp create mode 100644 src/test/crypto/gcmEncryptExtIV128.rsp create mode 100644 src/test/crypto/gcmEncryptExtIV256.rsp create mode 100644 src/test/crypto/t/001_testcrypto.pl create mode 100644 src/test/crypto/t/002_testkwp.pl create mode 100644 src/test/crypto/t/003_clusterkey.pl create mode 100644 src/test/crypto/t/004_buffers.pl create mode 100644 src/test/crypto/testcrypto.c diff --git a/src/test/Makefile b/src/test/Makefile index dbd3192874..617f1dfa29 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -17,6 +17,9 @@ SUBDIRS = perl regress isolation modules authentication recovery subscription ifeq ($(with_icu),yes) SUBDIRS += icu endif +ifeq ($(with_openssl),yes) +SUBDIRS += crypto +endif ifeq ($(with_gssapi),yes) SUBDIRS += kerberos endif diff --git a/src/test/README b/src/test/README index afdc767651..0955a764e3 100644 --- a/src/test/README +++ b/src/test/README @@ -11,6 +11,9 @@ which tests get run automatically. authentication/ Tests for authentication (but see also below) +crypto/ + Tests for cluster file encryption + examples/ Demonstration programs for libpq that double as regression tests via "make check" diff --git a/src/test/crypto/.gitignore b/src/test/crypto/.gitignore new file mode 100644 index 0000000000..7b92218ad2 --- /dev/null +++ b/src/test/crypto/.gitignore @@ -0,0 +1,4 @@ +# Generated by test suite +/tmp_check/ +/log/ +/testcrypto diff --git a/src/test/crypto/KWP_AD_128.txt b/src/test/crypto/KWP_AD_128.txt new file mode 100644 index 0000000000..d77aeab719 --- /dev/null +++ b/src/test/crypto/KWP_AD_128.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AD with AES-128 cipher function' information for test-files +# Seed = 0f0c6b6602d959d22ff2305478ea477a85d196d9695abf5b445010b45ce046c501bc04e6ca4faf46137adf6eac452d97314cd91137e33fc30cc6df316117fed3 +# Generated on Fri Apr 6 14:46:45 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 1dd51f0d3a0a784174ba81b2c9f89005 +C = e1bde6d2df3b8e48ca127f97b56b5dc2672b3736cc3157c7b80a0316ef1efbdbbce19fea23da831836ccd2e002b2c1dfad206b5cec358446b8434d7f4c39e65b0e0b50897642ffc34bfb3cb3e233aa9c1058ff0d4fd48e98bc8cc3d214c06d514dd97db2278093a308f91f4ae92626d85771fb1447b36a3467fff02ac7e81ddbd0fdbcd02d1acd4f053c989ef3dcc2c01e23bc2f6090f3e8c0ba5f0082341200b1c37b99daa9cb6fec78bce3429aec5badb9fd28fdbdbdc5d53570675a9e39535b4594095658ef950ecd79a162223b60d2eb91765e022dc6e1bbdd86f1bcc280ed9df350da08a801fa16a1bf2701947acfb08f19fdfcaa1d76f466a5de2458a78fb82f6af3e1be68f405a4289f25896f4c9830005c9e895c86e67eceab0ad544856071b8d9585835b5e85a07ab01515f7ab54f98dffb4ca49a15068eefc6a01f7f52fd1adbe3631c59f6f43f79d2b4f2a691e2b30bb1d43a848dc3ee39c7f2e50f0c9deb7ab51e33bf40903ac255bb1510fd61676a6c13c3c776b8aacc6cefb95e24973ebb11192e2692dd0c6a085b58f86e11cc28ee2194988c123e3666da7339c0a4ac6afbacc83f1f100fbb39efff7cc605c9213828224a17c476395aeb9bb0a3150fb8889a8c2a494c8c526203f261642bfa69a94b86de9e6d3d932fe20fffe4bd76d502c0d437a3e1d0d8727b7a8dc0e361967109e93566326b6c517663731c4c9bdd0295d8 +P = 1a4eed4bf5b8d2e2a58f1f1277f164cc32cdadaed848f76fe634034082ff9aa1711870bf3936d01a2aa48de30de5143b9148cf56f4490f9d480dda0b672e8e17a012cd26cec3c68837bd5b2f9beb13e0110f21c6c36343e09e027f39557d1596d4ca406e3e7aa113e9bb8623106bae25f0ea23d46bc29970ba2596f83fe4f73a6f978a4d949fa7c271570a2ae5d2b50792d5ab5c43d455f359fb83c35ca3da37cd73cd66b6adce94d78ecdeabf667daa47ea70799af299e1d898ccf3fca6c42c6fff8cf2ec992f596fed4a0cdb502a00f9b5689302931d15cba691e2f8079a0411332438b714ace5234b91e4aebee8f8dda0e1968c2016fed350430a65d8d206c9436f40b79ce03083b8dc207d6960be1ce97007ed22a388ebb7b3d8f7d2b7d9f8f49731fbcb21e21db0cdd15674c795d5af2b2cd727f83e634e8c47157ed0c6873a5c9419e683f16f4a7827b444967812f9d1adb9201b89a0e66bbcf0591465f5d7036a21cdda0e10099feb819dfc37fdd3105120044dab716882d3971f312e3f4459006fd5a1eab08ff63edf6718f47ddaa37f7f40c9c372995f3aec97bc45e287b64fc8cf5559ab04a4d4d3ed482f5d61d3abd99cc87ee406da3ab9c9cd22ba3b8d191b26754aa94a2412f39e332d77fe72210adb0cbb5c96adebdbde036f1f1aaafad74a7ac2594f81efa734054e2e16dc931d49b970b81756862705fcd4 + +COUNT = 1 +K = b3fa008b5947ce58dfbd354dd01f2d43 +C = 55cd8e45138f477ce0a84f07bd28a93d7d628bb4860207a2f6dc4256bd79843e32c856a4fa831d1603699d49e6c36291b60aa80635900cc6c78cf0a2ddc457beb41782de0de03f08a064df90b41f2e98ce61185d735380403fe56b68f8343a801a14afb8a7ba79684dc2a585110da83e9a836cae1fd9e1a220dd6dc922b4f02b15ca88d43ab61e1da24a9b3cb99c4e5024ce5667f4841ca2a305b1f4c1ae9fb63d1d4dcb83870755a1a646b16c088e612d82ba2bf0e7e2fa0e8035c3baeb595f1ac9bb49b01f6f71392e217c049c0e9bd794b9aa2383cf59ee0a90f965610c65ecd629a17cba2bdf2458e3a8e1a9d219cb66eb9ec8e5226b34f95003064952523920a0b4e94ec8ecd1bdca8a65fe46ed25fd4d076e46fa62a8cde6eabc593045d17cef996ebbeca4b537f65c4f683a10baeb4c42b9867bbb49ca7ea1c5437bc114948c542cffced9bb1ebe3c946eb24ff55be89be004596ba648b264167217d267b881020b905f508e4f0e1a58eca051d56ff30d91891838c574c3de54e3feafcdf514740ddc94ba92cb85fe86033e67f14d90be7a0222e4bd1624cea8894df66a36a8e848dfe9168d8024b7ba5636afbcf6b945a53e6b2778f229af7dc2e59bebbf8bdbdfde1e21465f6b6344b13afa0e5ceac212b3b88932f21b1ae04268476597c92e64ff7c14b9ef678f10a35b56cd70ba03063f94aed97b0a6cf883d1f07facfa37b6e5b070 +P = a067ab39cede4ac6c6cb7630cba48c52a794ac8ebec037125bcd97d1a3c52a8ed64764899f9035a6944d0605a5d977172a55bbf86cd81aef5d6bafb1ac86bfa65da2b3c39bf5da94a98f7b6dbc5df16a7b38061e0665ad16b20fb6aedc9ce7f6d3497c3c55cea92e6343f21251092ef2ea307b35f999683298098bedaea847d1ccbf8bda18dc477e8d49fee4e357273396ad2245703485b97b5a7d97057bad875a3e76b67ad5adbc6ef3b8ba9a1786aa93149f0f8dd166535acbf93f1b9839754d537da3fae1ab02973427c3f353fe9aa6c5a100bf0e6ccb08dc1fdb0fc363a95c77c5758d440db0a70f0340a4c488de51e1ecb932ce2fcb2c95ea28c9f55695d97ba1765c8f11e523ae3e4e1efceb69000a192c047ab197f4840c664c035064ecc12926fd3bca0527a160b5b5a2bbaf5db11437f2c38a1c7535e87f552b9f04f2fdd309a826e4ec7708217022fb075cdfc6cc23e9301e33068caa69ef746f357b09ccc098443a3a2979a225e70be1e722e8d6fbb57d0dded2456c1d47eeb0af2241f769836026fec8fc51d97c4abbe9710a4aa5b95aaac83bee57e1333fa244ccc971b6260a9be16e31cc2fd283fec1b247a7340d149fe5309acb47c9cdb955b7bcc4df277eaf611e8af281ff0bcd64b4534309282d1b5cb14efa93141869d67ce7e418f06bb4c2feebcb7a1151aea2eb8bc2fc4dcee53de9b2fb1803490caf + +COUNT = 2 +K = 4b4c43c9de4fb4a2a7a7adafeabe2dbd +C = 6e4d08b8124f7d3e23303fac1a842014f95e3d71c438f8f1990307842796dc5e404ad81802e35c183fe000390a12c81ee684c5cf26c1d90e414cfffe6931b0f352936fcf0b31429eb5c7612cc359a15371390e518cf5c6a6bff1bb0348d14e2c39b98c9f30672ed2af1d96296df8b5567db25b9510a2083461810e119735490058ed1b46b7fdfa885041d8749f90a072b43ba49f2f51fbcda0dbf3cf99fca1d8f46330e5f6fe079d6679cfa26214c8831b782aaa023a2e0ea91050d277dab876aa6865f2bb3fc1a4a77db52f6179d5e5325993280948b6b7002b572829641d35ed3d735d8423e5b24673c4570ca25064fc2c2ad4840632536bcfaf2a7a814f3eaed92b4d501bc51c1719a0d8d8f420b66db845682bb41c88038cfedf13417143a3a701b521a9bf0bb639875a728c3b5ce6ca7e7a45bc75285c193902e6b5e7a4c6e720493d3937bf485e587bff894f70fd6165a1d0129cc673a992e0a4f5489d228a066b1df60002ec0521924f8d672cd1452fec927e58e75807b2a390256f920743fa4d0fc8f59f2469a595ef65095ca0c80adfc843e9e69b6d4a3f824af47b2bfbf2a7a6c1b650378f096f6f0bfabc752c8f279d4f45d56d09dce97962c119de3a64d83b93ea55066f24d4238a229ae86e6a7857af1d8aba823370a72fe358046049a84a70213ef31d9e77a722def8e21480e79b71299438070946bd459a7251707446c911e381 +FAIL + +COUNT = 3 +K = 96ab719a3d08df2393ebc330e151dab1 +C = d50ae797f6c3418f388a7513d693c6dd665e858767531fbccd3eb1aabe796690ec8fbb757d88b169adf5c136de50ff0f2cfdd8389f812382578aee0b0b61e13c6a2bc500640fe1585f068eee0d1fa3420220e23090e24e3248fe16f4e0c7c0e996a21b4947ddd08fd3ccc1f036651be4f48ee1ffb486cdc05911244480548221d8da1f2bc37dece080e51b2cdd1ddebf37213a4dfa1b252e567243d9cec8c89eb8db544e7c389a2e13f1b91d860df3cbcec3e85c93276c2a9a5fa080efc85e9bad3bfe2d9bb06498dd8b3720456bfabd3c69b345f6954872baa1d43b9f7ceb92ae9ad77b270d0b94c79275a48874dafb136105f5553529687b6aeeaa521790b9376c9f88ace94049235cd52c4387ad210442513dd5e07171519d58b1294fb8ac1f60ac68b8f07b418e1bb0598601ec38b9a9b137dd87d0c8a41089d17ca1c720fd0e7e3b81b85a373753bed0f5e29586f84cb29e1d88c379c965c50f6a803ddfac2e1555beb9c208a3821f53bead8f120f4ef4a1490b730a0b8a2f1869c6b985520d709bdc0e5fce44316b8aa2448a2743761bf77bdbbfdab6a721a8ec79f38f7e7321a80a2cd3a35a912eaac5eace85c4cad3c6685b88be4517cd1c20971b85bd9e8eb6e52869e014831dff7585a163f5a4dbf1d59160104da90a9cfcc8d6a0324942b40fde4319a32442d83ebbf5d7a36e9495be2ffd0e7faec1b66c96f71843750b8a051b7170 +P = 3a3b9e6de537458875e59204ef7565b6dde796e5ab11c83f7a361b8143f0f7a7eadb5b53c6efa6d199f759cad5c029004024eabbaff717bafb95646dc31a8f6063b9f8faaea650dfa8803bfa0c79091f299a55f78611c2e0d015021d6c6d3abf3d85cac306740acc144201516b787421a77c78a566c6eadc88ecdeea4ff861b6db73f7b00f0a8f62faedefa58866fb368424d7267afdf5ff1279916d2f177408d780697e1c45e58a524bb0365858d2b5a42ee2bd9e8904134d04cf071e84db8a31804aa8bebc0b28dd621360385117764178fe74b29da3ac390ac4812fdc7eedf91fce6eaae3d03163435001ce42f55982daeda5cec5deb960b35df231463cbc26267746be628c53b55f4f21ef003816eb7bfc6c710efa03d0994a1b3c8595fc9293a2c101483798034d4ee7e3d5e07bbd897c9de4b8315e53cbd1f81bdecbd59d093c844a0ed1e3e9d238707a7b893ca453745223c67756d9062152b239ceec44c436e0896a59ea9ea8cf79a93b8b759389bb5e73c5f5330e26580d9777817400166d826008be5e8c7184ae2ecf8fb9dba92af3c747c74e1534c05395f9204b5e8481fdcf4dab5ea6224a8e0ee52576d467d930c0899d31a4e288e3eecb8cb7a3be3a66c79ae93033de5d0d422a6d54ab002d1a82f3f60db97834d9fa3782dd64cbec8ddcac2216a393dc263cea2705fd072ec82dfa1ddef9c588c49f17c275 + +COUNT = 4 +K = d8c221e426109cb5911d7d6f0836f4cd +C = d853d57eaacac8096346564eccd33281ee864fb290ce91eb717fa153ca00064e033635178c59860a567215b7320fab4a72ccf716501dbc9a44d5b3d501729674987d2413cad79dece055a9b0d47ec980331f4a236b31984f5d62f9d7f58c0f3afb81fe60f266652da65d06874334be065f56096e98536bd1f2120313b0905ffe2f2c3b6de265ab7800c42be810bed18548c08f9193b02a3981a922b32b618fd9a978439ea382bf2890ad1f30d115b2319276289cf4f7a9917b0c064180e79c8644f9ac880a793b4a8ee424dff32cf2b6ca46f52ff8bd8359ed18ea8aac23e63ae337f5baea9e2ff845a5fdc0b79d5767d47c2a1a536d889f553c52696cbe91ccd2ec671a0644689bdb0f4db7e5d58c854eb539b6b4cd9214e361a216d315b1b124b43c76c703c01d3bc3142f760a399ba4887a6e326a58ecaf56fd49ae128a86cda485eedc3da80b75b171e77cade00c903c1f216eefa845dfaef660fc5ecb6791ed53765683f44da6c4ed8a9ad9e995f7d920cdee8463e79b18e8874b0a2f573b1825f8a480b1ed1245c81f4ec097bf0a0504aab9bdaef27b67d98805a7ec687c8cbcbc92ad3ce58651162a1f57f8af427ea0a111dbd6e3c7f240eb6b2360650a72b9c1c4417b1d541dfcc2a8d6ce3e8c160d8d417e4efbdce809bfe30802696bfd52a0f40be4db9be247dfd867179d82390b55180ebc6ceba0a990e3f6d32eef9dfdb946706371 +P = 92814e18dbe6e83714c4a82ba3ac3baf682a8054eb36666c9546db040d40b8613fc560d97b05265fb19ade180efeb55dfce2cc5981ca222e66b547b78a42401710535b1181674fdd426cf2b0b55e5b7f0505f11307120d495cfb197a3de00569b3d39f93c27270df4725243d314a026549692b0e2b4079c60a8053f0f36e83aaa3494307b175fd40643c1bc264eed1c00f8b565f2a3aeeb78bd94970bd9267d21f5a9a1b07df09ed44a3bd4255a139a328235b921833f92904a74ec202b0eac65df1caed05dc84e52b06c1ecf0f7914324ac4d828b7de7189705308959be42401948e3bf4bdd50ce24101c67ae745a73a67d7e366b6b432ce67b05cefd149a17247010f42dae4de1b2ca42a8e71824cd32c5cb2e2055443ec3ff24339c774dc9207744b84e9203fda1f85595f961987d847ed33867f1ddfce0795e3f2d78c5d749a488a4997392b8c9022c810197c93186894faa55cb0b6775b57a7ba2729c617c9430a44098d5081e3c5c4908ddd1a475cf9211408a8ddbe19ee527ddb2596456e1b1481a09b04e091b1c14b7b2e41bb4434a906736e115cb25ada0950ac5d2845b4a9f1e95f4d80f64440e983324c3aa9f3ec8964f9da0d26aa47e86355aa80ad99d0e573fa9932da70bd65cb1a06d8cb77e455fe7cada4561e027ca1608132c2605b6d0489bba6b29f293951883c451f37bd545f6605364ddc75918df097e + +COUNT = 5 +K = 704eb91dd5ba3d85279cf47c01eca2a5 +C = 51c71fd7778aa3648f3e31e1db0c73cb1479372f2e35f65f00188f08f794993a2ee2bb7e91cd1a2b86e92b8ccca7277207fb525ab17600173fa28844ae27f093e0e5ae00585cc714dac90cbe9b6332cbe4cb689b2cd141c102c6881f5b71ec477c5f4a91f7bdcb5871aadd478f1a9ccc6e069b7283f4d70b26e8748eda6d443ab13804c543a44fe2fb366f90de35d83fbf6354a9a9ab4a93ff7d61cbc0bfe05d6102c9c393273e7d3a04d61eba771f05cee29e5dacb7abf34ec9159e121841e2c39848f604c8f743313cbdca828bfa4635a81136e7a37f230c0d3c814d35c2eaabdd94183312909ab3a09b87cce0c719408f837bf24bfb2dad87630aabc9eab35bdb9cc536198389aceec68e8779f9e1eee84392189823a68195b75bbb6d33addf580564e696a362928e2ac506b79480600bc2f9eaa3e96f323390d1d92cf3c6d4bd4147ada5634cff2bf2d97b259904a335eaf11ec3fc84dcd8e27f7538e0fac1dbe7cb4533f4fa58913535d957b90678fac58aa96694a8047ac774afe488ab429c6807e709351f8159dcfbf83b865aeeb26722ef64a537ce932b2cfa6d53ed6cc1ca8ab58748c06a753515fffc56e294f51ab257585b610d261c6fe12def38a1b5dceaa4681569124c679b20984ed2967740419b342e9010eabd291de026f6e829e4dba5300cb668191358ab58e178c29a0194a639233f9c28c50a609bc42f8fa6cd17bc58eccd +FAIL diff --git a/src/test/crypto/KWP_AD_256.txt b/src/test/crypto/KWP_AD_256.txt new file mode 100644 index 0000000000..6d28f39705 --- /dev/null +++ b/src/test/crypto/KWP_AD_256.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AD with AES-256 cipher function' information for test-files +# Seed = bf1ba8f321ce0abadb40026686c9d9e7f0c4a55388f2ea7cefc81aeb0e054d40c94f48093f2739580010d6ad6e6ce734f21e7338100b750ec9c7bb06bf46f7f4 +# Generated on Fri Apr 6 14:47:06 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 08f5c088acec18e6cf1f03a8f85d772e327e7fb07f8c2939eb554e84c42ab93d +C = dff30fd43647d4be54cf2dfd6187e2ddffb55267313f980fb09c833a9c2bfa558a95861711f0acb2a5c7e731ba22f24a9c4dfdd9e9b0216e9088f817a175b9835b0e17615687a20f68c067205626494cd04fbabc0b3eea7c0a4cd6236bc8b3e52e721dfc357fb8a3722bfcc4c690d8f63dbb864bb6e3a15805aea7270f8eb748deebaa2d066fcda11c2e67221f9a91d2c29a6c79ffae76aa80a2590b4f9e35f623fbf2f8ceb2a205493077556a186e25e5bd52dcff7bcc6909b37a66c1d1431be1b363bb40da25386eaaf5fcabc7be6422a04434a21d1d3105328e7c56770b9f59b03395e4138f5f06fc7e6b80dab87b08caa7bfffc45a095c15263efd3f06c651ded6f58074efc20620d704997fc84721a0a8e9e5b9f5cd330bbb156b31d9d1b1c260e4a24535f30404dc5b2dd6b35d916a1391b25a7d8790be09d85483ed1522074a2785812005bda10dd55acb245b3bd3d9bb777dd23f9b02538ba1a114ba53386d7ca4d9524b2f8a18e0ffb21580b560540bb2146f08f04974b90eb324547d56222df95f44bc6e5f183bef283e4816fb1b2933f9c7c6726a245a495e304d8318d0008c51b0be8090f8f668fbc3f31e073be4b9e97468f4dd8c798e9d682868df493db8a85738b58cfd005190f365849072577772672c6f82555c65046eb34e86fe61103327a063bacbbe33cea7eaa3d1de45471b7269e1b6b38608626e323447a3d5fe0599a6 +P = 8b68f66a3d2f59d419851b94d9a6f2f0e667f8125e11d463a6bc2cea46b12dcc40ce8018b204972c735fdd6d2d05b628f4905c6690f5ac5b1b51e12f3af2dc3ae9b9dab616f0a2a66a1ac197592fd5b15900547f32f54110b58d51a0340aa80e9eeb7b2e0eb97e80aa22ba918f2fe1c678c730ed5c3d8d24774f17d8ab6e01a06243d36e764df1dbb8af1faadbc55281f0242abd7a162c984fd0b05ab8b0bcaedffb2962024f009a8d7c9e71281c09f52ec0707ee3bbeb1ecb918be6ae3e9c1fabbcd3512af928db3ba6c109ff9e9839a616b2a53f092160a48222b84d53cd52490515ef93e1ebb33897263492ab8ec6fad2e633276ae367f76d7f926309478c0205d4f22506a451795dc98f5410d8f5d3e049cbedf381620861e7b4ae08f2d8a71abc1f230248cb636a2d7b4e7717ab2b7b5f2dc6e5b5a18e8043254208b50fd6f8929eaf974c48551233661ad67321b64d69245d536d9a8ca2a6a10966dddb9d2ce36641c9281c460ae524b077867258f638e6ac872cb5f5c6fb216b1ae60a9d0c5ea0dbcd060f255da26111175af4e9935df59ddade6a2a70cddff8cae6a98e4f3843c2dd59d09053b07b648a46f5de0eb21ebb192828279a386ea3eedf2cdc355d73d51111e8c1d522e059752bc56226a4225bcab713bfaaaec78167d7cfd33e913b26fda93ca7524aa8a8b17977c88ff9bc23ea810b4de59eac18d1523b + +COUNT = 1 +K = 94c4d5d70f881e58e10e7246cf812d40e2be258adb2b6c13c6603fc7daf7e85a +C = 6c07b5ffd1b9be182413ef8eae4a6eac657108a46008a0d898727f2711e6fa0ca60fd1d51fad683b57d4202fa2b0eb88b856e08b07155439bdb03890cbb7e0f228172bf297a4e0917dadaa5e89a287bb9ba6441c852c5b0cff5084e6c425aaf866815b3fc45f5f7fb5d14b270343e6a30f402e11d62e433a0d84f65684b2df78d4e7758bc0bf81783316905cdc3c1150ec47f225c966f7f339b2538970eb3b8a2c13f95df1310d6e3b2a1f8aed19105846557d8f0018fc0f17146bf836b654dec98e9ad639c7e4b2f922b4396e82c690cdecb65f5e0ea282dd6262f34346ff9adbc8b2f361ddd4356f0feadf7c750fc0580c4e12c00ee049d06eed2242b14727ef4d58386dc8df279a7bf8131c3befaea2f059ab757826e5e381d49a2f11b8cbc2b0021af4da7a779e5df0083edeb54348cc36ce96a19a3d7ff5bd2f19d05fef6b200e76399a02a991111832173353bff4ce1859ff534ae13290dd176ba8e1384ed24d9702dbff127e15e5c66618f94680271732d19f64552ed03df76dc9d46c3cfdd53a1b253992fbcbea6db006f16e8dd92406f0090ad9100856c6b71f7767fcb895136416b374285efe1c6506941911a380e2bf74ffd0f67e853f9ac7b5df6666b177a2908fda9add0eb798f8ccc52801535b2bdf9507f3fb3b46915aa889d62ac5909040a1a28856105dfe2e10d5cfbb569c380551fc8bbe7d83dc87ef7a92faa3fff4b1e2c2 +P = 85693a16ae69d751cfa6799b95a6396de2eabe7e4da74d734691d992cba353a39f3b9615c1325db5b0563ce1a846bb0f0534a86130ce6657736b9a9b35b0f8d89dd1b3a295131d2f3f57f94deef9606dad76a377d0b24e632b3680e4d3338f3e4484609e8063e9ec621297f55802d7c347e8085ba6e514884b8fc1ae109409c5c3a5bddf4daf034d300e31eccba07a9380f5325666c4a3aa12d60b30ca272fe03534aabe78ba0452a7e4648ebfd4645675629676be6f122a54b6b810cf9cc0c68b7c61470a537a5a664ec24dbb3eb4f9fa8355cc7ae8fef27a0146df5ccc585d8c106a1eeb64ad4c701fd5a54ef18295b07e9e47f7f7dd2f67d38ed776a5f0b28843cc4bb5d7fdbea9cb0088dee849ae232e4e016d8cf3681971e8a45d6b25451538212b91f30e17580a8107a7a95587a06d22d615f5475a5f616fdf2fab79152f2643054d96ba88f50888eb0f2f1f154c6fff53dd44c3613269751dca4fa86f45d6b1af9ad0159685223889529609e7003c8f3cab491fd6c1a020305da8f94ec833d721d9fac7e575c2a1bc26eb4fb5010c35ffbd39b98d857f12584f4ab7de92aa6d7e7148a0120cc6b3f7ae47a291ba1cf55a28d38d3a30dfc3917d663458cf840385ca81cf70acce45a5cd509f8387d450bbdd6fa51830cf9a7387887c620b86809c55a3eb322ca784a51693f1054759804314ae86048f0d9c99650a5a12 + +COUNT = 2 +K = d65338fd3771fd58c07b6b689577378939d439628529b92cd5625edd18afac76 +C = 1c429bf25c144a2cc649fbd60de5c26c31a0c352de99b34b86101c551994f082feffe1db8853de59b3e8593785eca100a71c5392f0c71eca9f411cbd87fc77ea1a96376dc13f6ad460a11e9cd5a829875a7b7dcd2ba4eaac08c5bb48ab5d4c338a6f8bc5e760739edcff2db116b5b1802e35f936d473db168edd12532a992bcc418a759cc9fd3f97f561623078af29d7ab489b7ec564ba981c188f11240dd9354c324f8d0cdf1c74252f0fc75e390e837b8be90a670f5803ee53eb75c3ce95b2853b2342e54f86dd9aeb308eb82ceb2bae7b3e0b364d17105eb61b3843f7206bdb6abb818efe0f0d3b1004e370191e8218cca14947aa8070f7c66fd0422b02ab4a1d94fa46197acd24e272c765667353e819588402feb85d7f00243521d0e7a9d9e70753d8b51d374ea9c8355536594bf05a6960ca7176a4b66086b055b099e315a23e042a7e0807316d7a11a657a6dc9043806e248a9af06570f710af65267d436a5fcb001104fe8a7c564afe075d85bc0a2ce3d33d8d93d5ab1e923f51d4ef26cbb6fd4a935a97cb115aed678e75d5d67fbfcd2362cb3d74ed6b9b9fb0cf82569a474a25e5aa39d22fe5cd301045203d9f93cf5c9e9e9451f1bf3566eec75fbd995cf8c640aa68fb04f5419344057fd1c0e655d750a68c523b0fab24cab03d7393ee3a5735039daed52895dfe7937f55d7ae9a8c0256e9d638a8598452f5329353a20c4bd9958c +P = 58b20979cba48a9dc95a8857f5bce433087ff93470fc62546e86e72dfaaf7b233ffe428802390c1db7cba00b1f23678aace4a16a237b41d26bcd83d471030929a34e8467f85eaef070b9b74a57f13e91b4e95a3c0b8dfea87d026196a10168c152c4ac42718989003b7e688ca43207034b674d3cbab6f57db6513f8883d27f2280c742896a62e7d0f3f20377e98a0688652d270887fdacb86daa086ffda17937e6d20e4a82667f80ac7749a889b0d748e906d653f569b86de2b42b5819ade9c92970d4caeeed8cd5759d56fd38205215bd8401b2a5a000990afe6c9bea8d091171e85ed83f45bb5b9a8d74cae897cc36f1eaf0122693990b1fb57d0025ad6d92c90885accb649368fe237c4cf017787609fb93c9ea5b413847a9fcf2d2ccb6283345a278619abf8dc351682928187bf92551a820939ec73928eb9930c48f7088ed0a367882f4a8b20d754c5f06bc82990da02227923eb8d1cb73c23793ea0d19bed4a9986f0d48d7835733d1ed3396ec3cf15e1854473b05535261251f4f0af8a0743b3298888bec2f7656493d05eb2d9b848e6802845fb9f7835b50d6a0f0e6cfdaf9b1afc6caa6573b3350256e6f23cc4681316705e33eb0a5f664b79be556cb1bbdd0208430cdc95a35f61facbe7ca2a9bd329e4a1fa42aab9bb02f6519a5672346a4cfac1b96a969317480dd995e339af888fc0e43692332d583fec6215d + +COUNT = 3 +K = 1726706350c11e6883955f24ea11ab247ce3b2ab54d05e67ad9770b5564483dd +C = b006f26a67d0e1e2cbeb5c23b6b300adc1526d1f17bbe964fe8237ae244878158e6b04cb488786b5258ac973c3a2eafd7fcf3a7ca6c825155659fbc53d112bc78b3a770cf059fdd5e68f2b4bfa36de3721231102e5041c947fba3d906bff39592ec3901a398da23035f1190e99b58659330cc2e856ee87ad4197dcc7d16e1f062275bced1ed5cd82163ae3e58da7368dc2aadac855385bd4fa0b8baadef608d0a5c27172d12b88c70b136eeccf37f36364361a990dc50815743cab1636e661bff04ca8345520c30b935a060b450526b1d6ac09170e5b0a327b88f42327b85c9a621d2ca745963c2815a2bfcf509d50b6058ed6e67f369b5608d2aa885238b67d1b8e0d83f9464aa473bf109350fcc02e360c2619236cbfbf895b607895530d8d3d2e41450750dad05b1c37ef15db7fb4707597ac252e8e58d4c1ab2713b427643d198164c908b5d8ff36e9700157284009c7b283633d8b27b378bb65eff8aa59b5fe5e6437a1d53a99c106c2c4d033d3d23950e313a10eb31d68524ae9f8e4f56437acf66db3e8f77407a15bbff4b393e5559908993146d93c673d2aeb7d4cb8fc8d0169de7ed6e2bbe6ce9958a0f5d201419e7acb17e47da827ba380d6b3ad3b5a8c2101c5fb501110c727169065f23297947f538ab3ec165d61edc1f6a9e1735e9b7fc06d4d3406cf8f9c6a68b196cf262324a986705fbc802cdd2e6b4ebcf68e6bb9e793ae644 +FAIL + +COUNT = 4 +K = 32e57ccfe7563dc0a20c14ee450837a33606c086ce1467fd7ec58467154338ab +C = 977d9c5f6861a69e13cd854299434e348cd0690b4d04e08e0598b47eea621bcd8a22838dc9c35a72c35fb1a6434718d02fd24cb4b3dd90b0430334a938a218467eeb4c373d446a539810bc3ce1e923b7c20d9f58ea931d4f964c79613bce67b268efc44bdb9bb00a68d60037949aec7a399493defb2a466e33d4831efd63ad1cb89e00b530626d2f0165975ddfc4cc5e0f968d3875de0f674b3a517df26480b02b6236ebb377118268cebb30ff1ddf0e280fe1bff61902a017e8decf60753c642f35faf0565303bfe651ec8f0193cf34d4af010c9925b8871f0f8c934a149d874a3b659f78ad148428aacaeab80b1b25dec8b0f7ce54406287bc802ac2c0cca3db4adcaa8400a8636ea339b62f5e94d5e32fd3d1183b374507a2af620ca1346dccc9f83a4fe855b1c0e91db9e7c532828d0944d9a81b553ebdf35e24119ed8164bd0260627ea011e93bc103f208c76498ddb8bca15fd05324da5473157feedd54592aacaee68852968eb54c69eb1ddf607917c57493ea380de0cc6ae304dc49cab80a31b8b456986dc367c70f144e52dd604c8d5edbce5de5efb30d9470bc883445b34fa4414f44bb94a64362a12b546665721fa6db82f0c947f015978412b2ce136c471c98b1f908315a16c83e9318e64508c7e179a4429195a9b1ccc211a1c1d4e4df15c5ebc7ab90926fcf7da03657159e440e93adea31ee35f72f2399f5fe2f8c560c8826e23 +P = 80de48ff805a88f3b359451bb6df61def9cb3551e64fdd3a3a70a3b6d238a69311a85bc5924e395ce92ef394b1e5dc301233e9a212f7fb86272c42ddf5f4857c38d0dd259dc1d663c0d729e033d9b0f7f01ab1f8f1b7192d40921ee0d4696a3e35663c5ffcff5ed167660bf6b4c00e619512a2e827be33c90eecc539e18acc8c76eb332b28b1cc502af571242342f63d155271da3211352128aa0af70c9ce78dfdf084a13049b7bb6f2bd10dd385b412d60bf1ccc9fae1208f39dc53db471a04d0dcc3703b4f7b95e72ea815b64a1499865a7ccc5b740999e76338e1b251c740d75274150a96def8760a08c5a8a6f58273b079c06ee09f79a976eaacc8a04c365bc61a786b496811121c386d274c413a2fbfae9464db6ea775233193395740fc9a5eca1a3820d33f6f7b38a83ccbacaac16479225e108acdf46ca35e573151963721b73b3e1c9a12effff0c3a622eb9f07bef7ae712c96ee3ba245597fb8d511698d6e819a967e0d1868c0c6055333b7c13a98cf63d6a5d87779a95345ca8b7e9e597ec588e96f8fc2a7f0a0b8f1543d9e362a911dfb1f03132a4e6af71b503c41814d6b684a26b8df00cdc657ae129a1f2a18cf4b78a3981de68296b1268609fe3ecb9928b90df4553be37319fc508096fa54b35e4822328569da60a6c660f30c61f02f4c5ab2527cf36cb7da8d7dade4c714ea3fc2da8f65b4199090e114dd + +COUNT = 5 +K = e3982db2032f2b4ce658fc44b76f5964c45cd31bf803708982ae599186fc3765 +C = 975e49a4b9a770957a1bb2be920a4f39b9cfd69ba46983d2473d631c08132b9bf61c44510b8aa8bd48c70a86276aa1149d8fdefad511d15d2e2037d9e920e640cb71a97663d19eb90d0b74d9764d03e17cda87ebec6e35ac2003cb75bf9192920d910188d78e2e664255fdf6c9190319d34adb858162ff0830f37fe1dd44003d3d5a1f9451949e368f46ad1977ce622daadf8483a1f60359992b9b366e8a81ffbe96cee45d3aef2fd0ad8c17cc34927af77a0d6d0c5deef3b4a25c82ec388667a493bb0599ac492b351246cbad6d283bf1820883afa48bd909eb7304b9fc5b7d960344309133aab7a85c49f7de396926f50bc83c95900cd049eac1b387aae7fcba5345496425f9216e1fd15c20da75fbb26da176149b40a701e15a7bacfe899e3ecc534ab8bc5b7bd081fb825b5f40fa57e363d7bce40020e73f638acfa097b89c50cb9edb0bd6d71d429b8003aa5dcb7d61792eb3bcac795954c625a104209b373c28cf02038c3318916edd2b818e6719ec154cfa56afb2f337d333069f915d0d35edd6c278fae23c4440c40be462a1dcab23758e4a7fbe8436493f58e890092ea71cb8bcf1336e9ee16b852ccc488f21682dc9f02bdf6c56fe8ad04d84a3c69d8d06dee3d126c0a75f142d0c90c256139acd4b719573e588b80b4540024a05a35044cf58d89673923a534c3816492e62379797cd6e6a7464da5eaaf11ee7b9c27b9b03d7b53c03 +P = 43d38ca132545b154995cff07082611cc47a6467a980654d2d1f1ccfb3bcd387e9d7ffa281b0e0b00dc8669207e0d8033e9e36613c98978f8644bb7e505fbf491dcefbe19589254c8abf859dd65cb94dfc99e7b9d3d1f0a31f21285963e1f7b45c7490a522ff887786f7940fb6192f5081ce7181944bdca5c5bfcf2589f9173a682b78fcdf971dd9f4e8529033e15cde560984dcf796914206973dcc13f8c9a24b25dd00c11166ec6ecf33c6ad9b487847abd7bd29b4f3b9c8dc93a6a5a31723dc03245884bfadc12b2fddcc82409d7b14660af808d4e8216157bb6ba03a319193ad4dacbd37ac884550962a4de26ae923f8d74f2f694fcd0aa74f2e809da4689aad9f2820684b3b423ec4a7da0ce4a1b599fc21bd2779653283b0ee81d7b0d9fd3f6d1e75bd71af9620630aa87b73f7b12e68ddbdfa02ae86ae06b0b1aee4a997d34f61b466348b92e36f83652763084a215c47dcf689df17e36b64bae3ca1a2cc22c837b5907236833c2c1e5f3ddb74165fb6f0633990122cbe4af8b5920b1bb6961cdb144ea8d7b245d0128ab76f4fc0189ba97385717e89e0f99c962ee8c2b6e55546a18be0ba3dbebf7e4140eed6aa3558c43115b65b6f6e8e8fb4b9cfbe0b6eac006603667b28cefb4dec037f33568a3c94d9e36539e91b3199d728521a9a6b82b96ff1c29dd1d10366d0510f1b9a9494cd104db2390530be3fb6abdb7 diff --git a/src/test/crypto/KWP_AE_128.txt b/src/test/crypto/KWP_AE_128.txt new file mode 100644 index 0000000000..a682d4bb32 --- /dev/null +++ b/src/test/crypto/KWP_AE_128.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AE with AES-128 cipher function' information for test-files +# Seed = 7c0f161159cc9e338308ddc59a655450a030832b3d4576c568d63725d57b020073aee9c77d9fec34eab358b83bf5aae4fb52803343bc03d472283edbebbcf75c +# Generated on Fri Apr 6 14:46:12 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 0e54956a24c7d4a343f90269fb18a17f +P = 817ddabdc5d215eee233adff97e92193c6beec52a71340477f70243a794ce954af51e356c9940e4ab198f0e68c543355f65ad179cb2d60dd369eaeb9ed141fb18c9e4054ac7fdc83506896990a4d20833d2d6e9a34938796ee67c9d7d23058544a4a35f2954103ce443a95a7e785602075ca0a73da37899e4568106bb2dbf1f901377d4d3380c70fa5175ebc550481ac6f15986a4407fde5c23ff317e37544c0a25f87117506597db5bb79850c86247b73a5d0090417d63e4c257ea0220c2c04db07a34f0ab7954e1dfa2007a1466795c4d0c2aa09ca3986c028185b43a466526594afc9c891c263a7c608304bc1957c9873f544dc71e6f847c48d32026ed03b2333825452ee7e12a50e1cd7d678319264c65f78001996d37fae7f9861fbd21cb506c2f8a3b0ee53c7debe17111b6e3f78a5c5677857b082c2c4943dfd1edf6337fea98a44fc25928361156ef38d865948b979cf6f4b46bd2119f12f0891cef7fc9d0638fd105fc05f9968d16948d1cb820751e82e44cb68e99d4f072ffd1577da6c0631b5827bec7e1b9ec72d18b74cf5f233e85013c1668ceb5d7a1f5e0f016b0ff726a0a9d41e2cea8e14a2f56492b14606d3fafd8ac141335f39f90d56863735628e8f17be90e100ef0785f3cd57db8b9d89a6b2189dc2ea00c285d2657983f8bd7883c215477e67a55556401f1d8b27d4e0d541c7fb7ace370c2e428884 +C = 876f3e53ba9cf4f6a521ac198bc813d0ede0f862ab6082e3e0a06ad82b4f279582f7c43bb63574608446bc2a05f401a68f74086cf2776b4b3df6b3679c2edfb91c024db54c6831e0752ae6f86c7596462de905ee0be908c1b9d043ecafe2ad1cbddb904e18ebc9b7a107031be3a87059516a3d1257812d9c801b0b9f21539e70c47150c128d87c5e58fa6e4371aedde69c7b5cd16b73ac422676328131f3ac48c602bb6e0741805aad9d23b33b3523b86cf0588cdf9dc6c4d5f9fa43d88ca17976eaf48fb37a41a598266da04144373df5631cc5126341c200a0c8499b29ae96e6e6e6c2bdf8d8903da62bf8ddae970569b695240e77f8ac5b191da5034008b6ef21936858e69bac372bbafd8794f6b03711503c1875528a9348681844edb199a0664d740f0f0b1f866c4248c80fe8b5700a3c4134cdddb17676e0cd37d6d81831a0f4adfba071bb0935502480eccd48b28be5954ea6c7d873b51b8bd2b709c5b6132ed31296510915073c18f7012f0eff6a9aad5340a19fd5e372d35260b718d9e4807b1954c24e6a4fd48e4dbb8f395474e99ab577367d2ab5ccaa18c947331047dc3986e213a878b41089aa221019dad4191a4feefd095f8606c2700a46d71cbb13efb6957df925ec26071c04d04d5a94e138e5fc5d1f059236aad76208077dcc607b1dd2086f9c04e33f955822b457eecd68bd5f24836ecedbac675e6ed93d8a787cb57ad68e + +COUNT = 1 +K = 6b8ba9cc9b31068ba175abfcc60c1338 +P = 8af887c58dfbc38ee0423eefcc0e032dcc79dd116638ca65ad75dca2a2459f13934dbe61a62cb26d8bbddbabf9bf52bbe137ef1d3e30eacf0fe456ec808d6798dc29fe54fa1f784aa3c11cf39405009581d3f1d596843813a6685e503fac8535e0c06ecca8561b6a1f22c578eefb691912be2e1667946101ae8c3501e6c66eb17e14f2608c9ce6fbab4a1597ed49ccb3930b1060f98c97d8dc4ce81e35279c4d30d1bf86c9b919a3ce4f0109e77929e58c4c3aeb5de1ec5e0afa38ae896df9121c72c255141f2f5c9a51be5072547cf8a3b067404e62f9615a02479cf8c202e7feb2e258314e0ebe62878a5c4ecd4e9df7dab2e1fa9a7b532c2169acedb7998d5cd8a7118848ce7ee9fb2f68e28c2b279ddc064db70ad73c6dbe10c5e1c56a709c1407f93a727cce1075103a4009ae2f7731b7d71756eee119b828ef4ed61eff164935532a94fa8fe62dc2e22cf20f168ae65f4b6785286c253f365f29453a479dc2824b8bdabd962da3b76ae9c8a720155e158fe389c8cc7fa6ad522c951b5c236bf964b5b1bfb098a39835759b95404b72b17f7dbcda936177ae059269f41ecdac81a49f5bbfd2e801392a043ef06873550a67fcbc039f0b5d30ce490baa979dbbaf9e53d45d7e2dff26b2f7e6628ded694217a39f454b288e7906b79faf4a407a7d207646f93096a157f0d1dca05a7f92e318fc1ff62ce2de7f129b187053 +C = aea19443d7f8ad7d4501c1ecadc6b5e3f1c23c29eca608905f9cabdd46e34a55e1f7ac8308e75c903675982bda99173a2ba57d2ccf2e01a02589f89dfd4b3c7fd229ec91c9d0c46ea5dee3c048cd4611bfeadc9bf26daa1e02cb72e222cf3dab120dd1e8c2dd9bd58bbefa5d14526abd1e8d2170a6ba8283c243ec2fd5ef07030b1ef5f69f9620e4b17a3639341005887b9ffc793533594703e5dcae67bd0ce7a3c98ca65815a4d067f27e6e66d6636cebb789732566a52ac3970e14c37310dc2fcee0e739a16291029fd2b4d534e30445474b26711a8b3e1ee3cc88b09e8b1745b6cc0f067624ecb232db750b01fe5457fdea77b251b10fe95d3eeedb083bdf109c41dba26cc9654f787bf95735ff07070b175cea8b62302e6087b91a0415474605691099f1a9e2b626c4b3bb7aeb8ead9922bc3617cb427c669b88be5f98aea7edb8b0063bec80af4c081f89778d7c7242ddae88e8d3aff1f80e575e1aab4a5d115bc27636fd14d19bc59433f697635ecd870d17e7f5b004dee4001cddc34ab6e377eeb3fb08e9476970765105d93e4558fe3d4fc6fe053aab9c6cf032f1116e70c2d65f7c8cdeb6ad63ac4291f93d467ebbb29ead265c05ac684d20a6bef09b71830f717e08bcb4f9d3773bec928f66eeb64dc451e958e357ebbfef5a342df28707ac4b8e3e8c854e8d691cb92e87c0d57558e44cd754424865c229c9e1abb28e003b6819400b + +COUNT = 2 +K = 5f0f2428ad83ea04642d8f90e31f00ad +P = 05f65580a602fc839c5372778bc495219ab159935acf3caed10958ca0e135cbc44a896aa84b1f6b05d65ac45f5930029a223977ca46c7717809e80d8eb91e7ccbc525e69ca0a85d3e45b2f9bcf0c86980b7f9f9eb20e9b558479aa8c814e6113635a9b8b0e264055da68198303ae91683eb8ed3351dea3b8280a504d163e1cf284834e3641a26868e23fcf460d279d3577115d8c4cf0f54374df26ddfbf0c1df5bf71fe509231177ac531514ed878d3ef44a03685dee906ab394bfb69cb56ff973f19fb466ce550ae7b03008feca7046bae160ad47f28c9004da68e594bcd7f9ca8674a645c50768b5ee0ec2088f66602a429f95492e09dc567c11083eb18da97d85968a86f6ad05054f4a9c4bcd57ece2511bdb38afd78084b0a088112b81b993a4d37674d97e8081795ee055d9c37d2907d859be201e73580ebd439620918e7c4b42243f43a59b475095205c40a1d37c0f87a5df19450ccd5f274f186b0f45f7bc153d30ce3c873d7bba5dea5a8bd348b2f0b4f25461927642a2991c3c4bc3ad34d09c104322bf27e8eaa0ca99c0dd47a3c4c618af02599192e362e75de53db336a68b223567bcddcd7feecde0aba91a1eac0ce6327d3c9c5be36698494b2b67ec0efd36b744d794f521bee251b31dd29852a94e6b95f143a53f93f85ec8c2e86e1026ec949aaf97fc9b8d728c13e1deab628e9624ca966992efae61dcd6e5 +C = efa9d549d0e975789a6c84b99991c98cf20db4de6789be6f9f7ae109e322a6f04a1a1c1fe582e4242a5e2f2b390827d7e52f1bd97a4e745920dda69a25fe93fa5de82c09dce8a2593106667820dfa1f55c8a1152ee02ed0c607ab986e5209859cbb36baf90ca63285c3128fdceb18e5145b9d3efe6bc7b8bac7837637ce5553d634b13fa3888201112248bc31e0785f2dbae1a8eaa7f4ebc0b00d21242f03acd4e4b0f4ad5e9bb1e25301a331a4b5535c1253b9ccc7800cb2e80602ec9ef858865a39e0ba28006da93f9c26c1fb9ee8258dfcd7e79f02ab0d9c2df9ade309fa6c604fcbeb38641b2968ff96349bcc34927cc479c30153125f7ad07068e5a4ffaba903d27488c94bfb694d8277251061a7dcc5b5fbf47ea88065ea1df007cbd4c1bdbc2e3609d53bf99bf2c7de8ef457fd7b85882210fb52c5093415dd2b37084b75f83d2a05a669867143fd95705f241fa794526bc65c5f6422410da4783340497bc54852125e3da579bfabc9ba98ec398487444b3da454f40d8b59d384cdd3454ad77f38d4ce657d10b4708dd4ecb0fcc3238aca57258e082e8cb86d802b9722388284f232811ec35b49ebba151d5b0771feeb6120d8bee8d55fa9c4de45d8f950752b3b9c6526fc4fa8fdd02c645fb053cef8293437111c85b07e98b3b38ccd3de93cd6fd08fdfb8a3a94b619f9acd226be3bd3ff906f14b1dda93cccae783c04ed545db685bf0 + +COUNT = 3 +K = 9b5c2e4a9e53d3e5f5670af2440439db +P = c4c726199cc27b81c90ee59f1f44420abad2a4fe0f24647952116ec46f364a3dad1b48ba5bc6c48c056e34764ed4b4a2da04e7e45ff477df8364ee9f116c82dcff7e9ae2db6a7722f2261b6cee1ef7cc66820184988e002886a6b636264e7704383543817e119ba3582d42aa00aa78dcf84d0ac92ff69f09046f89e7b608143699f9fda37b07d3e2949e935bba230e0a42c2d41d285042feaa480bcc1ef8236d615f449107fad97b5be514ae39d586b98ed45cc39784c30a3a256d62336002b1f7f6cedc3e5418d6f444bacd74d7bc89b3ec65c52191b4a2f3b3c7b682f254e6e909c5dcbc11d5def56792f5d2183e5e7119f268b0395e7f4f25b8574bc921da8419af30113bd35b11c21debb979fb698752264595777be7665419052cf3265117bce96b61f27da4072f828bd9315ef25a9ab9eb4f5319907cdbce6104054c8563a997cf477523478e6d23527f91997dd43ebaf629003530de4b8406bdbfae49c0d97920b745c78a665c438ff064d24803382f79fcc2e24265adaf707394b832685def125b69b1787719aaf92e518645ac22dd94d0dcb03f052c87954854d177e2caa6ab5d9cf3dc4ed4ceca90d2fa0b235efd1b7d17ba9758987d127b11204047f22e75158720feec549f29b67a8158e577325e178d6b757770d56a30b85d12ab5fb164a07a5f613b53d075c44cd3c1e3a311515aeed2427f70b3bc018dd796 +C = 09540dbbe97e983a3420ec1ef7d1195f9cd32acc93e9f31d07e68dd9d963f32bb23565a37d97e857e5e40eae27946ef83a4da10ec29ea001fbe892133a4814b04aa3ebabfa295a19429c7e6b263293d07a48f5b2beea2930aabe6268aec00a8fbcf9f4865d4658905413a0f17bb08bbc6466f8d3e1744d9ffe6132f94ce79bec4ffea05785525eb5e071ecf30081281df58f1b5295842967a486ee262a7c366978140bcb8562abf1a417c7b1ffe07b1748978408cfae4beec8290b312f15d3223fa7447f5a89e52ddfdc4f79cf1d9fe4ecad3e674e98136c457e36748da001970cceed0c61d2ea5f9ee5f9660ad561b9fb808be3d4455fe76ae9778a49526c8461f99dc2c271ff9aa389d3e6a546a82d15967a82e8ec9b64cdf25fb73afccd7cc2fde8d07ef4a1345976970997facd5be16566c7761d23f4a31909a5a31bce35e33596a861e426fb56d9ea13e5f6219aceda2450434e04c454ec8569ef9076856cb48f36618e690a710dfe8717526001c3be7eccb671dbb4430289924ae975a30b6fc7079f8c9cbc8a268f5defc03d94d5bae0474987b7ebd4c9e29eec9080e8d85776c570089fcd235372518dc38e481f8a5edc2105bb4191cd81a19e51c60252a18b3058ac1ff9368af59a2caf82a06357764fb3721287bd28204d77dc2eadb0913cabff7cd63c73488e92c818341563476e69a50fbb781f151e02f2df4a9fe05c8419a3393d67 + +COUNT = 4 +K = f2713273ad5b7add10d2b8d8b61d2f68 +P = 1c189d6d34b57c699dd8c79e30457c6735f3697aa6825e14257906e18413580727d676b6e667cd58e0a47b738abfd91144952940de60b161b0dd261d68af12573820f4f172b32d7c032ed7db3715fc6ceb92ec2d614a2005ea1aed14cc5015de143aaf6dc2db406e0fd31976562250e80eb79140bfc8d14fc79bea4159d66e1013704b474b0b1377220741dc55cecde0eed4e0931e58025e7fb672117a8288587cc91b1a41fb4d1e9e4a7ff59fdb814dbd9521dc148abfe16ab62fa45a3dc361cffb422482f4fd93466320a11d9a5438959947ba51d7761b2a3bb9e0b843ea6a71cba54a96bd14de8360ac8f16ece773d7bfb25df63c1aa67a7bcb587f618f2425bf9aa018eba5aebc2911b9fb6276e48d72dc74836a290fad0f5c545934f1231ed57c6944a464905fb08685e815ecddc8dbe4eeec1355f414fd2ce975c0dbe29bf4369e05c1582cfd9a7ddb7659247a0f6db38ba165f1f46a472760bcd2609ffc8bff49c6f54698063fa81305696f7ce25c95b237c6a4fecb3e48167060f556359609b73e738c5c3021d0e5bc62ed31a2a27f43dc417c0041f8a2402fa7bf76c72e2b290c060efd93ef65d0a893314b03c098a6674d0137f12eba3ebb63b3e65fd0aac89847bf9573d9eff0b1283843bd4edd6d459de6d559e48633d964dccd8b9fb5142596430b2373f9448606f9286157e810dd0f6a72843b0cb28ab452d7 +C = dbeedfe9aede047a07402d23b1b349a3e75490d3c404240c872470aa4f6b5a1723b6a0e455a8961fd25f88a9329b4fdc7d8e6bb2b3d7403bd65418f94b933467d882c1aaca702a47e7bf44916ce5eed0647250d572686c8a8c9f9437f2ceb13b361aef929d428c35cd4cf1d8970d98e75c46b4825c871003eceac2412adb6681f0b8a1d6d6bd7eb41c6e58ebe6909ab061d5e832f7580dce89eccb4ea2546204e895ccb54429fe388b0d9173922e08625d43f0a10743ad76230fa458b0971028b18e0f04f3c1049012a8bbc2f93934e0b51b943efd2fa4a14fdb795a1f21302529653583edcfa3c613ada2bf69a93e5a29a20a6f9d9df7891e3b438be4f9f714806cd57afcabc0d5f5753af804d885ca1ae23bb5fd2593fa361e92e7d12eea0805453056022256410369dab0e2e8d2d4c9ffda1d7df8ba5ee19c5bf62133d1ca9a713fa45157d327bc4e1d8e7a105012036939a5d5ae15a5fe8ef989c8323129b8b4e43402bb4f60f35483685893e9ea0b0dc4a8238148d0220cd9bdd7dc42590cc303f706cd81cc5f6ce761ffdc570c62cf734cfffe0ec7be17c7ae3ab34c65f1a53267b03de4da3a5f921cf01b068c4098c7594a3051bb03940cb6c5bac2aa04f95bd30bd15ccc9edb8e29091047de011ee9591cc4c83244750dceb8c2eedda797965c1243521c19228eb2d799775dd83ceab5aae082cfa743d58544b61b7104f3d10f632994a4 + +COUNT = 5 +K = 8967c8058742169ab39d2d528130df80 +P = cbe7ac6c232b327f7d9154acf334891398fa0a1ea64a4f15243f488ecc784749fec41fab7a447e2a2cc75a44fb548a46a46e51f4c245f64901f3acbf18f468a28367c51060c34762c8a88780044761579846557f285c97bcb9c966aa08d4cf352c26d8a2f308803393155437b91776ee74941da9db39ef64d9ac9261f2c7fb6349e52111a7c146d5a4ef6d5d00717ac93338fea5ff0c90cdefe0d66b039045330877d8596de4c35987ce70f660aa8777276eed1b128002dffd90ce03224d505950eb2c73f5e958df92ee1cd8dee667bbb8cc71f87e4ce880585f004160c798ff2cc195b09921c73951332bd437840bbadcd10730272d46b3f23963a7c0396275407bb8041a6e5ceed4d1dc2a65a70d4843660cc039ce94a4ce96573811d2b8be00e91dec7486484ccb050dfd6e319001ba8b8fef48b03d17902cb91792dc420c79e823616cc79af4e738dc077519015c734c1be0150921657f3139e59ba4ad8e75e0f7b51806a96b7249709faac3cdcd53bfed8f0326d19000ebc1ae05a43e96c1fa02f2cfadefaf871250aff6eade8ce8d4b8d5a686d553e8a66d96412f2f189fb724e67c8acdb229501414e45c430a9533d211a217eae0030ff10db12a399567e94c900ce839157ebee1f4f876675794cb3642ace55fbdb261f1369807f284f4e6f8a2c8330eed88e18b8ff564fb754b96f8c85d28dc7a0b18805ecc3ddffc +C = d29eb0bc54c43081030a3e1169f243eac73f1d0fb2b2f983fb25755684df4331a1932b02e237502c1b73a8d744619f521457b72fcd7401973ec48ee915054e5e12577ef1212d0e642c9dc7341cdaffd344dc480eaf9f29019355b07de2f05897cb91b00b4c53714609dea69f18c7c93429b73373311e0c4a45e2f2b3a6bd393b69f90a4125eb635a794563a68e092cc66730d236909cce6585d34f47a10c2968f161ed79c16c18a6c093c1fc455eb502770ee436dd48c091a4cefeaf449c5b27f2fd2f9150a601bccf8a37cda65b25917728ce3f1f12f55a51d9110c16132bde100b6479eb8a1ae670a9ca396f18a2ff19fb5543dbd3b189e51ba5f12516129071c997044df747f4ed70616e784bfb948a63211b224dfdfc5b9116b869fc5313c761d55c759a6e275e9ac53e83a5386e2cfa50f6c1b541cf752da800ff991ff896b79b9e691d67e96fffd166a06f0c300b7d9bb552f5ec7df6eea384751632ed9357df5a5a9fda931568930fedb0536f5491d4067a3951dc72903d1ca2f5f1b713e24cce5347b056e2f8829d34143032f53edb81a765d2ab3e6cbe12536cd85ac3230fea546ee50d991e13c562bea3d694d3990b1783a373a59fe12cfdb74fa0275e367d76bdef698dd33c30afadbbf0fe0dd7ddf288aec7a397e263a5d537df9f606f5c126565c06ed5e0bd6753d3eaf5002b472a1d7f4d7df7d9a8fe8aa2c595ecef8cfb3da3e7 diff --git a/src/test/crypto/KWP_AE_256.txt b/src/test/crypto/KWP_AE_256.txt new file mode 100644 index 0000000000..66d8523987 --- /dev/null +++ b/src/test/crypto/KWP_AE_256.txt @@ -0,0 +1,35 @@ +# CAVS 21.4 +# 'NIST SP 800-38F KWP-AE with AES-256 cipher function' information for test-files +# Seed = 6eb04695fd5dc2b0038aff221b225c302fc9cde3d6dbf4195d9d4d2756c8303906335e07679fee9a80e5d0d1580f7cdab8f0fdd78c0fa58f34afccdcba4820a5 +# Generated on Fri Apr 6 14:46:23 2018 +[PLAINTEXT LENGTH = 4096] + +COUNT = 0 +K = 20f31cded60b8ed8d9d3fd1e1fa6244e76c7cb7628bfd28a5d63ce8aa2c9494d +P = f07225202842c8dede42215301e44b9bb7e625d3812f74f9b6ddbcd024ebd1f33e2cbf280b9004941f3cbf86c880a2357f88f92a6dcf8dad9da7dddcd00f3635efdff0af4382024e93c2af66b991e565eacca6b886f07178c9b4adad6f0d6ada5ff6aa7cd0712519a947a8089cea5e1e3e40ffe1806010b0149f9ffc7c4dd3c31b3d08d5ae1997c52369393d58611dff9bec501c1ab35e6ed3e7f9445a34e211010a8236686f154e0a5ae3433d6a844eb3884961aa6592216d93952b46bb58a4195aa80966ad0ccd4a7e23823912556a90d5ee9c3bb952ecbb9d895dabd3b11ab4f2e3a6c2582de50403289230ef4dc46e7c0d870a3f0cba9d643a0349503c1b162ddb6350e699589eb47bd563999f55a1adb6b78b52f006901b0427ea7d3394bb0adae4637b4f1ad5d5425e2c8ff3083506d7ad7ba4c7405a778b0a3a11760c96900a5256956cc9710091d073a19f46a985d004651fe2b6448ed761bf9bc81619cf273a6783d868d090753bf01318be21afd88d9f3a961a69f93e9d9fb822c80acc7b48cf14a08b5b7ef15c66975721b7cde9761a145b679155472a44dea8fedc0f86ae7ebf6283ecfde5f2444b51569e6723a7a19e28cdf8dec6791ccc14af95abad018f741575b343cb1a20a2a9adf4248f99728069a1e2e78ad8966c41c9918fb7019ef56c153a183a6247d22d9956564bb03075cbfd1b43d96818b28484 +C = a5b63618fc0c4512960f00a1f226d9837a90480baea75265453b9553b12a58c72153080842d7f8710f317f88fbbbf97caf879ab4bf416ba767ee9aeb34357f4a2d0e8b9571054d98e28804a70bc4d74807f2bfd95ee955bfdbb6f4d6969a0c3c3b541a514647d5cd8c9740ac3496095c3f145c50c97ec98b935158fbdf89705d5330015e48ece89188b8c1bcb2ad6825d865b375a9b9056b743dac720feeac033c9f757f6fe73dd7c4a747661b64cf490a0dd43b547cd791a5d78dac97efcd355f7ebac248fa2a33e4fad640dc34e0d40b0d36588aa32f0864c9446739a6b44ff84666d723bd7d646c5172cda932fec34ddaaba342b02a9604087ef042a2be4774194b5d32cb3fb112438fbf2801050b5424635fa2d3d3fb10332965c73e6669e65195310a3a30602640e9809179cdfc50de585aa1c0072423c626815d281a06eac3b6ffa137716318e288e3f9970e415ef0451bdc557968febf9eb6772c1f77cb8e95701246d9c567048142bb25e340351b87d7391822d9ee7fe51378bc0d08135f9f39cf44b348b87937939dc61f430dfe308cada632722e23aed5a0699e039cf0563ab8025163744b136a13ce3c62c748c89f5e17540f105e7c6ec9ba13515b504342f9e6dc7d65b9a633d8c0b5c9fa858dbb9b3a594406d478a81bb9abfa289730408c1e303c663a61d5caca00f615065312580042862397b9aa8c80ca812887664c439c8c68 + +COUNT = 1 +K = 85c8aa6d9c83cbcb11550c95cb9fa991d49dc89fbe2531e10694269f38e9a309 +P = 58ffd9e73b5f64cb951b7df799a505d842ef0cca6d4b21027680793565a67d47db0bc872a7640691216c044fbfc0be1cd33ffe9fc73c5ad8cf38bcfe6d4b7b76639ef4200ced0eff2d361e9c948075f09398fbe5ef24d370a5a16b515c102d415813cbce56f6a8f7b5227db7aab33c4d8162ac3c178e2344f3c7e123f0e60142112b960137c0176c450248167e9714db8566e2d4af731a7ef3f88730a4be4cb99bf46eba39527ff279be67aee1f4af25b3396aef5883fe7086fed7285af5216c7945d9bd61d907a84095d6fad383a7a90ad4061f7241a84358cf9fbc7048c010c7652bf47f9ff58f1c0cfd62a5fef92b02074ecbf4762af71eeda9aeb8497d7b33beeb4fdbd80b72f2c6dc9a12409cf7ee3716fc07515970411ed7e113b7ad9865d822a502976bba1d2ef254dda4acdf6f55a04c4ac5f6a5e011375af4cfd405d1220a10465a4cd01d160d179fc7ce614ef4023d55dad363e3639f1eb8fec716f21d267ca21b0587186d162b4a3cd23f80e33ca330a8c3bd756e70c2d457306cd54be9e0433bd9416efd861ed42fcde10ccc98389652b5c14cff6c73341320c93ee457f9e3f320c5c5564fe50eeecf2d7a62a85639ddac4abdad7495e8926111b18f7866e6b3f95c788d9124ab9788210b1e25524427e86049726200052404d9b3585477bfc11f12c4fa761ed86b7449a5471ce18f964acee913946c0997980f +C = d4112af26f37a7e2e9cdc0fd460cf4cf4f3985fc820348e63a095751ce30b2fa5d6e89a8d37b8ddd3dc2af8fa64e0db729f526bab89562ad7f2bbb9ba02668033ebf6eb742904f37fbf1d87b4d23b1dfc815529a6e3f24bb31ba1d4354820a47e0d7b0acb4e495a50912824715d24cdeab0be3597dc535326ee073eb39b925c7699115dfdffd4162b84afc61048c7802a76ddafec243b91791a26138f202bf365a0b0863555d48c953c50cce5e09fede4a97571d3ebf0582c57c39680714750fd2227a632274adece8a4bd13413825b51969b3e965ad417c96415f4041d8f0d0a3df04cbb24fcfbeb95b401be7ea840da43606cd5bd56bfde26034fe94aadc78bedfaa4eed77e64cdc3d059e3765ff6346e9344f0e0a183085d78711b13d4ddc356a669693d2ad090bb67e8013403c51c81951f8a6241269d3423a739b44215bfa403f8927fc1d100587d602a7efc505f28edfe51ae8df407361fa06c09d06c7e960cb7b7041f3603faa6b4767a8a3c3d1c8ab05f75143a3061396c7289503ec29237eea7a364eedb77f66da308fe8e194c9b1784364072dbcc8aa1145db1a03f3862c38cc1a08b569f8576623915a644700ab43f7eff7e59684eea5595c31b13778824704b4aaa9c1b5d2acc4a6cd2ae21c5b0b32bc1433d536e8bd7a4e6c5e6d1ceb70dcd2ac215bd4619aa1ea8c184033b3279cc9fd094336c5e6f5112bd9f36fbebf304ab04f + +COUNT = 2 +K = 5309c5dff8fb03b007c28bda0a219d329080120b430f4b5af9df60ebbe20890c +P = 22b75fa402aa5b665aa71db0e24e76a6796329fea9128dffbbe03764e23d32443cadfbb6d557e833f841ecf5cc7c9200de81b2c58fc72a1a9451ce51c75abde6504b7868d1864a5824d6ef5dbd57c80e488a011102f7a321639fa57c40a5eb8479cbbc7428bb5497dd6295e270b53b00eaff17c2aac199a637e2baabce9c3f5af8d8c4698eea5390d7bdaa92ddf8190361a2452df2f44220b92519449605069661e0360cfef6841b734c8bdac87feeac243693a4e114995e1985d5c93e512d0f2fa27f12db8803e0aafd974d968cfb82cd9f0d4b09f34a440cff85e09a63462fe9820d5f1ba6bc6a489f30ed570e44c06668b6a4d3b977a6b7fdff3e781f0b2c7a1361108eb95f569c3ac5dfb1352cfe0965d9bea70f923d9f65cccfa83a9272947b3563f81c92f7f5caff203137b034af82847de0653bfee01aea476c9112267f97507b6949dd1d0d7510fa280432159a0b3bd67a8e628ef11599be4459238161914520b556fd5d91e92349169bd2ec175a9e728ca78b56eb2bacd56fd2af2696d1b116f5776e9df459c1023180a18f5d6e3718ac402bb0fdb806cd7b1cda6fc107e635f6a756005df79b720f3f274deffb31552cf238ec93182370d0ebe9f0b819e2e7225bc2b77e702d5df2825fe5d42005f1e39d67414ea539810db552049641ca22bd6539f1efefcb90dd479bf93080cb9df86a9aa99048f950297b8343 +C = c6a64b49464f5db516dd39f3d1dbc444528b5da57016b5f2a5649965a4781962264dee68d4c78e987f9446cdc815aabeff2590d3e2679a2583dbfd3aa6b8bd3bcd86d66d598daa6e766a6ab8fc6fe98a1bc89b5a137a342a59dfb52c7e7b2feb2a33d5f936503121476a9eee8d5c60b6c4c9aa7f8c56ddee5b4a9d7908890d3aa98f45cce20a074896aacefb6d7520e84b6675d02b819f37929746298d2c81407c20328a1ff4ee6da8298fd639fa9bba4ba27da5e19520434e27ae4bef59b73c3660aba0353701573d602d8135442a14cea4dbca3732a3fc8e5ac831929ad48ca30d4ee0f068f54faafb50bc7c6a784e81cf86b29225200f985af914ea6ed166e08c04e6324a372550603c98bb6c010ea7219699e76e4cdd9c4b8e6c9200d10c51e93a57f25b0019ae436dde220868596e96ebea3086089ece51b93a812c8120b8ba60435720ed690a2d1ff7c024cd514cf73bc7cb121a83222e4fba2dca65aaafd8cb51ac096511f83327458c53119f2462caf6cde5ae679fafd4f96756aa496ae6657ce417b2963c09463ab810ef51b4942dd1ecd120d69ae76f8928f3bcf20465f68e526beb7cef276533270f08136c4088182dcd626ffb2c3c4a357d09affb6b935e671bc827779ec6f260acf2c01f1faad57b162fb50579ed45909be1cb2a4a4dd07effcae6a2342022e0bad504261bb8d5d3b73eea932b0135fbb76fd6b8f8d3b6435db6d3 + +COUNT = 3 +K = 8027a28b92435ac111a67ba0f4af5cdd50e6bbf965cb9077189143440db0327d +P = 0d8f70b229ad687ab5f55f0b0ade81a4bcff76dba57aef40abb8074b9745ad74efcc8d8f2202b92a9788f8049c864e5973403279dd02e165b772ed7444795ebfdafd8e13a9a22ac1103d0c15d8d279c617dafff98dbdb9bcbea923a00967287ea9accdfcc90b2b4047021115d5454d057cf3a27a44988a8fa71578ab7751a5d3bbabd1a335248564f5e334c3925119a29c219aff6e71b3963f603b0f6e8213219fa56a36e33066f49409c643dfa1ab7478630d623c7b77efe07b0ad4811a522ed81a1c905831689747efb4edab65062cea3724c3f9decad27290c4ca0ee9c68f1ad9ce3981a0b712acbe2de17eefb7f9bfb3efd66aae691e63e5d64233715eae8957bce7924fe2f259309ddfe050cfca3cf76c52f96f8b7608cc833bb3876f35e37f6a83ac0a21f0c0fe859133c17c5f7422c72741137b8c3b5c3ed3e7737d5d8a29c063f1490558982496b7c852ec95bc452113c48bba93394dc1df0eee1f8dadd38715e93a1d7065c528f08525cd84583e250327320405bcf18d8011e3d864978f948b2839bb6d1d7bcef4b5b6871243a5c93a2afc1f7fc7d06c8307a025ebb522cf0fa1e1e9f8163f20854514504f4f873d757faa7938bb1ef91000df4ea64bb985d903136fcb9c4e9467561cd564f61146ac519c08dfa41f22b773cc821cd1252a3ba237c1e8c1c128ba40e0b75e67adf12cc448e47fb4037850b014f9c3 +C = c1e82d374a1d7c07b1f061656fd58b5df0a04c099f3390e98d2cd460b2e2713eb7274b9f646fc453a5d26c3fc0b9203521de72dc78292cf4d61095febeb0748658dc6764fdee4fa498b93a6bc053b3d81e10564cc9681aff2296ae0864fe3db495298ac4174f6ce06ae6b5bf86985c4603700757ef44c82c0e03beb7f2841df2dc4dd8022d5dc7ea2f711daacc1c8d1805798f646f1041146891e88edc4db9904f122c74656a5ea1694f8e6073920481e6cd30b20fac4cad7e03f58183f6b0beff8dde3ab4f781a1dc36ce0905e41e77be65c83dc9b6ab69d5d4c15a1de3cae6b081ea9568578207d678df77e20196483c2e00d2dba604784495949285a623f33e07e20687aadb86d57adb9935865e6467801a10add2ac7dfb6f8ba98e7635d6766eed38ca8fa8101f93018261b872a2a899065918b672423288cdaa417560701600424da6db15e0ebb7a5d1d0a7e1cc8aca714ac6bde51f779033563b49dcaeb8e0330112902b5ff0ebbdd00aa371c902092566244bd9a1e5e978af39adbe13a7067d056d04b718133575e420058529a42ec032cc5fa62035e872cec7acb3c32eb8b02e0cdd81c59afdba65b673c828b980756e8ff6b0160a1e796e221ed40c342863848b76e4390ef730abd3edc73ba975c037e6ba5e58162d7a60cd55b61643e9a8d4f6e5efa2485da7c45991ec4e8d4c6d9432e630e703ad1fc2f644d7577248ef7c7c974ce7 + +COUNT = 4 +K = 1a5b5f43929215da78d1d8c8fd4ba4508a66273be95c85a892da606eed04d6da +P = bd762d13e2385d333e435ecd792b8b3f13936d3b8372466fa7f65aa537bbd0bca184ec570acdb99ce59633bace041fb1374961be99276ff1d4b82cd49d78b89585aaa6cb1bf18498f4d01f2fc5e41e2b4fa56cedf5fe69e39eb1bed303d0795a715bb7eece6b389b3ae81b23b4f053137d54c03c2bdc68c79184fcca1758652c61a5e3c19d49b3af1a8946403b254da0f9ef227d2be6d0c528e1a7b78a55a2e79d79d6e420e9a3e8b40bb3ee74935be7c8e935414209be01532c991a871b90de0b35e963989475fbcfc37d81cbac674b1b29da1d0f9e78a6c3ef55575a8d3d7637b93895bae528d85130d059415d7c456a5c8d94897e567d6d6748e7c2689c98331ee5a7e1e25909901ace55828737e7464cb670399af3f806b996f416550d256f354dcdd1c332b4ca9eaef70114a7bc171efa08b88f1afd0429b2cf15366105cb812596a05e22b06debe1c69b06d904fe6cce284531ebefbae44b9c9eb156dedd01a4a73a1e631fcabc349e07636741580a916b35560d02e1cbbbea0a335b395487df8f972cf8600b48237e7b63bad5f08c85b85043eac0ca93d34df56cc486857d6eb3de0c332ee1ded06927e8acdaa3fce78636eb06c0f805788189c56023c8a1f5a2427c6f0f8b993f517a9b4a3bc3fa9c3e6318256dc49941c1e92973e3ae4814b87135c9ee767348045ef7877027c8ec441c50b1cab98bf556352333aa +C = bed34ebd1c70d52665103651110f847d83e19a551f5156c9720e2d1b038ed5d5ac0be8cd97ad3d725a8fa3c127939cbe1a6946558015bda4c7fffff7e00662c4b0ae3d5477bf61e175f8e4ca14cf53ccc558b8fb5f371748329b6cfbfb277cd0828a2394ec99f943775b4a2c1c285127a2d4677ba3b918154908badde2a74b58f73ac8f92b440214a3cc990f1517372b2e53272db9bb191cc7711ea31ffba2de2b646e8e312f4eedbe9e51e182603cf075ff69bc4d845410bf66d20215c160df05ab007b49a5396bf69d702d193aa0e2a2f9d62b7f9fb46cdf11864314101c7b7072f5ba8262c32b12b14225edba48b440501163d027e7fcf2292c9b06ce19dd0eabc6b6639769544d71bf068b8e03c4834c4a785a6b2b927c077f839f29da61c6c52c39ef4354839eff993068b67438266e10e9a421a90a103835829320fb1ad9e7e0cf73bdd22aa268436f4df0d973c401f8bb20f55585954079b6de5a2707ab5ce3b9f42af09b7c1996a45a15af12d041f5fe0c055b969a9055f7656cc2806745e84836903804799ffcc15580f8a50362f4dab9694e8ebf6f32cacbc3ba3501959ffbf33db0d231722715705e685df9202f0466c3ac0f1361a71905e23b0aa9fcac76b01d6402598861084477338b5d0463a641e2576f76aab2de95016cb31d37d8f4617e3908a179b43e3d50c516bff1e388bfc0ca519430f85a0de5b0e274cd0bd021f20c34 + +COUNT = 5 +K = 2a919617d24baf991779de6c06e68bbc4894a4e9a702214ba3eabfbd0015a7c0 +P = 8d01e11206e3b9285cfc73c2ad6b238536fefbe824f19588685601bf94e1ee2515c4ac281aa04feef877694840d15f2a6722661159a7216548d768b4754d65f1f1c522daf2baebbf88f638fb324b182e61d315f5f12b04c486022749f6a222d87c7d635c52605c7efc1853cbba466d138d3cda22979e7742e4fc8168f5139a72818109d2124c20ddcecc74fb388f34151e96fd33e8b269a7d7e51b6fc6ebb427219163c3d9f7de4a6831a7f8b596b4b490789f1104fa160115b6e8d191d1db885e42812c01724170e765634fdb8e05555cc7a85c3d8cc7a8e299b4f3708a4a5189871037c0d04216dca8e6127b449ec6004e7f076430d1f870f0d2c0c6812c810b4665f7fc07fbb04b56bd956b81267d2408c31be8dabade28a3c40c609eee3458b7340715272ee26df8d875d1139511649b629df067a8994e1fbcf7ec60f565e8da257db1b7ef3b2b765c77124c8120bf26db8c47f493d2afbbf8b89efe2e33c21c2a2a1141c0aa5b9dc1eeae50714304ff35a7cd34da963f5249bdce8e2c2a3026672bcd2ebf311f446cbfb7171fd35b446f0270320b30b554f96573eac72ea0594306d43c5ed48a6c2d01a363e88646a8759117075596fe17f18c0b36d00b7976098fddeb7abc4e34bf6d6b35238f81b59d169db4578a96c2bcfe78bd583fb31fc2e7fc97c6f3460147a5730bdf31ca197482acc341b7a719c4e1716cde56 +C = eb62ee28d9929093e3c6dbc0592c35dd0ffbe8faa37173195da391b84b1ffb342b9bd9f0df0947c33c7c74a071e7d71312f91c8e1f11d680865563f2d9604dea06892f2adaf2028246b74ecc82686bd7c369426a09f23012b69a89e7e66e7cf008b41dbf04f72cf950fd8f6ed63b3392de53b6feb34e45df3153e4aa6fdd685fcd7e3239f0509f474d8f6a58abfbd36bf14f93fecf913b2f9a0f0763c39cb6828651434ef0520cee88f46daef7932e5ce1549774ec106c2aba54cff64067e0202167f5a3ba1cd6396bda08023d2cfa9100043b44b8d90e2862d7d2a7f6e3f5eb6976dac2d5dbd2adeb69858ca391fdd4a582d3cb79a2d9fb57ad5b2cbd157b1e36ed49fb848e9960b0dd1c715b701f1d379027999fe8e35769e27a9ef60a45aeee56c7a3ebb39f6c50796d1236e721cd0e5d931e6a85f28fd87d652c3e5b706e2e97c12c4b33fc7df4585d60d0326267d8d252ab8970a2528be086d7ad18ee4b1531cc13b9c88cb3a9188a47b8a72cd276eb4d7d726a290398a98be0083e9917f349d2c8137a6fcd2232baedb6edf075d8e938319a12bfc31726b13d0e1a6c18ae33e11ad1f451fb9693f1774aec30fa7b473c98cbaf4dad1b3743d118ce69620503a753b87e4ae25f36564c2538d6650291087ae04d181e4fc44b6804b5bb0c6c8a0aa04a49895ea64d13f7d253e62216736a716258bc98c63349b5b19f5ae9aec3a176f98cb298 diff --git a/src/test/crypto/Makefile b/src/test/crypto/Makefile new file mode 100644 index 0000000000..9b5fce5b30 --- /dev/null +++ b/src/test/crypto/Makefile @@ -0,0 +1,39 @@ +#------------------------------------------------------------------------ +# +# Makefile for test crypto programs +# +# Copyright (c) 1998-2021, PostgreSQL Global Development Group +# +# src/test/crypto/Makefile +# +#------------------------------------------------------------------------ + +PGFILEDESC = "testcrypto - test PG crypto routines" +PGAPPICON=win32 + +subdir = src/test/crypto +top_builddir = ../../.. +include $(top_builddir)/src/Makefile.global + +export with_ssl + +OBJS = \ + $(WIN32RES) \ + testcrypto.o + +PROGS = testcrypto + +all: $(PROGS) + +testcrypto: $(OBJS) | submake-libpgport + $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X) + +check: temp-install $(PROGS) + $(prove_check) + +installcheck: $(PROGS) + $(prove_installcheck) + +clean distclean maintainer-clean: + rm -f $(PROGS) $(OBJS) + rm -rf tmp_check log diff --git a/src/test/crypto/README b/src/test/crypto/README new file mode 100644 index 0000000000..3750346a83 --- /dev/null +++ b/src/test/crypto/README @@ -0,0 +1,33 @@ +src/test/crypto/README + +Regression tests for cluster file encryption +============================================ + +This directory contains scripts for testing cluster file encryption. +The first two tests test the encryption/decryption using AES128 and +AES256 in GCM mode and the Key Wrap with Padding (KWP) method. + +The third test tests that the data encryption keys can be encrypted and +decrypted, and tests cluster key rotation via pg_alterckey. The fourth +test checks that the database files are encrypted. + +Running the tests +================= + +Run + make check +or + make installcheck +You can use "make installcheck" if you previously did "make install". +In that case, the code in the installation tree is tested. With +"make check", a temporary installation tree is built from the current +sources and then tested. + +Either way, this test initializes, starts, and stops a test Postgres +cluster. + +Requirements +============ + +OpenSSL must be compiled into the server for these test to run; if not +the tests are skipped. diff --git a/src/test/crypto/gcmDecrypt128.rsp b/src/test/crypto/gcmDecrypt128.rsp new file mode 100644 index 0000000000..53f694d37a --- /dev/null +++ b/src/test/crypto/gcmDecrypt128.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Decrypt with keysize 128 test information +# Generated on Fri Aug 31 11:28:04 2012 + +[Keylen = 128] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = 137f40bc82b01b34047a407f40d5c434 +IV = 1f3de3a272227db8a88ba0cf7b23f7e57f84ce370b289e2f774a0c6184f44f9df9b3e645ae506e3d18cdc819fdc3b31f77d2dae7ee295ba2c776e6f1fefecec740021c4aa1118a6ed3813720411f27ace5c11437a78b3d626b5f421e2f3ffcd0677930e7878110a0cad535c057261c63f0531b538d8bf04f491c16e871f071ac +CT = f6336f2001a5edc519aa64071fabec2fbe4654ff334871e9cb8b12c24ffb7ec9f0d316e43bbdb568b65608f937f24f64eab19b +AAD = +Tag = ea3d8cad +PT = 60881890126afdf881dba4c1a7bd0f9d8c240073517b21c622bfa70c1f05d59544b9dccdd908bd924ca60d23697fe64aab927c + +Count = 1 +Key = 5e49fac6608ee4157a9f0ffde503893c +IV = 53d854bba28eb9d9db71e68b5916f9e73964a0f95c6524411fb18f64977d012d611a15fe8d46ec49bcabf060cfc7989c4a8272567c660c0881f944066b88311992b5d52cd644bf07e9f01d2249e5f9b00ae1af9f2739da627b3a8838e28116b02e7d171d1938edc6dd5d7da3dac63ed148b367961b809bf423510f0a3f44a8d5 +CT = 23bff066410784e0999a3d0fe742ee28ce8095fe822fae0f50b0b17d70c8cdcd705c83e108fcef2d64c5b4db110e9a72d3c4e2 +AAD = +Tag = 01aaf4e6 +FAIL + +Count = 2 +Key = 174b47de82c64196af18214d9d6dce2f +IV = db9653e9ca4ff27f8180f45eaf4861541d30b3b147d2b9b64709983dd08312b5fa709a7bc6afc591fbdd08fa0f21f06a25758ee199a6c6415cffdd9777280e620d7bfaa6d551875c069f5672a730aa68137cf33f86d46cbd548248e68f589c2e699c554ae4680417e9440e5514c1cee03205324f789d953d63518b023f65abe6 +CT = f8b0c1317c5f3550363762dc7fcc847376a7917e1505cddafeb5451ecf4464ef52afd797697f1b0b1e8d3a2376600d6acc1c88 +AAD = +Tag = f6fa70c6 +PT = 523986d433fbcca153af7765d177e2bad72ba52743440250db451e2da643913cb25b33fa364011a8f8d5cb563b461d46952923 + +Count = 3 +Key = 5194e88020a501ddf21a7ba5525bff70 +IV = 444b0037fba6cb9d4f4e33df32289d44f262d8f535564a1921e6a4315611b98a4e1074b37114758b421ca44efda91a48ba8674c695c259649c0bd3c11bcdc8363d3d113a534bd790988a44e72559fa70a5e2deeb9da187e2fa89c7b636a000a38d7bfec2edf2831ddfc9326f0c54956c149e5ffba3f556c8f2a749dc6ed5407a +CT = 31398e58d6e27c50e85c87a4dbfd73134f456ff042c9f869b47a41e3c10db7b24cffc103a2185a696fdc1f2d389cbf41379f04 +AAD = +Tag = 2d28343a +FAIL + +Count = 4 +Key = de250ebe0ea1212480f7193934d1ae84 +IV = 255f658334f1c5258d613ef92da7242079795e9c04c5c4e800aa1c03b6abe77f4dc28cf614278900b52f1de23550fc9c0ace8e8dd91a25d0240281dcd684dcb786839f376496692f3e35a5946bbf7b39cabf7d782f4e77eec53a875d2a5e3106238dae0d048e8b9ec7b05961d9d68489bcbe2805ab0f9cd42b5f8f62b4b28b00 +CT = 6eace8c7c10c713870c676e557c6b49134af26011af157de1e2c6ce0769fab666b9a7844756d6b948ccdf1df77fa98119c83a0 +AAD = +Tag = 87b81d39 +FAIL + +Count = 5 +Key = d0d9c4aa70e5bcad8dd9d72d14c21c54 +IV = da20388ed0709e5af5a52a5293bc4060f02ed6d05d96f9ac3a05c14118a3613ff790b721ff2c888d63c04b9b28a7138ca00d0ebbfcb11b70868f5b90c0dce79c7220fb90d5f3b6e22c6ae3640e7dfaca72ec12813820debdb69f90c8393645d7e96cc36bdf62ecc4c0ec181c64593c65c8f707d6db61b16c6a5c2f05f8851692 +CT = 2f08e5a8027a88f0e223429e6749dbdf91ff9742c7c6130962ae837a434a7247600df8e2004fcf3e81f8ef64e6a23bda002f3e +AAD = +Tag = ef6c0d39 +FAIL + +Count = 6 +Key = 464609f485abe0bf4930102f9637fdee +IV = d0a788fa1494d47bd1cddd55f88515229de5c1a2ca7dca054e08bf35d3c0aba677671d5290c3375fabe40de14e15465138d298edea5efe1ac62452dfee86920fec8e46577bae1c0b43d77cb6b8d1de6ba4557d7410796bede796b8b87b5771e30e05a8bfa55ed92194d506892c33f8ad6038ee8896b98772d1a88a7a40ceb650 +CT = 5e81bd883fbedb3e074ba273751081752c3aeb7a62b08091b768f8f8e6c0913cf633358fc26885150adf65bd1d708d9e45eff2 +AAD = +Tag = 7e715546 +FAIL + +Count = 7 +Key = 018319ad18e5d877dbdd871316fa6610 +IV = 941f8715bbba7b1fdfa98c8c05546a50f86cd02b654862af6c1a70cf4d79618b37abd1f06b7a29bbed74e728a8c78a0ec8ef8312f2552fc4571f48024ec5c6d44c21d69311cb1ba390cda615f246cb175a9e8a522a20b671e6df973b8924fd003c5dab7e59af63f20edf56e9dbc9f98fcac25cc692cd87a3d8fbf8d4b5f150cf +CT = c5a681b2b2e5f68e0cb698150aab9d13e93e900e0e6fb20170a4104839a7e0a5775bb3f20ae814edd109530552ee2b41cf5986 +AAD = +Tag = 75a966e1 +FAIL + +Count = 8 +Key = 3556566a1cdec721469cc86412ade697 +IV = 731a29cc083422da193724a57c8126b587c3817fbc6b76bee106abb3ac48722bfd0cc15deb573ca73f2bc10ad7b60024f297bc52c31a7021bda91c5246c781a95d5d18978c685f096e9310d6f5ed151122262f7d3f5a2b265321683cd72e0693ff608e378199264eea876df44659817997b323e43457e70809c74737d015cf26 +CT = 753174df0aff575fcea79e54e5b8347e69175070a00d58720529b173e9dc64677fdf459ad72385b03244d90810670304129f2e +AAD = +Tag = 41147153 +PT = 7588b757f08b75fc799fc825034ac60585e24c56db94920b6982bbae86030e7a021ddc89a5131e5371e9295e414d3503ec11e3 + +Count = 9 +Key = 2ec96059d1313923fe522bed1e942491 +IV = aafb096a63195cfc959bf93785a48cb7a78b98c60f861c9646129222643d249e50d46f706d670662f60c12a7f0a6ad65b908ede9a56b27c46f07d427754c11549154632e9b5ffa25c9306add4c0b13c7191abd59430a65ccc42c5420ec98093ad6ec6ad1c4d3bb895ca3546236c2c78cd075b4b1f44a6a03d115b91df02b96b9 +CT = 61c6b8bd44c9c6eb075fa9bba7ac0156bc64a6326d32afedb51f35daf232dfd068dbfd0505933519a753bce6de4560b79b31f1 +AAD = +Tag = 26f53f5c +PT = e4b86f8b7684d6a300cdca848ee8185c41fcfe061e61af6f3a563ceccab58751f1884b322019983d99150db217a1e92a8f15bb + +Count = 10 +Key = 48cbb46539dca9faa97b69b32015886f +IV = 1d3152ccf3ff05a81837ac7c55f791c78aba8c95149c80de8724b1126dda847b983a1bfe54ac5206d5915dd0796944e4daf432b14a72c5fcf81822ce1e3e7ade345528641b94a2fdc90afac098b9e4b0b5ad159ba275dcbcc11fec04bba0f12db9b3d1473ea0ed7825d293fd0b5d2d07795b3de922bba5660778adbdb2cf8539 +CT = b2658e897b85d9acf437f9e895189954134c543aaa91fa82c586cb5bb1d96e04b189df64df5bfcb63a567e444f4653aee3fdbf +AAD = +Tag = 48d21620 +PT = ef58f7ad664e36dce523408e497acb871e9a579fe4688be6425adf41c4c41db85e905e9279b93a037648802dfc273b387e7216 + +Count = 11 +Key = c5c2e83c4e5b336e30257c95c8f7f75d +IV = 269c29118eb9cb4ae8fd8ff5b3a5d61f4a63a719416bf2cc5225544cabdae24741239ba7910d1a32364c341169a215264ae62964322b6ac64002d09455f11d9658a9454cc6bafa82dba5b5365b2243e854470dbe74a3043b5f5f82ebefadaca4d468317c83076691c188af6e800dc296cdd941fc1aaed8840874f8cbbf96827e +CT = 891da6e799429c772edfe93d9fc3ad20c25fd5a172d272d0ee6b7063b59b611e4a2805923f182abf99373fbe4a1759339bb13d +AAD = +Tag = 6759d819 +PT = 474a2e7aa4a5422bf17f257b584059fc7abe2d25e7243b434a6f936f5a0eaed692616a43a2aef8b427d83ddf1bece29d96c581 + +Count = 12 +Key = f4d7a0e84b7f024988f0507098415616 +IV = 9d58c03fe45a97d4c556bafa4292766c55bd84d0e467c6c26b0e42162015f3f506c9fa06caac63a3ef6a3f0ea62ca4e122bf453a767b2b00d6277a990329b76073c64ef260421d887eaf89b6deb488a0528608ee8c38faa5f5976b2b71d7e29a08ccbeb2342ec15c46232f3c8867ada7cb8cc8aed0728dc59d706913ba62124a +CT = 11358bd3ea694574c0bde99aafc2621732fcd978626d3a3ef124e98a2c3dcb8261580306b51ffa9557d379952ee31bffd9ade0 +AAD = +Tag = cc46943b +PT = 4a289261f4ced6fbe544fa27d430f04c72ccfdbbe0f93e881fcc1b29a19e706a579a98304cb2b03cfb3aacbe187dc69f923ee5 + +Count = 13 +Key = 31237ffe81db56b9541133156fd0e7d8 +IV = feea389ed83bddb360f42351cfd56321bdd0a8fbbdcc8787725a236365eafe5dbc9c6e9ec8e9ea0f74e623c33a574ea4223a43b1550add0c75d8348315add72ee9561e1750b69defd46214bcc507d4db4c66c6da23612aabc2ebc9b0e6d5aeee270c013a553ddbc7d7d7ac62ecb33a5a6e3db859471ca3d3686203c0c96eed2e +CT = b965d7bb29a8811872c894d535e001d534a7d6e99eea5bd14ebf39c5e79c623a67d549bafbd0dcee1494245c04854f34de677e +AAD = +Tag = e6fac1d4 +FAIL + +Count = 14 +Key = b0e576fd7cc85f7c92d44e4b8c149b3c +IV = 3855af97c5b740baebda1532c196d187b8af0da761489ee9e267fcbeb720bf6f73cd743b69d942f3f44893d68a5c70174a1863dfb831ced0ffbee668a03e1066e8b3dc03dc1471c3a848b3787c0645b20add2fc0cc37e5e0aa57ef08fc69c53897030fcb5579a831ee53c76df2f75d2de5bea93e9ddd8e4e1383bfa1a7c7aa77 +CT = 624aad7fd047a61c09638b8dd5065d7c00960035440303a03ff1b19caac02baa3b6835581f2b66c2381006798c9d5d63d65684 +AAD = +Tag = e6142617 +PT = 0fb24ea93a14e3d76c4d7a991cb55e6bde4083504ac9f3a00d4b0ef53d3929953b53f5ce820c41b9aa75a985e09986d3f4baee diff --git a/src/test/crypto/gcmDecrypt256.rsp b/src/test/crypto/gcmDecrypt256.rsp new file mode 100644 index 0000000000..af72933a60 --- /dev/null +++ b/src/test/crypto/gcmDecrypt256.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Decrypt with keysize 256 test information +# Generated on Fri Aug 31 11:31:25 2012 + +[Keylen = 256] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = 60c9f83fb0ddbdc727e70bf9eb1acc13b1b63e3056e64db7c2ac55c4f2068273 +IV = c33d34a3673b93bb78dd1e00f877c4e6e4cf628438b9effa61cfe81e159155cc9ca7c1418917527ed3f0a51daf2bedbdaca20fad687a7dd086ae086c8ff5094e9b31fd71bd6f8f1f1adbf96bb2690663386c37d7bce891137897aeef70be10a453cef7e31c1b8c0a24ac1baeaf08a46aac445ad5a8103804825fde86dd4720b4 +CT = df6586921250aacd9d25f432977e92b09ddf89a9403c83a80890ff15ce9c4559145ecd85d86f1573bbc1b48992859d22fc13b6 +AAD = +Tag = b0dc70f0 +FAIL + +Count = 1 +Key = 8d7db520bf88c96d46778991a4f0b6de9aa7fd5d35cb6188a6f355499072af5c +IV = 488a50706bfd8ec7fb4c508511bf4c897c8566ef289b5e58a4c59bcbf16b5ae85fbccaee4a1cc0d1ec74156ae911d36d497f5ee71f1fa51649819c9cb88cf65d62d2abb65d621c202bcb33d8d68018a858d04e79deb62b3486658730735a1c87829acb49e73301902c116c9b6ce110f23a6b1a4dd657e47a328e017c19f0ee52 +CT = 497e1a39ba1b38d263bcbf19cc2900ca4070ad37ec12bfdd30139a7068a889825eaac5012cb5c2dcc710a220cc658dcf069f60 +AAD = +Tag = f20b9885 +FAIL + +Count = 2 +Key = 1ae1e42ee656973f5628e3cd11f0494dc8a563cbf5fbc5880cc2dc6d787bc9b9 +IV = 23e3c948cac6eba2ed11d667783557917f066ff6b93ab9409df9c7c84b27d26817dbdebb9fa9d0a64bbc572bfb2c7ef7f0c836528c9bd692505c8c5e522f57aecf6b479723449398e5b1f45cadd81264c5aca8059562d69deba26395034b4b01325d072dce92e540c159dba92d3e41e2d0947d873ad48f9f0b00f4807d420aac +CT = 3839fae7008b88250b602cbdf295e932e3c4e3710d397a2b9a37289104efde75f73302b2820f14664c064e8dec45ae49a74036 +AAD = +Tag = bce334cd +FAIL + +Count = 3 +Key = 8af302bc8684cb91b4d7a6088cf8c94f9f6e027ba046c2b508956ba3c88f2d65 +IV = 3218736e931392e6510b91210a6a6a27680740ba8924062ea176048d6b42f44ed04a46ce31843b735ef4f63dd1d85643f28fb335d21fc2e3c673e97e6b845e363362d32844c9054a165f40658267bb177b74797a8828b1eea723d51b571d93748c758ea5c328103612b109e008f743f9505034ed3c42ab3dc310c20938f8627c +CT = 1f1b133a1a7b58625fc77021f8ad1751bfa2b8addc0a9837dd5c44632cffe5ecc2e9e54b90cafb6cf8b652a8d2da116ecda3f5 +AAD = +Tag = a673129d +FAIL + +Count = 4 +Key = 13a7942b5a5cecb2bdc0e8b0348d4db5a98572544ee31918ea625b0691c10779 +IV = d92b4d05a549b296e18c90a8da55ec5bff3547a679697c489a1d49dc02bfe2dd85c8f050b32c389c4f857eb4b663f53354bcfe9c3a7e30019f2e3994421bcf3a3d1cc093768eed71bad5139f3f3078514d80a4a41d1284b5dc43ce07efac9c475d6ba2acb66dee50cdc62c463a05ca396e72d189f50d44ffb70d2c6112c6ef0e +CT = afe058cf694d64706302b405243db77c7ae2fe4f33c6427416f8992ba92754c69d4e7c1a89e9b6987f2bc0a7b568dca9c9d273 +AAD = +Tag = e0bfbdb0 +PT = d544d114e3d9ab8aa2b9ec588a112f780a6df74d637be3cd34fdefe14506f26281cacd2b98c26fc4adff837a7bd72173b962a2 + +Count = 5 +Key = 7272e6ca6d6d76c483df9a55c6d07bd54fd8fad50b529ed52154959acf01b64a +IV = 5c55bb8f4fe797ce34c0e281c3b04ba0bce8689493451ea569ba8cbacc74ea36ccf319776f77cb4d7f901fd0ff23cd28ff0ca77ad9d4adb0329fb68a60ff004a1c5b12111d2dd705ab1f7734178f14dbb356cfc0c5c208b91c277235f35afe8c2d46ebf43bd5e0a653e67e0c086ebcfca32a56d56dd5f810f562f769cce2794c +CT = 7f378bc30cf2774f21078f42b5d6b66aa355c8c073d3a70f06775f3c7e5948539ec08a2cc50cae6f2ad9680ba47bac190c3068 +AAD = +Tag = e612f4b2 +FAIL + +Count = 6 +Key = 82b1d2ffb53fc79f5ef88742a28eabcf404074836fe28b5b202ce7d5c68f6ebe +IV = 33f834fe23b9639d30de763faf7c1a71568c5dea9d5d253f28723ccb3306a3cac3cac3beca638067a3485ff743b5133577633ec88dae0aec4fec08e894ab5d61c411f0939772df2fa66d5775f74b3ff36ee61695d7cd2726b9be4df80750011477705948b1276db0cafede5d7ac73ccdf01b73a5492a02c43b89632a501f6694 +CT = d9d92b33a10f4252fff828b57ca5f5f118885df0825be80ea5725a874b7e8721af40bd221e7f5c2c8b005d77af6266cd36ddd2 +AAD = +Tag = 86e48fa3 +FAIL + +Count = 7 +Key = 5264831eaebdde1eadf741dbfd585cb0ef6437d1365bd5848d9cb3a22f57d420 +IV = ede74a8f53eac5dac276bc72518255831b616c9fb50a617eacdcdfa50e197d2941004f785f00f8c600e239cda77c8c06088793a674efb8759c98604dc0143e06665dc7e21d5031fd4751a7cd1b947304645e0987ec7e765db80a743122fbcaef9ec83849e8eee8d011dab67fb54317caddcfc472f585e93df91b1edce9695908 +CT = 9ad126b39dc2066542dd30c8fe81cd750b72123d74aa162113c6b0cf10a9cdb217d921e8f03b400f1ff719fc704f44e26ad463 +AAD = +Tag = 5d7bed4e +FAIL + +Count = 8 +Key = 6c2a43ce5610eab9dc40f43f035f7eed6651789dfdd166d4f106c95cef2a67ec +IV = 60e3a8ddb899108c11550a461720bdbf9adef26c300f098c73c3767621b06eac4f5619b9855d96e4d972ddd38f4538f8e25b7524b46c6341e8780e22c3b42ccf43f41fddfc5680432b64fb4025b378204045bb2d7ea56f4340a4018a4c99eb8b91012b28024d1b2bdb603fa10a28130e84bce38384fbb7c43548c0072c5c657d +CT = e073e948ddfc414948b12b4540d43dfeb9cbfa525b3cacccd21da89ecfb254c840722b9179057cb3ee69358f05e4ad0e41a543 +AAD = +Tag = ac0497a5 +FAIL + +Count = 9 +Key = 7c36a9bb3033bc6f7395155eadf0e07c8e5b3441d0ad66b21625d4950760386c +IV = c8393fb1d80ce92801a4fd906a568f7f404a82b02096e859e70e46d1ca5e231a073c5acbaa4cb4c33581e6887c402753bd55f95c76e68bfcbb1cb21bd37ab7a226e03d03e9dca6589c3020f5f916c50676e8c387f9b1710579a728ba7e7b60955ee5e383bb75d2b9d0f2abc72c02edd925bb32dc5a994f032e9a856931eb1ca3 +CT = 559b2ff3f5fd147b9889146f9fdea6758e5e0c716395cf1caf577dc2707764833099bda0910626c62bbb1ca010b66c54114982 +AAD = +Tag = cd2ed4fc +PT = 178745b297a23a897ec5cfe3a9e373befbdb840d9eb657885ad0423628c4a18f934e6fb57974a52436c517f4463cc5f9370c54 + +Count = 10 +Key = 981afbf7e7b74f08d186616d1f71b682bccc3cff6c5560696d267ad455d111f6 +IV = a2d07ac3ef29978c44ebbd83e1ad330a8fcfad8213fe2e924390015bc966a944a0a76831189a011094ec4ef98535efeb56b871e7e1aa36748e639dd5f9d1bf3286a1b2965bfc029faa0f855622c30cad67331bd11dbcea51e397185cbc4f0f0341fd8e744d2f09b2e3c2bd03af15850dbe2a701855ed4247f97acf9754f5e4f9 +CT = cbb1f9a5bd84c4b1b8df2714f87db878f7d2658cc7c37f75d784e2157687398a391ecdfd1119e087bd12f6af79db50ae7711bc +AAD = +Tag = ad794c97 +PT = 06ea96ad8e6044978ea676056df8c647b7bdfce3923750983cdca875089841612737e6fe078496d77906b9606532b309851cdd + +Count = 11 +Key = 03183678896e28b84e16ac41ebb14f4f436efe386ee6df4e8ad2a7aaf11f17d6 +IV = 718ec99fa1b9b1d29a06ca3973d9c0323b14a2cc34cfa2816481aa2da97b435b0a075a2ed6412bb482bea23df9deddd16944492b1756c65138c3d189b8d2d695150667f46edce88755e868a2d90bf13f170d9b6bb29d9210f3c9f507663756866ede0b362aa5c859e15cd96da4f8c7f7852b3924bdf35ff3a515ba5150e1b017 +CT = 6ecd41a492ae5d6295e9c18290c9a36999c79c87f8b69ff20cb42ccb7c6678baaf159c75ecfb15cb87db99a3236734001545d2 +AAD = +Tag = cad99689 +FAIL + +Count = 12 +Key = 2ca8b01d1cbb8d392bae40bd8a51205a9020be27a23533da51dfe1ad0c4c1d41 +IV = 5b5edcc2f17942afb9577c3d2ed7d5ecaf009ac3ebac985fcf1e0fac0dfdfe747fdfe3d05795337baf41cea3b26e4f35caee1c13fc52d1192da145f376b4ea810ce7dc94845a9ca9184203c3b8e803e7a9bfcbf4a310c85b28b04a007e8e9bd14ff0ae28a1966918a6e22ae8415334e7df0d530b0507a24f755f70f117581820 +CT = c420656fb66e89d5b10fbe3ec0929286683ddc4a34cbaca638493f5c09673609814127709b6b1bb765902f6857761a8d57d98f +AAD = +Tag = 23f1dc74 +FAIL + +Count = 13 +Key = 25cb38a4b7ff73bb632ecae5f75d46e45a108ffaf3ec2d6ad39d3af4b3c64ca5 +IV = 372510e18f877e0f74c1cc54b19d265b27a452cbe91339bb720aa1bdfaf9bbe5365c571ce8f01d2e96aef8bf089c3f4402f186213be72b46b200337c9ebf943bf3d2db1f68c8e655534d9198825737e623745c26f6b0a82585660a7cc3985a271dea9b20f93653701a8d383bbf3155864809decc03ffefc9ce018379d12d8bfc +CT = ea01ae67abee8f8552ad260ca9d08ea5b35b53667a3455718545e007e5ac0c62c1ff0c5b06f8c031079fce5f2367889a6a068e +AAD = +Tag = c6a365f1 +PT = 3bc70116886ed9b4ef795e45c6ec8ea65f6285b3449174f89ceb1294ea73dae9b2f037107f57355be7242abb7da818c98d2755 + +Count = 14 +Key = 19bb98022f5d140cdbb5b1c02aae8eeec1e96dc6eb489d70967588b6f414330a +IV = 6d7b41c7f949f8ff3e9e18ff7af3d67eff5ddaa62eefdbc0b0a49dfb6fb07582998250d1c8e609d57510c859333a268f7e89bca06adf1646cdeb2e592bc86769aba402410cbd71f572dbe065beb37d8766ac61c12e7ac322d213407e073d4bb3c28848c42959cab21f9e39d7f4ff8debd50f40bfff96cbf81af07fbffb6bb2b0 +CT = 60a77e3d27fcea5e505221382d82e9ee39c2bfaa01d7d6ce0d293e7fc7bd0d7f900afa9a7f080c33c04cedd76573a914409e39 +AAD = +Tag = b561ee30 +FAIL diff --git a/src/test/crypto/gcmEncryptExtIV128.rsp b/src/test/crypto/gcmEncryptExtIV128.rsp new file mode 100644 index 0000000000..69c74ca6c5 --- /dev/null +++ b/src/test/crypto/gcmEncryptExtIV128.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Encrypt with keysize 128 test information +# Generated on Fri Aug 31 11:23:06 2012 + +[Keylen = 128] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = f8fe56171fa546a34b1b28e0b1d31cfb +IV = 960d57f1336271e069c12f11044dd5a5bea996fc0290d37b5b2f47c8df3ae3ee37214a6871d963b830aec266026364984cfe31eb88c2a6229f5594ca9d3b6d26c7fadb91a0282cdd0a321714b745dd5e161e7cd192420cf2eacd552c4df5cee8fb5f0e06b7c353017b4b9523ce56899db770c344da720327817ba823a8f71382 +PT = fd229158f18f5b8c2a96c86fa3d8084014660eb2314bbab4ca09fa72c3a98b6faa2ebb83a1809de9ccbc8973d23af34014fb27 +AAD = +CT = 4dc0fd07c86ec84c264f0544456bfee14f688af2109455d73aa1e58a3354727e05387c94568edf352f8342a6156c64d87c44d7 +Tag = ac350afd + +Count = 1 +Key = 803007b69146363999afad4433c0f3b5 +IV = 2fc7688faf0b3783094294526ac07c38a73ff961ab41f20deb66edcabfe68e094aa0f1fc52318706a0e2f9b3901a768346494e846cf17b662d05a3788d77c1468408a49ffe5c0cf68b3b8b26193dfd84c63c4631eebf0c7974283e05e39494d9aaad038018a6e999912b1f92681375214e634f5937cb32ccc4face42d013980c +PT = bb311134866deba57fe506445c5a312ea1ba1ba16469731c1647c6a482bc84fa8349d82bbd01d3edb6cfe8f25b37ff8ec9d621 +AAD = +CT = aa1d6ae4151aaec1030a6f0297c48e67b84d1c397e9eaf0c5c8c3d252bf8638bf8591342c5c20c7f88f41140b0334f55cb7b04 +Tag = 30076c5b + +Count = 2 +Key = bfb96f58535443205ce281e03bdc5c38 +IV = 3961f47e9e800f1c72526d73fe372cd6e69a7eb1d2692c58bf4297bb05503edf95e074a7cf2644981f72421f229d93866a6e1fcf5c13953b39bf36b56fcbd4c09b505e550b5ba0e9bc26efbe0b9621a47b81842caa8c945fa5bb606f0dab824a4bb5a2625668a916e47f0a0d8a995bcae6940e120724f6d53629545da5456008 +PT = 2c07d76ceac2a77809906bebd3452e2f898ea5467d47be1f17573b3f7fc7d11c9b868d1d1a24010b63dabe9c6c6b4e123df559 +AAD = +CT = ca443acf08d121e1ae4221013ec40dca2237d035e89ae67040602132972417e07a7c770d75d96fbb5b8a38e048abb15bb978d2 +Tag = f61f75ee + +Count = 3 +Key = f3a44f15d104f81b4bb263eecf806737 +IV = bbf1b0646991c2b9735066dad5860fcc08a6b92944a5e90dfb120acf2a75403d3175f5e61a1d84b89a0c1bdd3b3414450faf6ffb8820ab1ea01a2b3cc05f1cd1de9bd48ee1308ddc7d87e6db33d3a171e7f63fce6b8e0417359afa833f6b5f293195bedf444ac56103ee0c8706a69c08fe59a95c8474f28a4a12661905f8781c +PT = 62d05ea0b0ddff1c0418b01a21267230ebbf23b63a6c14caa769c9148150c2454c055cbe4a72a08e7cd8dcb456e1ec17bc3a63 +AAD = +CT = df08c4d223a168d4dea9445f5b88d40ef0a796de3e77a6a116bfed840ccbe7b988345d070640f62f5757878420b5c50dd9f567 +Tag = 1518c2d1 + +Count = 4 +Key = ccd58a6017ac344ce5f8ebebdbe03593 +IV = b2c53f013021f494f6637876f62ce5b5dec6d548ffe58d0952aa8fd8fd5c8d2b835165e1b0ca01e72c19f962e38cd4458229a3415d7b4f9afddf5bb63215999b750c07a080677ba4e40f6c5e42038882503c9923a6eb2cf0d3b82f9f94e624f9938830fc22430f16f6c93c362cb3c11cb05d63becdb4c572f03431e7108369c2 +PT = 0b89a455a2470b3b2b7e04afac15c45a0742061494b78f88c57f2505e1f5804f35c0a829f8b6443e427fb6ecd374642217b6f7 +AAD = +CT = a8b6203b914391c61dbc123efc6902b892107ea9724341a22ecdd0deb16c48d885d606f68724cb43a956c07ef9ef654c042906 +Tag = 02aeaa63 + +Count = 5 +Key = 50bcf114df40a431c0e1e88033154a25 +IV = 0ffcd9b14c6bf5f630c86b41cb3cf96004f3fa4fd48ee87b7235d34be0be1fdddeaa79abb3c0c9198a1eea0ccf04e8cfc8d24e4badbf438c59a70b435fbf07d44f55b75e5e48fa0f3d7714aaf9e34b430640614646d0008014ad432a464a252c66584c922b29a90cb2c2e4237c8545f913cd2ed3910a4f075062a55b71228411 +PT = 9f5206b2afe824ac4303d58a97255dea6f026b8531651105db9695f09acb16afd9488928060219307fd41edc16e49f01b7d646 +AAD = +CT = 0143c0a035ce29d1418acb2561dfe88c74f24a9808a8672427797cc0ba01693c2d66c0e365961cfd58fd039bf08fb4c2b1be29 +Tag = 552e27c7 + +Count = 6 +Key = 8dbf11f923374b8e8be93788de939806 +IV = 2fcb130962ba3e3eb8e31a7a26e2082a643f39d67cce11d8b2ba8a782f63d4df375b21b1fedaf67bd8d73b2208937ae941afc99420ebbe328214fe6a456bf00979d5ebfb22b79fd3cdea81056747bf4e4ded33f2f26f2d228965128a3d0a32696db44e4aff6ca5467d3c749830a5d2e9a41b4ffa3a422e5bb870cee84f4b64af +PT = 1c57186740c4901e022e63b2e7b085f9cb60c83763e357a591c1968277920ade1987334e88ed9c3c96665e37fe492a975153d0 +AAD = +CT = ce4c7e5fb87ed02424a13cd2cfcfaacf67f1072198dedc594db26d6453991821861ba6cc843c6d2750e846e162415123b7e01e +Tag = ab10b040 + +Count = 7 +Key = 6e88bed99ebe380c1f8297f46019d8a0 +IV = 80fb82c0016d054491f396a7722217b0a07bfd0fc954a7141bc1e2e7958cb24541a21492ec85d3c744489f93ae3abb9af101a78f2366226080389d29eef564d5205f377ab0902043bbf7ba64c30c9d2c945cd6f29654738106dd282194fa02344ec177b5547531061b31cebfac4a2b0f46b68e44c8c89f6942f9c9c13e50c58e +PT = 6c90ea3dad3e172ae6784cf1b7c08fc04d0f9b4372f1103d11393a8f02e2f495b53c57cf46f1df6f55b2fdb184fd2b7d590402 +AAD = +CT = f55d67b524b8e633019e9b1736f3db1a254e53cd71fecb48f2dceffa62256a0bfc3775f6506db52db6c6eb91971f5bb5688325 +Tag = b4585815 + +Count = 8 +Key = 99de57ce03d1db62a751c3b2c7d38f3c +IV = 2e74c283e216ab5ac9a1b214dd9280431d7ff942a17715fadc3a22978fa1ccc0743f969358baea3d79abe93388a7db82d82ad6a917bf67795fb4360543d7f22f7ee49f41029ebb87573ac03ddb7e279f1846f4a88b85bea63e2c9b9ecf6f91777434b3def0d4a42d3e025eb43a666a28d5f6c834c7e7991897bf051915e646c6 +PT = 72dc7e5533e862efe0d23c62095506b11c9b256d8d18d11511aa1ed4eae67b0017ec74e322f3a7a18e7d199e7093cadf26680a +AAD = +CT = af31d880d1820a35c9248ee0b1aa0da31339f90182e60451493bec8d4dec3baf922268741c2717831b8365bebf072aa7931ab4 +Tag = b1700396 + +Count = 9 +Key = acdc98a8baa0f003c130ce196135334d +IV = b7c9885d302842c41a2880b9382584806e8cb55b49183b80bc50403c82cfaf0d28a00fef813ba5b7e35b80dc1f0b7c3b0669a3bc739f499d77ed9cc47a467ca62fc34c5bf4c374ff396c01a472c3dfdab394e6926545a1c20363960c72dfad3eebf9a970e6579e3eff7a38f6bcf0373a8494d450d12445f9ff62c233dc1d2379 +PT = 30f1c7fb5fc152dad6911623ca4af1eb495e108ed94b6e6cc19eabeaa7b85262ea3cc4dc5297aa6f7cf504ac6e07db5db550e6 +AAD = +CT = b2cbae4df8898ca223213824a5c08e16eda81f063916e813bf2d0d8c7e8a75b2d0a9f6de91e08d5422970534331cf1dd53fa8e +Tag = 8ef08260 + +Count = 10 +Key = 75d651a377ccdc0e743d73b8205fb38c +IV = c7296796ef031d372284f7b1481a13862aec243792ca73f40e11bfe39da28984f11d591d3294c833babf05a1f19b603f4f4a9ca1102f201c1405b6cb45facc8ef408541963abcafa907b2eb8e5c1c404b0e4884a48bbb2d43add4dc1c44c526295bcfbd8f2b7041ed49189e835cdf4ed00dcbb450eef4070482f5e8b52360966 +PT = 13ab743d9db511857f0f79a84742c225b3692a9aee8a63697d42fe50d74fe028ccf95e18a2dd2d9b778392ee7a5d2f23b399eb +AAD = +CT = 09c3f153ef712fcb3b5fca2b9bf7f25740fb748bec64bc35576ed01682030ae2728d4282140264819e8c4dedd48e29199a6236 +Tag = ae374dd4 + +Count = 11 +Key = 15414fda594aa87a3e7af69df769adbf +IV = 7f021a78293e7cb4dd0af221efda3149b0cab87241b597865267cbab5aad530ea4aa4b10815ba9318a45fcb22fd0e6d692d7beecc2042fa2791f6cca5f9916b0bbae79e9d91133aa54d15a1397f8b063695a3d36b8e573a866fc94964f39016e9490c37189cbb0638db09548a91688d73e2c0d4542f5bd08e03ac0d75e36f519 +PT = bc8263ccf50d0224d088546bc16e2577925567ca52d98ec45cb43b190159bbd5f0326d4498a8a88c0ea0b0a79420b906cd5115 +AAD = +CT = 34b07942793b066b74b7fb8a4ce71a04f9b29e40dca351f5b6e0939bc2819cd95e69bd58163a4df9729e3d6220a6df60a4ba9e +Tag = bcf4aed9 + +Count = 12 +Key = 4d73cae96ded98e1688104a63f462c76 +IV = 331b51fd88cd4731e0eca051717b642f86ea6d6941f9a7331ff361e0edc3f9ea4c013d585f3eff70004a696c4b51d7c589ca97e5fd30f4b2f99c0f3ac83769c2397e10669b7b83aef714a2388638b8941efde8e631098dd78742772f484edd568fefe26b9d981b437e4e4f3ad25445e1aa8c8608c655a5090d2cac158ee67e98 +PT = bf55d48ae5015c39bb167782cab391510d7b7d698e9d3faacd7d409fe4d86fea0a6f61128b6aa305ac0fe4cee4d582dd30e717 +AAD = +CT = 3cf29f4d50feac5cda4c7cef91d563b2573096b7f32c723355d8d59b8e1a0e229dc2f6114f1db6bdc26212043d153709597c1d +Tag = 76fcb529 + +Count = 13 +Key = 7045a9d7caae2c0a39f58998720974ec +IV = a9028008f708ae19dd28e75a02c6e84a05096779781ce0a908047152208468cc4b2a57d25608767f936cf70fbe82dd8a61497a180e1fc967caf4e6310ed850082e6919c922e021ec070dc64b040ed9edbbf5883676630d69e48953068b2bd006bd6d5417038604ee5aed04980c9ed2316c531ce6a3a73dfad90c04c58596d5e9 +PT = 635b946047c38533bb2cb4c9799b44f6eae0e63626901b0741f6dcf3c2bb02270343c7708a72dfe303b20f7805cb732386b341 +AAD = +CT = aebb8c1e914ea9bc1ada9b01f84cc8dbbc611f2cd386d5fc89497d37e5a469b28fe2fdf0ab0f1c882dcce50620b1b18a2d8343 +Tag = 27bcaf06 + +Count = 14 +Key = 79d2a35efbf03f57b66e875c232e10d5 +IV = 7cc96b48afff401adf4bab2ebedd021377b18a819f3c3af39fda42e24c5d62e67ec30f8bcab00263dc5a9bb06cbfe1750c98555901d34d775fdcc86841bd08fbeb44ba68ee794dc351a29a1a9de576d83c17a730d50db79cab88d538a441bb9ff6aa073a2a976de820ab5cc61f834753220d4e472a275dcd13db3e51a23a84a5 +PT = d91e1b2811b3e3894b46c563e6ea0b4a33990ba4fce8a354c941e1effc5691671de5d97c4c1a35e3730b43584944695f00544a +AAD = +CT = 6a45beb05c0dba6c38c997f8c37ef07c7cf78eacff6ff4dd6fa000e745e8053d2d270a746994f29c8628f41fc7fecdac158655 +Tag = 36caee75 diff --git a/src/test/crypto/gcmEncryptExtIV256.rsp b/src/test/crypto/gcmEncryptExtIV256.rsp new file mode 100644 index 0000000000..3faa8956bb --- /dev/null +++ b/src/test/crypto/gcmEncryptExtIV256.rsp @@ -0,0 +1,129 @@ +# CAVS 14.0 +# GCM Encrypt with keysize 256 test information +# Generated on Fri Aug 31 11:26:26 2012 + +[Keylen = 256] +[IVlen = 1024] +[PTlen = 408] +[AADlen = 0] +[Taglen = 32] + +Count = 0 +Key = 0df25c2bc9444b4a01e26d357a3ac0635fb6ff2e65ce1e759aae491a17772243 +IV = 54652573ba189cffed3bcfa60efbfb417eb4b0e8de80c7e53765d018cbddfae74617269eb35f29faf628d28c40737f9d9e1eb0b8757c984d94340ecc5ddd108f0a5a0e96335ea805950d378fea7569b98693e88bc1cfd9f6eb8d25de177122fc774a5b1957bcb80e92230c12fe401a8e00d0c04897e234644bda36ae761ea619 +PT = b29af460c6a5dbe56f1d67751346d7182c93413a6c328c6d85176cd8dea8ecce1cad3063c8708c0be9ae73d42bbb10421e73f1 +AAD = +CT = 3c786a3d0c8945bf320a21ca63f3b8bf5c6bf56a8412f836d7894e42c9e0695a8e41bf59b23fc52d17f8b341183f1cdce02e22 +Tag = 0b31bffe + +Count = 1 +Key = 1b6b7b8d00e543f0a17bd0bf595319a4a1f8a55ff41ce2380381d4e83c83243b +IV = 791d9546f180b838b50bb7b66b68f7f1aa09a2cef411a1dbf2e64c5ed026613200ac8f0e5b961925853621a1d4339322ea4b7bbc4adcbc008efeaaee0dc948a916b22ff8693a7d441620d0ff67680b23567f9582e22eec529408c6d0a00de1bd1ee5a11ed7fa29b7567f990e412ce90ee12d5b1d8ba8c2b528d0d9963fdc49a5 +PT = 0a76090de676d71d9f5ee8511864f0c9440a0fa12b5155a5bffc36127bc957b293d4fb624af3b956385783124f28f6f3c0f0e8 +AAD = +CT = fdaa9ef2c65d7666b61367f843863a3b273249192551b5633cfaf84a5ab9ecba42916395177f9a16c1ad385e77393cd93d71b3 +Tag = fd854d63 + +Count = 2 +Key = 261a0382c739093634502303d60dd0d2a568b5155147d661e7789f7bd70de82c +IV = e22330a2981b9b60354a740c49dcd17c9016cff50423977f7fc8500fc36a81610c979eb37a1f9c4d54d11b790906492205263178dca6d269a230a595ee95edeb60e45d92a2d6169877dfe5514b23db143dbcf3ca44e13cf5b7402e4f95a9e6760451be2d57d1d5fefb015ffe4b69456d87338865ee8775d1cce238f75c345dad +PT = 42c8c79988c52bcf38865a6f341a5165294af0947bc4fa597c6f648a1172070851c4ee154604d9a21c8d53f7eba77fbb72cf2b +AAD = +CT = 0fb368563c6ad0c097207239a791bc685274d434c4a54cb0ea97a98b4cee9eb100c4cd626ba82c338a3f51cb29391b1da3a44f +Tag = a9607e42 + +Count = 3 +Key = 5f3d6b5c76cb7ad0be3d771f1f7d1bafd88c8c7c7c84922f89f45f3b7453eb44 +IV = 7072d302e4acb642090c5d48e4549a8823725be1389316df0152e68c41f77937eadc4ee1d164101717cdcdd3bc9be3c2669c0b1394953090ac787fb117500b6275122f608e70a3f5063b21fae42ca4b00724b21c50e27c37a77d4befb118018ada7999888442e1271410f6b804c36a27a41e95a438792de4f06fd7122b177cc3 +PT = 00d82cc00c07d3cf0fd9b31ddc091d351aab9b58af07d2c59f3e28c7202ef6e3ec35726cbdd14f8f1985a770d092470f115be7 +AAD = +CT = 0aab0dbb31c113b33b116659294e0af5497c987af870ae8ce55288d48e8a4f1b3e8377cb34d06460daf3cc99fbefb90724a39f +Tag = 26e56b13 + +Count = 4 +Key = 1d4b6b6ba43269d46ca5c0fab38b61f1d2bf3d24dae46181fac73420a8e6194f +IV = 01c283a90c588dafc585f437dc111e94dd8ea98a622a5d2e554a87086ec10e8dac9b205fae70be3c024b3f9fbdc26208a6e44d082ad92a51a7fa1f861d0e93e7e74d4e41426e70b2e7ef7fbcbc5a302aed4bc8a42963ca53334259f1924e74225c3bf9aa2ffdf97a2be6474c72f6dedba7c454e36fbc3537596a63b61a4cf3ce +PT = 4eb036d945c14de26c4e0c83d446dacb57e91b5bdec2ef612bbed8135e57ce62b28843fbac555580ef23c74959eb869a017872 +AAD = +CT = 24d67f2e94294255b9fd90c16b67c7a8b3a47a42782954aa15e80ffe732d64342b1ec65aa66fc4d6b8fe9dfcdbf4d4c1a979e7 +Tag = 37358e86 + +Count = 5 +Key = cbfbb15a4fae4af3de55ffcaa4b8cdbd515ecc72fd50060b22acc8fad57a5f4b +IV = 86e47ba98d8ed360262f94afc364df373d1e44c788b4ee5ed7542822858aaeca2b3c07b5464d6f7edefca9759107bbb64c086d526cc4c9a6b9a7cd6dbb50acec2297597612fd436d8c8ddbc83664305b214cf2c3b6fab3e545c499c7f1cbba1897041475f94952d8a4fc10110986b5ccc7a10c1d9e6427dabba8d942ada9290a +PT = 5470a60f2818ffef374be3cc591084096a4a8caa0dac0024c12d304382c301264526b02efb674dfbcc3be5c818a1d88a7c19f1 +AAD = +CT = a6a8ebadfe5c5ac754e4a2a9cdc25fdf3588c69a287b0f9aba2347bfcdc110a1134426a45c62d577e52a8d61fae39ea38bad51 +Tag = 97406b4b + +Count = 6 +Key = 3ced5c63078a36f6b02d3bec0debacd19021c8dbc501a9d86f556c4ad2cdef8a +IV = b9629801cba22f7493b4f6394620d49a76ec686e524f5ebcb3a76b5a189473484060cbb01bcbf10048427ef21527626085c8a75aa5264b6338abbc26171c2c3a44f6b3b5c3fd05a892c8290a8f99be962deea48d7ae4e626c45a45ffda5efaad6e54e98ba876b039a5dde3d6061f217e57da4774acfaf1f5da9495083aad4dad +PT = c29788107799077ac6dbccad29a346727f263676a8510fbcadb9b5bf53df978b3382fdfc1b5c3312eaa0f7621b6efdaeebe1c0 +AAD = +CT = 5811d4a96169b39cd8f8ff3f931efdda68550f17558b48de8bb5adf455af5179c8c5ff4a73f363f8819daf846427132daf6a17 +Tag = 06e8e7fa + +Count = 7 +Key = 39ae4fdfe56b74a71325a1a685a6593b44936892890cd05d2719a420c97a7c64 +IV = 8f110d0fd20ece7f35c35b2eed32316fd742ae9346fc6907bd749361a4436427f80185b376b16a36fb95429cfaf2e22e46c210442fe5efc14985a9d9d847c3ceb5db02e0d999acbc3ba0afdfecbfaf65024258cc7f6fc8e3f568cbedec1c7eabe3ff3ab3c7331722b6400429d46b54820bd1f88ac03cfba5cbd0812d91342c4b +PT = 8b086d11be7ff55312addad86b49585ea38ea1ee7c4200964cd269a4bb5cfe0f518e6f9b733efb4ba3ce35ce2e803b0ad47d24 +AAD = +CT = be46d6e61dc11b2fc8ac4c9c5f49ceca0fb6fe1fc7221c7cc5d8ba254a92282500b1b31528314035cd125578de960b3bafb69d +Tag = 7c00aa14 + +Count = 8 +Key = c244e9afd90ec810acb7e586cf7a06386e6892e01c7d111d5c1455fc95250d1b +IV = 6ff9103f9751ff4743d856c5cd54491e1a537384260fbfe076f772ad0d66ff6ddd0aaff57023885b0a4d60d2b25c80b1cdc54802607770a61a2503c23cf26f1fe529573c8d4745b19dedaff5769a6a796c01540776d4fa99be9057ad87cdc973e7938640f9497753e88c4cdc358c1cfa06f1ddd826c33f44c55e2d183927baaa +PT = cd1296133cefe6f5cf6f8ec68b37172bfb793a8582d92a539f24f3582569bfcfebb706fe9f276716b185558fbfd6fe8ea99a1b +AAD = +CT = dd1694b178f15612c454885b3100c576c7b68206c57898161d4fdc51e75a428840c5cfee104c3d85fcbd92edbd1d80f22d8e64 +Tag = 741953f5 + +Count = 9 +Key = 785ea9cd2403ecaeec3e4940dce7c41ec012203a2610c780bca5d15af64748c5 +IV = fbc177ec9d47f69e2fcaba9fef7de30735e46a5d20dc66bf66c8a76a382051d780f58dccae8e2054bbe437a5bc0814381bac2b0efa99202aa1f1bf7f51b842907dc9b60f83987c31eae086e26e2018243bdd47a291b523ac6905b40dfb442ba239c876cdfaa581b2ef0683456ac944829a97b663ecd48c116d06f1f054ddfcdb +PT = a3465ad9a4009e26c39acb1d424b7ba6556a74dcc2b78a5ab65b5d07c2a97f382aadb7415395fa6fc90bd137f6894a75f70907 +AAD = +CT = 2b3eb5ab251282da1795a25f9b43ccaa5a27643d042fc315ad662947b0c5cddeb5848e6c69869ae5e81f5c76729bbaceda9889 +Tag = 64173526 + +Count = 10 +Key = 0becf9c7a069db5b9a25f2871fb0594e452126262ec1c48bca3d3024d85d0c51 +IV = e05643a72ab0b1fc42a6f17302d6446fc507bbb4f0ad59178f5084530f02534df2e673e92e67802629f93221bba545a13fb9143eb2a2ee4bce047be621a9d96a450a19951d93c527eb698ac7e132e4f00985dff91e079ae791549d37da3105c77ef5d8cb7649f1aa761b5a5fbc9e0d7dfb5aade98f3df4a641cca02f33eed55c +PT = 70ba9822801760e5ec647a41e27cf6069978be6a28be0b1da0f3661e124847823dec7ec6af737d6dea597ce5bd5baa5d6f2651 +AAD = +CT = 0d45b5ad77226524374a3bca3680528787cefd9978b3f1407688886ce28fea86e7d9005d7d6acde2a7c2fb158479918c06bba4 +Tag = 43dfd859 + +Count = 11 +Key = d1f374702b4481b83e7c7783853c1850e887b0c80cd28c0686b4cab6adc744c2 +IV = 5e15c44cef36c1c89e02b2979f4e7d275fe5973a80580753f6fd51ea477ba84346a030b90b22a9adfff02096bc0b1691c37ee21cdc1b5f4862696d195859cd2bf0b423f5d19da5e1475bcc99f96b8c7c51fe85930aea0e97304d4c025b52bd386666d0537036360e939a6fbdfd6ce3b012e551d333fa74f3cbb9d33a59477364 +PT = 996428a232b1b9ac81ff5260eb77938f0d531a4a0ed3bd774c72c18128af72e964f8d05fb7ec4e0ac8e37056a48c85713a3a01 +AAD = +CT = aa9c103ae51bf556f7c57f9948d28c859f458a74e22f039fea183e9e262b023bdac9bb5a2f167b34cde4a694c555eb0c905f34 +Tag = d1ed9cf0 + +Count = 12 +Key = a6a69524d66f015b4afb746ad0942410baf06d1aba18ef1a8b40e35633f04ce1 +IV = 5e1df758c63728b1269a1d3f610b6e3724bc7ff797dcb4a7aca9dfe1c8e1717ce4281d1c5b4b33d0aafcee342a4f4eb30089eeab2b0470d3f9b709622bde4654d41b3bbc6bf59c11edea28f26b099d83d4fefdc63ab7218221238fe1c230b1290235465b445f60eaa9822eaa8da2c08f6f2fdaaef87dacf74c7e5d6cea191c74 +PT = 0ffc6c3090817fc1a0e6f3802269e263e40d17772fabf5cbee905962878d77c8bc4223e5671bd9f310d8db56ceaebae41fc79e +AAD = +CT = f24537db3fe27eb07f1d2b9b7059dfae97df86d4e609930491c4ec3154462df308ba1c85204dd754521d8de9619603f90f8c44 +Tag = 360adc36 + +Count = 13 +Key = ff0007317f9ecc43c1601708614eb8d443a407318cbf3c085c0a9a7c67faafa8 +IV = 8186f92bd394ecfbf4bcbb9e7e442bd6faa1e59f8785d9aec82551fee38aa10212a8477be3055927379a906902c153598d50c63e316c4e4f3956ef04ce40e5b6f1901fb5d08a23b913e5ad53512b02bb75ce96f8c50a2cebdfa058ded996fe1dc5370ee50c6d90e948129d544dc89c2f28dda8429bb338d6aed0d9557f96889e +PT = e7f4063319fd31e07192a5fc9e486fc0a2a3470671a46356ab0d32989803259c0dae103a4033c533fbde585866fc2af5eba151 +AAD = +CT = 8f86644614e4f6483f28eec02fab33e7cbbd56381a8f878522c2015b91d48652472e356f608c62361939dc983e1cd364a28665 +Tag = 83335b3a + +Count = 14 +Key = 43646d9ddebec23447febe71596e6f9b2387965db1faf3f06feec8bc5c808342 +IV = 0d3774607261b07427dc77f0dcf57b026226dbd2c1df1b9d74598582da44c677af36a6bd80b6bc1f000632c84c5dfa701f8b51b4da228d340d8b4ccc4d2f5d7b5fad00809133eae9250ecd18d7a8741bb57c394396feb81c20bec23de8520f883b8e22b362dde6e6bd9dad73e8919695384a04c09a28bdb6de2fa356b0d259b1 +PT = 2aab9a484b817b759bd9d876967c90160a18208cfa753e7bccd4f73a715aaa6acc6ce666e97bc22fbcf11f263dfed332418707 +AAD = +CT = dd80974e31ad15ba26ec8a8dacfa5f57f0bc69f7113cc6c39fb038ddce37ffeee3f789a02cd86b0418fe16ca104b5fbdb26432 +Tag = 75c35f4c diff --git a/src/test/crypto/t/001_testcrypto.pl b/src/test/crypto/t/001_testcrypto.pl new file mode 100644 index 0000000000..586a37c5bd --- /dev/null +++ b/src/test/crypto/t/001_testcrypto.pl @@ -0,0 +1,137 @@ +# Reads and parses .rsp files and runs them through testcrypto +# +# Test vectors downloaded from: +# +# https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/cavp-testing-block-cipher-modes +# +# Specifically GCM Test Vectors (SP 800-38D): +# +# https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip +# +# Note that the AADlen > 0 cases were removed from our set, since we don't support that and +# the test code will just skip them currently, and we also don't bother testing 192-bit, +# but one could download the full set and run all of the files through if they wished and +# all should pass (they did when this test suite was written originally). + +use strict; +use warnings; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 56; +} +else +{ + plan skip_all => 'SSL not supported by this build'; +} + + +# XXX add CTR tests here + +my $algo = "AES-GCM"; +my @rspfiles = ( + "gcmDecrypt128.rsp", "gcmDecrypt256.rsp", + "gcmEncryptExtIV128.rsp", "gcmEncryptExtIV256.rsp"); + +note "running tests"; + +foreach my $rspfile (@rspfiles) +{ + open(my $in_rspfile, '<', $rspfile) || die; + my %testrun; + my %lengths; + + while (my $line = <$in_rspfile>) + { + + chomp($line); + + # Remove CR, if it's there. + $line =~ s/\r$//; + + # Skip comments + if ($line =~ /^[[:space:]]*#/) { next; } + + # If we hit a blank, time to run a test + if ($line =~ /^[[:space:]]*$/) + { + if (%testrun) + { + my @testargs; + + # Set up the command to run + push(@testargs, ("$ENV{TESTDIR}/testcrypto", '-a', $algo)); + + if ($testrun{'Key'}) + { + push(@testargs, ('-k', $testrun{'Key'})); + } + + if ($testrun{'IV'}) + { + push(@testargs, ('-i', $testrun{'IV'})); + } + + if ($testrun{'CT'}) + { + push(@testargs, ('-c', $testrun{'CT'})); + } + + if ($testrun{'AAD'}) + { + # Don't currently support AAD + undef(%testrun); + next; + } + + if ($testrun{'Tag'}) + { + push(@testargs, ('-t', $testrun{'Tag'})); + } + + if ($testrun{'PT'}) + { + push(@testargs, ('-p', $testrun{'PT'})); + } + + if ($testrun{fail}) + { + command_exit_is(\@testargs, 1, + "Run $testrun{Count} of Keylen: $lengths{Keylen}, IVlen: $lengths{IVlen}, PTlen: $lengths{PTlen}, AADlen: $lengths{AADlen}, Taglen: $lengths{Taglen}" + ); + } + else + { + command_ok(\@testargs, + "Run $testrun{Count} of Keylen: $lengths{Keylen}, IVlen: $lengths{IVlen}, PTlen: $lengths{PTlen}, AADlen: $lengths{AADlen}, Taglen: $lengths{Taglen}" + ); + } + undef(%testrun); + undef(%lengths); + } + else + { + next; + } + } + + # Grab length information, just to have. + if ($line =~ /^\[([A-Za-z]*) = ([0-9]*)]$/) + { + $lengths{$1} = $2; + next; + } + + if ($line =~ /^([A-Za-z]*) = ([a-f0-9]*)$/) + { + $testrun{$1} = $2; + } + + if ($line =~ /^FAIL$/) + { + $testrun{fail} = 1; + } + } +} diff --git a/src/test/crypto/t/002_testkwp.pl b/src/test/crypto/t/002_testkwp.pl new file mode 100644 index 0000000000..25911cfd9f --- /dev/null +++ b/src/test/crypto/t/002_testkwp.pl @@ -0,0 +1,126 @@ +# Reads and parses .rsp files and runs them through testcrypto +# +# (Partial) Test vectors downloaded from: +# +# https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/cavp-testing-block-cipher-modes +# +# Specifically Key Wrap Test Vectors (SP 800-38F): +# +# https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/kwtestvectors.zip +# +# We don't include the 192-bit tests, though they all worked when this test suite was written. +# We also don't include the _inv tests as those aren't supported in OpenSSL yet. + +use strict; +use warnings; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 20; +} +else +{ + plan skip_all => 'SSL not supported by this build'; +} + + +my $algo; +# my @txtfiles = ("KWP_AD_128.txt", "KWP_AD_192.txt", "KWP_AD_256.txt", "KWP_AE_128.txt", "KWP_AE_192.txt", "KWP_AE_256.txt"); +my @txtfiles = + ("KWP_AD_128.txt", "KWP_AD_256.txt", "KWP_AE_128.txt", "KWP_AE_256.txt"); + +note "running tests"; + +foreach my $txtfile (@txtfiles) +{ + open(my $in_txtfile, '<', $txtfile) || die; + my %testrun; + my %lengths; + + if ($txtfile =~ /^KWP_/) + { + $algo = 'AES-KWP'; + } + + while (my $line = <$in_txtfile>) + { + + chomp($line); + + # Remove CR, if it's there. + $line =~ s/\r$//; + + # Skip comments + if ($line =~ /^[[:space:]]*#/) { next; } + + # If we hit a blank, time to run a test + if ($line =~ /^[[:space:]]*$/) + { + if (%testrun) + { + my @testargs; + + # Set up the command to run + push(@testargs, ("$ENV{TESTDIR}/testcrypto", '-a', $algo)); + + if ($testrun{'K'}) + { + push(@testargs, ('-k', $testrun{'K'})); + } + + if ($testrun{'C'}) + { + push(@testargs, ('-c', $testrun{'C'})); + } + + if ($testrun{'P'}) + { + push(@testargs, ('-p', $testrun{'P'})); + } + + if ($testrun{fail}) + { + command_exit_is(\@testargs, 1, + "Run $testrun{COUNT} of Plaintext Length: $lengths{'PLAINTEXT LENGTH'}" + ); + } + else + { + command_ok(\@testargs, + "Run $testrun{COUNT} of Plaintext Length: $lengths{'PLAINTEXT LENGTH'}" + ); + } + undef(%testrun); + undef(%lengths); + } + else + { + next; + } + } + + # Grab length information, just to have. + if ($line =~ /^\[([A-Za-z ]*) = ([0-9]*)]$/) + { + $lengths{$1} = $2; + next; + } + + if ($line =~ /^([A-Z]) = ([a-f0-9]*)$/) + { + $testrun{$1} = $2; + } + + if ($line =~ /^COUNT = ([0-9]*)$/) + { + $testrun{COUNT} = $1; + } + + if ($line =~ /^FAIL$/) + { + $testrun{fail} = 1; + } + } +} diff --git a/src/test/crypto/t/003_clusterkey.pl b/src/test/crypto/t/003_clusterkey.pl new file mode 100644 index 0000000000..5b8a255ace --- /dev/null +++ b/src/test/crypto/t/003_clusterkey.pl @@ -0,0 +1,93 @@ +# Test cluster file encryption key managment +# + +use strict; +use warnings; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 6; +} +else +{ + plan skip_all => "tests cannot run without OpenSSL"; +} + +# generate two cluster file encryption keys of random hex digits +my ($rand_hex, $rand_hex2); +$rand_hex .= sprintf("%x", rand 16) for 1 .. 64; +$rand_hex2 .= sprintf("%x", rand 16) for 1 .. 64; + +# initialize cluster using the first cluster key +my $node = get_new_node('node'); +$node->init( + extra => [ + '--file-encryption-method', 'AES256', + '--cluster-key-command', "echo $rand_hex" + ]); + +# Set wal_level to 'replica'; encryption can't use 'minimal' +$node->append_conf('postgresql.conf', 'wal_level=replica'); + +$node->start; + +# check encryption method +my $file_encryption_method = + $node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq 'AES256', 'file_encryption_method is valid'); + +# record pg_proc count +my $old_pg_proc_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM pg_proc;'); +ok($old_pg_proc_count > 0, 'pg_proc count is valid'); + +# create permanent table +$node->safe_psql('postgres', + 'CREATE TABLE perm_table (x) AS SELECT * FROM generate_series(1, 100);'); + +# create unlogged table +# Non-permanent tables like unlogged tables use a special nonce bit, so test those here. +$node->safe_psql('postgres', + 'CREATE UNLOGGED TABLE unlog_table (x) AS SELECT * FROM generate_series(1, 200);' +); + +# We can run pg_alterckey and change the cluster_key_command here +# without affecting the running server. +system_or_bail( + 'pg_alterckey', + "echo $rand_hex", + "echo $rand_hex2", + $node->data_dir); + +$node->safe_psql('postgres', + "ALTER SYSTEM SET cluster_key_command TO 'echo $rand_hex2'"); + +$node->stop; + +# start/stop with new cluster key +$node->start; + +# check encryption method +$file_encryption_method = + $node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq 'AES256', 'file_encryption_method is valid'); + +# check pg_proc count +my $new_pg_proc_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM pg_proc;'); +ok($new_pg_proc_count == $old_pg_proc_count, 'old/new pg_proc counts match'); + +# check permanent table count +my $perm_table_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM perm_table;'); +ok($perm_table_count == 100, 'perm_table_count count matches'); + +# check unlogged table count +my $unlog_table_count = + $node->safe_psql('postgres', 'SELECT COUNT(*) FROM unlog_table;'); +ok($unlog_table_count == 200, 'unlog_table_count count matches'); + +$node->stop; diff --git a/src/test/crypto/t/004_buffers.pl b/src/test/crypto/t/004_buffers.pl new file mode 100644 index 0000000000..a1379b4772 --- /dev/null +++ b/src/test/crypto/t/004_buffers.pl @@ -0,0 +1,157 @@ +# Test cluster file encryption buffer management + +# This tests that an encrypted server actually encrypts the database +# files. It does this by checking for strings in the database files in +# both non-encrypted and encrypted clusters. We test a system table, a +# permanent relation, and a unlogged/non-permanent table. +# (Non-permanent relations use a special nonce bit, which is why we test +# it here.) + +use strict; +use warnings; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_ssl} eq 'openssl') +{ + plan tests => 17; +} +else +{ + plan skip_all => "tests cannot run without OpenSSL"; +} + +my %file_match_count; + +sub get_cluster_file_contents +{ + my $node = shift(); + my %relnode; + + # get postgres database oid + my $postgres_db_oid = $node->safe_psql('postgres', + "SELECT oid FROM pg_database WHERE datname = 'postgres';"); + ok($postgres_db_oid != 0, 'retrieving postgres database oid'); + + # get pg_proc relfilenode + $relnode{pg_proc} = + $node->safe_psql('postgres', "SELECT pg_relation_filenode('pg_proc');"); + ok($relnode{pg_proc} != 0, 'retrieving pg_proc relfilenode'); + + # create permanent table + $node->safe_psql('postgres', + "CREATE TABLE perm_table (x) AS SELECT 'aaaaaaaa' FROM generate_series(1, 100);" + ); + + # get permanent table relfilenode + $relnode{perm} = $node->safe_psql('postgres', + "SELECT pg_relation_filenode('perm_table');"); + ok($relnode{perm} != 0, 'retrieving permanent table relfilenode'); + + # create unlogged table + $node->safe_psql('postgres', + "CREATE UNLOGGED TABLE unlog_table (x) AS SELECT 'bbbbbbbb' FROM generate_series(1, 200);" + ); + + # get unlogged table relfilenode + $relnode{unlog} = $node->safe_psql('postgres', + "SELECT pg_relation_filenode('unlog_table');"); + ok($relnode{unlog} != 0, 'retrieving unlogged table relfilenode'); + + my $file_contents = + slurp_file($node->basedir . + '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{pg_proc}); + # () converts to list context + $file_match_count{pg_proc} = () = $file_contents =~ m/pg_[a-z]{3,}/g; + + $file_contents = + slurp_file($node->basedir . + '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{perm}); + $file_match_count{perm} = () = $file_contents =~ m/a{8,}/g; + + $file_contents = + slurp_file($node->basedir . + '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{unlog}); + $file_match_count{unlog} = () = $file_contents =~ m/b{8,}/g; +} + +# +# Test with disabled encryption +# + +# initialize cluster +my $non_encrypted_node = get_new_node('non_encrypted_node'); +$non_encrypted_node->init(); + +$non_encrypted_node->start; + +# check encryption is disabled +my $file_encryption_method = + $non_encrypted_node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq '', 'file_encryption_method is valid'); + +get_cluster_file_contents($non_encrypted_node); + +# record pg_proc count +my $query_pg_proc_count = $non_encrypted_node->safe_psql('postgres', + "SELECT COUNT(*) FROM pg_proc WHERE proname ~ '^pg_[a-z]{3,}';"); +ok($query_pg_proc_count > 0, 'pg_proc count is valid'); + +# check pg_proc count +ok($file_match_count{pg_proc} != $query_pg_proc_count, + 'SQL/file pg_proc counts match'); + +# check permanent table count +ok($file_match_count{perm} != 100, 'perm_table count matches'); + +# check unlogged table count +ok($file_match_count{unlog} != 200, 'unlog_table count matches'); + +$non_encrypted_node->stop; + + +#--------------------------------------------------------------------------- + +# +# Test with enabled encryption +# + +my $rand_hex; +$rand_hex .= sprintf("%x", rand 16) for 1 .. 64; + +# initialize cluster using a cluster key +my $encrypted_node = get_new_node('encrypted_node'); +$encrypted_node->init( + extra => [ + # We tested AES256 in 003, so use AES128 + '--file-encryption-method', 'AES128', + '--cluster-key-command', "echo $rand_hex" + ]); + +# Set wal_level to 'replica'; encryption can't use 'minimal' +$encrypted_node->append_conf('postgresql.conf', 'wal_level=replica'); + +$encrypted_node->start; + +# check encryption method +$file_encryption_method = + $encrypted_node->safe_psql('postgres', 'SHOW file_encryption_method;'); +ok($file_encryption_method eq 'AES128', 'file_encryption_method is valid'); + +get_cluster_file_contents($encrypted_node); + +# Because the files are encrypted, we should get zero matches for all +# comparisons below. However, technically the encrypted data might +# match the desired string, so we allow one such match. + +# check pg_proc +ok($file_match_count{pg_proc} <= 1, 'pg_proc is encrypted'); + +# check permanent table +ok($file_match_count{perm} <= 1, 'perm_table is encrypted'); + +# check unlogged table +ok($file_match_count{unlog} <= 1, 'unlog_table is encrypted'); + +$encrypted_node->stop; diff --git a/src/test/crypto/testcrypto.c b/src/test/crypto/testcrypto.c new file mode 100644 index 0000000000..85bf921c09 --- /dev/null +++ b/src/test/crypto/testcrypto.c @@ -0,0 +1,458 @@ +/*------------------------------------------------------------------------- + * + * testcrypto.c + * A utility to test our encryption / decryption routines. + * + * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/test/crypto/testcrypto.c + * + *------------------------------------------------------------------------- + */ + +#define FRONTEND 1 + +#define EXITSUCCESS 0 +#define EXITDECRYPTFAIL 1 +#define EXITFAILURE 2 + +#include "postgres_fe.h" + +#include <signal.h> +#include <string.h> +#include <unistd.h> +#include <sys/stat.h> + +#include "common/hex.h" +#include "common/cipher.h" +#include "common/logging.h" +#include "getopt_long.h" +#include "pg_getopt.h" + +static const char *progname; + +static void +usage(const char *progname) +{ + printf(_("%s tests encryption/decryption routines in PG common library.\n\n"), progname); + printf(_("Usage:\n")); + printf(_(" %s [OPTION]\n"), progname); + printf("\n"); + printf(_(" Performs one encryption and one decryption run.\n")); + printf("\n"); + printf(_(" Encrypts the provided plaintext (or the empty string if none given) and generates a tag, if using AES-GCM, using the key and IV given.\n")); + printf(_(" After encryption, compares provided ciphertext to resulting ciphertext.\n")); + printf(_(" Compares provided tag, if any, to resulting tag.\n")); + printf(_(" If no tag is provided, then the tag created during encryption is used during decryption.\n")); + printf("\n"); + printf(_(" Decrypts the provided ciphertext (or the empty string if none given) using the key, and IV + tag given if using AES-GCM.\n")); + printf(_(" After successful decryption (requires tag to match for AES-GCM), compares provided plaintext to resulting plaintext.\n")); + printf(_(" Exits with '1' if decryption fails.\n")); + printf("\n"); + printf(_(" Exits with '2' for any other failure.\n")); + printf("\n"); + printf(_(" Key is always required, IV is required for AES-GCM mode.\n")); + printf("\n"); + printf(_(" Algorithms supported are AES-GCM and AES-KWP.\n")); + printf("\n"); + printf(_("\nOptions:\n")); + printf(_(" -a, --algorithm=ALG Crypto algorithm to use\n")); + printf(_(" -i, --init-vector=IV Initialization vector to use\n")); + printf(_(" -k, --key=KEY Key to use, in hex\n")); + printf(_(" -p, --plain-text=PT Plain text to encrypt\n")); + printf(_(" -c, --cipher-text=CT Cipher text to decrypt\n")); + printf(_(" -t, --tag=TAG Tag to use for decryption\n")); + printf(_(" -T, --tag-length=LEN Length of tag to use for encryption\n")); + printf(_(" -v, --verbose verbose output\n")); + printf(_(" -V, --version output version information, then exit\n")); + printf(_(" -?, --help show this help, then exit\n")); + printf("\n"); + printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT); +} + + +int +main(int argc, char *argv[]) +{ + char *algorithm = NULL, + *iv_hex = NULL, + *key_hex = NULL, + *plaintext_hex = NULL, + *ciphertext_hex = NULL, + *tag_hex = NULL; + + unsigned char *plaintext = NULL, + *ciphertext = NULL, + *key = NULL, + *iv = NULL, + *tag = NULL, + *tag_result = NULL, + *result = NULL; + + int verbose = 0, + plaintext_len = 0, + ciphertext_len = 0, + key_len = 0, + iv_len = 0, + tag_len = 16, + result_len = 0, + blocksize = 0, + cipher = PG_CIPHER_AES_GCM; + + PgCipherCtx *ctx = NULL; + + static struct option long_options[] = { + {"algorithm", required_argument, NULL, 'a'}, + {"init-vector", required_argument, NULL, 'i'}, + {"key", required_argument, NULL, 'k'}, + {"plain-text", required_argument, NULL, 'p'}, + {"cipher-text", required_argument, NULL, 'c'}, + {"tag", required_argument, NULL, 't'}, + {"tag-length", required_argument, NULL, 'T'}, + {"verbose", required_argument, NULL, 'v'}, + {NULL, 0, NULL, 0} + }; + + int c; + + pg_logging_init(argv[0]); + set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("testcrypto")); + progname = get_progname(argv[0]); + + if (argc > 1) + { + if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0) + { + usage(progname); + exit(EXITSUCCESS); + } + if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0) + { + puts("testcrypto (PostgreSQL) " PG_VERSION); + exit(EXITSUCCESS); + } + } + + /* Process command-line argument */ + + while ((c = getopt_long(argc, argv, "a:i:k:p:c:t:T:v", long_options, NULL)) != -1) + { + switch (c) + { + case 'a': + algorithm = pg_strdup(optarg); + break; + + case 'i': + iv_hex = pg_strdup(optarg); + break; + + case 'k': + key_hex = pg_strdup(optarg); + break; + + case 'p': + plaintext_hex = pg_strdup(optarg); + break; + + case 'c': + ciphertext_hex = pg_strdup(optarg); + break; + + case 't': + tag_hex = pg_strdup(optarg); + break; + + case 'T': + tag_len = atoi(optarg); + break; + + case 'v': + verbose = 1; + break; + + default: + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname); + exit(EXITFAILURE); + } + } + + /* Complain if any arguments remain */ + if (optind < argc) + { + pg_log_error("too many command-line arguments (first is \"%s\")", + argv[optind]); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + + /* Check options passed in */ + if (algorithm) + { + if (strcmp(algorithm, "AES-GCM") == 0) + cipher = PG_CIPHER_AES_GCM; + else if (strcmp(algorithm, "AES-KWP") == 0) + cipher = PG_CIPHER_AES_KWP; + else + { + pg_log_error("Unsupported algorithm selected."); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + } + + if (key_hex == NULL) + { + pg_log_error("Key must be provided"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + else + { + size_t key_hex_len = strlen(key_hex); + + key_len = pg_hex_dec_len(key_hex_len); + + key = pg_malloc0(key_len); + pg_hex_decode(key_hex, key_hex_len, (char *) key, key_len); + } + + if (cipher == PG_CIPHER_AES_GCM && iv_hex == NULL) + { + pg_log_error("Initialization vector must be provided"); + fprintf(stderr, _("Try \"%s --help\" for more information.\n"), + progname); + exit(EXITFAILURE); + } + else if (cipher == PG_CIPHER_AES_GCM) + { + size_t iv_hex_len = strlen(iv_hex); + + iv_len = pg_hex_dec_len(iv_hex_len); + + iv = pg_malloc0(iv_len); + pg_hex_decode(iv_hex, iv_hex_len, (char *) iv, iv_len); + } + + if (plaintext_hex) + { + size_t plaintext_hex_len = strlen(plaintext_hex); + + plaintext_len = pg_hex_dec_len(plaintext_hex_len); + + plaintext = pg_malloc0(plaintext_len); + pg_hex_decode(plaintext_hex, plaintext_hex_len, (char *) plaintext, + plaintext_len); + } + + /* + * OpenSSL 1.1.1d and earlier crashes on some zero-length plaintext and + * ciphertext strings. It crashes on an encryption call to + * EVP_EncryptFinal_ex(() in GCM mode of zero-length strings if plaintext + * is NULL, even though plaintext_len is zero. Setting plaintext to + * non-NULL allows it to work. In KWP mode, zero-length strings fail if + * plaintext_len = 0 and plaintext is non-NULL (the opposite). OpenSSL + * 1.1.1e+ is fine with all options. + */ + else if (cipher == PG_CIPHER_AES_GCM) + { + plaintext_len = 0; + plaintext = pg_malloc0(1); + } + + if (ciphertext_hex) + { + size_t ciphertext_hex_len = strlen(ciphertext_hex); + + ciphertext_len = pg_hex_dec_len(ciphertext_hex_len); + + ciphertext = pg_malloc0(ciphertext_len); + pg_hex_decode(ciphertext_hex, ciphertext_hex_len, + (char *) ciphertext, ciphertext_len); + } + /* see OpenSSL 1.1.1d item above, though crash only happens in GCM mode */ + else if (cipher == PG_CIPHER_AES_GCM) + { + ciphertext_len = 0; + ciphertext = pg_malloc0(1); + } + + if (cipher == PG_CIPHER_AES_GCM) + tag_result = pg_malloc0(tag_len); + + if (tag_hex) + { + size_t tag_hex_len = strlen(tag_hex); + + tag_len = pg_hex_dec_len(tag_hex_len); + + tag = pg_malloc0(tag_len); + pg_hex_decode(tag_hex, tag_hex_len, (char *) tag, tag_len); + } + else + tag = tag_result; + + if (verbose) + { + printf("Alrogithm: %d\n", cipher); + printf("Key length: %d (%d bits)\n", key_len, key_len * 8); + printf("IV length: %d (%d bits)\n", iv_len, iv_len * 8); + printf("Tag length: %d (%d bits)\n", tag_len, tag_len * 8); + printf("Plaintext length: %d\n", plaintext_len); + printf("Ciphertext length: %d\n", ciphertext_len); + } + + /* + * Encryption + * + * We run through the encryption even if there wasn't a plaintext + * provided- in that case we just encrypt the empty string. + */ + ctx = pg_cipher_ctx_create(cipher, key, key_len, true); + if (!ctx) + { + pg_log_error("Error creating encryption context, be sure key is of supported length"); + exit(EXITFAILURE); + } + + blocksize = pg_cipher_blocksize(ctx); + + /* If we were provided with a plaintext input */ + if (plaintext_len != 0) + { + /* Encryption might result in as much as input length + blocksize */ + result_len = plaintext_len + blocksize; + result = palloc0(result_len); + + if (ciphertext_hex == NULL) + { + ciphertext = result; + ciphertext_len = plaintext_len; + } + } + + if (cipher == PG_CIPHER_AES_GCM) + { + if (!pg_cipher_encrypt(ctx, cipher, + plaintext, plaintext_len, + result, &result_len, + iv, iv_len, + tag_result, tag_len)) + { + pg_log_error("Error during encryption."); + exit(EXITFAILURE); + } + } + else if (cipher == PG_CIPHER_AES_KWP) + { + if (!pg_cipher_keywrap(ctx, + plaintext, plaintext_len, + result, &result_len)) + { + pg_log_error("Error during encryption."); + exit(EXITFAILURE); + } + } + + if (verbose || ciphertext == NULL) + { + uint64 result_hex_len = pg_hex_enc_len(result_len); + char *result_hex = palloc0(result_hex_len + 1); + + pg_hex_encode((char *) result, result_len, result_hex, result_hex_len); + result_hex[result_hex_len] = '\0'; + + printf("ciphertext: %s\n", result_hex); + + if (cipher == PG_CIPHER_AES_GCM) + { + result_hex_len = pg_hex_enc_len(tag_len); + result_hex = palloc0(result_hex_len + 1); + + pg_hex_encode((char *) tag_result, tag_len, result_hex, result_hex_len); + result_hex[result_hex_len] = '\0'; + + printf("tag: %s\n", result_hex); + } + } + + /* + * Report on non-matching results, but still go through the decryption + * routine to make sure that we get the correct result, and then error + * out. + */ + if (plaintext_len != 0 && ciphertext != NULL && memcmp(ciphertext, result, plaintext_len) != 0) + pg_log_error("Provided ciphertext does not match"); + + if (cipher == PG_CIPHER_AES_GCM && tag != tag_result && memcmp(tag, tag_result, tag_len) != 0) + pg_log_error("Provided tag does not match"); + + /* + * If a ciphertext was provided then use that as the max size of our + * plaintext result. We shouldn't ever get a result larger. + */ + if (ciphertext_len != 0) + { + result_len = ciphertext_len; + result = palloc0(result_len); + } + + /* + * Decryption + * + * We run through the decryption even if there wasn't a ciphertext + * provided- in that case we just decrypt the empty string. + */ + ctx = pg_cipher_ctx_create(cipher, key, key_len, false); + if (!ctx) + { + pg_log_error("Error creating decryption context, be sure key is of supported length"); + exit(EXITFAILURE); + } + + if (cipher == PG_CIPHER_AES_GCM) + { + if (!pg_cipher_decrypt(ctx, cipher, + ciphertext, ciphertext_len, + result, &result_len, + iv, iv_len, + tag, tag_len)) + { + pg_log_error("Error during decryption."); + exit(EXITDECRYPTFAIL); + } + } + else if (cipher == PG_CIPHER_AES_KWP) + { + if (!pg_cipher_keyunwrap(ctx, + ciphertext, ciphertext_len, + result, &result_len)) + { + pg_log_error("Error during decryption."); + exit(EXITDECRYPTFAIL); + } + } + + if (verbose || plaintext == NULL) + { + uint64 result_hex_len = pg_hex_enc_len(result_len); + char *result_hex = palloc0(result_hex_len + 1); + + pg_hex_encode((char *) result, result_len, result_hex, result_hex_len); + result_hex[result_hex_len] = '\0'; + + printf("plaintext: %s\n", result_hex); + } + + if (ciphertext_len != 0 && plaintext != NULL && memcmp(plaintext, result, plaintext_len) != 0) + { + pg_log_error("Provided plaintext does not match"); + exit(EXITFAILURE); + } + + exit(EXITSUCCESS); +} -- 2.31.1 [application/octet-stream] v2-0010-cfe-10-hint_over_cfe-09-test-squash-commit.patch (17.6K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/11-v2-0010-cfe-10-hint_over_cfe-09-test-squash-commit.patch) download | inline diff: From a396672cf4c919b273a8ad8a103e659c1b2e7d1b Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:03 -0400 Subject: [PATCH v2 10/12] cfe-10-hint_over_cfe-09-test squash commit --- doc/src/sgml/config.sgml | 7 +- src/backend/access/rmgrdesc/xlogdesc.c | 6 +- src/backend/access/transam/xlog.c | 4 + src/backend/access/transam/xloginsert.c | 24 +++++ src/backend/replication/logical/decode.c | 1 + src/backend/storage/buffer/bufmgr.c | 131 ++++++++++++++++------- src/bin/pg_alterckey/pg_alterckey.c | 76 +++++++++++-- src/include/access/xlog.h | 14 +-- src/include/access/xloginsert.h | 2 + src/include/catalog/pg_control.h | 1 + 10 files changed, 214 insertions(+), 52 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 4e95f7e22a..b1b3914933 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3164,9 +3164,10 @@ include_dir 'conf.d' </para> <para> - If data checksums are enabled, hint bit updates are always WAL-logged - and this setting is ignored. You can use this setting to test how much - extra WAL-logging would occur if your database had data checksums + If data checksums or cluster file encryption is enabled, + hint bit updates are always WAL-logged and this setting is + ignored. You can use this setting to test how much extra + WAL-logging would occur if your database had data checksums enabled. </para> diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c index 3fd7185f21..e08c364d03 100644 --- a/src/backend/access/rmgrdesc/xlogdesc.c +++ b/src/backend/access/rmgrdesc/xlogdesc.c @@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record) appendStringInfoString(buf, xlrec->rp_name); } - else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT) + else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT || + info == XLOG_ENCRYPTION_LSN) { /* no further information to print */ } @@ -196,6 +197,9 @@ xlog_identify(uint8 info) case XLOG_FPI_FOR_HINT: id = "FPI_FOR_HINT"; break; + case XLOG_ENCRYPTION_LSN: + id = "ENCRYPTION_LSN"; + break; } return id; diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index e247376ead..38e46bbbf7 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -8012,6 +8012,10 @@ xlog_redo(XLogReaderState *record) UnlockReleaseBuffer(buffer); } } + else if (info == XLOG_ENCRYPTION_LSN) + { + /* nothing to do here */ + } else if (info == XLOG_BACKUP_END) { /* nothing to do here, handled in xlogrecovery_redo() */ diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c index f811523448..b4bbacae26 100644 --- a/src/backend/access/transam/xloginsert.c +++ b/src/backend/access/transam/xloginsert.c @@ -1081,6 +1081,30 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std) return recptr; } +/* + * This function returns either a WAL or fake LSN, for use by encryption. + */ +XLogRecPtr +LSNForEncryption(bool use_wal_lsn) +{ + if (use_wal_lsn) + { + int dummy = 0; + + Assert(FileEncryptionEnabled); + /* + * Records other than SWITCH_WAL must have content. We use an integer 0 to + * follow the restriction. + */ + XLogBeginInsert(); + XLogSetRecordFlags(XLOG_MARK_UNIMPORTANT); + XLogRegisterData((char *) &dummy, sizeof(dummy)); + return XLogInsert(RM_XLOG_ID, XLOG_ENCRYPTION_LSN); + } + else + return GetFakeLSNForUnloggedRel(); +} + /* * Write a WAL record containing a full image of a page. Caller is responsible * for writing the page to disk after calling this routine. diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index 2cc0ac9eb0..6922f6c0e2 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -162,6 +162,7 @@ xlog_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf) case XLOG_FPI_FOR_HINT: case XLOG_FPI: case XLOG_OVERWRITE_CONTRECORD: + case XLOG_ENCRYPTION_LSN: break; default: elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info); diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 73d30bf619..256f468474 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -3991,11 +3991,12 @@ IncrBufferRefCount(Buffer buffer) * This is essentially the same as MarkBufferDirty, except: * * 1. The caller does not write WAL; so if checksums are enabled, we may need - * to write an XLOG_FPI_FOR_HINT WAL record to protect against torn pages. + * to write an XLOG_FPI_FOR_HINT record to protect against torn pages, or + * XLOG_ENCRYPTION_LSN to generate a new LSN for the page. * 2. The caller might have only share-lock instead of exclusive-lock on the - * buffer's content lock. + * buffer's content lock. * 3. This function does not guarantee that the buffer is always marked dirty - * (due to a race condition), so it cannot be used for important changes. + * (due to a race condition), so it cannot be used for important changes. */ void MarkBufferDirtyHint(Buffer buffer, bool buffer_std) @@ -4041,53 +4042,111 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * If we need to protect hint bit updates from torn writes, WAL-log a * full page image of the page. This full page image is only necessary * if the hint bit update is the first change to the page since the - * last checkpoint. + * last checkpoint. If cluster file encryption is enabled, we also + * need to generate new page LSNs for all other cases of page writes. * * We don't check full_page_writes here because that logic is included * when we call XLogInsert() since the value changes dynamically. */ - if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + if (XLogHintBitIsNeeded()) { /* - * If we must not write WAL, due to a relfilelocator-specific - * condition or being in recovery, don't dirty the page. We can - * set the hint, just not dirty the page as a result so the hint - * is lost when we evict the page or shutdown. + * If we must not write WAL during recovery so don't dirty the page. + * We can set the hint, just not dirty the page as a result so the + * hint is lost when we evict the page or shutdown. * * See src/backend/storage/page/README for longer discussion. */ if (RecoveryInProgress() || - RelFileLocatorSkippingWAL(BufTagGetRelFileLocator(&bufHdr->tag))) + (RelFileLocatorSkippingWAL(BufTagGetRelFileLocator(&bufHdr->tag)) && + !FileEncryptionEnabled)) return; /* - * If the block is already dirty because we either made a change - * or set a hint already, then we don't need to write a full page - * image. Note that aggressive cleaning of blocks dirtied by hint - * bit setting would increase the call rate. Bulk setting of hint - * bits would reduce the call rate... - * - * We must issue the WAL record before we mark the buffer dirty. - * Otherwise we might write the page before we write the WAL. That - * causes a race condition, since a checkpoint might occur between - * writing the WAL record and marking the buffer dirty. We solve - * that with a kluge, but one that is already in use during - * transaction commit to prevent race conditions. Basically, we - * simply prevent the checkpoint WAL record from being written - * until we have marked the buffer dirty. We don't start the - * checkpoint flush until we have marked dirty, so our checkpoint - * must flush the change to disk successfully or the checkpoint - * never gets written, so crash recovery will fix. - * - * It's possible we may enter here without an xid, so it is - * essential that CreateCheckPoint waits for virtual transactions - * rather than full transactionids. + * Non-BM_PERMANENT objects don't need full page images because + * they are not restored. WAL-skipped relfilenodes should never + * have full page images generated. */ - Assert((MyProc->delayChkptFlags & DELAY_CHKPT_START) == 0); - MyProc->delayChkptFlags |= DELAY_CHKPT_START; - delayChkptFlags = true; - lsn = XLogSaveBufferForHint(buffer, buffer_std); + if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT && + !RelFileLocatorSkippingWAL(BufTagGetRelFileLocator(&bufHdr->tag))) + { + /* + * If the block is already dirty because we either made a change + * or set a hint already, then we don't need to write a full + * page image. Note that aggressive cleaning of blocks dirtied + * by hint bit setting would increase the call rate. Bulk + * setting of hint bits would reduce the call rate... + * + * We must issue the WAL record before we mark the buffer + * dirty. Otherwise we might write the page before we write + * the WAL. That causes a race condition, since a checkpoint + * might occur between writing the WAL record and marking the + * buffer dirty. We solve that with a kluge, but one that is + * already in use during transaction commit to prevent race + * conditions. Basically, we simply prevent the checkpoint WAL + * record from being written until we have marked the buffer + * dirty. We don't start the checkpoint flush until we have + * marked dirty, so our checkpoint must flush the change to disk + * successfully or the checkpoint never gets written, so crash + * recovery will fix. + * + * It's possible we may enter here without an xid, so it + * is essential that CreateCheckpoint waits for virtual + * transactions rather than full transactionids. + */ + MyProc->delayChkptFlags |= DELAY_CHKPT_START; + delayChkptFlags = true; + lsn = XLogSaveBufferForHint(buffer, buffer_std); + } + + /* + * Above, for hint bit changes, we might have generated a new page + * LSN and a full-page WAL record for a page's first-clean-to-dirty + * during a checkpoint for permanent, non-WAL-skipped relfilenodes. + * If we didn't (the lsn variable is invalid), and we are + * doing cluster file encryption, we must generate a new + * page LSN here for either non-permanent relations or page + * non-first-clean-to-dirty during a checkpoint. (Cluster file + * encryption does not support WAL-skip relfilenodes.) We must + * update the page LSN even if the page with the hint bit change is + * later overwritten in the file system with an earlier version of + * the page during crash recovery. + * + * XXX Can we rely on the full page write above with no lock being + * held to avoid torn pages? Above, the LSN and page image are + * tied together, but here is just the page LSN update. + */ + if (XLogRecPtrIsInvalid(lsn) && FileEncryptionEnabled) + { + /* + * For cluster file encryption we need a new page LSN because + * the LSN is used, with the page number and permanent flag, as + * part of the nonce, and the nonce must be unique for every + * page write. If we reencrypt a page with hint bit changes + * using the same nonce as previous writes, it would expose the + * hint bit change locations. To avoid this, we write a simple + * WAL record to advance the lsn, which can then be assigned to + * the page below. + * + * Above we are relying on the full page writes to revert + * any partial pages writes caused by this LSN change for + * permanent, non-WAL-skip relfilenodes. For non-permanent + * relations, they crash recover as empty. For WAL-skip + * relfilenodes, they recover with their original contents, so + * that works too. + */ + /* XXX Do we need the checkpoint delay here? */ + MyProc->delayChkptFlags |= DELAY_CHKPT_START; + delayChkptFlags = true; + /* + * XXX We probably don't need to replay this WAL on the primary + * since the full page image is restored, but do we have + * to replay this on the repicas (for relations that are + * replicated)? + */ + lsn = LSNForEncryption( + pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT); + } } buf_state = LockBufHdr(bufHdr); diff --git a/src/bin/pg_alterckey/pg_alterckey.c b/src/bin/pg_alterckey/pg_alterckey.c index 7e3d4ad915..c699c59761 100644 --- a/src/bin/pg_alterckey/pg_alterckey.c +++ b/src/bin/pg_alterckey/pg_alterckey.c @@ -38,10 +38,9 @@ #include "common/file_perm.h" #include "common/file_utils.h" -#include "common/hex.h" #include "common/restricted_token.h" -#include "crypto/kmgr.h" #include "common/logging.h" +#include "crypto/kmgr.h" #include "getopt_long.h" #include "pg_getopt.h" @@ -84,6 +83,9 @@ static void bzero_keys_and_exit(exit_action action); static void reencrypt_data_keys(void); static void install_new_keys(void); +static uint64 hex_decode(const char *src, size_t len, char *dst); + + static void usage(const char *progname) { @@ -511,8 +513,8 @@ retrieve_cluster_keys(void) (char *) cluster_key_hex, ALLOC_KMGR_CLUSTER_KEY_LEN, live_path, terminal_fd); - if (pg_hex_decode(cluster_key_hex, cluster_key_len, - (char *) old_cluster_key, KMGR_CLUSTER_KEY_LEN) != + if (hex_decode(cluster_key_hex, cluster_key_len, + (char *) old_cluster_key) != KMGR_CLUSTER_KEY_LEN) { pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); @@ -534,8 +536,8 @@ retrieve_cluster_keys(void) (char *) cluster_key_hex, ALLOC_KMGR_CLUSTER_KEY_LEN, new_path, terminal_fd); - if (pg_hex_decode(cluster_key_hex, cluster_key_len, - (char *) new_cluster_key, KMGR_CLUSTER_KEY_LEN) != + if (hex_decode(cluster_key_hex, cluster_key_len, + (char *) new_cluster_key) != KMGR_CLUSTER_KEY_LEN) { pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN); @@ -722,3 +724,65 @@ bzero_keys_and_exit(exit_action action) /* return 0 or 1 */ exit(action != SUCCESS_EXIT); } + +/* + * HEX + */ + +static const int8 hexlookup[128] = { + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, +}; + +static inline char +get_hex(const char *cp) +{ + unsigned char c = (unsigned char) *cp; + int res = -1; + + if (c < 127) + res = hexlookup[c]; + + if (res < 0) + pg_fatal("invalid hexadecimal digit: \"%s\"",cp); + + return (char) res; +} + +static uint64 +hex_decode(const char *src, size_t len, char *dst) +{ + const char *s, + *srcend; + char v1, + v2, + *p; + + srcend = src + len; + s = src; + p = dst; + while (s < srcend) + { + if (*s == ' ' || *s == '\n' || *s == '\t' || *s == '\r') + { + s++; + continue; + } + v1 = get_hex(s) << 4; + s++; + if (s >= srcend) + pg_fatal("invalid hexadecimal data: odd number of digits"); + + v2 = get_hex(s); + s++; + *p++ = v1 | v2; + } + + return p - dst; +} diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 8f219fbacc..78def5753a 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -107,13 +107,15 @@ extern PGDLLIMPORT int wal_level; /* * Is a full-page image needed for hint bit updates? * - * Normally, we don't WAL-log hint bit updates, but if checksums are enabled, - * we have to protect them against torn page writes. When you only set - * individual bits on a page, it's still consistent no matter what combination - * of the bits make it to disk, but the checksum wouldn't match. Also WAL-log - * them if forced by wal_log_hints=on. + * Normally, we don't WAL-log hint bit updates, but if checksums or encryption + * is enabled, we have to protect them against torn page writes. When you + * only set individual bits on a page, it's still consistent no matter what + * combination of the bits make it to disk, but the checksum wouldn't match. + * Cluster file encryption requires a new LSN for hint bit changes, and can't + * tolerate torn pages. Also WAL-log them if forced by wal_log_hints=on. */ -#define XLogHintBitIsNeeded() (DataChecksumsEnabled() || wal_log_hints) +#define XLogHintBitIsNeeded() \ + (DataChecksumsEnabled() || FileEncryptionEnabled || wal_log_hints) /* Do we need to WAL-log information required only for Hot Standby and logical replication? */ #define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA) diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h index 001ff2f521..dfb501a7c3 100644 --- a/src/include/access/xloginsert.h +++ b/src/include/access/xloginsert.h @@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forknum, BlockNumber startblk, BlockNumber endblk, bool page_std); extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std); +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + extern void InitXLogInsert(void); #endif /* XLOGINSERT_H */ diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index db7de77287..1c97ffd2cd 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -78,6 +78,7 @@ typedef struct CheckPoint #define XLOG_FPI 0xB0 /* 0xC0 is used in Postgres 9.5-11 */ #define XLOG_OVERWRITE_CONTRECORD 0xD0 +#define XLOG_ENCRYPTION_LSN 0xE0 /* -- 2.31.1 [application/octet-stream] v2-0011-cfe-11-gist_over_cfe-10-hint-squash-commit.patch (7.1K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/12-v2-0011-cfe-11-gist_over_cfe-10-hint-squash-commit.patch) download | inline diff: From 3e8db4616f5d567c98a33554bd6151676f7ea95e Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:05 -0400 Subject: [PATCH v2 11/12] cfe-11-gist_over_cfe-10-hint squash commit --- src/backend/access/gist/README | 10 ++++++++++ src/backend/access/gist/gist.c | 21 +++++++++++++++------ src/backend/access/gist/gistbuild.c | 12 +++++++++--- src/backend/access/gist/gistvacuum.c | 20 ++++++++++++++------ 4 files changed, 48 insertions(+), 15 deletions(-) diff --git a/src/backend/access/gist/README b/src/backend/access/gist/README index efb2730e18..c4d908fcb2 100644 --- a/src/backend/access/gist/README +++ b/src/backend/access/gist/README @@ -477,6 +477,16 @@ value. The page is not recycled, until that XID is no longer visible to anyone. That's much more conservative than necessary, but let's keep it simple. +GiST and Encryption +------------------- + +GiST uses LSNs and NSNs for concurrency. This means that unlogged and +temporary relations also need LSNs. GiST has a mechanism to assign LSNs +to such relations, but sometimes uses fixed LSNs and or calls +gistGetFakeLSN(), which can cause duplicate LSNs to be used. Therefore, +when encryption is enabled and fake LSNs are needed, the GiST code calls +LSNForEncryption(). See src/backend/crypto/README for more details. + Authors: Teodor Sigaev <[email protected]> diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 30069f139c..74e7e485e6 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -501,13 +501,16 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, * we don't need to be able to detect concurrent splits yet.) */ if (is_build) - recptr = GistBuildLSN; + recptr = !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(rel)); else { if (RelationNeedsWAL(rel)) recptr = gistXLogSplit(is_leaf, dist, oldrlink, oldnsn, leftchildbuf, markfollowright); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else recptr = gistGetFakeLSN(rel); } @@ -568,7 +571,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, MarkBufferDirty(leftchildbuf); if (is_build) - recptr = GistBuildLSN; + recptr = !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(rel)); else { if (RelationNeedsWAL(rel)) @@ -586,6 +590,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, deloffs, ndeloffs, itup, ntup, leftchildbuf); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else recptr = gistGetFakeLSN(rel); } @@ -1665,6 +1671,8 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel) if (ndeletable > 0) { + XLogRecPtr recptr; + TransactionId latestRemovedXid = InvalidTransactionId; if (XLogStandbyInfoActive() && RelationNeedsWAL(rel)) @@ -1690,16 +1698,17 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel) /* XLOG stuff */ if (RelationNeedsWAL(rel)) { - XLogRecPtr recptr; - recptr = gistXLogDelete(buffer, deletable, ndeletable, latestRemovedXid); - PageSetLSN(page, recptr); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - PageSetLSN(page, gistGetFakeLSN(rel)); + recptr = gistGetFakeLSN(rel); + + PageSetLSN(page, recptr); END_CRIT_SECTION(); } diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index fb0f466708..bf5162b410 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -48,6 +48,8 @@ #include "utils/rel.h" #include "utils/tuplesort.h" +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + /* Step of index tuples for check whether to switch to buffering build mode */ #define BUFFERING_MODE_SWITCH_CHECK_STEP 256 @@ -307,7 +309,8 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo) GISTInitBuffer(buffer, F_LEAF); MarkBufferDirty(buffer); - PageSetLSN(page, GistBuildLSN); + PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(index))); UnlockReleaseBuffer(buffer); @@ -458,7 +461,9 @@ gist_indexsortbuild(GISTBuildState *state) gist_indexsortbuild_flush_ready_pages(state); /* Write out the root */ - PageSetLSN(levelstate->pages[0], GistBuildLSN); + RelationGetSmgr(state->indexrel); + PageSetLSN(levelstate->pages[0], !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(state->indexrel))); PageSetChecksumInplace(levelstate->pages[0], GIST_ROOT_BLKNO); smgrwrite(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, GIST_ROOT_BLKNO, levelstate->pages[0], true); @@ -655,7 +660,8 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state) if (blkno != state->pages_written) elog(ERROR, "unexpected block number to flush GiST sorting build"); - PageSetLSN(page, GistBuildLSN); + PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : + LSNForEncryption(RelationIsPermanent(state->indexrel))); PageSetChecksumInplace(page, blkno); smgrextend(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, blkno, page, true); diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c index 0aa6e58a62..35265ae0e7 100644 --- a/src/backend/access/gist/gistvacuum.c +++ b/src/backend/access/gist/gistvacuum.c @@ -24,6 +24,8 @@ #include "storage/lmgr.h" #include "utils/memutils.h" +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + /* Working state needed by gistbulkdelete */ typedef struct { @@ -180,6 +182,8 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, vstate.callback_state = callback_state; if (RelationNeedsWAL(rel)) vstate.startNSN = GetInsertRecPtr(); + else if (FileEncryptionEnabled) + vstate.startNSN = LSNForEncryption(RelationIsPermanent(rel)); else vstate.startNSN = gistGetFakeLSN(rel); @@ -359,6 +363,8 @@ restart: */ if (ntodelete > 0) { + XLogRecPtr recptr; + START_CRIT_SECTION(); MarkBufferDirty(buffer); @@ -367,16 +373,15 @@ restart: GistMarkTuplesDeleted(page); if (RelationNeedsWAL(rel)) - { - XLogRecPtr recptr; - recptr = gistXLogUpdate(buffer, todelete, ntodelete, NULL, 0, InvalidBuffer); - PageSetLSN(page, recptr); - } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - PageSetLSN(page, gistGetFakeLSN(rel)); + recptr = gistGetFakeLSN(rel); + + PageSetLSN(page, recptr); END_CRIT_SECTION(); @@ -663,8 +668,11 @@ gistdeletepage(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, if (RelationNeedsWAL(info->index)) recptr = gistXLogPageDelete(leafBuffer, txid, parentBuffer, downlink); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(info->index)); else recptr = gistGetFakeLSN(info->index); + PageSetLSN(parentPage, recptr); PageSetLSN(leafPage, recptr); -- 2.31.1 [application/octet-stream] v2-0012-cfe-12-rel_over_cfe-11-gist-squash-commit.patch (33.0K, ../../CAOxo6XLmZGvzgeEttHSJfHKQrY-rWk0xyK-JFcVe9qT4DbBf_Q@mail.gmail.com/13-v2-0012-cfe-12-rel_over_cfe-11-gist-squash-commit.patch) download | inline diff: From 48a043538028826cc6e29fa41b0b265bb29a5987 Mon Sep 17 00:00:00 2001 From: Bruce Momjian <[email protected]> Date: Fri, 25 Jun 2021 16:57:08 -0400 Subject: [PATCH v2 12/12] cfe-12-rel_over_cfe-11-gist squash commit --- contrib/basic_archive/basic_archive.c | 2 +- contrib/bloom/blinsert.c | 3 + src/backend/access/gist/gist.c | 51 ++++---- src/backend/access/gist/gistbuild.c | 12 ++ src/backend/access/hash/hashpage.c | 1 + src/backend/access/heap/rewriteheap.c | 7 + src/backend/access/nbtree/nbtree.c | 3 + src/backend/access/nbtree/nbtsort.c | 4 +- src/backend/access/spgist/spginsert.c | 6 + src/backend/access/transam/xlog.c | 2 + src/backend/bootstrap/bootstrap.c | 1 + src/backend/catalog/storage.c | 7 +- src/backend/crypto/Makefile | 1 + src/backend/crypto/bufenc.c | 176 ++++++++++++++++++++++++++ src/backend/postmaster/postmaster.c | 2 + src/backend/storage/buffer/bufmgr.c | 22 +++- src/backend/storage/buffer/localbuf.c | 8 +- src/backend/storage/file/copydir.c | 28 +++- src/backend/storage/file/reinit.c | 2 +- src/backend/storage/page/bufpage.c | 51 +++++++- src/backend/tcop/postgres.c | 2 + src/include/access/gist.h | 5 +- src/include/crypto/bufenc.h | 28 ++++ src/include/storage/bufpage.h | 18 ++- src/include/storage/copydir.h | 2 +- 25 files changed, 396 insertions(+), 48 deletions(-) create mode 100644 src/backend/crypto/bufenc.c create mode 100644 src/include/crypto/bufenc.h diff --git a/contrib/basic_archive/basic_archive.c b/contrib/basic_archive/basic_archive.c index 9f221816bb..7a59e34a42 100644 --- a/contrib/basic_archive/basic_archive.c +++ b/contrib/basic_archive/basic_archive.c @@ -275,7 +275,7 @@ basic_archive_file_internal(const char *file, const char *path) * Copy the file to its temporary destination. Note that this will fail * if temp already exists. */ - copy_file(unconstify(char *, path), temp); + copy_file(unconstify(char *, path), temp, false); /* * Sync the temporary file to disk and move it to its final destination. diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c index dd26d6ac29..3204950817 100644 --- a/contrib/bloom/blinsert.c +++ b/contrib/bloom/blinsert.c @@ -176,7 +176,10 @@ blbuildempty(Relation index) * XLOG_DBASE_CREATE* or XLOG_TBLSPC_CREATE record. Therefore, we need * this even when wal_level=minimal. */ + PageEncryptInplace(metapage, INIT_FORKNUM, RelationIsPermanent(index), + BLOOM_METAPAGE_BLKNO); PageSetChecksumInplace(metapage, BLOOM_METAPAGE_BLKNO); + smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, BLOOM_METAPAGE_BLKNO, (char *) metapage, true); log_newpage(&(RelationGetSmgr(index))->smgr_rlocator.locator, INIT_FORKNUM, diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 74e7e485e6..3472afe363 100644 --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -503,17 +503,14 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, if (is_build) recptr = !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(rel)); + else if (RelationNeedsWAL(rel)) + recptr = gistXLogSplit(is_leaf, + dist, oldrlink, oldnsn, leftchildbuf, + markfollowright); + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); else - { - if (RelationNeedsWAL(rel)) - recptr = gistXLogSplit(is_leaf, - dist, oldrlink, oldnsn, leftchildbuf, - markfollowright); - else if (FileEncryptionEnabled) - recptr = LSNForEncryption(RelationIsPermanent(rel)); - else - recptr = gistGetFakeLSN(rel); - } + recptr = gistGetFakeLSN(rel); for (ptr = dist; ptr; ptr = ptr->next) PageSetLSN(ptr->page, recptr); @@ -573,28 +570,26 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate, if (is_build) recptr = !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(rel)); - else + else if (RelationNeedsWAL(rel)) { - if (RelationNeedsWAL(rel)) - { - OffsetNumber ndeloffs = 0, - deloffs[1]; + OffsetNumber ndeloffs = 0, + deloffs[1]; - if (OffsetNumberIsValid(oldoffnum)) - { - deloffs[0] = oldoffnum; - ndeloffs = 1; - } - - recptr = gistXLogUpdate(buffer, - deloffs, ndeloffs, itup, ntup, - leftchildbuf); + if (OffsetNumberIsValid(oldoffnum)) + { + deloffs[0] = oldoffnum; + ndeloffs = 1; } - else if (FileEncryptionEnabled) - recptr = LSNForEncryption(RelationIsPermanent(rel)); - else - recptr = gistGetFakeLSN(rel); + + recptr = gistXLogUpdate(buffer, + deloffs, ndeloffs, itup, ntup, + leftchildbuf); } + else if (FileEncryptionEnabled) + recptr = LSNForEncryption(RelationIsPermanent(rel)); + else + recptr = gistGetFakeLSN(rel); + PageSetLSN(page, recptr); if (newblkno) diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c index bf5162b410..b54d50c440 100644 --- a/src/backend/access/gist/gistbuild.c +++ b/src/backend/access/gist/gistbuild.c @@ -464,6 +464,12 @@ gist_indexsortbuild(GISTBuildState *state) RelationGetSmgr(state->indexrel); PageSetLSN(levelstate->pages[0], !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(state->indexrel))); + /* Make sure LSNs are vaild, and if encryption, are not constant. */ + Assert(!XLogRecPtrIsInvalid(PageGetLSN(levelstate->pages[0])) && + (!FileEncryptionEnabled || + PageGetLSN(levelstate->pages[0]) != GistBuildLSN)); + PageEncryptInplace(levelstate->pages[0], MAIN_FORKNUM, RelationIsPermanent(state->indexrel), + GIST_ROOT_BLKNO); PageSetChecksumInplace(levelstate->pages[0], GIST_ROOT_BLKNO); smgrwrite(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, GIST_ROOT_BLKNO, levelstate->pages[0], true); @@ -662,6 +668,12 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state) PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN : LSNForEncryption(RelationIsPermanent(state->indexrel))); + /* Make sure LSNs are vaild, and if encryption, are not constant. */ + Assert(!XLogRecPtrIsInvalid(PageGetLSN(page)) && + (!FileEncryptionEnabled || + PageGetLSN(page) != GistBuildLSN)); + PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(state->indexrel), + blkno); PageSetChecksumInplace(page, blkno); smgrextend(RelationGetSmgr(state->indexrel), MAIN_FORKNUM, blkno, page, true); diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c index d2edcd4617..4ba41ce288 100644 --- a/src/backend/access/hash/hashpage.c +++ b/src/backend/access/hash/hashpage.c @@ -1025,6 +1025,7 @@ _hash_alloc_buckets(Relation rel, BlockNumber firstblock, uint32 nblocks) zerobuf.data, true); + PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(rel), lastblock); PageSetChecksumInplace(page, lastblock); smgrextend(RelationGetSmgr(rel), MAIN_FORKNUM, lastblock, zerobuf.data, false); diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c index a34e9b352d..c163c7ba4b 100644 --- a/src/backend/access/heap/rewriteheap.c +++ b/src/backend/access/heap/rewriteheap.c @@ -324,6 +324,9 @@ end_heap_rewrite(RewriteState state) state->rs_buffer, true); + PageEncryptInplace(state->rs_buffer, MAIN_FORKNUM, + RelationIsPermanent(state->rs_new_rel), + state->rs_blockno); PageSetChecksumInplace(state->rs_buffer, state->rs_blockno); smgrextend(RelationGetSmgr(state->rs_new_rel), MAIN_FORKNUM, @@ -690,8 +693,12 @@ raw_heap_insert(RewriteState state, HeapTuple tup) * need for smgr to schedule an fsync for this write; we'll do it * ourselves in end_heap_rewrite. */ + PageEncryptInplace(page, MAIN_FORKNUM, + RelationIsPermanent(state->rs_new_rel), + state->rs_blockno); PageSetChecksumInplace(page, state->rs_blockno); + /* XXX - is this still needed? */ smgrextend(RelationGetSmgr(state->rs_new_rel), MAIN_FORKNUM, state->rs_blockno, (char *) page, true); diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c index b52eca8f38..e60582e7e3 100644 --- a/src/backend/access/nbtree/nbtree.c +++ b/src/backend/access/nbtree/nbtree.c @@ -163,7 +163,10 @@ btbuildempty(Relation index) * XLOG_DBASE_CREATE* or XLOG_TBLSPC_CREATE record. Therefore, we need * this even when wal_level=minimal. */ + PageEncryptInplace(metapage, INIT_FORKNUM, RelationIsPermanent(index), + BTREE_METAPAGE); PageSetChecksumInplace(metapage, BTREE_METAPAGE); + smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, BTREE_METAPAGE, (char *) metapage, true); log_newpage(&RelationGetSmgr(index)->smgr_rlocator.locator, INIT_FORKNUM, diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c index 501e011ce1..70a94115cb 100644 --- a/src/backend/access/nbtree/nbtsort.c +++ b/src/backend/access/nbtree/nbtsort.c @@ -661,13 +661,15 @@ _bt_blwritepage(BTWriteState *wstate, Page page, BlockNumber blkno) { if (!wstate->btws_zeropage) wstate->btws_zeropage = (Page) palloc0(BLCKSZ); - /* don't set checksum for all-zero page */ + /* don't set checksum or encryption for all-zero page */ smgrextend(RelationGetSmgr(wstate->index), MAIN_FORKNUM, wstate->btws_pages_written++, (char *) wstate->btws_zeropage, true); } + PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(wstate->index), + blkno); PageSetChecksumInplace(page, blkno); /* diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c index c6821b5952..b36bc41854 100644 --- a/src/backend/access/spgist/spginsert.c +++ b/src/backend/access/spgist/spginsert.c @@ -168,6 +168,8 @@ spgbuildempty(Relation index) * of their existing content when the corresponding create records are * replayed. */ + PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index), + SPGIST_METAPAGE_BLKNO); PageSetChecksumInplace(page, SPGIST_METAPAGE_BLKNO); smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, SPGIST_METAPAGE_BLKNO, (char *) page, true); @@ -177,6 +179,8 @@ spgbuildempty(Relation index) /* Likewise for the root page. */ SpGistInitPage(page, SPGIST_LEAF); + PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index), + SPGIST_ROOT_BLKNO); PageSetChecksumInplace(page, SPGIST_ROOT_BLKNO); smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, SPGIST_ROOT_BLKNO, (char *) page, true); @@ -186,6 +190,8 @@ spgbuildempty(Relation index) /* Likewise for the null-tuples root page. */ SpGistInitPage(page, SPGIST_LEAF | SPGIST_NULLS); + PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index), + SPGIST_NULL_BLKNO); PageSetChecksumInplace(page, SPGIST_NULL_BLKNO); smgrwrite(RelationGetSmgr(index), INIT_FORKNUM, SPGIST_NULL_BLKNO, (char *) page, true); diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 38e46bbbf7..fd217e97f4 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -71,6 +71,7 @@ #include "common/file_utils.h" #include "crypto/kmgr.h" #include "executor/instrument.h" +#include "crypto/bufenc.h" #include "miscadmin.h" #include "pg_trace.h" #include "pgstat.h" @@ -4832,6 +4833,7 @@ BootStrapXLOG(void) WriteControlFile(); BootStrapKmgr(); + InitializeBufferEncryption(); if (terminal_fd != -1) { diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 87918a4d28..ffaf9c7cc4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -29,6 +29,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/link-canary.h" +#include "crypto/bufenc.h" #include "libpq/pqsignal.h" #include "miscadmin.h" #include "nodes/makefuncs.h" diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c index d708af19ed..925195d5a3 100644 --- a/src/backend/catalog/storage.c +++ b/src/backend/catalog/storage.c @@ -486,8 +486,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst, smgrread(src, forkNum, blkno, buf.data); - if (!PageIsVerifiedExtended(page, blkno, - PIV_LOG_WARNING | PIV_REPORT_STAT)) + if (!PageIsVerifiedExtended(page, forkNum, + relpersistence == RELPERSISTENCE_PERMANENT, + blkno, PIV_LOG_WARNING | PIV_REPORT_STAT)) { /* * For paranoia's sake, capture the file path before invoking the @@ -514,6 +515,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst, if (use_wal) log_newpage(&dst->smgr_rlocator.locator, forkNum, blkno, page, false); + PageEncryptInplace(page, forkNum, + relpersistence == RELPERSISTENCE_PERMANENT, blkno); PageSetChecksumInplace(page, blkno); /* diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile index c27362029d..8985a66875 100644 --- a/src/backend/crypto/Makefile +++ b/src/backend/crypto/Makefile @@ -13,6 +13,7 @@ top_builddir = ../../.. include $(top_builddir)/src/Makefile.global OBJS = \ + bufenc.o \ kmgr.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/crypto/bufenc.c b/src/backend/crypto/bufenc.c new file mode 100644 index 0000000000..0213fb23f5 --- /dev/null +++ b/src/backend/crypto/bufenc.c @@ -0,0 +1,176 @@ +/*------------------------------------------------------------------------- + * + * bufenc.c + * + * Copyright (c) 2020, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/crypto/bufenc.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "miscadmin.h" +#include "lib/stringinfo.h" + +#include "access/gist.h" +#include "access/xlog.h" +#include "crypto/bufenc.h" +#include "storage/bufpage.h" +#include "storage/fd.h" + +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + +/* + * We use the page LSN, page number, and permanent-bit to indicate if a fake + * LSN was used to create a nonce for each page. + */ +#define BUFENC_IV_SIZE 16 + +static unsigned char buf_encryption_iv[BUFENC_IV_SIZE]; + +PgCipherCtx *BufEncCtx = NULL; +PgCipherCtx *BufDecCtx = NULL; + +static void set_buffer_encryption_iv(Page page, BlockNumber blkno, + bool relation_is_permanent); + +void +InitializeBufferEncryption(void) +{ + const CryptoKey *key; + + if (!FileEncryptionEnabled) + return; + + key = KmgrGetKey(KMGR_KEY_ID_REL); + + BufEncCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR, + (unsigned char *) key->key, + (key->klen), true); + if (!BufEncCtx) + elog(ERROR, "cannot intialize encryption context"); + + BufDecCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR, + (unsigned char *) key->key, + (key->klen), false); + if (!BufDecCtx) + elog(ERROR, "cannot intialize decryption context"); +} + +/* Encrypt the given page with the relation key */ +void +EncryptPage(Page page, bool relation_is_permanent, BlockNumber blkno) +{ + unsigned char *ptr = (unsigned char *) page + PageEncryptOffset; + bool is_gist_page_or_similar; + + int enclen; + + Assert(BufEncCtx != NULL); + + /* + * Permanent pages have valid LSNs, and non-permanent pages usually have + * invalid (not set) LSNs. (One exception are GiST fake LSNs, see below.) + * However, we need valid ones on all pages for encryption. There are too + * many places that set the page LSN for permanent pages to do the same + * for non-permanent pages, so we just set it here. + * + * Also, while permanent relations get new LSNs every time the page is + * modified, for non-permanent relations do not, so we just update the LSN + * here before it is encrypted. + * + * GiST indexes uses LSNs, which are also stored in NSN fields, to detect + * page splits. Therefore, we allow the GiST code to assign LSNs and we + * don't change them here. + */ + + /* Permanent relations should already have valid LSNs. */ + Assert(!XLogRecPtrIsInvalid(PageGetLSN(page)) || !relation_is_permanent); + + /* + * Check if the page has a special size == GISTPageOpaqueData, a valid + * GIST_PAGE_ID, no invalid GiST flag bits are set, and a valid LSN. This + * is true for all GiST pages, and perhaps a few pages that are not. The + * only downside of guessing wrong is that we might not update the LSN for + * some non-permanent relation page changes, and therefore reuse the IV, + * which seems acceptable. + */ + is_gist_page_or_similar = + (PageGetSpecialSize(page) == MAXALIGN(sizeof(GISTPageOpaqueData)) && + GistPageGetOpaque(page)->gist_page_id == GIST_PAGE_ID && + (GistPageGetOpaque(page)->flags & ~GIST_FLAG_BITMASK) == 0 && + !XLogRecPtrIsInvalid(PageGetLSN(page))); + + if (!relation_is_permanent && !is_gist_page_or_similar) + PageSetLSN(page, LSNForEncryption(relation_is_permanent)); + + set_buffer_encryption_iv(page, blkno, relation_is_permanent); + if (unlikely(!pg_cipher_encrypt(BufEncCtx, PG_CIPHER_AES_CTR, + (const unsigned char *) ptr, /* input */ + SizeOfPageEncryption, + ptr, /* length */ + &enclen, /* resulting length */ + buf_encryption_iv, /* iv */ + BUFENC_IV_SIZE, + NULL, 0))) + elog(ERROR, "cannot encrypt page %u", blkno); + + Assert(enclen == SizeOfPageEncryption); +} + +/* Decrypt the given page with the relation key */ +void +DecryptPage(Page page, bool relation_is_permanent, BlockNumber blkno) +{ + unsigned char *ptr = (unsigned char *) page + PageEncryptOffset; + int enclen; + + Assert(BufDecCtx != NULL); + + set_buffer_encryption_iv(page, blkno, relation_is_permanent); + if (unlikely(!pg_cipher_decrypt(BufDecCtx, PG_CIPHER_AES_CTR, + (const unsigned char *) ptr, /* input */ + SizeOfPageEncryption, + ptr, /* output */ + &enclen, /* resulting length */ + buf_encryption_iv, /* iv */ + BUFENC_IV_SIZE, + NULL, 0))) + elog(ERROR, "cannot decrypt page %u", blkno); + + Assert(enclen == SizeOfPageEncryption); +} + +/* Construct iv for the given page */ +static void +set_buffer_encryption_iv(Page page, BlockNumber blkno, + bool relation_is_permanent) +{ + unsigned char *p = buf_encryption_iv; + + MemSet(buf_encryption_iv, 0, BUFENC_IV_SIZE); + + /* page lsn (8 byte) */ + memcpy(p, &((PageHeader) page)->pd_lsn, sizeof(PageXLogRecPtr)); + p += sizeof(PageXLogRecPtr); + + /* block number (4 byte) */ + memcpy(p, &blkno, sizeof(BlockNumber)); + p += sizeof(BlockNumber); + + /* + * Mark use of fake LSNs in IV so if the real and fake LSN counters + * overlap, the IV will remain unique. XXX Is there a better value? + */ + if (!relation_is_permanent) + *p++ = 0x80; + + /* + * The maximum required counter for AES-CTR is 2048, which fits in the + * last three bytes. + */ +} diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 1673717da9..c463b27d90 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -98,6 +98,7 @@ #include "common/ip.h" #include "common/pg_prng.h" #include "common/string.h" +#include "crypto/bufenc.h" #include "crypto/kmgr.h" #include "lib/ilist.h" #include "libpq/auth.h" @@ -1354,6 +1355,7 @@ PostmasterMain(int argc, char *argv[]) } InitializeKmgr(); + InitializeBufferEncryption(); if (terminal_fd != -1) close(terminal_fd); diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 256f468474..17ae644022 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -39,6 +39,7 @@ #include "catalog/catalog.h" #include "catalog/storage.h" #include "catalog/storage_xlog.h" +#include "crypto/bufenc.h" #include "executor/instrument.h" #include "lib/binaryheap.h" #include "miscadmin.h" @@ -1029,7 +1030,9 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, } /* check for garbage data */ - if (!PageIsVerifiedExtended((Page) bufBlock, blockNum, + if (!PageIsVerifiedExtended((Page) bufBlock, forkNum, + relpersistence == RELPERSISTENCE_PERMANENT, + blockNum, PIV_LOG_WARNING | PIV_REPORT_STAT)) { if (mode == RBM_ZERO_ON_ERROR || zero_damaged_pages) @@ -2893,12 +2896,24 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln) */ bufBlock = BufHdrGetBlock(buf); + if (FileEncryptionEnabled) + { + /* + * Technically BM_PERMANENT could indicate an init fork, but that's + * okay since forkNum would also tell us not to encrypt init forks. + */ + bufToWrite = PageEncryptCopy((Page) bufBlock, buf->tag.forkNum, + buf_state & BM_PERMANENT, buf->tag.blockNum); + bufToWrite = PageSetChecksumCopy((Page) bufToWrite, buf->tag.blockNum); + } + else + bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum); + /* * Update page checksum if desired. Since we have only shared lock on the * buffer, other processes might be updating hint bits in it, so we must * copy the page to private storage if we do checksumming. */ - bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum); if (track_io_timing) INSTR_TIME_SET_CURRENT(io_start); @@ -3543,6 +3558,9 @@ FlushRelationBuffers(Relation rel) errcallback.previous = error_context_stack; error_context_stack = &errcallback; + /* XXX should we be writing a copy of the page here? */ + PageEncryptInplace(localpage, bufHdr->tag.forkNum, + RelationIsPermanent(rel), bufHdr->tag.blockNum); PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); smgrwrite(RelationGetSmgr(rel), diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 30d67d1c40..09b3ec71d9 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -159,7 +159,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } return bufHdr; } - + #ifdef LBDEBUG fprintf(stderr, "LB ALLOC (%u,%d,%d) %d\n", smgr->smgr_rlocator.locator.relNumber, forkNum, blockNum, @@ -217,6 +217,12 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, /* Find smgr relation for buffer */ oreln = smgropen(BufTagGetRelFileLocator(&bufHdr->tag), MyBackendId); + /* + * Technically BM_PERMANENT could indicate an init fork, but that's + * okay since forkNum would also tell us not to encrypt init forks. + */ + PageEncryptInplace(localpage, bufHdr->tag.forkNum, + buf_state & BM_PERMANENT, bufHdr->tag.blockNum); PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); /* And write... */ diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c index 8a866191e1..d2f4d13ec0 100644 --- a/src/backend/storage/file/copydir.c +++ b/src/backend/storage/file/copydir.c @@ -22,12 +22,15 @@ #include <unistd.h> #include <sys/stat.h> +#include "crypto/bufenc.h" #include "common/file_utils.h" #include "miscadmin.h" #include "pgstat.h" #include "storage/copydir.h" #include "storage/fd.h" +extern XLogRecPtr LSNForEncryption(bool use_wal_lsn); + /* * copydir: copy a directory * @@ -72,7 +75,7 @@ copydir(char *fromdir, char *todir, bool recurse) copydir(fromfile, tofile, true); } else if (xlde_type == PGFILETYPE_REG) - copy_file(fromfile, tofile); + copy_file(fromfile, tofile, false); } FreeDir(xldir); @@ -115,7 +118,7 @@ copydir(char *fromdir, char *todir, bool recurse) * copy one file */ void -copy_file(char *fromfile, char *tofile) +copy_file(char *fromfile, char *tofile, bool encrypt_init_file) { char *buffer; int srcfd; @@ -123,9 +126,8 @@ copy_file(char *fromfile, char *tofile) int nbytes; off_t offset; off_t flush_offset; - /* Size of copy buffer (read and write requests) */ -#define COPY_BUF_SIZE (8 * BLCKSZ) + int copy_buf_size = (encrypt_init_file) ? BLCKSZ : 8 * BLCKSZ; /* * Size of data flush requests. It seems beneficial on most platforms to @@ -140,7 +142,7 @@ copy_file(char *fromfile, char *tofile) #endif /* Use palloc to ensure we get a maxaligned buffer */ - buffer = palloc(COPY_BUF_SIZE); + buffer = palloc(copy_buf_size); /* * Open the files @@ -178,7 +180,7 @@ copy_file(char *fromfile, char *tofile) } pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ); - nbytes = read(srcfd, buffer, COPY_BUF_SIZE); + nbytes = read(srcfd, buffer, copy_buf_size); pgstat_report_wait_end(); if (nbytes < 0) ereport(ERROR, @@ -186,6 +188,20 @@ copy_file(char *fromfile, char *tofile) errmsg("could not read file \"%s\": %m", fromfile))); if (nbytes == 0) break; + /* + * When we copy an init fork page to be part of an empty unlogged + * relation, its real LSN must be replaced with a fake one, and the + * page encrypted. + */ + if (encrypt_init_file) + { + Page page = (Page) buffer; + + Assert(nbytes == BLCKSZ); + PageSetLSN(page, LSNForEncryption(false)); + PageEncryptInplace(page, MAIN_FORKNUM, false, offset / BLCKSZ); + } + errno = 0; pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_WRITE); if ((int) write(dstfd, buffer, nbytes) != nbytes) diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c index 647c458b52..2e57943768 100644 --- a/src/backend/storage/file/reinit.c +++ b/src/backend/storage/file/reinit.c @@ -312,7 +312,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op) /* OK, we're ready to perform the actual copy. */ elog(DEBUG2, "copying %s to %s", srcpath, dstpath); - copy_file(srcpath, dstpath); + copy_file(srcpath, dstpath, true); } FreeDir(dbspace_dir); diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c index 8b617c7e79..1255566f57 100644 --- a/src/backend/storage/page/bufpage.c +++ b/src/backend/storage/page/bufpage.c @@ -17,6 +17,7 @@ #include "access/htup_details.h" #include "access/itup.h" #include "access/xlog.h" +#include "crypto/bufenc.h" #include "pgstat.h" #include "storage/checksum.h" #include "utils/memdebug.h" @@ -85,7 +86,8 @@ PageInit(Page page, Size pageSize, Size specialSize) * to pgstat. */ bool -PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags) +PageIsVerifiedExtended(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno, int flags) { PageHeader p = (PageHeader) page; size_t *pagebytes; @@ -108,6 +110,8 @@ PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags) checksum_failure = true; } + PageDecryptInplace(page, forknum, relation_is_permanent, blkno); + /* * The following checks don't prove the header is correct, only that * it looks sane enough to allow into the buffer pool. Later usage of @@ -1544,3 +1548,48 @@ PageSetChecksumInplace(Page page, BlockNumber blkno) ((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blkno); } + +char * +PageEncryptCopy(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno) +{ + static char *pageCopy = NULL; + + /* If we don't need a checksum, just return the passed-in data */ + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum)) + return (char *) page; + + /* + * We allocate the copy space once and use it over on each subsequent + * call. The point of palloc'ing here, rather than having a static char + * array, is first to ensure adequate alignment for the checksumming code + * and second to avoid wasting space in processes that never call this. + */ + if (pageCopy == NULL) + pageCopy = MemoryContextAlloc(TopMemoryContext, BLCKSZ); + + memcpy(pageCopy, (char *) page, BLCKSZ); + EncryptPage(pageCopy, relation_is_permanent, blkno); + return pageCopy; +} + +void +PageEncryptInplace(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno) +{ + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum)) + return; + + EncryptPage(page, relation_is_permanent, blkno); +} + + +void +PageDecryptInplace(Page page, ForkNumber forknum, bool relation_is_permanent, + BlockNumber blkno) +{ + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum)) + return; + + DecryptPage(page, relation_is_permanent, blkno); +} diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 5ee6c43394..6f52dfe7c5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -35,6 +35,7 @@ #include "commands/async.h" #include "commands/prepare.h" #include "common/pg_prng.h" +#include "crypto/bufenc.h" #include "crypto/kmgr.h" #include "jit/jit.h" #include "libpq/libpq.h" @@ -4076,6 +4077,7 @@ PostgresMain(const char *dbname, const char *username) if (!IsUnderPostmaster) { InitializeKmgr(); + InitializeBufferEncryption(); if (terminal_fd != -1) close(terminal_fd); diff --git a/src/include/access/gist.h b/src/include/access/gist.h index a3337627b8..3e116a66d8 100644 --- a/src/include/access/gist.h +++ b/src/include/access/gist.h @@ -41,7 +41,7 @@ #define GISTNProcs 11 /* - * Page opaque data in a GiST index page. + * Page opaque data flags in a GiST index page. */ #define F_LEAF (1 << 0) /* leaf page */ #define F_DELETED (1 << 1) /* the page has been deleted */ @@ -51,6 +51,9 @@ #define F_HAS_GARBAGE (1 << 4) /* some tuples on the page are dead, * but not deleted yet */ +/* Specifies the bits that can be set in the GiST flags field */ +#define GIST_FLAG_BITMASK 0x1F + /* * NSN (node sequence number) is a special-purpose LSN which is stored on each * index page in GISTPageOpaqueData and updated only during page splits. By diff --git a/src/include/crypto/bufenc.h b/src/include/crypto/bufenc.h new file mode 100644 index 0000000000..cc86ceb821 --- /dev/null +++ b/src/include/crypto/bufenc.h @@ -0,0 +1,28 @@ +/*------------------------------------------------------------------------- + * + * bufenc.h + * + * Portions Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/crypto/bufenc.h + * + *------------------------------------------------------------------------- + */ +#ifndef BUFENC_H +#define BUFENC_H + +#include "storage/bufmgr.h" +#include "crypto/kmgr.h" + +/* Cluster encryption encrypts only main forks */ +#define PageNeedsToBeEncrypted(forknum) \ + (FileEncryptionEnabled && (forknum) == MAIN_FORKNUM) + + +extern void InitializeBufferEncryption(void); +extern void EncryptPage(Page page, bool relation_is_permanent, + BlockNumber blkno); +extern void DecryptPage(Page page, bool relation_is_permanent, + BlockNumber blkno); + +#endif /* BUFENC_H */ diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h index 2708c4b683..d06c88d98c 100644 --- a/src/include/storage/bufpage.h +++ b/src/include/storage/bufpage.h @@ -15,6 +15,7 @@ #define BUFPAGE_H #include "access/xlogdefs.h" +#include "common/relpath.h" #include "storage/block.h" #include "storage/item.h" #include "storage/off.h" @@ -168,6 +169,8 @@ typedef struct PageHeaderData } PageHeaderData; typedef PageHeaderData *PageHeader; +#define PageEncryptOffset offsetof(PageHeaderData, pd_special) +#define SizeOfPageEncryption (BLCKSZ - PageEncryptOffset) /* * pd_flags contains the following flag bits. Undefined bits are initialized @@ -471,8 +474,8 @@ do { \ ((overwrite) ? PAI_OVERWRITE : 0) | \ ((is_heap) ? PAI_IS_HEAP : 0)) -#define PageIsVerified(page, blkno) \ - PageIsVerifiedExtended(page, blkno, \ +#define PageIsVerified(page, relation_is_permanent, blkno) \ + PageIsVerifiedExtended(page, MAIN_FORKNUM, relation_is_permanent, blkno, \ PIV_LOG_WARNING | PIV_REPORT_STAT) /* @@ -486,7 +489,10 @@ StaticAssertDecl(BLCKSZ == ((BLCKSZ / sizeof(size_t)) * sizeof(size_t)), "BLCKSZ has to be a multiple of sizeof(size_t)"); extern void PageInit(Page page, Size pageSize, Size specialSize); -extern bool PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags); +extern bool PageIsVerifiedExtended(Page page, ForkNumber forknum, + bool relation_is_permanent, + BlockNumber blkno, + int flags); extern OffsetNumber PageAddItemExtended(Page page, Item item, Size size, OffsetNumber offsetNumber, int flags); extern Page PageGetTempPage(Page page); @@ -506,5 +512,11 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum, Item newtup, Size newsize); extern char *PageSetChecksumCopy(Page page, BlockNumber blkno); extern void PageSetChecksumInplace(Page page, BlockNumber blkno); +extern char *PageEncryptCopy(Page page, ForkNumber forknum, + bool relation_is_permanent, BlockNumber blkno); +extern void PageEncryptInplace(Page page, ForkNumber forknum, + bool relation_is_permanent, BlockNumber blkno); +extern void PageDecryptInplace(Page page, ForkNumber forknum, + bool relation_is_permanent, BlockNumber blkno); #endif /* BUFPAGE_H */ diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h index 50a26edeb0..7fdfb71067 100644 --- a/src/include/storage/copydir.h +++ b/src/include/storage/copydir.h @@ -14,6 +14,6 @@ #define COPYDIR_H extern void copydir(char *fromdir, char *todir, bool recurse); -extern void copy_file(char *fromfile, char *tofile); +extern void copy_file(char *fromfile, char *tofile, bool encrypt_init_file); #endif /* COPYDIR_H */ -- 2.31.1 ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-03 14:09 ` Re: Moving forward with TDE Aleksander Alekseev <[email protected]> 2022-11-03 22:06 ` Re: Moving forward with TDE David Christensen <[email protected]> @ 2022-11-04 08:42 ` Dilip Kumar <[email protected]> 2022-11-17 16:34 ` Re: Moving forward with TDE David Christensen <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Dilip Kumar @ 2022-11-04 08:42 UTC (permalink / raw) To: David Christensen <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Stephen Frost <[email protected]> On Fri, Nov 4, 2022 at 3:36 AM David Christensen <[email protected]> wrote: > > > Unless somebody in the community remembers open questions/issues with > > TDE that were never addressed I suggest simply iterating with our > > usual testing/reviewing process. For now I'm going to change the > > status of the CF entry [1] to "Waiting for Author" since the patchset > > doesn't pass the CI [2]. > > Thanks, enclosed is a new version that is rebased on HEAD and fixes a > bug that the new pg_control_init() test picked up. I was looking into the documentation patches 0001 and 0002, I think the explanation is very clear. I have a few questions/comments +By not using the database id in the IV, CREATE DATABASE can copy the +heap/index files from the old database to a new one without +decryption/encryption. Both page copies are valid. Once a database +changes its pages, it gets new LSNs, and hence new IV. How about the WAL_LOG method for creating a database? because in that we get the new LSN for the pages in the new database, so do we reencrypt, if yes then this documentation needs to be updated otherwise we might need to add that code. +changes its pages, it gets new LSNs, and hence new IV. Using only the +LSN and page number also avoids requiring pg_upgrade to preserve +database oids, tablespace oids, and relfilenodes. I think this line needs to be changed, because now we are already preserving dbid/tbsid/relfilenode. So even though we are not using those in IV there is no point in saying we are avoiding that requirement. I will review the remaining patches soon. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-03 14:09 ` Re: Moving forward with TDE Aleksander Alekseev <[email protected]> 2022-11-03 22:06 ` Re: Moving forward with TDE David Christensen <[email protected]> 2022-11-04 08:42 ` Re: Moving forward with TDE Dilip Kumar <[email protected]> @ 2022-11-17 16:34 ` David Christensen <[email protected]> 0 siblings, 0 replies; 9+ messages in thread From: David Christensen @ 2022-11-17 16:34 UTC (permalink / raw) To: Dilip Kumar <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Stephen Frost <[email protected]> Hi Dilip, Thanks for the feedback here. I will review the docs changes and add to my tree. Best, David ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> @ 2022-11-15 19:07 ` Jacob Champion <[email protected]> 2022-11-15 19:39 ` Re: Moving forward with TDE David Christensen <[email protected]> 1 sibling, 1 reply; 9+ messages in thread From: Jacob Champion @ 2022-11-15 19:07 UTC (permalink / raw) To: David Christensen <[email protected]>; +Cc: pgsql-hackers; Stephen Frost <[email protected]> On Mon, Oct 24, 2022 at 9:29 AM David Christensen <[email protected]> wrote: > I would love to open a discussion about how to move forward and get > some of these features built out. The historical threads here are > quite long and complicated; is there a "current state" other than the > wiki that reflects the general thinking on this feature? Any major > developments in direction that would not be reflected in the code from > May 2021? I don't think the patchset here has incorporated the results of the discussion [1] that happened at the end of 2021. For example, it looks like AES-CTR is still in use for the pages, which I thought was already determined to be insufficient. The following next steps were proposed in that thread: > 1. modify temporary file I/O to use a more centralized API > 2. modify the existing cluster file encryption patch to use XTS with a > IV that uses more than the LSN > 3. add XTS regression test code like CTR > 4. create WAL encryption code using CTR Does this patchset need review before those steps are taken (or was there additional conversation/work that I missed)? Thanks, --Jacob [1] https://www.postgresql.org/message-id/flat/20211013222648.GA373%40momjian.us ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-15 19:07 ` Re: Moving forward with TDE Jacob Champion <[email protected]> @ 2022-11-15 19:39 ` David Christensen <[email protected]> 2022-11-15 20:08 ` Re: Moving forward with TDE Jacob Champion <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: David Christensen @ 2022-11-15 19:39 UTC (permalink / raw) To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers; Stephen Frost <[email protected]> > On Nov 15, 2022, at 1:08 PM, Jacob Champion <[email protected]> wrote: > > On Mon, Oct 24, 2022 at 9:29 AM David Christensen > <[email protected]> wrote: >> I would love to open a discussion about how to move forward and get >> some of these features built out. The historical threads here are >> quite long and complicated; is there a "current state" other than the >> wiki that reflects the general thinking on this feature? Any major >> developments in direction that would not be reflected in the code from >> May 2021? > > I don't think the patchset here has incorporated the results of the > discussion [1] that happened at the end of 2021. For example, it looks > like AES-CTR is still in use for the pages, which I thought was > already determined to be insufficient. Good to know about the next steps, thanks. > The following next steps were proposed in that thread: > >> 1. modify temporary file I/O to use a more centralized API >> 2. modify the existing cluster file encryption patch to use XTS with a >> IV that uses more than the LSN >> 3. add XTS regression test code like CTR >> 4. create WAL encryption code using CTR > > Does this patchset need review before those steps are taken (or was > there additional conversation/work that I missed)? This was just a refresh of the old patches on the wiki to work as written on HEAD. If there are known TODOs here this then that work is still needing to be done. I was going to take 2) and Stephen was going to work on 3); I am not sure about the other two but will review the thread you pointed to. Thanks for pointing that out. David ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-15 19:07 ` Re: Moving forward with TDE Jacob Champion <[email protected]> 2022-11-15 19:39 ` Re: Moving forward with TDE David Christensen <[email protected]> @ 2022-11-15 20:08 ` Jacob Champion <[email protected]> 2022-11-17 16:02 ` Re: Moving forward with TDE David Christensen <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Jacob Champion @ 2022-11-15 20:08 UTC (permalink / raw) To: David Christensen <[email protected]>; +Cc: pgsql-hackers; Stephen Frost <[email protected]> On Tue, Nov 15, 2022 at 11:39 AM David Christensen <[email protected]> wrote: > Good to know about the next steps, thanks. You're welcome! > This was just a refresh of the old patches on the wiki to work as written on HEAD. If there are known TODOs here this then that work is still needing to be done. > > I was going to take 2) and Stephen was going to work on 3); I am not sure about the other two but will review the thread you pointed to. Thanks for pointing that out. I've attached the diffs I'm carrying to build this under meson (as well as -Wshadow; my removal of the two variables probably needs some scrutiny). It looks like the testcrypto executable will need substantial changes after the common/hex.h revert. --Jacob From eb4b55f5d03e362cf340f322c0cefbf95f53657a Mon Sep 17 00:00:00 2001 From: Jacob Champion <[email protected]> Date: Mon, 14 Nov 2022 16:02:57 -0800 Subject: [PATCH] WIP: fix meson Note that the testcrypto executable needs to be rewritten. --- src/backend/bootstrap/bootstrap.c | 2 -- src/backend/crypto/kmgr.c | 1 - src/backend/crypto/meson.build | 14 ++++++++++++++ src/backend/meson.build | 1 + src/bin/meson.build | 1 + src/bin/pg_alterckey/meson.build | 16 ++++++++++++++++ src/common/meson.build | 3 +++ 7 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 src/backend/crypto/meson.build create mode 100644 src/bin/pg_alterckey/meson.build diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index ffaf9c7cc4..07b2f15343 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -256,8 +256,6 @@ BootstrapModeMain(int argc, char *argv[], bool check_only) break; case 'K': { - int i; - /* method 0/disabled cannot be specified */ for (i = DISABLED_ENCRYPTION_METHOD + 1; i < NUM_ENCRYPTION_METHODS; i++) diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c index a83effbca8..b4248965f8 100644 --- a/src/backend/crypto/kmgr.c +++ b/src/backend/crypto/kmgr.c @@ -141,7 +141,6 @@ BootStrapKmgr(void) if (bootstrap_old_key_datadir == NULL) { unsigned char *bootstrap_keys_wrap[KMGR_NUM_DATA_KEYS]; - int key_lens[KMGR_NUM_DATA_KEYS]; PgCipherCtx *cluster_key_ctx; /* Create KEK encryption context */ diff --git a/src/backend/crypto/meson.build b/src/backend/crypto/meson.build new file mode 100644 index 0000000000..2b72b26ebf --- /dev/null +++ b/src/backend/crypto/meson.build @@ -0,0 +1,14 @@ +backend_sources += files( + 'bufenc.c', + 'kmgr.c', +) + +install_data( + 'ckey_aws.sh.sample', + 'ckey_direct.sh.sample', + 'ckey_passphrase.sh.sample', + 'ckey_piv_nopin.sh.sample', + 'ckey_piv_pin.sh.sample', + 'ssl_passphrase.sh.sample', + install_dir: dir_data / 'auth_commands', +) diff --git a/src/backend/meson.build b/src/backend/meson.build index 37562bae13..2162aef554 100644 --- a/src/backend/meson.build +++ b/src/backend/meson.build @@ -9,6 +9,7 @@ subdir('backup') subdir('bootstrap') subdir('catalog') subdir('commands') +subdir('crypto') subdir('executor') subdir('foreign') subdir('jit') diff --git a/src/bin/meson.build b/src/bin/meson.build index 5fd5a9d2f9..8dd9133213 100644 --- a/src/bin/meson.build +++ b/src/bin/meson.build @@ -1,4 +1,5 @@ subdir('initdb') +subdir('pg_alterckey') subdir('pg_amcheck') subdir('pg_archivecleanup') subdir('pg_basebackup') diff --git a/src/bin/pg_alterckey/meson.build b/src/bin/pg_alterckey/meson.build new file mode 100644 index 0000000000..72ec8274c3 --- /dev/null +++ b/src/bin/pg_alterckey/meson.build @@ -0,0 +1,16 @@ +pg_alterckey_sources = files( + 'pg_alterckey.c', +) + +if host_system == 'windows' + pg_alterckey_sources += rc_bin_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'pg_alterckey', + '--FILEDESC', 'pg_alterckey - alter the cluster key',]) +endif + +pg_alterckey = executable('pg_alterckey', + pg_alterckey_sources, + dependencies: [frontend_code, libpq], + kwargs: default_bin_args, +) +bin_targets += pg_alterckey diff --git a/src/common/meson.build b/src/common/meson.build index 1c9b8a3a01..102b4d7b23 100644 --- a/src/common/meson.build +++ b/src/common/meson.build @@ -12,6 +12,7 @@ common_sources = files( 'ip.c', 'jsonapi.c', 'keywords.c', + 'kmgr_utils.c', 'kwlookup.c', 'link-canary.c', 'md5_common.c', @@ -34,12 +35,14 @@ common_sources = files( if ssl.found() common_sources += files( + 'cipher_openssl.c', 'cryptohash_openssl.c', 'hmac_openssl.c', 'protocol_openssl.c', ) else common_sources += files( + 'cipher.c', 'cryptohash.c', 'hmac.c', 'md5.c', -- 2.25.1 Attachments: [text/plain] fix-meson.patch.txt (3.9K, ../../CAAWbhmiVCyYXp6TDkc+Ci9F0Ph8cdGDFA1D3MTVZimuqp5v3ag@mail.gmail.com/2-fix-meson.patch.txt) download | inline diff: From eb4b55f5d03e362cf340f322c0cefbf95f53657a Mon Sep 17 00:00:00 2001 From: Jacob Champion <[email protected]> Date: Mon, 14 Nov 2022 16:02:57 -0800 Subject: [PATCH] WIP: fix meson Note that the testcrypto executable needs to be rewritten. --- src/backend/bootstrap/bootstrap.c | 2 -- src/backend/crypto/kmgr.c | 1 - src/backend/crypto/meson.build | 14 ++++++++++++++ src/backend/meson.build | 1 + src/bin/meson.build | 1 + src/bin/pg_alterckey/meson.build | 16 ++++++++++++++++ src/common/meson.build | 3 +++ 7 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 src/backend/crypto/meson.build create mode 100644 src/bin/pg_alterckey/meson.build diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index ffaf9c7cc4..07b2f15343 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -256,8 +256,6 @@ BootstrapModeMain(int argc, char *argv[], bool check_only) break; case 'K': { - int i; - /* method 0/disabled cannot be specified */ for (i = DISABLED_ENCRYPTION_METHOD + 1; i < NUM_ENCRYPTION_METHODS; i++) diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c index a83effbca8..b4248965f8 100644 --- a/src/backend/crypto/kmgr.c +++ b/src/backend/crypto/kmgr.c @@ -141,7 +141,6 @@ BootStrapKmgr(void) if (bootstrap_old_key_datadir == NULL) { unsigned char *bootstrap_keys_wrap[KMGR_NUM_DATA_KEYS]; - int key_lens[KMGR_NUM_DATA_KEYS]; PgCipherCtx *cluster_key_ctx; /* Create KEK encryption context */ diff --git a/src/backend/crypto/meson.build b/src/backend/crypto/meson.build new file mode 100644 index 0000000000..2b72b26ebf --- /dev/null +++ b/src/backend/crypto/meson.build @@ -0,0 +1,14 @@ +backend_sources += files( + 'bufenc.c', + 'kmgr.c', +) + +install_data( + 'ckey_aws.sh.sample', + 'ckey_direct.sh.sample', + 'ckey_passphrase.sh.sample', + 'ckey_piv_nopin.sh.sample', + 'ckey_piv_pin.sh.sample', + 'ssl_passphrase.sh.sample', + install_dir: dir_data / 'auth_commands', +) diff --git a/src/backend/meson.build b/src/backend/meson.build index 37562bae13..2162aef554 100644 --- a/src/backend/meson.build +++ b/src/backend/meson.build @@ -9,6 +9,7 @@ subdir('backup') subdir('bootstrap') subdir('catalog') subdir('commands') +subdir('crypto') subdir('executor') subdir('foreign') subdir('jit') diff --git a/src/bin/meson.build b/src/bin/meson.build index 5fd5a9d2f9..8dd9133213 100644 --- a/src/bin/meson.build +++ b/src/bin/meson.build @@ -1,4 +1,5 @@ subdir('initdb') +subdir('pg_alterckey') subdir('pg_amcheck') subdir('pg_archivecleanup') subdir('pg_basebackup') diff --git a/src/bin/pg_alterckey/meson.build b/src/bin/pg_alterckey/meson.build new file mode 100644 index 0000000000..72ec8274c3 --- /dev/null +++ b/src/bin/pg_alterckey/meson.build @@ -0,0 +1,16 @@ +pg_alterckey_sources = files( + 'pg_alterckey.c', +) + +if host_system == 'windows' + pg_alterckey_sources += rc_bin_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'pg_alterckey', + '--FILEDESC', 'pg_alterckey - alter the cluster key',]) +endif + +pg_alterckey = executable('pg_alterckey', + pg_alterckey_sources, + dependencies: [frontend_code, libpq], + kwargs: default_bin_args, +) +bin_targets += pg_alterckey diff --git a/src/common/meson.build b/src/common/meson.build index 1c9b8a3a01..102b4d7b23 100644 --- a/src/common/meson.build +++ b/src/common/meson.build @@ -12,6 +12,7 @@ common_sources = files( 'ip.c', 'jsonapi.c', 'keywords.c', + 'kmgr_utils.c', 'kwlookup.c', 'link-canary.c', 'md5_common.c', @@ -34,12 +35,14 @@ common_sources = files( if ssl.found() common_sources += files( + 'cipher_openssl.c', 'cryptohash_openssl.c', 'hmac_openssl.c', 'protocol_openssl.c', ) else common_sources += files( + 'cipher.c', 'cryptohash.c', 'hmac.c', 'md5.c', -- 2.25.1 ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Moving forward with TDE 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-15 19:07 ` Re: Moving forward with TDE Jacob Champion <[email protected]> 2022-11-15 19:39 ` Re: Moving forward with TDE David Christensen <[email protected]> 2022-11-15 20:08 ` Re: Moving forward with TDE Jacob Champion <[email protected]> @ 2022-11-17 16:02 ` David Christensen <[email protected]> 0 siblings, 0 replies; 9+ messages in thread From: David Christensen @ 2022-11-17 16:02 UTC (permalink / raw) To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers; Stephen Frost <[email protected]> Hi Jacob, Thanks, I've added this patch in my tree [1]. (For now, just adding fixes and the like atop the original separate patches, but will eventually get things winnowed down into probably the same 12 parts the originals were reviewed in. Best, David [1] https://github.com/pgguru/postgres/tree/tde ^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2022-11-17 16:34 UTC | newest] Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-10-24 16:29 Moving forward with TDE David Christensen <[email protected]> 2022-11-03 14:09 ` Aleksander Alekseev <[email protected]> 2022-11-03 22:06 ` David Christensen <[email protected]> 2022-11-04 08:42 ` Dilip Kumar <[email protected]> 2022-11-17 16:34 ` David Christensen <[email protected]> 2022-11-15 19:07 ` Jacob Champion <[email protected]> 2022-11-15 19:39 ` David Christensen <[email protected]> 2022-11-15 20:08 ` Jacob Champion <[email protected]> 2022-11-17 16:02 ` David Christensen <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox