pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedKey management with tests
114+ messages / 13 participants
[nested] [flat]
* Key management with tests
@ 2021-01-01 04:50 Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-01 04:50 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
I have completed the key management patch with tests created by Stephen
Frost. Original patch by Masahiko Sawada. It requires the hex
reorganization patch first. The key patch is now 2.1MB because of the
tests, so attaching it here seems unwise:
https://github.com/postgres/postgres/compare/master...bmomjian:hex.diff
https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
I will add it to the commitfest. I think we need to figure out how much
of the tests we want to add.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-01 06:07 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-01 06:07 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote:
> I have completed the key management patch with tests created by Stephen
> Frost. Original patch by Masahiko Sawada. It requires the hex
> reorganization patch first. The key patch is now 2.1MB because of the
> tests, so attaching it here seems unwise:
>
> https://github.com/postgres/postgres/compare/master...bmomjian:hex.diff
> https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
>
> I will add it to the commitfest. I think we need to figure out how much
> of the tests we want to add.
I am getting regression test errors using OpenSSL 1.1.1d 10 Sep 2019
with zero-length input data (no -p), while Stephen is able for those
tests to pass. This needs more research, plus I think higher-level
tests.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-07 15:02 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 3 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-07 15:02 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote:
> On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote:
> > I have completed the key management patch with tests created by Stephen
> > Frost. Original patch by Masahiko Sawada. It requires the hex
> > reorganization patch first. The key patch is now 2.1MB because of the
> > tests, so attaching it here seems unwise:
> >
> > https://github.com/postgres/postgres/compare/master...bmomjian:hex.diff
> > https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
> >
> > I will add it to the commitfest. I think we need to figure out how much
> > of the tests we want to add.
>
> I am getting regression test errors using OpenSSL 1.1.1d 10 Sep 2019
> with zero-length input data (no -p), while Stephen is able for those
> tests to pass. This needs more research, plus I think higher-level
> tests.
I have found the cause of the failure, which I added as a C comment:
/*
* OpenSSL 1.1.1d and earlier crashes on some zero-length plaintext
* and ciphertext strings. It crashes on an encryption call to
* EVP_EncryptFinal_ex(() in GCM mode of zero-length strings if
* plaintext is NULL, even though plaintext_len is zero. Setting
* plaintext to non-NULL allows it to work. In KW/KWP mode,
* zero-length strings fail if plaintext_len = 0 and plaintext is
* non-NULL (the opposite). OpenSSL 1.1.1e+ is fine with all options.
*/
else if (cipher == PG_CIPHER_AES_GCM)
{
plaintext_len = 0;
plaintext = pg_malloc0(1);
}
All the tests pass now. The current src/test directory is 19MB, and
adding these tests takes it to 23MB, or a 20% increase. That seems like
a lot. It is testing 128-bit and 256-bit keys --- should we do fewer
tests, or just test 256, or use gzip to compress the tests by 50%?
(Does every platform have gzip?)
My next step is to add the high-level tests.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-07 19:08 Alvaro Herrera <alvherre@alvh.no-ip.org>
parent: Bruce Momjian <bruce@momjian.us>
2 siblings, 1 reply; 114+ messages in thread
From: Alvaro Herrera @ 2021-01-07 19:08 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On 2021-Jan-07, Bruce Momjian wrote:
> All the tests pass now. The current src/test directory is 19MB, and
> adding these tests takes it to 23MB, or a 20% increase. That seems like
> a lot. It is testing 128-bit and 256-bit keys --- should we do fewer
> tests, or just test 256, or use gzip to compress the tests by 50%?
> (Does every platform have gzip?)
So the tests are about 95% of the patch ... do we really need that many
tests?
--
Álvaro Herrera
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-08 00:39 Bruce Momjian <bruce@momjian.us>
parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-08 00:39 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote:
> On 2021-Jan-07, Bruce Momjian wrote:
>
> > All the tests pass now. The current src/test directory is 19MB, and
> > adding these tests takes it to 23MB, or a 20% increase. That seems like
> > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer
> > tests, or just test 256, or use gzip to compress the tests by 50%?
> > (Does every platform have gzip?)
>
> So the tests are about 95% of the patch ... do we really need that many
> tests?
No, I don't think so. Stephen imported the entire NIST test suite. It
was so comperhensive, it detected several OpenSSL bugs for zero-length
strings, which I already reported, but we would never be encrypting
zero-length strings, so there wasn't a lot of value to it.
Anyway, I think we need to figure out how to trim. The first part would
be to figure out whether we need 128 _and_ 256-bit tests, and then see
what items are really useful. Stephen, do you have any ideas on that?
We currently have 10296 tests, and I think we could get away with 100.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-08 00:46 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
2 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-08 00:46 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Jan 7, 2021 at 10:02:14AM -0500, Bruce Momjian wrote:
> My next step is to add the high-level tests.
Here is the high-level script, and the log output. I used the
pg_upgrade test.sh as a model.
It uses "CFE DEBUG" lines that are already in the code to compare the
initdb encryption with the other initdb decryption and pg_ctl
decryption. It was easier than I thought.
What it does not do is to test the file descriptor passing from
/dev/tty, or the sample scripts. This seems acceptable to me since I
test them and they rarely change.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
test.sh (application/x-sh, 4.5K) download
log (text/plain, 2.1K) download
make[1]: Leaving directory '/usr/local/src/gen/pgsql/postgresql/src/common'
MAKE= PATH="/pgtop/tmp_install/usr/local/pgsql/bin:$PATH" LD_LIBRARY_PATH="/pgtop/tmp_install/usr/local/pgsql/lib" bindir=/pgtop/tmp_install//usr/local/pgsql/bin /bin/sh test.sh
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
Cluster file encryption is enabled.
creating directory /pgtop/src/test/crypto/tmp_check/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
/pgtop/tmp_install/usr/local/pgsql/bin/pg_ctl -D /pgtop/src/test/crypto/tmp_check/data -l logfile start
waiting for server to start.... done
server started
waiting for server to shut down.... done
server stopped
PASSED
rm -rf '/usr/local/src/gen/pgsql/postgresql/src/test/crypto'/tmp_check
/bin/mkdir -p '/usr/local/src/gen/pgsql/postgresql/src/test/crypto'/tmp_check
cd . && TESTDIR='/usr/local/src/gen/pgsql/postgresql/src/test/crypto' PATH="/pgtop/tmp_install/usr/local/pgsql/bin:$PATH" LD_LIBRARY_PATH="/pgtop/tmp_install/usr/local/pgsql/lib" PGPORT='65432' PG_REGRESS='/usr/local/src/gen/pgsql/postgresql/src/test/crypto/../../../src/test/regress/pg_regress' REGRESS_SHLIB='/pgtop/src/test/regress/regress.so' /usr/bin/prove -I ../../../src/test/perl/ -I . t/*.pl
t/001_testcrypto.pl .. ok
t/002_testkw.pl ...... ok
All tests successful.
Files=2, Tests=10296, 138 wallclock secs ( 1.81 usr 0.22 sys + 26.55 cusr 24.02 csys = 52.60 CPU)
Result: PASS
* Re: Key management with tests
@ 2021-01-08 20:33 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-08 20:33 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote:
> > On 2021-Jan-07, Bruce Momjian wrote:
> >
> > > All the tests pass now. The current src/test directory is 19MB, and
> > > adding these tests takes it to 23MB, or a 20% increase. That seems like
> > > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer
> > > tests, or just test 256, or use gzip to compress the tests by 50%?
> > > (Does every platform have gzip?)
> >
> > So the tests are about 95% of the patch ... do we really need that many
> > tests?
>
> No, I don't think so. Stephen imported the entire NIST test suite. It
> was so comperhensive, it detected several OpenSSL bugs for zero-length
> strings, which I already reported, but we would never be encrypting
> zero-length strings, so there wasn't a lot of value to it.
I ran the entire test suite locally to ensure everything worked, but I
didn't actually include all of it in the PR which you merged- I had
already reduced it quite a bit by removing all 'additional
authenticated data' test cases (which the tests will automatically skip
and which we haven't implemented support for in the common library
wrappers) and by removing the 192-bit cases. This reduced the overall
test set by about 2/3rd's or so, as I recall.
> Anyway, I think we need to figure out how to trim. The first part would
> be to figure out whether we need 128 _and_ 256-bit tests, and then see
> what items are really useful. Stephen, do you have any ideas on that?
> We currently have 10296 tests, and I think we could get away with 100.
Yeah, it's probably still too much, but I don't have any particularly
justifiable suggestions as to exactly what we should remove or what we
should keep.
Perhaps it'd make sense to try and cover the cases that are more likely
to be issues between our wrapper functions and OpenSSL, and not stress
too much about constantly testing cases that should really be up to
OpenSSL. As such, I'd propose:
- Add back in some 192-bit tests, so we cover all three bit lengths.
- Add back in some additional authenticated test cases, just to make
sure that, until/unless we implement support, the test code properly
skips over those.
- Keep tests for various length plaintext/ciphertext (including 0-byte
cases, so we make sure those work, since they really should).
- Keep at least one test for each length of tag that's included in the
test suite.
I'm not sure how many tests we'd end up with from that, but my swag /
gut feeling is that it'd probably be on the order of 100ish and a small
enough set that it won't dwarf the rest of the patch.
Would be nice if we had a way for some buildfarm animal or something to
pull in the entire suite and test it, imv.. If anyone wants to
volunteer, I'd be happy to explain how to make that happen (it's not
hard though- download/unzip the files, drop them in the directory,
update the test script to add all the files into the array).
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-08 20:34 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
2 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-08 20:34 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings Bruce,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote:
> > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote:
> > > I have completed the key management patch with tests created by Stephen
> > > Frost. Original patch by Masahiko Sawada. It requires the hex
> > > reorganization patch first. The key patch is now 2.1MB because of the
> > > tests, so attaching it here seems unwise:
> > >
> > > https://github.com/postgres/postgres/compare/master...bmomjian:hex.diff
> > > https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
> > >
> > > I will add it to the commitfest. I think we need to figure out how much
> > > of the tests we want to add.
> >
> > I am getting regression test errors using OpenSSL 1.1.1d 10 Sep 2019
> > with zero-length input data (no -p), while Stephen is able for those
> > tests to pass. This needs more research, plus I think higher-level
> > tests.
>
> I have found the cause of the failure, which I added as a C comment:
>
> /*
> * OpenSSL 1.1.1d and earlier crashes on some zero-length plaintext
> * and ciphertext strings. It crashes on an encryption call to
> * EVP_EncryptFinal_ex(() in GCM mode of zero-length strings if
> * plaintext is NULL, even though plaintext_len is zero. Setting
> * plaintext to non-NULL allows it to work. In KW/KWP mode,
> * zero-length strings fail if plaintext_len = 0 and plaintext is
> * non-NULL (the opposite). OpenSSL 1.1.1e+ is fine with all options.
> */
> else if (cipher == PG_CIPHER_AES_GCM)
> {
> plaintext_len = 0;
> plaintext = pg_malloc0(1);
> }
>
> All the tests pass now. The current src/test directory is 19MB, and
> adding these tests takes it to 23MB, or a 20% increase. That seems like
> a lot. It is testing 128-bit and 256-bit keys --- should we do fewer
> tests, or just test 256, or use gzip to compress the tests by 50%?
> (Does every platform have gzip?)
Thanks a lot for working on this and figuring out what the issue was and
fixing it! That's great that we got all those cases passing for you
too.
Thanks again,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-08 21:17 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-08 21:17 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote:
> > No, I don't think so. Stephen imported the entire NIST test suite. It
> > was so comperhensive, it detected several OpenSSL bugs for zero-length
> > strings, which I already reported, but we would never be encrypting
> > zero-length strings, so there wasn't a lot of value to it.
>
> I ran the entire test suite locally to ensure everything worked, but I
> didn't actually include all of it in the PR which you merged- I had
> already reduced it quite a bit by removing all 'additional
> authenticated data' test cases (which the tests will automatically skip
> and which we haven't implemented support for in the common library
> wrappers) and by removing the 192-bit cases. This reduced the overall
> test set by about 2/3rd's or so, as I recall.
Wow, so that was reduced!
> > Anyway, I think we need to figure out how to trim. The first part would
> > be to figure out whether we need 128 _and_ 256-bit tests, and then see
> > what items are really useful. Stephen, do you have any ideas on that?
> > We currently have 10296 tests, and I think we could get away with 100.
>
> Yeah, it's probably still too much, but I don't have any particularly
> justifiable suggestions as to exactly what we should remove or what we
> should keep.
>
> Perhaps it'd make sense to try and cover the cases that are more likely
> to be issues between our wrapper functions and OpenSSL, and not stress
> too much about constantly testing cases that should really be up to
> OpenSSL. As such, I'd propose:
>
> - Add back in some 192-bit tests, so we cover all three bit lengths.
> - Add back in some additional authenticated test cases, just to make
> sure that, until/unless we implement support, the test code properly
> skips over those.
> - Keep tests for various length plaintext/ciphertext (including 0-byte
> cases, so we make sure those work, since they really should).
> - Keep at least one test for each length of tag that's included in the
> test suite.
Makes sense. I did a simplistic trim-down to 90 tests but it still was
40% of the patch; attached. The hex strings are very long.
> I'm not sure how many tests we'd end up with from that, but my swag /
> gut feeling is that it'd probably be on the order of 100ish and a small
> enough set that it won't dwarf the rest of the patch.
>
> Would be nice if we had a way for some buildfarm animal or something to
> pull in the entire suite and test it, imv.. If anyone wants to
> volunteer, I'd be happy to explain how to make that happen (it's not
> hard though- download/unzip the files, drop them in the directory,
> update the test script to add all the files into the array).
Yes, do we have a place to store more comprehensive tests outside of our
git tree? Has this been done before?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
cryptotest.tgz (application/x-gtar-compressed, 73.7K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-08 21:18 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-08 21:18 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 8, 2021 at 03:34:23PM -0500, Stephen Frost wrote:
> > All the tests pass now. The current src/test directory is 19MB, and
> > adding these tests takes it to 23MB, or a 20% increase. That seems like
> > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer
> > tests, or just test 256, or use gzip to compress the tests by 50%?
> > (Does every platform have gzip?)
>
> Thanks a lot for working on this and figuring out what the issue was and
> fixing it! That's great that we got all those cases passing for you
> too.
Yes, I was relieved. The pattern of when zero-length strings fail in
which modes is still very odd, but at least it reports an error, so it
isn't returning incorrect data.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-08 21:24 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 114+ messages in thread
From: Stephen Frost @ 2021-01-08 21:24 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote:
> > > Anyway, I think we need to figure out how to trim. The first part would
> > > be to figure out whether we need 128 _and_ 256-bit tests, and then see
> > > what items are really useful. Stephen, do you have any ideas on that?
> > > We currently have 10296 tests, and I think we could get away with 100.
> >
> > Yeah, it's probably still too much, but I don't have any particularly
> > justifiable suggestions as to exactly what we should remove or what we
> > should keep.
> >
> > Perhaps it'd make sense to try and cover the cases that are more likely
> > to be issues between our wrapper functions and OpenSSL, and not stress
> > too much about constantly testing cases that should really be up to
> > OpenSSL. As such, I'd propose:
> >
> > - Add back in some 192-bit tests, so we cover all three bit lengths.
> > - Add back in some additional authenticated test cases, just to make
> > sure that, until/unless we implement support, the test code properly
> > skips over those.
> > - Keep tests for various length plaintext/ciphertext (including 0-byte
> > cases, so we make sure those work, since they really should).
> > - Keep at least one test for each length of tag that's included in the
> > test suite.
>
> Makes sense. I did a simplistic trim-down to 90 tests but it still was
> 40% of the patch; attached. The hex strings are very long.
I don't think we actually need to stress over the size of the test data
relative to the size of the patch- it's not like it's all that much perl
code. I can appreciate that we don't want to add megabytes worth of
test data to the git repo though.
> > I'm not sure how many tests we'd end up with from that, but my swag /
> > gut feeling is that it'd probably be on the order of 100ish and a small
> > enough set that it won't dwarf the rest of the patch.
> >
> > Would be nice if we had a way for some buildfarm animal or something to
> > pull in the entire suite and test it, imv.. If anyone wants to
> > volunteer, I'd be happy to explain how to make that happen (it's not
> > hard though- download/unzip the files, drop them in the directory,
> > update the test script to add all the files into the array).
>
> Yes, do we have a place to store more comprehensive tests outside of our
> git tree? Has this been done before?
Not that I'm aware of.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-09 18:17 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-09 18:17 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote:
> On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote:
> > I have completed the key management patch with tests created by Stephen
> > Frost. Original patch by Masahiko Sawada. It requires the hex
> > reorganization patch first. The key patch is now 2.1MB because of the
> > tests, so attaching it here seems unwise:
> >
> > https://github.com/postgres/postgres/compare/master...bmomjian:hex.diff
> > https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
> >
> > I will add it to the commitfest. I think we need to figure out how much
> > of the tests we want to add.
>
> I am getting regression test errors using OpenSSL 1.1.1d 10 Sep 2019
> with zero-length input data (no -p), while Stephen is able for those
> tests to pass. This needs more research, plus I think higher-level
> tests.
I know we are still working on the hex patch (dest-len) and the crypto
tests, but I wanted to post this so people can see where we are, and we
can get some current cfbot testing.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
hex.diff.gz (application/gzip, 3.6K) download
key.diff.gz (application/gzip, 120.8K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-10 01:08 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-10 01:08 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sat, Jan 9, 2021 at 01:17:36PM -0500, Bruce Momjian wrote:
> On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote:
> > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote:
> > > I have completed the key management patch with tests created by Stephen
> > > Frost. Original patch by Masahiko Sawada. It requires the hex
> > > reorganization patch first. The key patch is now 2.1MB because of the
> > > tests, so attaching it here seems unwise:
> > >
> > > https://github.com/postgres/postgres/compare/master...bmomjian:hex.diff
> > > https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
> > >
> > > I will add it to the commitfest. I think we need to figure out how much
> > > of the tests we want to add.
> >
> > I am getting regression test errors using OpenSSL 1.1.1d 10 Sep 2019
> > with zero-length input data (no -p), while Stephen is able for those
> > tests to pass. This needs more research, plus I think higher-level
> > tests.
>
> I know we are still working on the hex patch (dest-len) and the crypto
> tests, but I wanted to post this so people can see where we are, and we
> can get some current cfbot testing.
Here is an updated version that covers all the possible
testing/configuration options.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
hex.diff.gz (application/gzip, 3.6K) download
hex..key.diff.gz (application/gzip, 120.9K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-10 02:45 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-10 02:45 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>; Thomas Munro <thomas.munro@gmail.com>
On Sat, Jan 9, 2021 at 08:08:16PM -0500, Bruce Momjian wrote:
> On Sat, Jan 9, 2021 at 01:17:36PM -0500, Bruce Momjian wrote:
> > I know we are still working on the hex patch (dest-len) and the crypto
> > tests, but I wanted to post this so people can see where we are, and we
> > can get some current cfbot testing.
>
> Here is an updated version that covers all the possible
> testing/configuration options.
Does anyone know why the cfbot applied the patch listed second first
here?
http://cfbot.cputube.org/patch_31_2925.log
Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed
it would apply attachments in the order they appear in the email.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-10 05:04 Thomas Munro <thomas.munro@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Thomas Munro @ 2021-01-10 05:04 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian <bruce@momjian.us> wrote:
> Does anyone know why the cfbot applied the patch listed second first
> here?
>
> http://cfbot.cputube.org/patch_31_2925.log
>
> Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed
> it would apply attachments in the order they appear in the email.
It sorts the filenames (in this case after decompressing step removes
the .gz endings). That works pretty well for the patches that "git
format-patch" spits out, but it's a bit hit and miss with cases like
yours.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-10 14:51 Bruce Momjian <bruce@momjian.us>
parent: Thomas Munro <thomas.munro@gmail.com>
0 siblings, 2 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-10 14:51 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 10, 2021 at 06:04:12PM +1300, Thomas Munro wrote:
> On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian <bruce@momjian.us> wrote:
> > Does anyone know why the cfbot applied the patch listed second first
> > here?
> >
> > http://cfbot.cputube.org/patch_31_2925.log
> >
> > Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed
> > it would apply attachments in the order they appear in the email.
>
> It sorts the filenames (in this case after decompressing step removes
> the .gz endings). That works pretty well for the patches that "git
> format-patch" spits out, but it's a bit hit and miss with cases like
> yours.
OK, here they are with numeric prefixes. It was actually tricky to
figure out how to create a squashed format-patch based on another branch.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
01-hex.diff.gz (application/gzip, 3.6K) download
02-hex..key.diff.gz (application/gzip, 120.9K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 11:12 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 2 replies; 114+ messages in thread
From: Masahiko Sawada @ 2021-01-11 11:12 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 10, 2021 at 11:51 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Sun, Jan 10, 2021 at 06:04:12PM +1300, Thomas Munro wrote:
> > On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian <bruce@momjian.us> wrote:
> > > Does anyone know why the cfbot applied the patch listed second first
> > > here?
> > >
> > > http://cfbot.cputube.org/patch_31_2925.log
> > >
> > > Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed
> > > it would apply attachments in the order they appear in the email.
> >
> > It sorts the filenames (in this case after decompressing step removes
> > the .gz endings). That works pretty well for the patches that "git
> > format-patch" spits out, but it's a bit hit and miss with cases like
> > yours.
>
> OK, here they are with numeric prefixes. It was actually tricky to
> figure out how to create a squashed format-patch based on another branch.
>
Thank you for attaching the patches. It passes all cfbot tests, great.
Looking at the patch, it supports three algorithms but only
PG_CIPHER_AES_KWP is used in the core for now:
+/*
+ * Supported symmetric encryption algorithm. These identifiers are passed
+ * to pg_cipher_ctx_create() function, and then actual encryption
+ * implementations need to initialize their context of the given encryption
+ * algorithm.
+ */
+#define PG_CIPHER_AES_GCM 0
+#define PG_CIPHER_AES_KW 1
+#define PG_CIPHER_AES_KWP 2
+#define PG_MAX_CIPHER_ID 3
Are we in the process of experimenting which algorithms are better? If
we support one algorithm that is actually used in the core, we would
reduce the tests as well.
FWIW, I've written a PoC patch for buffer encryption to make sure the
kms patch would be workable with other components using the encryption
key managed by kmgr.
Overall it’s good. While the buffer encryption patch is still PoC
quality and there are some problems regarding nonce generation we need
to deal with, it easily can use the relation key managed by the kmgr
to encrypt/decrypt buffers.
Regards,
--
Masahiko Sawada
EnterpriseDB: https://www.enterprisedb.com/
Attachments:
0003-Poc-buffer-encryption.patch (application/octet-stream, 23.6K) download
From fba051add0f0ec73770db415bba5a1c486d41c27 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Mon, 11 Jan 2021 19:37:12 +0900
Subject: [PATCH 3/3] Poc: buffer encryption.
---
contrib/bloom/blinsert.c | 1 +
src/backend/access/hash/hashpage.c | 1 +
src/backend/access/heap/rewriteheap.c | 1 +
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/nbtree/nbtsort.c | 1 +
src/backend/access/spgist/spginsert.c | 3 +
src/backend/access/transam/xloginsert.c | 13 +++
src/backend/bootstrap/bootstrap.c | 3 +
src/backend/catalog/storage.c | 2 +-
src/backend/crypto/Makefile | 1 +
src/backend/crypto/bufenc.c | 138 ++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 2 +
src/backend/storage/buffer/bufmgr.c | 25 ++++-
src/backend/storage/page/bufpage.c | 49 ++++++++-
src/backend/tcop/postgres.c | 2 +
src/common/cipher_openssl.c | 26 +++++
src/include/access/xloginsert.h | 1 +
src/include/common/cipher.h | 3 +-
src/include/crypto/bufenc.h | 27 +++++
src/include/crypto/kmgr.h | 2 +
src/include/storage/bufpage.h | 11 +-
21 files changed, 306 insertions(+), 7 deletions(-)
create mode 100644 src/backend/crypto/bufenc.c
create mode 100644 src/include/crypto/bufenc.h
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index 32b5d62e1f..d474af753c 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -177,6 +177,7 @@ blbuildempty(Relation index)
* XLOG_DBASE_CREATE or XLOG_TBLSPC_CREATE record. Therefore, we need
* this even when wal_level=minimal.
*/
+ PageEncryptInplace(metapage, INIT_FORKNUM, BLOOM_METAPAGE_BLKNO);
PageSetChecksumInplace(metapage, BLOOM_METAPAGE_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, BLOOM_METAPAGE_BLKNO,
(char *) metapage, true);
diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c
index 49a9867787..7ee505c6ba 100644
--- a/src/backend/access/hash/hashpage.c
+++ b/src/backend/access/hash/hashpage.c
@@ -1025,6 +1025,7 @@ _hash_alloc_buckets(Relation rel, BlockNumber firstblock, uint32 nblocks)
true);
RelationOpenSmgr(rel);
+ PageEncryptInplace(page, MAIN_FORKNUM, lastblock);
PageSetChecksumInplace(page, lastblock);
smgrextend(rel->rd_smgr, MAIN_FORKNUM, lastblock, zerobuf.data, false);
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 29ffe40670..d752331706 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -325,6 +325,7 @@ end_heap_rewrite(RewriteState state)
true);
RelationOpenSmgr(state->rs_new_rel);
+ PageEncryptInplace(state->rs_buffer, MAIN_FORKNUM, state->rs_blockno);
PageSetChecksumInplace(state->rs_buffer, state->rs_blockno);
smgrextend(state->rs_new_rel->rd_smgr, MAIN_FORKNUM, state->rs_blockno,
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index ba79a7f3e9..b2969dd07e 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -175,6 +175,7 @@ btbuildempty(Relation index)
* XLOG_DBASE_CREATE or XLOG_TBLSPC_CREATE record. Therefore, we need
* this even when wal_level=minimal.
*/
+ PageEncryptInplace(metapage, INIT_FORKNUM, BTREE_METAPAGE);
PageSetChecksumInplace(metapage, BTREE_METAPAGE);
smgrwrite(index->rd_smgr, INIT_FORKNUM, BTREE_METAPAGE,
(char *) metapage, true);
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index c904fc8ef7..04da2d865b 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -665,6 +665,7 @@ _bt_blwritepage(BTWriteState *wstate, Page page, BlockNumber blkno)
true);
}
+ PageEncryptInplace(page, MAIN_FORKNUM, blkno);
PageSetChecksumInplace(page, blkno);
/*
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 2e1d8a33d1..84187583d2 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -168,6 +168,7 @@ spgbuildempty(Relation index)
* of their existing content when the corresponding create records are
* replayed.
*/
+ PageEncryptInplace(page, INIT_FORKNUM, SPGIST_METAPAGE_BLKNO);
PageSetChecksumInplace(page, SPGIST_METAPAGE_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_METAPAGE_BLKNO,
(char *) page, true);
@@ -177,6 +178,7 @@ spgbuildempty(Relation index)
/* Likewise for the root page. */
SpGistInitPage(page, SPGIST_LEAF);
+ PageEncryptInplace(page, INIT_FORKNUM, SPGIST_ROOT_BLKNO);
PageSetChecksumInplace(page, SPGIST_ROOT_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_ROOT_BLKNO,
(char *) page, true);
@@ -186,6 +188,7 @@ spgbuildempty(Relation index)
/* Likewise for the null-tuples root page. */
SpGistInitPage(page, SPGIST_LEAF | SPGIST_NULLS);
+ PageEncryptInplace(page, INIT_FORKNUM, SPGIST_NULL_BLKNO);
PageSetChecksumInplace(page, SPGIST_NULL_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_NULL_BLKNO,
(char *) page, true);
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..d7738c8834 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -1190,6 +1190,19 @@ log_newpage_range(Relation rel, ForkNumber forkNum,
}
}
+/* Write a no-op dummy xlog */
+XLogRecPtr
+log_noop(void)
+{
+ bool dummy;
+
+ XLogBeginInsert();
+
+ /* register dummy data as the record have to contain data */
+ XLogRegisterData((char *) &dummy, sizeof(bool));
+ return XLogInsert(RM_XLOG_ID, XLOG_NOOP);
+}
+
/*
* Allocate working buffers needed for WAL record construction.
*/
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index e2231f1430..9eba3735a5 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -28,6 +28,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_type.h"
#include "common/link-canary.h"
+#include "crypto/bufenc.h"
#include "crypto/kmgr.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
@@ -528,6 +529,8 @@ BootstrapModeMain(void)
InitPostgres(NULL, InvalidOid, NULL, InvalidOid, NULL, false);
+ InitializeBufferEncryption();
+
/* Initialize stuff for bootstrap-file processing */
for (i = 0; i < MAXATTR; i++)
{
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..c876c536a5 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -443,7 +443,7 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
smgrread(src, forkNum, blkno, buf.data);
- if (!PageIsVerifiedExtended(page, blkno,
+ if (!PageIsVerifiedExtended(page, forkNum, blkno,
PIV_LOG_WARNING | PIV_REPORT_STAT))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile
index c27362029d..8985a66875 100644
--- a/src/backend/crypto/Makefile
+++ b/src/backend/crypto/Makefile
@@ -13,6 +13,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ bufenc.o \
kmgr.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/crypto/bufenc.c b/src/backend/crypto/bufenc.c
new file mode 100644
index 0000000000..4401303fe9
--- /dev/null
+++ b/src/backend/crypto/bufenc.c
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * bufenc.c
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/crypto/bufenc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "lib/stringinfo.h"
+
+#include "storage/bufpage.h"
+#include "storage/fd.h"
+#include "crypto/kmgr.h"
+#include "crypto/bufenc.h"
+
+/*-------------------------------------------------------------------------
+ * We use both page LSN and page number to create a nonce for each page. Page
+ * LSN is 8 byte, page number is 4 byte, and the maximum required counter for
+ * AES-CTR is 2048, which fits in 3 byte. Since the length of IV is 16 byte
+ * it's fine. Using the LSN and page number as part of the nonce has
+ * three benefits:
+ *
+ * 1. We don't need to decrypt/re-encrypt during CREATE DATABASE since the page
+ * contents are the same in both places, and once one database changes its pages,
+ * it gets a new LSN, and hence a new nonce.
+ * 2. For each change of an 8k page, we get a new nonce, so we are not encrypting
+ * different data with the same nonce/IV.
+ * 3. We avoid requiring pg_upgrade to preserve database oids, tablespace oids,
+ * relfilenodes.
+ *
+ * Different two pages could have the same page LSN when heap update expires the
+ * old tuple and add the new tuple to the another page. By using the page number
+ * as a part of nonce we can get an unique nonce even in this case. We assume that
+ * we don't use the same LSN in different relations.
+ *-------------------------------------------------------------------------
+ */
+#define BUFENC_IV_SIZE 16
+
+static unsigned char buf_encryption_iv[BUFENC_IV_SIZE];
+
+PgCipherCtx *BufencCtx = NULL;
+PgCipherCtx *BufdecCtx = NULL;
+
+static void set_buffer_encryption_iv(Page page, BlockNumber blkno);
+
+void
+InitializeBufferEncryption(void)
+{
+ const CryptoKey *key;
+
+ if (!DataEncryptionEnabled())
+ return;
+
+ key = KmgrGetKey(KMGR_KEY_ID_REL);
+
+ BufencCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR,
+ (unsigned char *) key->key,
+ (file_encryption_keylen / 8), true);
+ if (!BufencCtx)
+ elog(ERROR, "could not intialize encryption context");
+
+ BufdecCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR,
+ (unsigned char *) key->key,
+ (file_encryption_keylen / 8), false);
+ if (!BufdecCtx)
+ elog(ERROR, "could not intialize decryption context");
+}
+
+/* Encrypt the given page with the relation key */
+void
+EncryptPage(Page page, BlockNumber blkno)
+{
+ unsigned char *ptr = (unsigned char *) page + PageEncryptOffset;
+ int enclen;
+
+ Assert(BufencCtx != NULL);
+
+ set_buffer_encryption_iv(page, blkno);
+ if (unlikely(!pg_cipher_encrypt(BufencCtx,
+ (const unsigned char *) ptr, /* input */
+ SizeOfPageEncryption,
+ ptr, /* length */
+ &enclen, /* resulting length */
+ buf_encryption_iv, /* iv */
+ BUFENC_IV_SIZE,
+ NULL, 0)))
+ elog(ERROR, "could not encrypt page %u", blkno);
+ Assert(enclen == SizeOfPageEncryption);
+}
+
+/* Decrypt the given page with the relation key */
+void
+DecryptPage(Page page, BlockNumber blkno)
+{
+ unsigned char *ptr = (unsigned char *) page + PageEncryptOffset;
+ int enclen;
+
+ Assert(BufdecCtx != NULL);
+
+ set_buffer_encryption_iv(page, blkno);
+ if (unlikely(!pg_cipher_decrypt(BufdecCtx,
+ (const unsigned char *) ptr, /* input */
+ SizeOfPageEncryption,
+ ptr, /* output */
+ &enclen, /* resulting length */
+ buf_encryption_iv, /* iv */
+ BUFENC_IV_SIZE,
+ NULL, 0)))
+ elog(ERROR, "could not decrypt page %u", blkno);
+ Assert(enclen == SizeOfPageEncryption);
+}
+
+/* Construct iv for the given page */
+static void
+set_buffer_encryption_iv(Page page, BlockNumber blkno)
+{
+ unsigned char *p = buf_encryption_iv;
+
+ MemSet(buf_encryption_iv, 0, BUFENC_IV_SIZE);
+
+ /* page lsn (8 byte) */
+ memcpy(p, &((PageHeader) page)->pd_lsn, sizeof(PageXLogRecPtr));
+ p += sizeof(PageXLogRecPtr);
+
+ /* block number (4 byte) */
+ memcpy(p, &blkno, sizeof(BlockNumber));
+ p += sizeof(BlockNumber);
+
+ /* remainig 4 bytes are the space for the counter, already set to 0 */
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index e2f8e07c88..c0424f3f36 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -100,6 +100,7 @@
#include "common/file_perm.h"
#include "common/ip.h"
#include "common/string.h"
+#include "crypto/bufenc.h"
#include "crypto/kmgr.h"
#include "lib/ilist.h"
#include "libpq/auth.h"
@@ -1333,6 +1334,7 @@ PostmasterMain(int argc, char *argv[])
RemovePgTempFiles();
InitializeKmgr();
+ InitializeBufferEncryption();
if (terminal_fd != -1)
close(terminal_fd);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 71b5852224..c86b6ea068 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -37,6 +37,7 @@
#include "access/xlog.h"
#include "catalog/catalog.h"
#include "catalog/storage.h"
+#include "crypto/bufenc.h"
#include "executor/instrument.h"
#include "lib/binaryheap.h"
#include "miscadmin.h"
@@ -918,7 +919,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
}
/* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, blockNum,
+ if (!PageIsVerifiedExtended((Page) bufBlock, forkNum, blockNum,
PIV_LOG_WARNING | PIV_REPORT_STAT))
{
if (mode == RBM_ZERO_ON_ERROR || zero_damaged_pages)
@@ -2792,12 +2793,15 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
*/
bufBlock = BufHdrGetBlock(buf);
+ bufToWrite = PageEncryptCopy((Page) bufBlock, buf->tag.forkNum,
+ buf->tag.blockNum);
+
/*
* Update page checksum if desired. Since we have only shared lock on the
* buffer, other processes might be updating hint bits in it, so we must
* copy the page to private storage if we do checksumming.
*/
- bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum);
+ bufToWrite = PageSetChecksumCopy((Page) bufToWrite, buf->tag.blockNum);
if (track_io_timing)
INSTR_TIME_SET_CURRENT(io_start);
@@ -3280,6 +3284,8 @@ FlushRelationBuffers(Relation rel)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ PageEncryptInplace(localpage, bufHdr->tag.forkNum,
+ bufHdr->tag.blockNum);
PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
smgrwrite(rel->rd_smgr,
@@ -3677,6 +3683,21 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
{
dirtied = true; /* Means "will be dirtied by this action" */
+ /*
+ * We will dirty the page but the page lsn is not changed if we
+ * doesn't write a backup block. We don't want to encrypt the
+ * different bits stream with the same combination of nonce and key
+ * since in buffer encryption the page lsn is a part of nonce.
+ * Therefore we WAL-log no-op record just to move page lsn forward if
+ * we doesn't write a backup block, even when this is not the first
+ * modification in this checkpoint round.
+ */
+ if (XLogRecPtrIsInvalid(lsn) && DataEncryptionEnabled())
+ {
+ lsn = log_noop();
+ Assert(!XLogRecPtrIsInvalid(lsn));
+ }
+
/*
* Set the page LSN if we wrote a backup block. We aren't supposed
* to set this when only holding a share lock but as long as we
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index 9ac556b4ae..3a480e7b82 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -17,6 +17,7 @@
#include "access/htup_details.h"
#include "access/itup.h"
#include "access/xlog.h"
+#include "crypto/bufenc.h"
#include "pgstat.h"
#include "storage/checksum.h"
#include "utils/memdebug.h"
@@ -85,7 +86,8 @@ PageInit(Page page, Size pageSize, Size specialSize)
* to pgstat.
*/
bool
-PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags)
+PageIsVerifiedExtended(Page page, ForkNumber forknum, BlockNumber blkno,
+ int flags)
{
PageHeader p = (PageHeader) page;
size_t *pagebytes;
@@ -108,6 +110,9 @@ PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags)
checksum_failure = true;
}
+ if (DataEncryptionEnabled())
+ PageDecryptInplace(page, forknum, blkno);
+
/*
* The following checks don't prove the header is correct, only that
* it looks sane enough to allow into the buffer pool. Later usage of
@@ -1427,3 +1432,45 @@ PageSetChecksumInplace(Page page, BlockNumber blkno)
((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blkno);
}
+
+char *
+PageEncryptCopy(Page page, ForkNumber forknum, BlockNumber blkno)
+{
+ static char *pageCopy = NULL;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum))
+ return (char *) page;
+
+ /*
+ * We allocate the copy space once and use it over on each subsequent
+ * call. The point of palloc'ing here, rather than having a static char
+ * array, is first to ensure adequate alignment for the checksumming code
+ * and second to avoid wasting space in processes that never call this.
+ */
+ if (pageCopy == NULL)
+ pageCopy = MemoryContextAlloc(TopMemoryContext, BLCKSZ);
+
+ memcpy(pageCopy, (char *) page, BLCKSZ);
+ EncryptPage(pageCopy, blkno);
+ return pageCopy;
+}
+
+void
+PageEncryptInplace(Page page, ForkNumber forknum, BlockNumber blkno)
+{
+ if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum))
+ return;
+
+ EncryptPage(page, blkno);
+}
+
+
+void
+PageDecryptInplace(Page page, ForkNumber forknum, BlockNumber blkno)
+{
+ if (PageIsNew(page) || PageNeedsToBeEncrypted(forknum))
+ return;
+
+ DecryptPage(page, blkno);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0463da6d46..36759c0190 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -43,6 +43,7 @@
#include "commands/async.h"
#include "commands/prepare.h"
#include "crypto/kmgr.h"
+#include "crypto/bufenc.h"
#include "executor/spi.h"
#include "jit/jit.h"
#include "libpq/libpq.h"
@@ -3978,6 +3979,7 @@ PostgresMain(int argc, char *argv[],
if (!IsUnderPostmaster)
{
InitializeKmgr();
+ InitializeBufferEncryption();
if (terminal_fd != -1)
close(terminal_fd);
diff --git a/src/common/cipher_openssl.c b/src/common/cipher_openssl.c
index 28f613cecc..9a6b4ddbc1 100644
--- a/src/common/cipher_openssl.c
+++ b/src/common/cipher_openssl.c
@@ -34,6 +34,7 @@ static ossl_EVP_cipher_func get_evp_aes_gcm(int klen);
static ossl_EVP_cipher_func get_evp_aes_kw(int klen);
static EVP_CIPHER_CTX *ossl_cipher_ctx_create(int cipher, unsigned char *key, int klen,
bool enc);
+static ossl_EVP_cipher_func get_evp_aes_ctr(int klen);
/*
* Return a newly created cipher context. 'cipher' specifies cipher algorithm
@@ -343,6 +344,26 @@ get_evp_aes_kwp(int klen)
}
}
+/*
+ * Returns the correct cipher functions for OpenSSL based
+ * on the key length requested.
+ */
+static ossl_EVP_cipher_func
+get_evp_aes_ctr(int klen)
+{
+ switch (klen)
+ {
+ case PG_AES128_KEY_LEN:
+ return EVP_aes_128_ctr;
+ case PG_AES192_KEY_LEN:
+ return EVP_aes_192_ctr;
+ case PG_AES256_KEY_LEN:
+ return EVP_aes_256_ctr;
+ default:
+ return NULL;
+ }
+}
+
/*
* Initialize and return an EVP_CIPHER_CTX. Returns NULL if the given
* cipher algorithm is not supported or on failure.
@@ -389,6 +410,11 @@ ossl_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc)
if (!func)
goto failed;
break;
+ case PG_CIPHER_AES_CTR:
+ func = get_evp_aes_ctr(klen);
+ if (!func)
+ goto failed;
+ break;
default:
goto failed;
}
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..43c4f1f611 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -59,6 +59,7 @@ extern void log_newpages(RelFileNode *rnode, ForkNumber forkNum, int num_pages,
extern XLogRecPtr log_newpage_buffer(Buffer buffer, bool page_std);
extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
+extern XLogRecPtr log_noop(void);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
extern void InitXLogInsert(void);
diff --git a/src/include/common/cipher.h b/src/include/common/cipher.h
index 7d446e86b6..9c4a630cd6 100644
--- a/src/include/common/cipher.h
+++ b/src/include/common/cipher.h
@@ -27,7 +27,8 @@
#define PG_CIPHER_AES_GCM 0
#define PG_CIPHER_AES_KW 1
#define PG_CIPHER_AES_KWP 2
-#define PG_MAX_CIPHER_ID 3
+#define PG_CIPHER_AES_CTR 3
+#define PG_MAX_CIPHER_ID 4
/* AES128/192/256 various length definitions */
#define PG_AES128_KEY_LEN (128 / 8)
diff --git a/src/include/crypto/bufenc.h b/src/include/crypto/bufenc.h
new file mode 100644
index 0000000000..97302a4f3f
--- /dev/null
+++ b/src/include/crypto/bufenc.h
@@ -0,0 +1,27 @@
+/*-------------------------------------------------------------------------
+ *
+ * bufenc.h
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/crypto/bufenc.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef BUFENC_H
+#define BUFENC_H
+
+#include "storage/bufmgr.h"
+#include "crypto/kmgr.h"
+
+#define DataEncryptionEnabled() (file_encryption_keylen > 0)
+
+/* Cluster encryption encrypts only main fork */
+#define PageNeedsToBeEncrypted(forknum) \
+ (DataEncryptionEnabled() && ((forknum) == MAIN_FORKNUM))
+
+extern void InitializeBufferEncryption(void);
+extern void EncryptPage(Page page, BlockNumber blkno);
+extern void DecryptPage(Page page, BlockNumber blkno);
+
+#endif /* BUFENC_H */
diff --git a/src/include/crypto/kmgr.h b/src/include/crypto/kmgr.h
index 386ac1cb4a..6a728ebefc 100644
--- a/src/include/crypto/kmgr.h
+++ b/src/include/crypto/kmgr.h
@@ -16,6 +16,8 @@
#include "storage/relfilenode.h"
#include "storage/bufpage.h"
+#define DataEncryptionEnabled() (file_encryption_keylen > 0)
+
/* GUC parameters */
extern int file_encryption_keylen;
extern char *cluster_key_command;
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 359b749f7f..fc855cc3b5 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -15,6 +15,7 @@
#define BUFPAGE_H
#include "access/xlogdefs.h"
+#include "common/relpath.h"
#include "storage/block.h"
#include "storage/item.h"
#include "storage/off.h"
@@ -164,6 +165,8 @@ typedef struct PageHeaderData
} PageHeaderData;
typedef PageHeaderData *PageHeader;
+#define PageEncryptOffset offsetof(PageHeaderData, pd_special)
+#define SizeOfPageEncryption (BLCKSZ - PageEncryptOffset)
/*
* pd_flags contains the following flag bits. Undefined bits are initialized
@@ -419,7 +422,7 @@ do { \
((is_heap) ? PAI_IS_HEAP : 0))
#define PageIsVerified(page, blkno) \
- PageIsVerifiedExtended(page, blkno, \
+ PageIsVerifiedExtended(page, MAIN_FORKNUM, blkno, \
PIV_LOG_WARNING | PIV_REPORT_STAT)
/*
@@ -433,7 +436,8 @@ StaticAssertDecl(BLCKSZ == ((BLCKSZ / sizeof(size_t)) * sizeof(size_t)),
"BLCKSZ has to be a multiple of sizeof(size_t)");
extern void PageInit(Page page, Size pageSize, Size specialSize);
-extern bool PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags);
+extern bool PageIsVerifiedExtended(Page page, ForkNumber forknum, BlockNumber blkno,
+ int flags);
extern OffsetNumber PageAddItemExtended(Page page, Item item, Size size,
OffsetNumber offsetNumber, int flags);
extern Page PageGetTempPage(Page page);
@@ -452,5 +456,8 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern char *PageEncryptCopy(Page page, ForkNumber forknum, BlockNumber blkno);
+extern void PageEncryptInplace(Page page, ForkNumber forknum, BlockNumber blkno);
+extern void PageDecryptInplace(Page page, ForkNumber forknum, BlockNumber blkno);
#endif /* BUFPAGE_H */
--
2.27.0
* Re: Key management with tests
@ 2021-01-11 17:22 Bruce Momjian <bruce@momjian.us>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-11 17:22 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote:
> On Sun, Jan 10, 2021 at 11:51 PM Bruce Momjian <bruce@momjian.us> wrote:
> > OK, here they are with numeric prefixes. It was actually tricky to
> > figure out how to create a squashed format-patch based on another branch.
>
> Thank you for attaching the patches. It passes all cfbot tests, great.
Yeah, I saw that. :-) I head to learn a lot about how to create
squashed format-patches on non-master branches. I have now automated it
so it will be easy going forward.
> Looking at the patch, it supports three algorithms but only
> PG_CIPHER_AES_KWP is used in the core for now:
>
> +/*
> + * Supported symmetric encryption algorithm. These identifiers are passed
> + * to pg_cipher_ctx_create() function, and then actual encryption
> + * implementations need to initialize their context of the given encryption
> + * algorithm.
> + */
> +#define PG_CIPHER_AES_GCM 0
> +#define PG_CIPHER_AES_KW 1
> +#define PG_CIPHER_AES_KWP 2
> +#define PG_MAX_CIPHER_ID 3
>
> Are we in the process of experimenting which algorithms are better? If
> we support one algorithm that is actually used in the core, we would
> reduce the tests as well.
I think we are only using KWP (Key Wrap with Padding) because that is
for wrapping keys:
https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/K...
I am not sure about KW. I think we are using GCM for the WAP/heap/index
pages. Stephen would know more.
> FWIW, I've written a PoC patch for buffer encryption to make sure the
> kms patch would be workable with other components using the encryption
> key managed by kmgr.
Wow, it is a small patch --- nice.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 17:54 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-11 17:54 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote:
> > Looking at the patch, it supports three algorithms but only
> > PG_CIPHER_AES_KWP is used in the core for now:
> >
> > +/*
> > + * Supported symmetric encryption algorithm. These identifiers are passed
> > + * to pg_cipher_ctx_create() function, and then actual encryption
> > + * implementations need to initialize their context of the given encryption
> > + * algorithm.
> > + */
> > +#define PG_CIPHER_AES_GCM 0
> > +#define PG_CIPHER_AES_KW 1
> > +#define PG_CIPHER_AES_KWP 2
> > +#define PG_MAX_CIPHER_ID 3
> >
> > Are we in the process of experimenting which algorithms are better? If
> > we support one algorithm that is actually used in the core, we would
> > reduce the tests as well.
>
> I think we are only using KWP (Key Wrap with Padding) because that is
> for wrapping keys:
>
> https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/K...
Yes.
> I am not sure about KW. I think we are using GCM for the WAP/heap/index
> pages. Stephen would know more.
KW was more-or-less 'for free' and there were tests for it, which is why
it was included. Yes, GCM would be for WAL/heap/index pages, it
wouldn't be appropriate to use KW or KWP for that. Using KW/KWP for the
key wrapping also makes the API simpler- and therefore easier for other
implementations to be written which provide the same API.
> > FWIW, I've written a PoC patch for buffer encryption to make sure the
> > kms patch would be workable with other components using the encryption
> > key managed by kmgr.
>
> Wow, it is a small patch --- nice.
I agree that the actual encryption patch, for just the main heap/index,
won't be too bad. The larger part will be dealing with all of the
temporary files we create that have user data in them... I've been
contemplating a way to try and make that part of the patch smaller
though and hopefully that will bear fruit and we can avoid having to
change a lof of, eg, reorderbuffer.c and pgstat.c.
There's a few places where we need to be sure to be updating the LSN for
both logged and unlogged relations properly, including dealing with
things like the magic GIST "GistBuildLSN" fake-LSN too, and we will
absolutely need to have a bit used in the IV to distinguish if it's a
real LSN or an unlogged LSN.
Although, another approach and one that I've discussed a bit with Bruce,
is to have more keys- such as a key for temporary files, and perhaps
even a key for logged relations and a different for unlogged.. Or
perhaps sets of keys for each which automatically are rotating every X
number of GB based on the LSN... Which is a big part of why key
management is such an important part of this effort.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 18:04 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-11 18:04 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote:
> Although, another approach and one that I've discussed a bit with Bruce,
> is to have more keys- such as a key for temporary files, and perhaps
> even a key for logged relations and a different for unlogged.. Or
Yes, we have to make sure the nonce (computed as LSN/pageno) is never
reused, so if we have several LSN usage "spaces", they need different
data keys.
> perhaps sets of keys for each which automatically are rotating every X
> number of GB based on the LSN... Which is a big part of why key
> management is such an important part of this effort.
Yes, this would avoid the need to failover to a standby for data key
rotation.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 18:23 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 114+ messages in thread
From: Stephen Frost @ 2021-01-11 18:23 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote:
> > Although, another approach and one that I've discussed a bit with Bruce,
> > is to have more keys- such as a key for temporary files, and perhaps
> > even a key for logged relations and a different for unlogged.. Or
>
> Yes, we have to make sure the nonce (computed as LSN/pageno) is never
> reused, so if we have several LSN usage "spaces", they need different
> data keys.
Right, or ensure that the actual IV used is distinct (such as by using
another bit in the IV to distinguish logged-vs-unlogged), but it seems
saner to just use a different key, ultimately.
> > perhaps sets of keys for each which automatically are rotating every X
> > number of GB based on the LSN... Which is a big part of why key
> > management is such an important part of this effort.
>
> Yes, this would avoid the need to failover to a standby for data key
> rotation.
Yes, and it avoids the issue of using a single key for too much, which
is also a concern. The remaining larger issues are to figure out a
place to put the tag for each page, and the relatively simple matter of
programming a mechanism to cache the keys we're commonly using (current
key for encryption, recently used keys for decryption) since we'll
eventually get to a point of having written out more data than we are
going to keep keys in memory for.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 19:06 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-11 19:06 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote:
> Greetings,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote:
> > > Although, another approach and one that I've discussed a bit with Bruce,
> > > is to have more keys- such as a key for temporary files, and perhaps
> > > even a key for logged relations and a different for unlogged.. Or
> >
> > Yes, we have to make sure the nonce (computed as LSN/pageno) is never
> > reused, so if we have several LSN usage "spaces", they need different
> > data keys.
>
> Right, or ensure that the actual IV used is distinct (such as by using
> another bit in the IV to distinguish logged-vs-unlogged), but it seems
> saner to just use a different key, ultimately.
Yes, we have eight unused bit in the Nonce right now.
> > > perhaps sets of keys for each which automatically are rotating every X
> > > number of GB based on the LSN... Which is a big part of why key
> > > management is such an important part of this effort.
> >
> > Yes, this would avoid the need to failover to a standby for data key
> > rotation.
>
> Yes, and it avoids the issue of using a single key for too much, which
> is also a concern. The remaining larger issues are to figure out a
> place to put the tag for each page, and the relatively simple matter of
> programming a mechanism to cache the keys we're commonly using (current
> key for encryption, recently used keys for decryption) since we'll
> eventually get to a point of having written out more data than we are
> going to keep keys in memory for.
I thought the LSN range would be stored with the keys, so there is no
need to tag the LSN on each page.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 19:19 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-11 19:19 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote:
> > Yes, and it avoids the issue of using a single key for too much, which
> > is also a concern. The remaining larger issues are to figure out a
> > place to put the tag for each page, and the relatively simple matter of
> > programming a mechanism to cache the keys we're commonly using (current
> > key for encryption, recently used keys for decryption) since we'll
> > eventually get to a point of having written out more data than we are
> > going to keep keys in memory for.
>
> I thought the LSN range would be stored with the keys, so there is no
> need to tag the LSN on each page.
Yes, LSN range would be stored with the keys in some fashion (maybe just
the start of a particular LSN range would be in the filename of the key
for that range...). The 'tag' that I'm referring to there is one of the
outputs from the GCM encryption and is what provides the integrity /
authentication of the encrypted data to be able to detect if it's been
modified. Unfortunately, while the page checksum will continue to be
used and available for checking against disk corruption, it's not
sufficient. Hence, ideally, we'd find a spot to stick the 128-bit tag
on each page.
Given that, clearly, it's not possible to go from an unencrypted cluster
to an encrypted cluster without rewriting the entire cluster, we aren't
bound to maintain the on-disk page format, we should be able to
accomadate including the tag somewhere. Unfortuantely, it doesn't seem
quite as trivial as I'd hoped since there are parts of the code which
make assumptions about the page beyond perhaps what they should be, but
I'm still hopeful that it won't be *too* hard to do.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 20:22 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-11 20:22 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote:
> Greetings,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote:
> > > Yes, and it avoids the issue of using a single key for too much, which
> > > is also a concern. The remaining larger issues are to figure out a
> > > place to put the tag for each page, and the relatively simple matter of
> > > programming a mechanism to cache the keys we're commonly using (current
> > > key for encryption, recently used keys for decryption) since we'll
> > > eventually get to a point of having written out more data than we are
> > > going to keep keys in memory for.
> >
> > I thought the LSN range would be stored with the keys, so there is no
> > need to tag the LSN on each page.
>
> Yes, LSN range would be stored with the keys in some fashion (maybe just
> the start of a particular LSN range would be in the filename of the key
> for that range...). The 'tag' that I'm referring to there is one of the
Oh, that tag, yes, we need to add that to each page. I thought you mean
an LSN-range-key tag.
> outputs from the GCM encryption and is what provides the integrity /
> authentication of the encrypted data to be able to detect if it's been
> modified. Unfortunately, while the page checksum will continue to be
> used and available for checking against disk corruption, it's not
> sufficient. Hence, ideally, we'd find a spot to stick the 128-bit tag
> on each page.
Agreed. Would checksums be of any value with GCM?
> Given that, clearly, it's not possible to go from an unencrypted cluster
> to an encrypted cluster without rewriting the entire cluster, we aren't
> bound to maintain the on-disk page format, we should be able to
> accomadate including the tag somewhere. Unfortuantely, it doesn't seem
> quite as trivial as I'd hoped since there are parts of the code which
> make assumptions about the page beyond perhaps what they should be, but
> I'm still hopeful that it won't be *too* hard to do.
OK, thanks. Are there other page improvements we should make when we
are requiring a page rewrite?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-11 20:52 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-11 20:52 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote:
> > outputs from the GCM encryption and is what provides the integrity /
> > authentication of the encrypted data to be able to detect if it's been
> > modified. Unfortunately, while the page checksum will continue to be
> > used and available for checking against disk corruption, it's not
> > sufficient. Hence, ideally, we'd find a spot to stick the 128-bit tag
> > on each page.
>
> Agreed. Would checksums be of any value with GCM?
The value would be to allow testing of the database integrity, to the
amount allowed by the checksum, to be done without having access to the
encryption keys, and because there's not much else we'd be using those
bits for if we didn't.
> > Given that, clearly, it's not possible to go from an unencrypted cluster
> > to an encrypted cluster without rewriting the entire cluster, we aren't
> > bound to maintain the on-disk page format, we should be able to
> > accomadate including the tag somewhere. Unfortuantely, it doesn't seem
> > quite as trivial as I'd hoped since there are parts of the code which
> > make assumptions about the page beyond perhaps what they should be, but
> > I'm still hopeful that it won't be *too* hard to do.
>
> OK, thanks. Are there other page improvements we should make when we
> are requiring a page rewrite?
This is an interesting question but ultimately I don't think we should
be looking at this from the perspective of allowing arbitrary changes to
the page format. The challenge is that much of the page format, today,
is defined by a C struct and changing the way that works would require a
great deal of code to be modified and turn this into a massive effort,
assuming we wish to have the same compiled binary able to work with both
unencrypted and encrypted clusters, which I do believe is a requirement.
The thought that I had was to, instead, try to figure out if we could
fudge some space by, say, putting a 128-bit 'hole' at the end of the
page and just move pd_special back, effectively making the page seem
'smaller' to all of the code that uses it, except for the code that
knows how to do the decryption. I ran into some trouble with that but
haven't quite sorted out what happened yet. Other ideas would be to put
it before pd_special, or maybe somewhere else, but a lot depends on the
code's expectations.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 00:32 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 114+ messages in thread
From: Masahiko Sawada @ 2021-01-12 00:32 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost <sfrost@snowman.net> wrote:
>
> Greetings,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote:
> > > Although, another approach and one that I've discussed a bit with Bruce,
> > > is to have more keys- such as a key for temporary files, and perhaps
> > > even a key for logged relations and a different for unlogged.. Or
> >
> > Yes, we have to make sure the nonce (computed as LSN/pageno) is never
> > reused, so if we have several LSN usage "spaces", they need different
> > data keys.
>
> Right, or ensure that the actual IV used is distinct (such as by using
> another bit in the IV to distinguish logged-vs-unlogged), but it seems
> saner to just use a different key, ultimately.
Agreed.
I think we also need to consider how to make sure nonce is unique when
making a page dirty by updating hint bits. Hint bit update changes the
page contents but doesn't change the page lsn if we already write a
full page write. In the PoC patch, I logged a dummy WAL record
(XLOG_NOOP) just to move the page lsn forward, but since this is
required even when changing the page is not the first time since the
last checkpoint we might end up logging too many dummy WAL records.
Regards,
--
Masahiko Sawada
EnterpriseDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 02:09 Bruce Momjian <bruce@momjian.us>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 02:09 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote:
> On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost <sfrost@snowman.net> wrote:
> > Right, or ensure that the actual IV used is distinct (such as by using
> > another bit in the IV to distinguish logged-vs-unlogged), but it seems
> > saner to just use a different key, ultimately.
>
> Agreed.
>
> I think we also need to consider how to make sure nonce is unique when
> making a page dirty by updating hint bits. Hint bit update changes the
> page contents but doesn't change the page lsn if we already write a
> full page write. In the PoC patch, I logged a dummy WAL record
> (XLOG_NOOP) just to move the page lsn forward, but since this is
> required even when changing the page is not the first time since the
> last checkpoint we might end up logging too many dummy WAL records.
This says:
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
wal_log_hints will be enabled automatically in encryption mode.
Does that help?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 03:40 Neil Chen <carpenter.nail.cz@gmail.com>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Neil Chen @ 2021-01-12 03:40 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi Stephen,
On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost <sfrost@snowman.net> wrote:
>
> This is an interesting question but ultimately I don't think we should
> be looking at this from the perspective of allowing arbitrary changes to
> the page format. The challenge is that much of the page format, today,
> is defined by a C struct and changing the way that works would require a
> great deal of code to be modified and turn this into a massive effort,
> assuming we wish to have the same compiled binary able to work with both
> unencrypted and encrypted clusters, which I do believe is a requirement.
>
> The thought that I had was to, instead, try to figure out if we could
> fudge some space by, say, putting a 128-bit 'hole' at the end of the
> page and just move pd_special back, effectively making the page seem
> 'smaller' to all of the code that uses it, except for the code that
> knows how to do the decryption. I ran into some trouble with that but
> haven't quite sorted out what happened yet. Other ideas would be to put
> it before pd_special, or maybe somewhere else, but a lot depends on the
> code's expectations.
>
>
I agree that we should not make too many changes to affect the use of
unencrypted clusters. But as a personal opinion only, I don't think it's a
good idea to add some "implicit" tricks. To provide an inspiration, can we
add a flag to mark whether the page format has been changed:
inline patch 1: src/include/storage/bufpage.h (+9/-1 lines) download inline-1.patch
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -181,8 +185,9 @@ typedef PageHeaderData *PageHeader;
#define PD_PAGE_FULL 0x0002 /* not enough free space for new tuple? */
#define PD_ALL_VISIBLE 0x0004 /* all tuples on page are visible to
* everyone */
+#define PD_PAGE_ENCRYPTED 0x0008 /* Is page encrypted? */
-#define PD_VALID_FLAG_BITS 0x0007 /* OR of all valid pd_flags bits */
+#define PD_VALID_FLAG_BITS 0x000F /* OR of all valid pd_flags bits */
/*
* Page layout version number 0 is for pre-7.3 Postgres releases.
@@ -389,6 +394,13 @@ PageValidateSpecialPointer(Page page)
#define PageClearAllVisible(page) \
(((PageHeader) (page))->pd_flags &= ~PD_ALL_VISIBLE)
+#define PageIsEncrypted(page) \
+ (((PageHeader) (page))->pd_flags & PD_PAGE_ENCRYPTED)
+#define PageSetEncrypted(page) \
+ (((PageHeader) (page))->pd_flags |= PD_PAGE_ENCRYPTED)
+#define PageClearEncrypted(page) \
+ (((PageHeader) (page))->pd_flags &= ~PD_PAGE_ENCRYPTED)
+
#define PageIsPrunable(page, oldestxmin) \
( \
AssertMacro(TransactionIdIsNormal(oldestxmin)), \
* Re: Key management with tests
@ 2021-01-12 12:40 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Masahiko Sawada @ 2021-01-12 12:40 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Stephen Frost <sfrost@snowman.net>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 11:09 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote:
> > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost <sfrost@snowman.net> wrote:
> > > Right, or ensure that the actual IV used is distinct (such as by using
> > > another bit in the IV to distinguish logged-vs-unlogged), but it seems
> > > saner to just use a different key, ultimately.
> >
> > Agreed.
> >
> > I think we also need to consider how to make sure nonce is unique when
> > making a page dirty by updating hint bits. Hint bit update changes the
> > page contents but doesn't change the page lsn if we already write a
> > full page write. In the PoC patch, I logged a dummy WAL record
> > (XLOG_NOOP) just to move the page lsn forward, but since this is
> > required even when changing the page is not the first time since the
> > last checkpoint we might end up logging too many dummy WAL records.
>
> This says:
>
> https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
>
> wal_log_hints will be enabled automatically in encryption mode.
>
> Does that help?
IIUC it helps but not enough. When wal_log_hints is enabled, we write
a full-page image when updating hint bits if it's the first time
change for the page since the last checkpoint. But I'm concerned that
what if we change hint bits again after the page is flushed. We would
mark the page as dirtied but not write any WAL, leaving the page lsn
as it is.
Regards,
--
Masahiko Sawada
EnterpriseDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 16:08 Stephen Frost <sfrost@snowman.net>
parent: Neil Chen <carpenter.nail.cz@gmail.com>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-12 16:08 UTC (permalink / raw)
To: Neil Chen <carpenter.nail.cz@gmail.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Neil Chen (carpenter.nail.cz@gmail.com) wrote:
> On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost <sfrost@snowman.net> wrote:
> > This is an interesting question but ultimately I don't think we should
> > be looking at this from the perspective of allowing arbitrary changes to
> > the page format. The challenge is that much of the page format, today,
> > is defined by a C struct and changing the way that works would require a
> > great deal of code to be modified and turn this into a massive effort,
> > assuming we wish to have the same compiled binary able to work with both
> > unencrypted and encrypted clusters, which I do believe is a requirement.
> >
> > The thought that I had was to, instead, try to figure out if we could
> > fudge some space by, say, putting a 128-bit 'hole' at the end of the
> > page and just move pd_special back, effectively making the page seem
> > 'smaller' to all of the code that uses it, except for the code that
> > knows how to do the decryption. I ran into some trouble with that but
> > haven't quite sorted out what happened yet. Other ideas would be to put
> > it before pd_special, or maybe somewhere else, but a lot depends on the
> > code's expectations.
>
> I agree that we should not make too many changes to affect the use of
> unencrypted clusters. But as a personal opinion only, I don't think it's a
> good idea to add some "implicit" tricks. To provide an inspiration, can we
> add a flag to mark whether the page format has been changed:
Sure, of course we could add such a flag, but I don't see how that would
actually help with the issue?
> In this way, I think it has little effect on the unencrypted cluster, and
> we can also modify the page format as we wish. Of course, it's also
> possible that I didn't understand your design correctly, or there's
> something wrong with my idea. :D
No, we can't 'modify the page format as we wish'- if we change away from
using a C structure then we're going to be modifying quite a bit of
code which otherwise doesn't need to be changed. The proposed flag
doesn't actually make a different page format work, the only thing it
would do would be to allow some parts of the cluster to be encrypted and
other parts not be, but I don't know that that's actually a useful
capability or a good reason to use one of those bits. Having it handled
on a cluster level, at initdb time through pg_control, seems like it'd
work just fine.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 17:04 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 17:04 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 10, 2021 at 09:51:16AM -0500, Bruce Momjian wrote:
> On Sun, Jan 10, 2021 at 06:04:12PM +1300, Thomas Munro wrote:
> > On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian <bruce@momjian.us> wrote:
> > > Does anyone know why the cfbot applied the patch listed second first
> > > here?
> > >
> > > http://cfbot.cputube.org/patch_31_2925.log
> > >
> > > Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed
> > > it would apply attachments in the order they appear in the email.
> >
> > It sorts the filenames (in this case after decompressing step removes
> > the .gz endings). That works pretty well for the patches that "git
> > format-patch" spits out, but it's a bit hit and miss with cases like
> > yours.
>
> OK, here they are with numeric prefixes. It was actually tricky to
> figure out how to create a squashed format-patch based on another branch.
Here is an updated version built on top of Michael Paquier's patch
posted here:
https://www.postgresql.org/message-id/X/0IChOPHd+aYC1w@paquier.xyz
and included as my first attachment. This will give Michael's patch
cfbot testing too since the second attachment calls many of the first
attachment's functions.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
1-hex.diff.gz (application/gzip, 5.6K) download
2-key_over_1-hex.diff.gz (application/gzip, 120.9K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:03 Bruce Momjian <bruce@momjian.us>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 18:03 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote:
> > This says:
> >
> > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
> >
> > wal_log_hints will be enabled automatically in encryption mode.
> >
> > Does that help?
>
> IIUC it helps but not enough. When wal_log_hints is enabled, we write
> a full-page image when updating hint bits if it's the first time
> change for the page since the last checkpoint. But I'm concerned that
> what if we change hint bits again after the page is flushed. We would
> mark the page as dirtied but not write any WAL, leaving the page lsn
> as it is.
I updated the wiki to be:
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
wal_log_hints will be enabled automatically in encryption mode. However,
more than one hit change between checkpoints does not cause WAL
activity, which would cause the same LSN to be used for different pages
images.
I think one big question is that, since we are using a streaming cipher,
do we care about hint bit changes showing to users? I actually don't
know. If we do, some kind of dummy LSN record might be required, as you
suggested.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:11 Andres Freund <andres@anarazel.de>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 114+ messages in thread
From: Andres Freund @ 2021-01-12 18:11 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Stephen Frost <sfrost@snowman.net>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On 2021-01-12 13:03:14 -0500, Bruce Momjian wrote:
> I think one big question is that, since we are using a streaming cipher,
> do we care about hint bit changes showing to users? I actually don't
> know. If we do, some kind of dummy LSN record might be required, as you
> suggested.
That'd lead to a *massive* increase of WAL record volume. It's one thing
to WAL log hint bit writes once per page per checkpoint. It's another to
do so on every single hint bit write.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:11 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-12 18:11 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote:
> > > This says:
> > >
> > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
> > >
> > > wal_log_hints will be enabled automatically in encryption mode.
> > >
> > > Does that help?
> >
> > IIUC it helps but not enough. When wal_log_hints is enabled, we write
> > a full-page image when updating hint bits if it's the first time
> > change for the page since the last checkpoint. But I'm concerned that
> > what if we change hint bits again after the page is flushed. We would
> > mark the page as dirtied but not write any WAL, leaving the page lsn
> > as it is.
>
> I updated the wiki to be:
>
> https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
>
> wal_log_hints will be enabled automatically in encryption mode. However,
> more than one hit change between checkpoints does not cause WAL
> activity, which would cause the same LSN to be used for different pages
> images.
>
> I think one big question is that, since we are using a streaming cipher,
> do we care about hint bit changes showing to users? I actually don't
> know. If we do, some kind of dummy LSN record might be required, as you
> suggested.
I don't think there's any doubt that we need to make sure that the IV is
distinct and advancing the LSN to get a new one when needed for this
case seems like it's probably the way to do that. Hint bit change
visibility to users isn't really at issue here- we can't use the same IV
multiple times. The two options that we have are to either not actually
update the hint bit in such a case, or to make sure to change the
LSN/IV. Another option would be to, if we're able to make a hole to put
the GCM tag on to the page somewhere, further widen that hole to include
an additional space for a counter that would be mixed into the IV, to
avoid having to do an XLOG NOOP.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:15 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 18:15 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote:
> > I think one big question is that, since we are using a streaming cipher,
> > do we care about hint bit changes showing to users? I actually don't
> > know. If we do, some kind of dummy LSN record might be required, as you
> > suggested.
>
> I don't think there's any doubt that we need to make sure that the IV is
> distinct and advancing the LSN to get a new one when needed for this
> case seems like it's probably the way to do that. Hint bit change
> visibility to users isn't really at issue here- we can't use the same IV
> multiple times. The two options that we have are to either not actually
> update the hint bit in such a case, or to make sure to change the
> LSN/IV. Another option would be to, if we're able to make a hole to put
> the GCM tag on to the page somewhere, further widen that hole to include
> an additional space for a counter that would be mixed into the IV, to
> avoid having to do an XLOG NOOP.
Well, we have eight unused bits in the IV, so we could just increment
that for every hint bit change that uses the same LSN, and then force a
dummy WAL record when that 8-bit counter overflows --- that seems
simpler than logging hint bits.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:44 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-12 18:44 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote:
> > > I think one big question is that, since we are using a streaming cipher,
> > > do we care about hint bit changes showing to users? I actually don't
> > > know. If we do, some kind of dummy LSN record might be required, as you
> > > suggested.
> >
> > I don't think there's any doubt that we need to make sure that the IV is
> > distinct and advancing the LSN to get a new one when needed for this
> > case seems like it's probably the way to do that. Hint bit change
> > visibility to users isn't really at issue here- we can't use the same IV
> > multiple times. The two options that we have are to either not actually
> > update the hint bit in such a case, or to make sure to change the
> > LSN/IV. Another option would be to, if we're able to make a hole to put
> > the GCM tag on to the page somewhere, further widen that hole to include
> > an additional space for a counter that would be mixed into the IV, to
> > avoid having to do an XLOG NOOP.
>
> Well, we have eight unused bits in the IV, so we could just increment
> that for every hint bit change that uses the same LSN, and then force a
> dummy WAL record when that 8-bit counter overflows --- that seems
> simpler than logging hint bits.
Sure, as long as we have a place to store that information.. We need to
have the full IV available when we go to decrypt the page.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:46 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 18:46 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote:
> On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote:
> > I don't think there's any doubt that we need to make sure that the IV is
> > distinct and advancing the LSN to get a new one when needed for this
> > case seems like it's probably the way to do that. Hint bit change
> > visibility to users isn't really at issue here- we can't use the same IV
> > multiple times. The two options that we have are to either not actually
> > update the hint bit in such a case, or to make sure to change the
> > LSN/IV. Another option would be to, if we're able to make a hole to put
> > the GCM tag on to the page somewhere, further widen that hole to include
> > an additional space for a counter that would be mixed into the IV, to
> > avoid having to do an XLOG NOOP.
>
> Well, we have eight unused bits in the IV, so we could just increment
> that for every hint bit change that uses the same LSN, and then force a
> dummy WAL record when that 8-bit counter overflows --- that seems
> simpler than logging hint bits.
Sorry, I was incorrect. The IV is 16 bytes, made up of the LSN (8
bytes), and the page number (4 bytes). That leaves 4 bytes unused or
2^32 values for hint bit changes before we have to generate a dummy LSN
record.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:51 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 18:51 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> > Well, we have eight unused bits in the IV, so we could just increment
> > that for every hint bit change that uses the same LSN, and then force a
> > dummy WAL record when that 8-bit counter overflows --- that seems
> > simpler than logging hint bits.
>
> Sure, as long as we have a place to store that information.. We need to
> have the full IV available when we go to decrypt the page.
Oh, yeah, we would need that counter recorded since previously the IV
was made up of already-recorded information, i.e., the page LSN and page
number. However, the reason don't WAL-log hint bits always is because
we can afford to lose them, but in this case, any counter we need to
store will need to be WAL logged since we can't affort to lose that
counter value for decryption --- that gets us back to WAL-logging
something during hint bit changes. :-(
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 18:57 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-01-12 18:57 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > Well, we have eight unused bits in the IV, so we could just increment
> > > that for every hint bit change that uses the same LSN, and then force a
> > > dummy WAL record when that 8-bit counter overflows --- that seems
> > > simpler than logging hint bits.
> >
> > Sure, as long as we have a place to store that information.. We need to
> > have the full IV available when we go to decrypt the page.
>
> Oh, yeah, we would need that counter recorded since previously the IV
> was made up of already-recorded information, i.e., the page LSN and page
> number. However, the reason don't WAL-log hint bits always is because
> we can afford to lose them, but in this case, any counter we need to
> store will need to be WAL logged since we can't affort to lose that
> counter value for decryption --- that gets us back to WAL-logging
> something during hint bit changes. :-(
I don't think that's actually the case..? The hole I'm talking about is
there exclusively for post-encryption storage of the tag and maybe this
part of the IV and would be zero'd out in the FPIs that actually go into
the WAL (which would be encrypted with the WAL key, not the data key).
All we would need to be confident of is that if the page with the hint
bit update gets encrypted and written out that the IV counter gets
incremented and also written out as part of that write.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 19:32 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-12 19:32 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 01:57:11PM -0500, Stephen Frost wrote:
> Greetings,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote:
> > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > Well, we have eight unused bits in the IV, so we could just increment
> > > > that for every hint bit change that uses the same LSN, and then force a
> > > > dummy WAL record when that 8-bit counter overflows --- that seems
> > > > simpler than logging hint bits.
> > >
> > > Sure, as long as we have a place to store that information.. We need to
> > > have the full IV available when we go to decrypt the page.
> >
> > Oh, yeah, we would need that counter recorded since previously the IV
> > was made up of already-recorded information, i.e., the page LSN and page
> > number. However, the reason don't WAL-log hint bits always is because
> > we can afford to lose them, but in this case, any counter we need to
> > store will need to be WAL logged since we can't affort to lose that
> > counter value for decryption --- that gets us back to WAL-logging
> > something during hint bit changes. :-(
>
> I don't think that's actually the case..? The hole I'm talking about is
> there exclusively for post-encryption storage of the tag and maybe this
> part of the IV and would be zero'd out in the FPIs that actually go into
> the WAL (which would be encrypted with the WAL key, not the data key).
> All we would need to be confident of is that if the page with the hint
> bit update gets encrypted and written out that the IV counter gets
> incremented and also written out as part of that write.
OK, got it. I have added this to the wiki:
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
wal_log_hints will be enabled automatically in encryption mode. However,
more than one hit change between checkpoints does not cause WAL
activity, which would cause the same LSN to be used for different page
images. This means we need a page-stored counter, to be used in the four
unused bytes of the IV. This prevents multiple page writes during the
same checkpoint interval from using the same IV. Counter changes do not
need to be WAL logged since we either get the page from the WAL (which
is only encrypted with the WAL data key), or from disk, which is
durable.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-12 19:34 Andres Freund <andres@anarazel.de>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 114+ messages in thread
From: Andres Freund @ 2021-01-12 19:34 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi,
On 2021-01-11 20:12:00 +0900, Masahiko Sawada wrote:
> diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
> index 32b5d62e1f..d474af753c 100644
> --- a/contrib/bloom/blinsert.c
> +++ b/contrib/bloom/blinsert.c
> @@ -177,6 +177,7 @@ blbuildempty(Relation index)
> * XLOG_DBASE_CREATE or XLOG_TBLSPC_CREATE record. Therefore, we need
> * this even when wal_level=minimal.
> */
> + PageEncryptInplace(metapage, INIT_FORKNUM, BLOOM_METAPAGE_BLKNO);
> PageSetChecksumInplace(metapage, BLOOM_METAPAGE_BLKNO);
> smgrwrite(index->rd_smgr, INIT_FORKNUM, BLOOM_METAPAGE_BLKNO,
> (char *) metapage, true);
There's quite a few places doing encryption + checksum + smgwrite now. I
strongly suggest splitting that off into a helper routine in a
preparatory patch.
> @@ -528,6 +529,8 @@ BootstrapModeMain(void)
>
> InitPostgres(NULL, InvalidOid, NULL, InvalidOid, NULL, false);
>
> + InitializeBufferEncryption();
> +
> /* Initialize stuff for bootstrap-file processing */
> for (i = 0; i < MAXATTR; i++)
> {
Why are we initializing this here instead of postmaster? As far as I can
tell that just leads to redundant work instead of doing it once?
> +/*-------------------------------------------------------------------------
> + * We use both page LSN and page number to create a nonce for each page. Page
> + * LSN is 8 byte, page number is 4 byte, and the maximum required counter for
> + * AES-CTR is 2048, which fits in 3 byte. Since the length of IV is 16 byte
> + * it's fine. Using the LSN and page number as part of the nonce has
> + * three benefits:
> + *
> + * 1. We don't need to decrypt/re-encrypt during CREATE DATABASE since the page
> + * contents are the same in both places, and once one database changes its pages,
> + * it gets a new LSN, and hence a new nonce.
> + * 2. For each change of an 8k page, we get a new nonce, so we are not encrypting
> + * different data with the same nonce/IV.
> + * 3. We avoid requiring pg_upgrade to preserve database oids, tablespace oids,
> + * relfilenodes.
I think 3) also has a few minor downsides - by not including information
identifying a relation a potential attacker with access to the data
directory has more chances to get the database to decrypt data by
e.g. switching relation files around.
> @@ -2792,12 +2793,15 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
> */
> bufBlock = BufHdrGetBlock(buf);
>
> + bufToWrite = PageEncryptCopy((Page) bufBlock, buf->tag.forkNum,
> + buf->tag.blockNum);
> +
> /*
> * Update page checksum if desired. Since we have only shared lock on the
> * buffer, other processes might be updating hint bits in it, so we must
> * copy the page to private storage if we do checksumming.
> */
> - bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum);
> + bufToWrite = PageSetChecksumCopy((Page) bufToWrite, buf->tag.blockNum);
>
> if (track_io_timing)
> INSTR_TIME_SET_CURRENT(io_start);
So now we copy the page twice, not just once, if both checksums and
encryption is enabled? That doesn't seem right.
> @@ -3677,6 +3683,21 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
> {
> dirtied = true; /* Means "will be dirtied by this action" */
>
> + /*
> + * We will dirty the page but the page lsn is not changed if we
> + * doesn't write a backup block. We don't want to encrypt the
> + * different bits stream with the same combination of nonce and key
> + * since in buffer encryption the page lsn is a part of nonce.
> + * Therefore we WAL-log no-op record just to move page lsn forward if
> + * we doesn't write a backup block, even when this is not the first
> + * modification in this checkpoint round.
> + */
> + if (XLogRecPtrIsInvalid(lsn) && DataEncryptionEnabled())
> + {
> + lsn = log_noop();
> + Assert(!XLogRecPtrIsInvalid(lsn));
> + }
> +
Aren't you doing a WAL record while holding the buffer header lock here?
You can't do things like WAL insertions while holding a spinlock.
I don't see how it is safe / correct to use a noop record here. A noop
record isn't associated with the page, so WAL replay isn't going to
perform the same LSN modification.
Also, why is it OK to modify the LSN without, if necessary, logging an FPI?
> +char *
> +PageEncryptCopy(Page page, ForkNumber forknum, BlockNumber blkno)
> +{
> + static char *pageCopy = NULL;
> +
> + /* If we don't need a checksum, just return the passed-in data */
> + if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum))
> + return (char *) page;
Why is it OK to not encrypt new pages?
> +#define PageEncryptOffset offsetof(PageHeaderData, pd_special)
> +#define SizeOfPageEncryption (BLCKSZ - PageEncryptOffset)
I think you need a detailed explanation somewhere about what you're
doing here, and why it's a good idea.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-13 02:26 Neil Chen <carpenter.nail.cz@gmail.com>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 114+ messages in thread
From: Neil Chen @ 2021-01-13 02:26 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Thank you for your reply，
On Wed, Jan 13, 2021 at 12:08 AM Stephen Frost <sfrost@snowman.net> wrote:
>
> No, we can't 'modify the page format as we wish'- if we change away from
> using a C structure then we're going to be modifying quite a bit of
> code which otherwise doesn't need to be changed. The proposed flag
> doesn't actually make a different page format work, the only thing it
> would do would be to allow some parts of the cluster to be encrypted and
> other parts not be, but I don't know that that's actually a useful
> capability or a good reason to use one of those bits. Having it handled
> on a cluster level, at initdb time through pg_control, seems like it'd
> work just fine.
>
>
Yes, I realized that for cluster-level encryption, it would be unwise to
flag a single page(Unless we want to do it at relation-level). Forgive me
for not describing clearly, the 'modify the page' I said means the method
you mentioned, not modifying the C structure. My original motivation is to
avoid storing in an unconventional format without a description of the C
structure. However, as I just said, it seems that we should not set
the flag for a single page. Maybe it's enough to just add a comment
description?
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-14 00:49 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-14 00:49 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 01:46:53PM -0500, Bruce Momjian wrote:
> On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote:
> > Well, we have eight unused bits in the IV, so we could just increment
> > that for every hint bit change that uses the same LSN, and then force a
> > dummy WAL record when that 8-bit counter overflows --- that seems
> > simpler than logging hint bits.
>
> Sorry, I was incorrect. The IV is 16 bytes, made up of the LSN (8
> bytes), and the page number (4 bytes). That leaves 4 bytes unused or
> 2^32 values for hint bit changes before we have to generate a dummy LSN
> record.
I just did a massive update to the Transparent Data Encryption wiki page
to make it more readable and updated it with current decisions:
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 20:49 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-15 20:49 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 12, 2021 at 12:04:09PM -0500, Bruce Momjian wrote:
> On Sun, Jan 10, 2021 at 09:51:16AM -0500, Bruce Momjian wrote:
> > OK, here they are with numeric prefixes. It was actually tricky to
> > figure out how to create a squashed format-patch based on another branch.
>
> Here is an updated version built on top of Michael Paquier's patch
> posted here:
>
> https://www.postgresql.org/message-id/X/0IChOPHd+aYC1w@paquier.xyz
>
> and included as my first attachment. This will give Michael's patch
> cfbot testing too since the second attachment calls many of the first
> attachment's functions.
Now that Michael's hex encoding patch is committed, I am reposting my
key management patch without Michael's patch. It is improved since the
mid-December version:
* TAP tests for encrypt/decryption, wrapped key creation and decryption,
and KEK rotation
* built on top of new hex encoding functions in /common
* passes cfbot testing
* handles disabled OpenSSL library properly
* handles Windows builds properly
I also learned a lot about format-patch, cfbot testing, and TAP tests.
:-)
It still can't test everything, like prompting from /dev/tty. Also, if
we don't get data encryption into PG 14, we are going to need to hide
the user interface for some of this until it is useful. Prompting from
/dev/tty for the TLS private key passphrase already works and will be a
useful PG 14 feature, so that part of the API will be visible in PG 14.
I am planning to apply this next week.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
key.diff.gz (application/gzip, 120.9K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 21:23 Robert Haas <robertmhaas@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Robert Haas @ 2021-01-15 21:23 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian <bruce@momjian.us> wrote:
> I am planning to apply this next week.
I don't think that's appropriate. Several prominent community members
have told you that the patch, as committed the first time, needed a
lot more work. There hasn't been enough time between then and now for
you, or anyone, to do that amount of work. This patch needs detailed
and substantial review from senior community members, and multiple
rounds of feedback and improvement, before it should be considered for
commit.
I am not even sure there is a consensus on the design, without which
any commit is always premature.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 21:47 Bruce Momjian <bruce@momjian.us>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-15 21:47 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 04:23:22PM -0500, Robert Haas wrote:
> On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian <bruce@momjian.us> wrote:
> > I am planning to apply this next week.
>
> I don't think that's appropriate. Several prominent community members
> have told you that the patch, as committed the first time, needed a
> lot more work. There hasn't been enough time between then and now for
> you, or anyone, to do that amount of work. This patch needs detailed
> and substantial review from senior community members, and multiple
> rounds of feedback and improvement, before it should be considered for
> commit.
>
> I am not even sure there is a consensus on the design, without which
> any commit is always premature.
If people want changes, I need to hear about it here. I have address
everything people have mentioned in these threads so far.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 21:59 Robert Haas <robertmhaas@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 2 replies; 114+ messages in thread
From: Robert Haas @ 2021-01-15 21:59 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian <bruce@momjian.us> wrote:
> If people want changes, I need to hear about it here. I have address
> everything people have mentioned in these threads so far.
That does not match my perception of the situation.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 22:07 David G. Johnston <david.g.johnston@gmail.com>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 114+ messages in thread
From: David G. Johnston @ 2021-01-15 22:07 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 2:59 PM Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian <bruce@momjian.us> wrote:
> > If people want changes, I need to hear about it here. I have address
> > everything people have mentioned in these threads so far.
>
> That does not match my perception of the situation.
>
>
Looking at the Commitfest there are three authors and no reviewers. Given
the previous incident at minimum each of the people in the Commitfest
should add their approval to commit this patch to this thread. And while
committers get some leeway, in this case having a non-author review and
sign-off on it being ready-to-commit seems like it should be required.
David J.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 22:07 Bruce Momjian <bruce@momjian.us>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-15 22:07 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 04:59:17PM -0500, Robert Haas wrote:
> On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian <bruce@momjian.us> wrote:
> > If people want changes, I need to hear about it here. I have address
> > everything people have mentioned in these threads so far.
>
> That does not match my perception of the situation.
Well, that's not very specific, is it? You might be confusing the POC
data encryption patch that was posted in this thread with the key
management patch that I am working on.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-15 22:37 Andres Freund <andres@anarazel.de>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Andres Freund @ 2021-01-15 22:37 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi,
On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote:
> On Fri, Jan 15, 2021 at 04:23:22PM -0500, Robert Haas wrote:
> > On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian <bruce@momjian.us> wrote:
> > I don't think that's appropriate. Several prominent community members
> > have told you that the patch, as committed the first time, needed a
> > lot more work. There hasn't been enough time between then and now for
> > you, or anyone, to do that amount of work. This patch needs detailed
> > and substantial review from senior community members, and multiple
> > rounds of feedback and improvement, before it should be considered for
> > commit.
> >
> > I am not even sure there is a consensus on the design, without which
> > any commit is always premature.
>
> If people want changes, I need to hear about it here. I have address
> everything people have mentioned in these threads so far.
I don't even know how anybody is supposed to realistically review the
design or the patch:
This thread started at
https://postgr.es/m/20210101045047.GB30966%40momjian.us - there's no
reference to any discussion of the design at all and the supposed links
to code are dead.
The last version of the code that I see posted ([1]), has the useless
commit message of "key squash commit" - nothing else. There's no design
documentation included in the patch either, as far as I can tell.
Manually searching for the topic brings me to
https://www.postgresql.org/message-id/20201202213814.GG20285%40momjian.us
, a thread of 52 messages, which provides a bit more context, but
largely just references another thread and a wiki article. The link to
the other thread is into the middle of a 112 message thread.
The wiki page doesn't really describe a design either. It has a very
long todo, a bunch of implementation details, but no design.
Nor did 978f869b99 include much in the way of design description.
You cannot expect anybody to review a patch if developing some basic
understanding of the intended design requires reading hundreds of
messages in which the design evolved. And I don't think it's acceptable
to push it due to lack of further feedback, given this situation - the
lack of design description is a blocker in itself.
There's a few things that stand out on a very very brief scan:
- the patch badly needs to be split up into independently reviewable
pieces
- tests:
- wait, a .sh test script? No, we shouldn't add any more of those,
they're nightmare across platforms
- Do the tests actually do anything useful? It's not clear to me what
they are trying to achieve. En/Decrypting test vectors doesn't seem to
buy that much?
- the new pg_alterckey is completely untested
- the pg_upgrade paths is untested
- ..
- Without further comment BootStrapKmgr() does "copy cluster file
encryption keys from an old cluster?", but there's no explanation as
to why / when that's the case. Presumably pg_upgrade, but, uh, explain
that.
- pg_alterckey.c
- appears to create it's own cluster lock file, using its
own routine for doing so. How does that lock file interact with the
running server?
- retrieve_cluster_keys() is missing (void).
I think this is at the very least a month away from being committable,
even if the design were completely correct (which I do not know, see
above).
Greetings,
Andres Freund
[1] https://www.postgresql.org/message-id/20210115204926.GD8740%40momjian.us
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-16 00:21 Bruce Momjian <bruce@momjian.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-16 00:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 02:37:56PM -0800, Andres Freund wrote:
> On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote:
> > > I am not even sure there is a consensus on the design, without which
> > > any commit is always premature.
> >
> > If people want changes, I need to hear about it here. I have address
> > everything people have mentioned in these threads so far.
>
> I don't even know how anybody is supposed to realistically review the
> design or the patch:
>
> This thread started at
> https://postgr.es/m/20210101045047.GB30966%40momjian.us - there's no
> reference to any discussion of the design at all and the supposed links
> to code are dead.
You have to understand cryptography and Postgres internals to understand
the design, and I don't think it is realistic to explain that all to the
community. We did much of this in voice calls over months because it
was too much of a burden to explain all the cryptographic details so
everyone could follow along.
> The last version of the code that I see posted ([1]), has the useless
> commit message of "key squash commit" - nothing else. There's no design
> documentation included in the patch either, as far as I can tell.
>
> Manually searching for the topic brings me to
> https://www.postgresql.org/message-id/20201202213814.GG20285%40momjian.us
> , a thread of 52 messages, which provides a bit more context, but
> largely just references another thread and a wiki article. The link to
> the other thread is into the middle of a 112 message thread.
>
> The wiki page doesn't really describe a design either. It has a very
> long todo, a bunch of implementation details, but no design.
I am not sure what design document you are requesting. I thought the
TODO was that.
> Nor did 978f869b99 include much in the way of design description.
>
> You cannot expect anybody to review a patch if developing some basic
> understanding of the intended design requires reading hundreds of
> messages in which the design evolved. And I don't think it's acceptable
> to push it due to lack of further feedback, given this situation - the
> lack of design description is a blocker in itself.
OK, I will just move on to something else then. It is not worth the
feature to go into that kind of discussion again. I am willing to have
voice calls with individuals to explain the logic, but repeatedly
explaining it to the entire group I find unproductive. I don't think
another 400-email thread would help anyone.
> There's a few things that stand out on a very very brief scan:
> - the patch badly needs to be split up into independently reviewable
> pieces
I can do that, but there are enough complaints above that I feel it
would not be worth it.
> - tests:
> - wait, a .sh test script? No, we shouldn't add any more of those,
> they're nightmare across platforms
The script originatad from pg_upgrade. I don't know how to do things
like initdb and stuff another way, at least in our code.
> - Do the tests actually do anything useful? It's not clear to me what
> they are trying to achieve. En/Decrypting test vectors doesn't seem to
> buy that much?
Uh, that's because the key manager doesn't do anything useful yet.
> - the new pg_alterckey is completely untested
Wow, I was so excited I tested the data keys that I forgot to add the
pg_alterckey tests. My tests had that already. I have added it to the
attached patch.
> - the pg_upgrade paths is untested
Uh, I was waiting until we were actually encrypting some data to test
that.
> - ..
> - Without further comment BootStrapKmgr() does "copy cluster file
> encryption keys from an old cluster?", but there's no explanation as
> to why / when that's the case. Presumably pg_upgrade, but, uh, explain
> that.
Uh, the heap/index files are, in the future, encrypted with the keys of
the old cluster, so we just copy them to the new cluster and they keep
working. Potentially we could replace the WAL key at that point since
we don't move WAL from the old cluster to the new one, but we also need
a command-line tool to do that, so I figured I would just wait for that
to be done.
> - pg_alterckey.c
> - appears to create it's own cluster lock file, using its
> own routine for doing so. How does that lock file interact with the
> running server?
pg_alterckey runs fine while the old cluster is running, which is why I
used a new lock file. The keys are only read at db boot time.
> - retrieve_cluster_keys() is missing (void).
Oops, fixed.
> I think this is at the very least a month away from being committable,
> even if the design were completely correct (which I do not know, see
> above).
Those comments were very helpful, and I could certainly use more
feedback on the patch. Updated patch attached.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
key.diff.gz (application/gzip, 121.0K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-16 00:56 Andres Freund <andres@anarazel.de>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 114+ messages in thread
From: Andres Freund @ 2021-01-16 00:56 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi,
On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote:
> On Fri, Jan 15, 2021 at 02:37:56PM -0800, Andres Freund wrote:
> > On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote:
> > > > I am not even sure there is a consensus on the design, without which
> > > > any commit is always premature.
> > >
> > > If people want changes, I need to hear about it here. I have address
> > > everything people have mentioned in these threads so far.
> >
> > I don't even know how anybody is supposed to realistically review the
> > design or the patch:
> >
> > This thread started at
> > https://postgr.es/m/20210101045047.GB30966%40momjian.us - there's no
> > reference to any discussion of the design at all and the supposed links
> > to code are dead.
>
> You have to understand cryptography and Postgres internals to understand
> the design, and I don't think it is realistic to explain that all to the
> community. We did much of this in voice calls over months because it
> was too much of a burden to explain all the cryptographic details so
> everyone could follow along.
I think that's not at all acceptable. I don't mind hashing out details
on calls / off-list, but the design needs to be public, documented, and
reviewable. And if it's something the community can't understand, then
it can't get in. We're going to have to maintain this going forward.
I don't mean to say that we need to re-hash all design details from
scratch - but that there needs to be an explanation somewhere that
describes what's being done on a medium-high level, and what drove those
design decisions.
> > The last version of the code that I see posted ([1]), has the useless
> > commit message of "key squash commit" - nothing else. There's no design
> > documentation included in the patch either, as far as I can tell.
> >
> > Manually searching for the topic brings me to
> > https://www.postgresql.org/message-id/20201202213814.GG20285%40momjian.us
> > , a thread of 52 messages, which provides a bit more context, but
> > largely just references another thread and a wiki article. The link to
> > the other thread is into the middle of a 112 message thread.
> >
> > The wiki page doesn't really describe a design either. It has a very
> > long todo, a bunch of implementation details, but no design.
>
> I am not sure what design document you are requesting. I thought the
> TODO was that.
The TODO in https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
is a design document?
> > Nor did 978f869b99 include much in the way of design description.
> >
> > You cannot expect anybody to review a patch if developing some basic
> > understanding of the intended design requires reading hundreds of
> > messages in which the design evolved. And I don't think it's acceptable
> > to push it due to lack of further feedback, given this situation - the
> > lack of design description is a blocker in itself.
>
> OK, I will just move on to something else then. It is not worth the
> feature to go into that kind of discussion again. I am willing to have
> voice calls with individuals to explain the logic, but repeatedly
> explaining it to the entire group I find unproductive. I don't think
> another 400-email thread would help anyone.
Explaining something over voice doesn't help with people in a year or
five trying to understand the code and the design, so they can adapt it
when making half-related changes. Nor do I see why another 400 email
thread would be a necessary consequence of you explaining the design
that you came up with.
This isn't specific to this topic? I don't really understand why this
specific feature gets to avoid normal community development processes?
> > - tests:
> > - wait, a .sh test script? No, we shouldn't add any more of those,
> > they're nightmare across platforms
>
> The script originatad from pg_upgrade. I don't know how to do things
> like initdb and stuff another way, at least in our code.
We have had perl tap tests for quite a while now? And all new tests that
aren't regression / isolation tests are expected to be written in it.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-16 01:49 Bruce Momjian <bruce@momjian.us>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-16 01:49 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 04:56:24PM -0800, Andres Freund wrote:
> On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote:
> > You have to understand cryptography and Postgres internals to understand
> > the design, and I don't think it is realistic to explain that all to the
> > community. We did much of this in voice calls over months because it
> > was too much of a burden to explain all the cryptographic details so
> > everyone could follow along.
>
> I think that's not at all acceptable. I don't mind hashing out details
> on calls / off-list, but the design needs to be public, documented, and
> reviewable. And if it's something the community can't understand, then
> it can't get in. We're going to have to maintain this going forward.
OK, so we don't want it. That's fine with me.
> I don't mean to say that we need to re-hash all design details from
> scratch - but that there needs to be an explanation somewhere that
> describes what's being done on a medium-high level, and what drove those
> design decisions.
I thought the TODO list was that, and the email threads.
> > > The wiki page doesn't really describe a design either. It has a very
> > > long todo, a bunch of implementation details, but no design.
> >
> > I am not sure what design document you are requesting. I thought the
> > TODO was that.
>
> The TODO in https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements
> is a design document?
Yes.
> > > Nor did 978f869b99 include much in the way of design description.
> > >
> > > You cannot expect anybody to review a patch if developing some basic
> > > understanding of the intended design requires reading hundreds of
> > > messages in which the design evolved. And I don't think it's acceptable
> > > to push it due to lack of further feedback, given this situation - the
> > > lack of design description is a blocker in itself.
> >
> > OK, I will just move on to something else then. It is not worth the
> > feature to go into that kind of discussion again. I am willing to have
> > voice calls with individuals to explain the logic, but repeatedly
> > explaining it to the entire group I find unproductive. I don't think
> > another 400-email thread would help anyone.
>
> Explaining something over voice doesn't help with people in a year or
> five trying to understand the code and the design, so they can adapt it
> when making half-related changes. Nor do I see why another 400 email
> thread would be a necessary consequence of you explaining the design
> that you came up with.
I have underestimated the amount of discussion this has required
repeatedly, and I don't want to make that mistake again.
> This isn't specific to this topic? I don't really understand why this
> specific feature gets to avoid normal community development processes?
What is being avoided?
> > > - tests:
> > > - wait, a .sh test script? No, we shouldn't add any more of those,
> > > they're nightmare across platforms
> >
> > The script originatad from pg_upgrade. I don't know how to do things
> > like initdb and stuff another way, at least in our code.
>
> We have had perl tap tests for quite a while now? And all new tests that
> aren't regression / isolation tests are expected to be written in it.
What Perl tap tests run initdb and manage the cluster? I didn't find
any.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-16 04:20 Andres Freund <andres@anarazel.de>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 114+ messages in thread
From: Andres Freund @ 2021-01-16 04:20 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi,
On 2021-01-15 20:49:10 -0500, Bruce Momjian wrote:
> On Fri, Jan 15, 2021 at 04:56:24PM -0800, Andres Freund wrote:
> > On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote:
> > > You have to understand cryptography and Postgres internals to understand
> > > the design, and I don't think it is realistic to explain that all to the
> > > community. We did much of this in voice calls over months because it
> > > was too much of a burden to explain all the cryptographic details so
> > > everyone could follow along.
> >
> > I think that's not at all acceptable. I don't mind hashing out details
> > on calls / off-list, but the design needs to be public, documented, and
> > reviewable. And if it's something the community can't understand, then
> > it can't get in. We're going to have to maintain this going forward.
>
> OK, so we don't want it. That's fine with me.
That's not what I said...
> > This isn't specific to this topic? I don't really understand why this
> > specific feature gets to avoid normal community development processes?
>
> What is being avoided?
You previously pushed a patch without tests, now you want to push a
patch that was barely reviewed and also doesn't contain an explanation
of the design. I mean:
> > > You have to understand cryptography and Postgres internals to understand
> > > the design, and I don't think it is realistic to explain that all to the
> > > community. We did much of this in voice calls over months because it
> > > was too much of a burden to explain all the cryptographic details so
> > > everyone could follow along.
really is very far from the normal community process. Again, how is this
supposed to be maintained in the future, if it's based on a design
that's only understandable to the people on those phone calls?
> > We have had perl tap tests for quite a while now? And all new tests that
> > aren't regression / isolation tests are expected to be written in it.
>
> What Perl tap tests run initdb and manage the cluster? I didn't find
> any.
find . -name '*.pl'|xargs grep 'use PostgresNode;'
should give you a nearly complete list.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-16 04:39 Michael Paquier <michael@paquier.xyz>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 114+ messages in thread
From: Michael Paquier @ 2021-01-16 04:39 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 08:20:36PM -0800, Andres Freund wrote:
> On 2021-01-15 20:49:10 -0500, Bruce Momjian wrote:
>> What Perl tap tests run initdb and manage the cluster? I didn't find
>> any.
>
> find . -name '*.pl'|xargs grep 'use PostgresNode;'
>
> should give you a nearly complete list.
Just to add that all the perl modules we use for the tests are within
src/test/perl/. The coolest tests are within src/bin/ and src/test/.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-17 00:08 Tom Kincaid <tomjohnkincaid@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 114+ messages in thread
From: Tom Kincaid @ 2021-01-17 00:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
> > > I think that's not at all acceptable. I don't mind hashing out details
> > > on calls / off-list, but the design needs to be public, documented, and
> > > reviewable. And if it's something the community can't understand, then
> > > it can't get in. We're going to have to maintain this going forward.
> >
> > OK, so we don't want it. That's fine with me.
>
> That's not what I said...
>
I think the majority of us believe that it is important we take this
first step towards a solid TDE implementation in PostgreSQL that is
built around the community processes which involves general consensus.
Before this feature falls into the “we will never do it because we
will never build consensus" category and community PostgreSQL
potentially gets locked out of more deployment scenarios that require
this feature I would like to see if I can help with this current
attempt at it. I will share that I am concerned that if the people who
have been involved in this to date can’t get this in, it will never
happen.
Admittedly I am a novice on this topic, and the majority of the
PostgreSQL source code, however I am hopeful enough (those of you who
know me understand that I suffer from eternal optimism) that I am
going to attempt to help.
Is there a design document for a Postgres feature of this size and
scope that people feel would serve as a good example? Alternatively,
is there a design document template that has been successfully used in
the past? I could guess based on things I have observed reading this
list for many years. However, if there is something that those who are
deeply involved in the development effort feel would suffice as an
example of a "good design document" or a "good design template"
sharing it would be greatly appreciated.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-17 06:24 Amit Kapila <amit.kapila16@gmail.com>
parent: Tom Kincaid <tomjohnkincaid@gmail.com>
0 siblings, 2 replies; 114+ messages in thread
From: Amit Kapila @ 2021-01-17 06:24 UTC (permalink / raw)
To: Tom Kincaid <tomjohnkincaid@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid <tomjohnkincaid@gmail.com> wrote:
>
> > > > I think that's not at all acceptable. I don't mind hashing out details
> > > > on calls / off-list, but the design needs to be public, documented, and
> > > > reviewable. And if it's something the community can't understand, then
> > > > it can't get in. We're going to have to maintain this going forward.
> > >
> > > OK, so we don't want it. That's fine with me.
> >
> > That's not what I said...
> >
>
>
> I think the majority of us believe that it is important we take this
> first step towards a solid TDE implementation in PostgreSQL that is
> built around the community processes which involves general consensus.
>
> Before this feature falls into the “we will never do it because we
> will never build consensus" category and community PostgreSQL
> potentially gets locked out of more deployment scenarios that require
> this feature I would like to see if I can help with this current
> attempt at it. I will share that I am concerned that if the people who
> have been involved in this to date can’t get this in, it will never
> happen.
>
> Admittedly I am a novice on this topic, and the majority of the
> PostgreSQL source code, however I am hopeful enough (those of you who
> know me understand that I suffer from eternal optimism) that I am
> going to attempt to help.
>
> Is there a design document for a Postgres feature of this size and
> scope that people feel would serve as a good example? Alternatively,
> is there a design document template that has been successfully used in
> the past?
>
We normally write the design considerations and choices we made with
the reasons in README and code comments. Personally, I am not sure if
there is a need for any specific document per-se but a README and
detailed comments in the code should suffice what people are worried
about here. It is mostly from the perspective that other developers
reading the code, want to do bug-fix, or later enhance that code
should be able to understand it. One recent example I can give is
Peter's work on bottom-up deletion [1] which I have read today where I
find that the design is captured via README, appropriate comments in
the code, and documentation. This feature is quite different and
probably a lot more new concepts are being introduced but I hope that
will give you some clue.
[1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d168b666823b6e0bcf60ed19ce24fb5fb...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-17 06:58 Andres Freund <andres@anarazel.de>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 114+ messages in thread
From: Andres Freund @ 2021-01-17 06:58 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi,
On 2021-01-17 11:54:57 +0530, Amit Kapila wrote:
> On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid <tomjohnkincaid@gmail.com> wrote:
> > Admittedly I am a novice on this topic, and the majority of the
> > PostgreSQL source code, however I am hopeful enough (those of you who
> > know me understand that I suffer from eternal optimism) that I am
> > going to attempt to help.
> >
> > Is there a design document for a Postgres feature of this size and
> > scope that people feel would serve as a good example? Alternatively,
> > is there a design document template that has been successfully used in
> > the past?
> >
>
> We normally write the design considerations and choices we made with
> the reasons in README and code comments. Personally, I am not sure if
> there is a need for any specific document per-se but a README and
> detailed comments in the code should suffice what people are worried
> about here.
Right. It could be a README file, or a long comment at a start of one of
the files. It doesn't matter too much. What matters is that people that
haven't been on those phone call can understand the design and the
implications it has.
> It is mostly from the perspective that other developers
> reading the code, want to do bug-fix, or later enhance that code
> should be able to understand it.
I'd add the perspective of code reviewers as well.
> One recent example I can give is
> Peter's work on bottom-up deletion [1] which I have read today where I
> find that the design is captured via README, appropriate comments in
> the code, and documentation. This feature is quite different and
> probably a lot more new concepts are being introduced but I hope that
> will give you some clue.
>
> [1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d168b666823b6e0bcf60ed19ce24fb5fb...
This is a great example.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 00:50 Robert Haas <robertmhaas@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 114+ messages in thread
From: Robert Haas @ 2021-01-18 00:50 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bruce Momjian <bruce@momjian.us>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 15, 2021 at 7:56 PM Andres Freund <andres@anarazel.de> wrote:
> I think that's not at all acceptable. I don't mind hashing out details
> on calls / off-list, but the design needs to be public, documented, and
> reviewable. And if it's something the community can't understand, then
> it can't get in. We're going to have to maintain this going forward.
I agree. If the community is unable to clearly understand what
something is, and why we should have it, then we shouldn't have it --
even if the reason is that we're too dumb to understand, as Bruce
seems to be alleging. I don't really think I believe the theory that
community members by and large are too dumb to understand encryption.
Many features have provoked long and painful discussions about the
design and yet got into the tree in the end with documentation of that
design, and I don't see why that couldn't be done for this one, too. I
think it can and should, and the fact that the work hasn't been done
is one of several blockers for this patch. But even if I'm wrong, and
the real problem is that everyone except the select group of people on
these off-list phone calls are too stupid to understand this, then
that's still a reason not to accept the patch. The code that's in our
source tree is maintained by communal effort, and that means communal
understanding is important.
Frankly, it's more important in this particular case than in some
others. TDE is in great demand, so if it gets into the tree, it's
likely to get a lot of use. The preparatory patches, such as this one,
would at that point be getting a lot of use, too. That means many
people, not just hackers, will have to understand them and answer
questions about them. They are also likely to get a lot of scrutiny
from a security point of view, so we should have a way that we can be
confident that we know why we believe them to be secure. If a security
researcher shows up and says "your stuff is broken," we are not going
to get away with it "no it isn't, because we discussed it on a Friday
call with a closed group of people and decided it was OK." Our
reasoning is going to have to be documented. That doesn't guarantee
that it will be correct, but makes it possible to distinguish between
defects in design, defects in particular parts of the code, and
non-defects, which is otherwise impossible. Meanwhile, even if
security researches are as happy with our TDE implementation as they
could possibly be, a feature that changes the on-disk format can't
erase our ability to solve other problems with the database. Databases
using TDE are still going to have corruption, for example, but now a
corrupted page has a good chance of being completely unreadable rather
than just garbled. You certainly aren't going to be able to just run
pg_filedump on it. I think even if we do a great job explaining to
everybody what impact TDE and its preparatory patches are likely to
have on the system, there's likely to be a lot of cases where users
blame the database for eating their data when the real culprit is the
OS or the hardware, just because such cases are bound to get harder to
investigate, which could have a very negative effect on the
perceptions of PostgreSQL's quality. But if the TDE itself is magic
that only designated smart people on special calls can understand,
then it's going to be far worse, because that'll mean when any kind of
TDE problems comes up, nobody else can help debug anything.
While I would like to have TDE in PostgreSQL, I would not like to have
it on those terms.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 15:43 Bruce Momjian <bruce@momjian.us>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-18 15:43 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 17, 2021 at 07:50:13PM -0500, Robert Haas wrote:
> On Fri, Jan 15, 2021 at 7:56 PM Andres Freund <andres@anarazel.de> wrote:
> > I think that's not at all acceptable. I don't mind hashing out details
> > on calls / off-list, but the design needs to be public, documented, and
> > reviewable. And if it's something the community can't understand, then
> > it can't get in. We're going to have to maintain this going forward.
>
> I agree. If the community is unable to clearly understand what
> something is, and why we should have it, then we shouldn't have it --
> even if the reason is that we're too dumb to understand, as Bruce
I am not sure why you are brining intelligence into this discussion.
You have to understand Postgres internals and cryptography tradeoffs to
understand why some of the design decisions were made. It is a
knowledge issue, not an intelligence issue. The wiki page is the result
of those phone discussions.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 15:50 Bruce Momjian <bruce@momjian.us>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-18 15:50 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sun, Jan 17, 2021 at 11:54:57AM +0530, Amit Kapila wrote:
> > Is there a design document for a Postgres feature of this size and
> > scope that people feel would serve as a good example? Alternatively,
> > is there a design document template that has been successfully used in
> > the past?
>
> We normally write the design considerations and choices we made with
> the reasons in README and code comments. Personally, I am not sure if
> there is a need for any specific document per-se but a README and
> detailed comments in the code should suffice what people are worried
> about here. It is mostly from the perspective that other developers
> reading the code, want to do bug-fix, or later enhance that code
> should be able to understand it. One recent example I can give is
> Peter's work on bottom-up deletion [1] which I have read today where I
> find that the design is captured via README, appropriate comments in
> the code, and documentation. This feature is quite different and
> probably a lot more new concepts are being introduced but I hope that
> will give you some clue.
>
> [1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d168b666823b6e0bcf60ed19ce24fb5fb...
OK, I looked at that and it is good, and I see my patch is missing that.
Are people looking for me to take the wiki content, expand on it and tie
it to the code that will be applied, or something else like all the
various crypto options and why we chose what we did beyond what is
already on the wiki? I can easily go from what we have on the wiki to
implementation code steps, but the other part is harder to explain and
that is why I offered to talk to people via voice.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 15:51 Bruce Momjian <bruce@momjian.us>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-18 15:51 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sat, Jan 16, 2021 at 10:58:47PM -0800, Andres Freund wrote:
> Hi,
>
> On 2021-01-17 11:54:57 +0530, Amit Kapila wrote:
> > On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid <tomjohnkincaid@gmail.com> wrote:
> > > Admittedly I am a novice on this topic, and the majority of the
> > > PostgreSQL source code, however I am hopeful enough (those of you who
> > > know me understand that I suffer from eternal optimism) that I am
> > > going to attempt to help.
> > >
> > > Is there a design document for a Postgres feature of this size and
> > > scope that people feel would serve as a good example? Alternatively,
> > > is there a design document template that has been successfully used in
> > > the past?
> > >
> >
> > We normally write the design considerations and choices we made with
> > the reasons in README and code comments. Personally, I am not sure if
> > there is a need for any specific document per-se but a README and
> > detailed comments in the code should suffice what people are worried
> > about here.
>
> Right. It could be a README file, or a long comment at a start of one of
> the files. It doesn't matter too much. What matters is that people that
> haven't been on those phone call can understand the design and the
> implications it has.
OK, so does the wiki page contain most of what you want, but is missing
the connection between the design and the code?
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 17:06 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-18 17:06 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 18, 2021 at 10:50:37AM -0500, Bruce Momjian wrote:
> OK, I looked at that and it is good, and I see my patch is missing that.
> Are people looking for me to take the wiki content, expand on it and tie
> it to the code that will be applied, or something else like all the
> various crypto options and why we chose what we did beyond what is
> already on the wiki? I can easily go from what we have on the wiki to
> implementation code steps, but the other part is harder to explain and
> that is why I offered to talk to people via voice.
Just to clarify why voice calls can be helpful --- if you have to get
into "you have to understand X to understand Y", that's where a voice
call works best, because understanding X will require understanding
A/B/C, and everyone's missing pieces are different, so you have to
customize it for the individual.
You can explain some of this in a README, but trying to cover all of it
leads to a combinatorial problem of trying to explain everything.
Ideally the wiki page can be expanded so people can ask and answer all
posted issues, perhaps in a Q&A format. Someone could go through the
archives and post why certain decisions were made, and link to the
original emails.
I have to admit I was kind of baffled that the wiki page wasn't
sufficient, because it is one of the longest Postgres feature
explanations I have seen, but I now think the missing part is tying
the wiki contents to the code implementation. If that is it, please
confirm. If it is something else, also explain.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 17:42 Andres Freund <andres@anarazel.de>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 114+ messages in thread
From: Andres Freund @ 2021-01-18 17:42 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi,
On 2021-01-18 12:06:35 -0500, Bruce Momjian wrote:
> On Mon, Jan 18, 2021 at 10:50:37AM -0500, Bruce Momjian wrote:
> > OK, I looked at that and it is good, and I see my patch is missing that.
> > Are people looking for me to take the wiki content, expand on it and tie
> > it to the code that will be applied, or something else like all the
> > various crypto options and why we chose what we did beyond what is
> > already on the wiki? I can easily go from what we have on the wiki to
> > implementation code steps, but the other part is harder to explain and
> > that is why I offered to talk to people via voice.
>
> Just to clarify why voice calls can be helpful --- if you have to get
> into "you have to understand X to understand Y", that's where a voice
> call works best, because understanding X will require understanding
> A/B/C, and everyone's missing pieces are different, so you have to
> customize it for the individual.
I don't think anybody argued against having voice calls.
> You can explain some of this in a README, but trying to cover all of it
> leads to a combinatorial problem of trying to explain everything.
> Ideally the wiki page can be expanded so people can ask and answer all
> posted issues, perhaps in a Q&A format. Someone could go through the
> archives and post why certain decisions were made, and link to the
> original emails.
>
> I have to admit I was kind of baffled that the wiki page wasn't
> sufficient, because it is one of the longest Postgres feature
> explanations I have seen, but I now think the missing part is tying
> the wiki contents to the code implementation. If that is it, please
> confirm. If it is something else, also explain.
I don't think the wiki right now covers what's needed. The "Overview",
"Threat model" and "Scope of TDE" are a start, but beyond that it's
missing a bunch of things. And it's not in the source tree (we'll soon
have multiple versions of postgres with increasing levels of TDE
features, the wiki doesn't help with that)
Missing:
- talks about cluster wide encyrption being simpler, without mentioning
what it's being compared to, and what makes it simpler
- no differentiation from file system / block level encryption
- there's no explanation of which/why specific crypto primitives were
chosen, what the tradeoffs are
- no explanation which keys exists, stored where
- the key management patch introduces new files, not documented
- there's new types of lock files, possibility of interrupted
operations, ... - no documentation of what that means
- there's no documentation what "key wrapping" actually precisely is,
what the danger of the two-tier model is, ...
- are there dangers in not encrypting zero pages etc?
- ...
Personally, but I admit that there's legitimate reasons to differ on
that note, I don't think it's reasonable for a feature this invasive to
commit preliminary patches without the major subsequent patches being in
a shape that allows reviewing the whole picture.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 18:58 Bruce Momjian <bruce@momjian.us>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-18 18:58 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote:
> Personally, but I admit that there's legitimate reasons to differ on
> that note, I don't think it's reasonable for a feature this invasive to
> commit preliminary patches without the major subsequent patches being in
> a shape that allows reviewing the whole picture.
OK, if that is a requirement, I can't help anymore since there are
already complaints that the patch is too large to review, even if broken
into pieces. Please let me know what the community decides.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 19:00 Tom Kincaid <tomjohnkincaid@gmail.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 114+ messages in thread
From: Tom Kincaid @ 2021-01-18 19:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bruce Momjian <bruce@momjian.us>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
> > I have to admit I was kind of baffled that the wiki page wasn't
> > sufficient, because it is one of the longest Postgres feature
> > explanations I have seen, but I now think the missing part is tying
> > the wiki contents to the code implementation. If that is it, please
> > confirm. If it is something else, also explain.
>
> I don't think the wiki right now covers what's needed. The "Overview",
> "Threat model" and "Scope of TDE" are a start, but beyond that it's
> missing a bunch of things. And it's not in the source tree (we'll soon
> have multiple versions of postgres with increasing levels of TDE
> features, the wiki doesn't help with that)
>
Thanks, the versioning issue makes sense for the design document
needing to be part of the the source tree.
As I was reading the README for the patch Amit referenced and as I am
going through this patch, I feel the desire to incorporate diagrams.
Are design diagrams ever incorporated in the source tree as a part of
the design description of a feature? If not, any concerns about doing
that? I think that is likely where I can contribute the most.
> Missing:
> - talks about cluster wide encyrption being simpler, without mentioning
> what it's being compared to, and what makes it simpler
> - no differentiation from file system / block level encryption
> - there's no explanation of which/why specific crypto primitives were
> chosen, what the tradeoffs are
> - no explanation which keys exists, stored where
> - the key management patch introduces new files, not documented
> - there's new types of lock files, possibility of interrupted
> operations, ... - no documentation of what that means
> - there's no documentation what "key wrapping" actually precisely is,
> what the danger of the two-tier model is, ...
> - are there dangers in not encrypting zero pages etc?
> - ...
>
Some of the missing things you mention above are about the design of
TDE feature in general. However, this patch is about Key Management
which is going part of the larger TDE feature. So it feels as though
there is the need for a general design document about the overall
vision / approach for TDE and a specific design doc. for Key
Management. Is it appropriate to include both of those in the same
patch?
Something along the lines here is the overall design of TDE and here
is how the Key Management portion is designed and implemented. I guess
in that case, follow on patches for TDE could refer to the overall
design described in this patch.
>
>
> Personally, but I admit that there's legitimate reasons to differ on
> that note, I don't think it's reasonable for a feature this invasive to
> commit preliminary patches without the major subsequent patches being in
> a shape that allows reviewing the whole picture.
>
> Greetings,
>
> Andres Freund
--
Thomas John Kincaid
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 19:00 Andres Freund <andres@anarazel.de>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Andres Freund @ 2021-01-18 19:00 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On 2021-01-18 13:58:20 -0500, Bruce Momjian wrote:
> On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote:
> > Personally, but I admit that there's legitimate reasons to differ on
> > that note, I don't think it's reasonable for a feature this invasive to
> > commit preliminary patches without the major subsequent patches being in
> > a shape that allows reviewing the whole picture.
>
> OK, if that is a requirement, I can't help anymore since there are
> already complaints that the patch is too large to review, even if broken
> into pieces. Please let me know what the community decides.
Those aren't conflicting demands. Having later patches around to
validate the design of earlier patches doesn't necessitates that the
later patches need to be reviewed at the same time.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 21:38 Robert Haas <robertmhaas@gmail.com>
parent: Tom Kincaid <tomjohnkincaid@gmail.com>
0 siblings, 1 reply; 114+ messages in thread
From: Robert Haas @ 2021-01-18 21:38 UTC (permalink / raw)
To: Tom Kincaid <tomjohnkincaid@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Bruce Momjian <bruce@momjian.us>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 18, 2021 at 2:00 PM Tom Kincaid <tomjohnkincaid@gmail.com> wrote:
> Some of the missing things you mention above are about the design of
> TDE feature in general. However, this patch is about Key Management
> which is going part of the larger TDE feature. So it feels as though
> there is the need for a general design document about the overall
> vision / approach for TDE and a specific design doc. for Key
> Management. Is it appropriate to include both of those in the same
> patch?
To me, it wouldn't make sense to commit a full README for a TDE
feature that we don't have yet with a key management patch, but the
way that they'll interact with each other has to be clear. The
doc/database-encryption.sgml file that Bruce included in the patch is
a decent start on explaining the design, though I think it needs more
work and more details, perhaps including some of the things Andres
mentioned.
To be honest, after reading over that SGML documentation a bit, I'm
somewhat skeptical about whether it really makes sense to think about
committing the key management part separately. It seems to have no use
independent of the main feature, and it in fact embeds very specific
details of how the main feature is expected to work. For example, the
documentation says that key #0 will be used for data files, and key #1
for WAL. There seems to be no suggestion that the key management
portion of this can be used to manage encryption keys generally for
whatever purposes someone might have; it's all about the needs of a
particular TDE implementation. Typically, we would not commit
something like that separately, or only once the main patch was done,
with the two commits occurring in a relatively short time period.
Otherwise, as Bruce already noted, we can end up with something that
is documented and visible to users but doesn't actually work yet.
Some more specific comments on data-encryption.sgml:
* The documentation explains that the purpose of having a WAL key
separate from the data file key is so that the data file keys can
"eventually" be rotated. It's not clear whether this means that we
might eventually have that feature or that we might eventually be able
to rotate, after failing over. If this kind of thing is possible,
we'll eventually need documentation on how to do it.
* The reasons for use a DEK and a KEK are not explained. I realize
it's not an uncommon practice and that other systems do it, but I
think a few sentences of explanation wouldn't be a bad idea. Even if
we are supposing that hackers who want to have input into this feature
have to be knowledgeable about cryptography, I don't think we can
reasonably suppose that for users.
* "For example" is at one point followed by a period rather than a
colon or comma.
* In the "Internals" subsection, the last sentence doesn't seem to be
grammatical. I wonder if it's missing the word "or"'.
* The part about integrity-checking keys on startup isn't clear. It
makes it sound like we still have a copy of the KEK lying around
someplace against which we can compare, which I assume is not the case
since it would be really insecure.
* I think it's going to be pretty important that we can easily switch
to other cryptographic algorithms as they are discovered, so I don't
like the fact that this is tied specifically to AES. (In fact,
kmgr_utils.h makes it sound like we're specifically locked into
AES256, but that contradicts the documentation, so I guess there's
some clarification needed here about what exactly KMGR_CLUSTER_KEY_LEN
is doing.) As far as possible we should try to make this generic, like
supporting any cipher that SSL has which has property X. It seems
relatively inevitable that every currently popular cryptographic
algorithm will at some point in the future be judged weak and
worthless, just as has already happened with MD5 and some variants of
SHA, both of which used to be considered state of the art. It seems
equally inevitable that new and stronger algorithms will continued to
be devised, and we'll want to adopt those easily.
I'm not sure to what extent this a serious flaw in the patch and to
what extent it's just a matter of tweaking the wording of some things,
but I think this is actually an extremely critical design point where
we had better be certain we've got it right. Few things would be
sadder than to get a TDE feature and then have to rip it out again
because it couldn't be upgraded to work with newer crypto algorithms
with reasonable effort.
Notes on other parts of the documentation:
* The documentation for initdb -K doesn't list the valid values of the
parameter, only the default. Probably we should be specifying an
algorithm here and not just a bit count. Otherwise, like I say above,
what happens when AES gives way to something else? It'd be easy to say
-K BFT256 instead of -K AES256, but if AES is assumed and it's no
longer what we want them we have problems. This kind of thing probably
needs to be cleaned up in a bunch of places.
* I don't see the point of saying "a passphrase or PIN." We don't need
to document that your passphrase might happen to only contain digits.
* pg_alterckey's description of "repair" is hard to understand. It
doesn't really explain why or how this would be necessary, and it begs
the question of why we'd ever leave things in a state that requires
repair. This is sketched out in code comments elsewhere, but I think
at least some of it needs to be explained in the documentation as
well. (Incidentally, I don't think the comments at the top of
recover_failure will survive a visit from pgindent, though I might be
wrong about that.)
* The changes to config.sgml say "Sample script" instead of "Sample scripts".
* I don't think that the documentation of %R is very clear, or
adequate for someone to make effective use of it. If I wanted to use
%R, how would I ensure that a value is available?
* The changes to allfiles.sgml add pg_alterckey.sgml in the wrong
place and include an incorrect whitespace change.
* It's odd that "pg_alterckey" describes itself as "technically"
changing the KEK. Isn't that just what it does, not a technicality? I
imagine we'll ultimately need a way to change a DEK as well, because
otherwise the use of a separate key for the WAL wouldn't accomplish
the intended goal.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 22:47 Tom Kincaid <tomjohnkincaid@gmail.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 114+ messages in thread
From: Tom Kincaid @ 2021-01-18 22:47 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Bruce Momjian <bruce@momjian.us>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
I met with Bruce and Stephen this afternoon to discuss the feedback
we received so far (prior to Robert's note which I haven't fully
digested yet)
on this patch.
Here is what we plan to do:
1) Bruce is going to gather all the details from the Wiki and build a
README for the TDE Key Management patch. In addition, it will include
details about the implementation, the data structures involved and the
locks that are taken and general technical implementation approach.
2) Stephen is going to write up the overall design of TDE.
Between these two patches, we hope to cover what Andres is asking for
and what Robert is asking for in his reply on this thread which I
haven't fully digested yet.
Stephen's documentation patch will also make reference to Neil Chen's
TDE prototype for making use of this Key Management patch to encrypt
and
decrypt heap pages as well as index pages.
https://www.postgresql.org/message-id/CAA3qoJ=qtO5JcSBjqFDBT9iKUX9XKmC5bXCrd7rysE+XSMEuTg@mail.gmail...
3) Tom will work to find somebody who will sign up as a reviewer upon
the next submission of this patch. (Somebody who is not an author).
Could we get feedback if this feels like enough to get this patch
(which will include just the Key Management portion of TDE) to a state
where it can be reviewed and assuming the review issues are resolved
with consensus be committed?
On Mon, Jan 18, 2021 at 2:00 PM Andres Freund <andres@anarazel.de> wrote:
>
> On 2021-01-18 13:58:20 -0500, Bruce Momjian wrote:
> > On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote:
> > > Personally, but I admit that there's legitimate reasons to differ on
> > > that note, I don't think it's reasonable for a feature this invasive to
> > > commit preliminary patches without the major subsequent patches being in
> > > a shape that allows reviewing the whole picture.
> >
> > OK, if that is a requirement, I can't help anymore since there are
> > already complaints that the patch is too large to review, even if broken
> > into pieces. Please let me know what the community decides.
>
> Those aren't conflicting demands. Having later patches around to
> validate the design of earlier patches doesn't necessitates that the
> later patches need to be reviewed at the same time.
--
Thomas John Kincaid
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-18 23:21 Bruce Momjian <bruce@momjian.us>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-18 23:21 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 18, 2021 at 04:38:47PM -0500, Robert Haas wrote:
> To me, it wouldn't make sense to commit a full README for a TDE
> feature that we don't have yet with a key management patch, but the
> way that they'll interact with each other has to be clear. The
> doc/database-encryption.sgml file that Bruce included in the patch is
> a decent start on explaining the design, though I think it needs more
> work and more details, perhaps including some of the things Andres
> mentioned.
Sure.
> To be honest, after reading over that SGML documentation a bit, I'm
> somewhat skeptical about whether it really makes sense to think about
> committing the key management part separately. It seems to have no use
> independent of the main feature, and it in fact embeds very specific
For usefulness, it does enable passphrase prompting for the TLS private
key.
> details of how the main feature is expected to work. For example, the
> documentation says that key #0 will be used for data files, and key #1
> for WAL. There seems to be no suggestion that the key management
> portion of this can be used to manage encryption keys generally for
> whatever purposes someone might have; it's all about the needs of a
> particular TDE implementation. Typically, we would not commit
We originally were going to have SQL-level keys, but many felt they
weren't useful.
> something like that separately, or only once the main patch was done,
> with the two commits occurring in a relatively short time period.
> Otherwise, as Bruce already noted, we can end up with something that
> is documented and visible to users but doesn't actually work yet.
Yep, that is the risk.
> Some more specific comments on data-encryption.sgml:
>
> * The documentation explains that the purpose of having a WAL key
> separate from the data file key is so that the data file keys can
> "eventually" be rotated. It's not clear whether this means that we
> might eventually have that feature or that we might eventually be able
> to rotate, after failing over. If this kind of thing is possible,
> we'll eventually need documentation on how to do it.
I have clarified that saying "future release".
> * The reasons for use a DEK and a KEK are not explained. I realize
> it's not an uncommon practice and that other systems do it, but I
> think a few sentences of explanation wouldn't be a bad idea. Even if
> we are supposing that hackers who want to have input into this feature
> have to be knowledgeable about cryptography, I don't think we can
> reasonably suppose that for users.
I added a little about that in the docs.
> * "For example" is at one point followed by a period rather than a
> colon or comma.
Fixed.
> * In the "Internals" subsection, the last sentence doesn't seem to be
> grammatical. I wonder if it's missing the word "or"'.
Fixed.
> * The part about integrity-checking keys on startup isn't clear. It
> makes it sound like we still have a copy of the KEK lying around
> someplace against which we can compare, which I assume is not the case
> since it would be really insecure.
I rewored that entire section. See if it is better now.
> * I think it's going to be pretty important that we can easily switch
> to other cryptographic algorithms as they are discovered, so I don't
> like the fact that this is tied specifically to AES. (In fact,
> kmgr_utils.h makes it sound like we're specifically locked into
> AES256, but that contradicts the documentation, so I guess there's
> some clarification needed here about what exactly KMGR_CLUSTER_KEY_LEN
> is doing.) As far as possible we should try to make this generic, like
> supporting any cipher that SSL has which has property X. It seems
> relatively inevitable that every currently popular cryptographic
> algorithm will at some point in the future be judged weak and
> worthless, just as has already happened with MD5 and some variants of
> SHA, both of which used to be considered state of the art. It seems
> equally inevitable that new and stronger algorithms will continued to
> be devised, and we'll want to adopt those easily.
That is a nifty idea. Right now I just pass the integer length around,
and store it in pg_control, but if we define macros, we can easily
abstract this and easily allow for new methods. If others like that, I
will start on it now.
> I'm not sure to what extent this a serious flaw in the patch and to
> what extent it's just a matter of tweaking the wording of some things,
> but I think this is actually an extremely critical design point where
> we had better be certain we've got it right. Few things would be
> sadder than to get a TDE feature and then have to rip it out again
> because it couldn't be upgraded to work with newer crypto algorithms
> with reasonable effort.
Yep.
> Notes on other parts of the documentation:
>
> * The documentation for initdb -K doesn't list the valid values of the
> parameter, only the default. Probably we should be specifying an
Fixed.
> algorithm here and not just a bit count. Otherwise, like I say above,
> what happens when AES gives way to something else? It'd be easy to say
> -K BFT256 instead of -K AES256, but if AES is assumed and it's no
> longer what we want them we have problems. This kind of thing probably
> needs to be cleaned up in a bunch of places.
Again, I can do that if people like it.
> * I don't see the point of saying "a passphrase or PIN." We don't need
> to document that your passphrase might happen to only contain digits.
Well, PIN is what the Yubikey and PIV devices call it, so I thought I
should give specific examples of inputs.
> * pg_alterckey's description of "repair" is hard to understand. It
> doesn't really explain why or how this would be necessary, and it begs
> the question of why we'd ever leave things in a state that requires
> repair. This is sketched out in code comments elsewhere, but I think
> at least some of it needs to be explained in the documentation as
> well. (Incidentally, I don't think the comments at the top of
> recover_failure will survive a visit from pgindent, though I might be
> wrong about that.)
Fixed with rewording. Better?
> * The changes to config.sgml say "Sample script" instead of "Sample scripts".
Fixed.
> * I don't think that the documentation of %R is very clear, or
> adequate for someone to make effective use of it. If I wanted to use
> %R, how would I ensure that a value is available?
Fixed, use -R on server start.
> * The changes to allfiles.sgml add pg_alterckey.sgml in the wrong
> place and include an incorrect whitespace change.
Uh, the whitespace change was to align the column. I will review and
push that separately.
> * It's odd that "pg_alterckey" describes itself as "technically"
> changing the KEK. Isn't that just what it does, not a technicality? I
> imagine we'll ultimately need a way to change a DEK as well, because
> otherwise the use of a separate key for the WAL wouldn't accomplish
> the intended goal.
"technically" removed. I kind of wanted to say "in detail" or something
like that, but removing the word is fine. Change-only patch attached so
you can see the changes more easily.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EnterpriseDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
key_changes.diff (text/x-diff, 12.8K) download
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8f62e884b1..e6e4f346f1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1460,20 +1460,21 @@ include_dir 'conf.d'
<para>
The command must print the passphrase to the standard output
and exit with code 0. It can prompt from the terminal if
- <option>--authprompt</option> is used. In the parameter value,
- <literal>%R</literal> represents the file descriptor number opened
- to the terminal that started the server. A file descriptor is only
- available if enabled at server start. If <literal>%R</literal>
- is used and no file descriptor is available, the server will not
- start. Value <literal>%p</literal> is replaced by a pre-defined
- prompt string. (Write <literal>%%</literal> for a literal
- <literal>%</literal>.) Note that the prompt string will probably
- contain whitespace, so be sure to quote its use adequately.
- Newlines are stripped from the end of the output if present.
+ <option>--authprompt</option> is used. In the parameter
+ value, <literal>%R</literal> is replaced by a file descriptor
+ number opened to the terminal that started the server. A file
+ descriptor is only available if enabled at server start via
+ <option>-R</option>. If <literal>%R</literal> is specified and
+ no file descriptor is available, the server will not start. Value
+ <literal>%p</literal> is replaced by a pre-defined prompt string.
+ (Write <literal>%%</literal> for a literal <literal>%</literal>.)
+ Note that the prompt string will probably contain whitespace,
+ so be sure to quote its use adequately. Newlines are stripped
+ from the end of the output if present.
</para>
<para>
- Sample script can be found in
+ Sample scripts can be found in
<filename>$SHAREDIR/auth_commands</filename>,
where <literal>$SHAREDIR</literal> means the
<productname>PostgreSQL</productname> installation's shared-data
@@ -7874,15 +7875,16 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
server start.
</para>
<para>
- The command must print the cluster key to the standard output as
- 64 hexadecimal characters, and exit with code 0. The command
- can prompt for the passphrase or PIN from the terminal if
- <option>--authprompt</option> is used. In the parameter value,
- <literal>%R</literal> represents the file descriptor number opened
- to the terminal that started the server. A file descriptor is only
- available if enabled at server start. If <literal>%R</literal>
- is used and no file descriptor is available, the server will not
- start. Value <literal>%p</literal> is replaced by a pre-defined
+ The command must print the cluster key to the standard
+ output as 64 hexadecimal characters, and exit with code 0.
+ The command can prompt for the passphrase or PIN from the
+ terminal if <option>--authprompt</option> is used. In the
+ parameter value, <literal>%R</literal> is replaced by a file
+ descriptor number opened to the terminal that started the server.
+ A file descriptor is only available if enabled at server start
+ via <option>-R</option>. If <literal>%R</literal> is specified
+ and no file descriptor is available, the server will not start.
+ Value <literal>%p</literal> is replaced by a pre-defined
prompt string. Value <literal>%d</literal> is replaced by the
directory containing the keys; this is useful if the command
must create files with the keys, e.g., to store a cluster-level
diff --git a/doc/src/sgml/database-encryption.sgml b/doc/src/sgml/database-encryption.sgml
index cb62f1d975..8eece94322 100644
--- a/doc/src/sgml/database-encryption.sgml
+++ b/doc/src/sgml/database-encryption.sgml
@@ -24,8 +24,8 @@
Key one is used to encrypt write-ahead log (WAL) files. Two different
keys are used so that primary and standby servers can use different zero
(heap/index/temp) keys, but the same one (WAL) key, so that these keys
- can eventually be rotated by switching the primary to the standby
- and then changing the WAL key.
+ can (in a future release) be rotated by switching the primary to the
+ standby and then changing the WAL key.
</para>
<para>
@@ -36,7 +36,10 @@
each time the server is started. This key prevents anyone with access
to the database directories from decrypting the data because they do
not know the second-level key which encrypted the first-level keys
- which encrypted the database cluster files.
+ which encrypted the database cluster files. This key can be easily
+ changed via <command>pg_alterckey</command> without requiring any
+ changes to the the data files or <command>WAL</command> files, which
+ are encrypted with the data keys.
</para>
<sect1 id="encryption-file-encryption">
@@ -54,7 +57,7 @@
<filename>postgresql.conf</filename> must match for the database
cluster to start. Note that the cluster key command
passed to <command>initdb</command> must return a key of
- 64 hexadecimal characters. For example.
+ 64 hexadecimal characters. For example:
<programlisting>
initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
</programlisting>
@@ -71,7 +74,8 @@ initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
the key encryption key (KEK) supplied by the cluster key command before
being stored in the database directory. The key or passphrase that
derives the key must be supplied from the terminal or stored in a
- trusted key store, such as key vault software, hardware security module.
+ trusted key store, such as key vault software or a hardware security
+ module.
</para>
<para>
@@ -81,17 +85,22 @@ initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
<varname>cluster_key_command</varname> command will be executed
and the cluster key retrieved. The data encryption keys in the
<filename>pg_cryptokeys</filename> directory will then be decrypted
- using the supplied key and integrity-checked to ensure it
- matches the initdb-supplied key. If this check fails, the
- server will refuse to start.
+ using the supplied key and integrity-checked to ensure it matches the
+ initdb-supplied key. (If this check fails, the server will refuse
+ to start.) The cluster encryption key will then be removed from
+ system memory. The decrypted data encryption keys will remain in
+ shared memory until the server is stopped.
</para>
<para>
- The data encryption keys are randomly generated and can be 128, 192,
- or 256-bits in length. They are encrypted by the key encryption key
- (KEK) using Advanced Encryption Standard (<acronym>AES256</acronym>)
- encryption in Key Wrap Padded Mode, which also provides KEK
- authentication.
+ The data encryption keys are randomly generated and can be
+ 128, 192, or 256-bits in length. They are encrypted by the
+ key encryption key (KEK) using Advanced Encryption Standard
+ (<acronym>AES256</acronym>) encryption in Key Wrap Padded
+ Mode, which also provides KEK authentication; see <ulink
+ url="https://tools.ietf.org/html/rfc5649">RFC 5649</ulink>. While
+ 128-bit encryption is sufficient for most sites, 256-bit encryption
+ is thought to be more immune to future quantum cryptographic attacks.
</para>
</sect1>
</chapter>
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index 4e9504e11b..92bad3150c 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -189,7 +189,6 @@ Complete list of usable sgml source files in this directory.
<!ENTITY values SYSTEM "values.sgml">
<!-- applications and utilities -->
-<!ENTITY pgalterckey SYSTEM "pg_alterckey.sgml">
<!ENTITY clusterdb SYSTEM "clusterdb.sgml">
<!ENTITY createdb SYSTEM "createdb.sgml">
<!ENTITY createuser SYSTEM "createuser.sgml">
@@ -197,6 +196,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY dropuser SYSTEM "dropuser.sgml">
<!ENTITY ecpgRef SYSTEM "ecpg-ref.sgml">
<!ENTITY initdb SYSTEM "initdb.sgml">
+<!ENTITY pgalterckey SYSTEM "pg_alterckey.sgml">
<!ENTITY pgarchivecleanup SYSTEM "pgarchivecleanup.sgml">
<!ENTITY pgBasebackup SYSTEM "pg_basebackup.sgml">
<!ENTITY pgbench SYSTEM "pgbench.sgml">
diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml
index 25342d1c19..4596489146 100644
--- a/doc/src/sgml/ref/initdb.sgml
+++ b/doc/src/sgml/ref/initdb.sgml
@@ -242,7 +242,7 @@ PostgreSQL documentation
<listitem>
<para>
Specifies the number of bits for the file encryption keys. The
- default is 128 bits.
+ value values are 128 (the default), 192, and 256.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_alterckey.sgml b/doc/src/sgml/ref/pg_alterckey.sgml
index f80946dcc6..867c439b8e 100644
--- a/doc/src/sgml/ref/pg_alterckey.sgml
+++ b/doc/src/sgml/ref/pg_alterckey.sgml
@@ -78,14 +78,14 @@ PostgreSQL documentation
</para>
<para>
- Technically, <command>pg_alterckey</command> changes the key
- encryption key (<acronym>KEK</acronym>) which encrypts the data
- encryption keys; it does not change the data encryption keys. It does
- this by decrypting each data encryption key using the <replaceable
+ <command>pg_alterckey</command> changes the key encryption key
+ (<acronym>KEK</acronym>) which encrypts the data encryption keys;
+ it does not change the data encryption keys. It does this by
+ decrypting each data encryption key using the <replaceable
class="parameter">old_cluster_key_command</replaceable>,
re-encrypting it using the <replaceable
- class="parameter">new_cluster_key_command</replaceable>, and
- then writes the result back to the cluster directory.
+ class="parameter">new_cluster_key_command</replaceable>, and then
+ writes the result back to the cluster directory.
</para>
<para>
@@ -95,13 +95,26 @@ PostgreSQL documentation
arguments to specify retrieval of the old or new key.
</para>
+ <para>
+ <command>pg_alterckey</command> manages data encryption keys,
+ which are critical to allowing Postgres to access its decrypted
+ data. For this reason, it is very careful to preserve these
+ keys in most possible failure conditions, e.g., operating system
+ failure during cluster encryption key rotation.
+ </para>
+
<para>
When started, <command>pg_alterckey</command> repairs any files that
- remain from previous <command>pg_alterckey</command> failures before
- altering the cluster key. To perform only the repair task,
- use the <option>--repair</option> option. The server will not start
- if repair is needed, though a running server is unaffected by an
- unrepaired cluster key configuration.
+ remain from previous failures before altering the cluster encryption
+ key. During this repair phase, <command>pg_alterckey</command> will
+ either roll back the cluster key or roll forward the changes that
+ were previously requested. The server will not start if repair is
+ needed, though a running server is unaffected by an unrepaired cluster
+ key configuration. Therefore, if <command>pg_alterckey</command>
+ fails for any reason, it is recommended you run the command with
+ <option>--repair</option> to simply roll back or forward any previous
+ changes. This will report if it rolled the cluster key back or forward,
+ and then run the command again to change the cluster key if needed.
</para>
<para>
diff --git a/src/bin/pg_alterckey/pg_alterckey.c b/src/bin/pg_alterckey/pg_alterckey.c
index 6a809037b8..50e168306a 100644
--- a/src/bin/pg_alterckey/pg_alterckey.c
+++ b/src/bin/pg_alterckey/pg_alterckey.c
@@ -325,7 +325,7 @@ create_lockfile(void)
if (repair_mode)
printf("old lock file removed\n");
- /*
+ /* ----------
* pid is no longer running, so remove the lock file.
* This is not 100% safe from concurrent access, e.g.:
*
@@ -343,6 +343,7 @@ create_lockfile(void)
* even more error-prone, especially since this might happen
* on server start. Many PG tools seem to have problems with
* concurrent access.
+ * ----------
*/
unlink(pid_path);
@@ -380,6 +381,7 @@ create_lockfile(void)
}
/*
+ * ----------
* recover_failure
*
* A previous pg_alterckey might have failed, so it might need recovery.
@@ -406,6 +408,7 @@ create_lockfile(void)
* remove old and new X X X
*
* We don't handle the abnormal cases, just report an error.
+ * ----------
*/
static void
recover_failure(void)
* Re: Key management with tests
@ 2021-01-25 19:18 Bruce Momjian <bruce@momjian.us>
parent: Tom Kincaid <tomjohnkincaid@gmail.com>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-25 19:18 UTC (permalink / raw)
To: Tom Kincaid <tomjohnkincaid@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 18, 2021 at 05:47:34PM -0500, Tom Kincaid wrote:
> I met with Bruce and Stephen this afternoon to discuss the feedback
> we received so far (prior to Robert's note which I haven't fully
> digested yet)
> on this patch.
>
> Here is what we plan to do:
>
> 1) Bruce is going to gather all the details from the Wiki and build a
> README for the TDE Key Management patch. In addition, it will include
> details about the implementation, the data structures involved and the
> locks that are taken and general technical implementation approach.
...
> Could we get feedback if this feels like enough to get this patch
> (which will include just the Key Management portion of TDE) to a state
> where it can be reviewed and assuming the review issues are resolved
> with consensus be committed?
Attached is an updated patch that has the requested changes:
* broken into seven parts
* test script converted from shell to Perl
* added README for every new directory
* moved text from wiki to READMEs where appropriate
* included Robert's suggestions, including the ability to add
future non-AES crypto methods
* fixes for pg_alterckey PGDATA arg processing
The patch is attached, and is also here:
https://github.com/postgres/postgres/compare/master...bmomjian:key.patch
Questions:
* What changes do people want to this patch set?
* Do we want it applied, even though it might need to be hidden for PG
14?
* If not, how do people build on this patch? Using the commitfest
links or github URL?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
1-crypto.diff.gz (application/gzip, 9.4K) download
2-backend.diff.gz (application/gzip, 6.2K) download
3-common.diff.gz (application/gzip, 9.0K) download
4-pg_alterckey.diff.gz (application/gzip, 8.1K) download
5-bin.diff.gz (application/gzip, 9.3K) download
6-test.diff.gz (application/gzip, 82.5K) download
7-key.diff (text/x-diff, 7.9K) download
From b3251edebfac62c095d973189a652c3925db4a24 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Mon, 25 Jan 2021 14:15:57 -0500
Subject: [PATCH] key squash commit
---
doc/src/sgml/database-encryption.sgml (new) | 124 ++++++++++++++++++++
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/installation.sgml | 5 +-
doc/src/sgml/postgres.sgml | 1 +
4 files changed, 129 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/database-encryption.sgml b/doc/src/sgml/database-encryption.sgml
new file mode 100644
index 0000000000..e8e1e69ec7
--- /dev/null
+++ b/doc/src/sgml/database-encryption.sgml
@@ -0,0 +1,124 @@
+<!-- doc/src/sgml/database-encryption.sgml -->
+
+<chapter id="database-file-encryption">
+ <title>Cluster File Encryption</title>
+
+ <indexterm zone="database-file-encryption">
+ <primary>Cluster File Encryption</primary>
+ </indexterm>
+
+ <para>
+ The purpose of cluster file encryption is to prevent users with read
+ access to the directories used to store database files and write-ahead
+ log files from being able to access the data stored in those files.
+ For example, when using cluster file encryption, users who have read
+ access to the cluster directories for backup purposes will not be able
+ to decrypt the data stored in these files. It also protects against
+ decrypted data access after media theft.
+ </para>
+
+ <para>
+ File system write access can allow for unauthorized file system data
+ decryption if the writes can be used to weaken the system's security
+ and this weakened system is later supplied with externally-stored keys.
+ This also does not protect from users who have read access to system
+ memory.
+ </para>
+
+ <para>
+ Cluster file encryption uses two levels of encryption. The first level
+ is data encryption keys, specifically keys zero and one. Key zero is
+ the key used to encrypt database heap and index files which are stored in
+ the file system, plus temporary files created during database operation.
+ Key one is used to encrypt write-ahead log (WAL) files. Two different
+ keys are used so that primary and standby servers can use different zero
+ (heap/index/temp) keys, but the same one (WAL) key, so that these keys
+ can (in a future release) be rotated by switching the primary to the
+ standby and then changing the WAL key.
+ </para>
+
+ <para>
+ The second level of encryption is a key used to encrypt first-level
+ keys. This type of key is often referred to as a Key Encryption Key
+ (<acronym>KEK</acronym>). This key is <emphasis>not</emphasis> stored
+ in the file system, but provided at <command>initdb</command> time and
+ each time the server is started. This key prevents anyone with access
+ to the database directories from decrypting the data because they do
+ not know the second-level key which encrypted the first-level keys
+ which encrypted the database cluster files. This key can be easily
+ changed via <command>pg_alterckey</command> without requiring any
+ changes to the the data files or <command>WAL</command> files, which
+ are encrypted with the data keys.
+ </para>
+
+ <sect1 id="encryption-file-encryption">
+ <title>Initialization</title>
+
+ <para>
+ Cluster file encryption is enabled when
+ <productname>PostgreSQL</productname> is built
+ with <literal>--with-openssl</literal> and <xref
+ linkend="app-initdb-cluster-key-command"/> is specified
+ during <command>initdb</command>. The cluster key
+ provided by the <option>--cluster-key-command</option>
+ option during <command>initdb</command> and the one generated
+ by <xref linkend="guc-cluster-key-command"/> in the
+ <filename>postgresql.conf</filename> must match for the database
+ cluster to start. Note that the cluster key command
+ passed to <command>initdb</command> must return a key of
+ 64 hexadecimal characters. For example:
+<programlisting>
+initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
+</programlisting>
+ </para>
+ </sect1>
+
+ <sect1 id="key-encryption-key">
+ <title>Internals</title>
+
+ <para>
+ During the <command>initdb</command> process, if
+ <option>--cluster-key-command</option> is specified, two data-level
+ encryption keys are created. These two keys are then encrypted with
+ the key encryption key (KEK) supplied by the cluster key command before
+ being stored in the database directory. The key or passphrase that
+ derives the key must be supplied from the terminal or stored in a
+ trusted key store, such as key vault software or a hardware security
+ module.
+ </para>
+
+ <para>
+ If the <productname>PostgreSQL</productname> server has
+ been initialized to require a cluster key, each time the
+ server starts the <filename>postgresql.conf</filename>
+ <varname>cluster_key_command</varname> command will be executed
+ and the cluster key retrieved. The data encryption keys in the
+ <filename>pg_cryptokeys</filename> directory will then be decrypted
+ using the supplied key and integrity-checked to ensure it matches the
+ initdb-supplied key. (If this check fails, the server will refuse
+ to start.) The cluster encryption key will then be removed from
+ system memory. The decrypted data encryption keys will remain in
+ shared memory until the server is stopped.
+ </para>
+
+ <para>
+ The data encryption keys are randomly generated and can be 128, 192,
+ or 256-bits in length, depending on whether <literal>AES128</literal>,
+ <literal>AES192</literal>, or <literal>AES256</literal> is specified.
+ They are encrypted by the key encryption key (KEK) using Advanced
+ Encryption Standard (<acronym>AES256</acronym>) encryption in Key
+ Wrap Padded Mode, which also provides KEK authentication; see <ulink
+ url="https://tools.ietf.org/html/rfc5649">RFC 5649</ulink>. While
+ 128-bit encryption is sufficient for most sites, 256-bit encryption
+ is thought to be more immune to future quantum cryptographic attacks
+ </para>
+
+ <para>.
+ If you prefer to create the random keys on your own, you can create
+ a empty directory with a <filename>pg_cryptokeys/live</filename>
+ subdirectory, generate the keys there using your tools. and use the
+ <command>initdb</command> <option>--copy-encryption-keys</option>
+ to copy those keys into the newly-created cluster.
+ </para>
+ </sect1>
+</chapter>
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 38e8aa0bbf..b96f4ace6c 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -49,6 +49,7 @@
<!ENTITY wal SYSTEM "wal.sgml">
<!ENTITY logical-replication SYSTEM "logical-replication.sgml">
<!ENTITY jit SYSTEM "jit.sgml">
+<!ENTITY database-encryption SYSTEM "database-encryption.sgml">
<!-- programmer's guide -->
<!ENTITY bgworker SYSTEM "bgworker.sgml">
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index a53389b728..a1473bbf8d 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -976,8 +976,9 @@ build-postgresql:
<listitem>
<para>
Build with support for <acronym>SSL</acronym> (encrypted)
- connections. This requires the <productname>OpenSSL</productname>
- package to be installed. <filename>configure</filename> will check
+ connections and cluster file encryption. This requires the
+ <productname>OpenSSL</productname> package to be installed.
+ <filename>configure</filename> will check
for the required header files and libraries to make sure that
your <productname>OpenSSL</productname> installation is sufficient
before proceeding.
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 730d5fdc34..0ea7da604b 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -171,6 +171,7 @@ break is not needed in a wider output rendering.
&wal;
&logical-replication;
&jit;
+ &database-encryption;
®ress;
</part>
--
2.20.1
* Re: Key management with tests
@ 2021-01-25 23:12 Alvaro Herrera <alvherre@alvh.no-ip.org>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Alvaro Herrera @ 2021-01-25 23:12 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
In patch 1,
* The docs are not clear on what happens if --auth-prompt is not given
but an auth prompt is required for the program to work. Should it exit
with a status other than 0?
* BootStrapKmgr claims it is called by initdb, but that doesn't seem to
be the case.
* Also, BootStrapKmgr is the only one that checks USE_OPENSSL; what if a
with-openssl build inits the datadir, and then a non-openssl runs it?
What if it's the other way around? I think you'd get a failure in
stat() ...
* ... oh, KMGR_DIR_PID is used but not defined anywhere. Is it defined
in some later commit? If so, then I think you've chosen to split the
patch series wrong.
May I suggest to use "git format-patch" to produce the patch files? When
working with a series like this, trying to do patch handling manually
like you seem to be doing, is much more time-consuming and error prone.
For example, with a branch containing individual commits, you could use
git rebase -i origin/master -x "make install check-world"
or similar, so that each commit is built and tested individually.
--
Álvaro Herrera Valdivia, Chile
Al principio era UNIX, y UNIX habló y dijo: "Hello world\n".
No dijo "Hello New Jersey\n", ni "Hello USA\n".
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-26 00:09 Bruce Momjian <bruce@momjian.us>
parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-26 00:09 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 25, 2021 at 08:12:01PM -0300, Álvaro Herrera wrote:
> In patch 1,
>
> * The docs are not clear on what happens if --auth-prompt is not given
> but an auth prompt is required for the program to work. Should it exit
> with a status other than 0?
Uh, I think the docs talk about this:
It can prompt from the terminal if
option>--authprompt</option> is used. In the parameter
value, <literal>%R</literal> is replaced by a file descriptor
number opened to the terminal that started the server. A file
descriptor is only available if enabled at server start via
<option>-R</option>. If <literal>%R</literal> is specified and
no file descriptor is available, the server will not start.
The code is:
case 'R':
{
char fd_str[20];
if (terminal_fd == -1)
{
ereport(ERROR,
(errcode(ERRCODE_INTERNAL_ERROR),
errmsg("cluster key command referenced %%R, but --authprompt not specified")));
}
Does that help?
> * BootStrapKmgr claims it is called by initdb, but that doesn't seem to
> be the case.
Well, initdb starts the postmaster in --boot mode, and that calls
BootStrapKmgr(). Does that help?
> * Also, BootStrapKmgr is the only one that checks USE_OPENSSL; what if a
> with-openssl build inits the datadir, and then a non-openssl runs it?
> What if it's the other way around? I think you'd get a failure in
> stat() ...
Wow, I never considered that. I have added a check to InitializeKmgr().
Thanks.
> * ... oh, KMGR_DIR_PID is used but not defined anywhere. Is it defined
> in some later commit? If so, then I think you've chosen to split the
> patch series wrong.
OK, fixed. It is in include/common/kmgr_utils.c, which was in #3.
> May I suggest to use "git format-patch" to produce the patch files? When
> working with a series like this, trying to do patch handling manually
> like you seem to be doing, is much more time-consuming and error prone.
> For example, with a branch containing individual commits, you could use
> git rebase -i origin/master -x "make install check-world"
> or similar, so that each commit is built and tested individually.
I used "git format-patch". Are you asking for seven commits that then
generate seven files via one format-patch run? Or is the primary issue
that you want compile testing for each patch?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-26 03:27 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-26 03:27 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 25, 2021 at 07:09:44PM -0500, Bruce Momjian wrote:
> > May I suggest to use "git format-patch" to produce the patch files? When
> > working with a series like this, trying to do patch handling manually
> > like you seem to be doing, is much more time-consuming and error prone.
> > For example, with a branch containing individual commits, you could use
> > git rebase -i origin/master -x "make install check-world"
> > or similar, so that each commit is built and tested individually.
>
> I used "git format-patch". Are you asking for seven commits that then
> generate seven files via one format-patch run? Or is the primary issue
> that you want compile testing for each patch?
The attached patch meets both criteria. I also clarified the README on
how initdb calls those functions.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
key.diff.gz (application/gzip, 124.7K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-26 16:15 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-26 16:15 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Jan 25, 2021 at 10:27:18PM -0500, Bruce Momjian wrote:
> On Mon, Jan 25, 2021 at 07:09:44PM -0500, Bruce Momjian wrote:
> > > May I suggest to use "git format-patch" to produce the patch files? When
> > > working with a series like this, trying to do patch handling manually
> > > like you seem to be doing, is much more time-consuming and error prone.
> > > For example, with a branch containing individual commits, you could use
> > > git rebase -i origin/master -x "make install check-world"
> > > or similar, so that each commit is built and tested individually.
> >
> > I used "git format-patch". Are you asking for seven commits that then
> > generate seven files via one format-patch run? Or is the primary issue
> > that you want compile testing for each patch?
>
> The attached patch meets both criteria. I also clarified the README on
> how initdb calls those functions.
This version fixes OpenSSL detection and improves docs for initdb
interactions.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
key.diff.gz (application/gzip, 124.7K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-26 20:24 Robert Haas <robertmhaas@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Robert Haas @ 2021-01-26 20:24 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 26, 2021 at 11:15 AM Bruce Momjian <bruce@momjian.us> wrote:
> This version fixes OpenSSL detection and improves docs for initdb
> interactions.
Hi,
I'm wondering whether you've considered storing all the keys in one
file instead of a file per key. The reason I ask is that it seems to
me that the key rotation procedure would be a lot simpler if it were
all in one file. You could just create a temporary file and atomically
rename it over the existing file. If you see a temporary file you're
always free to remove it. This is a lot simpler than what you have
right now. The "repair" concept pretty much goes away completely,
which seems nice. Granted I don't know exactly how to store multiple
keys in one file, but I bet there's some way to do it.
The way in which you are posting these patches is quite unlike what
most people do when posting patches to this list. You seem to have
generated a bunch of patches using 'git format-patch' but then
concatenated them all together in a single file. It would be helpful
if you could do this more like the way that is now standard on this
list. Not only that, but the patches don't have meaningful commit
messages in them, and don't seem to be meaningfully split for easy
review. They just say things like 'crypto squash commit'. Compare this
to for example what I did on the "cleaning up a few CLOG-related
things" thread where the commits appear in a logical sequence, and
each one has a meaningful commit message. Or here's an example from
someone else --
http://postgr.es/m/be72abfa-e62e-eb81-4e70-1b57fe6dc9e2@amazon.com --
and note the inclusion of authorship information in the commit
messages, so that the source of the code can be easily understood.
The README in src/backend/crypto does not explain how the scripts in
that directory are intended to be used. If I want to use AWS Secrets
Manager with this feature, I can see that I should use
ckey_aws.sh.sample as a basis for that integration, but I don't know
what I should do with the script because the README says nothing about
it. I am frankly pretty doubtful about the idea of shipping a bunch of
/bin/sh scripts as a best practice; for one thing, that's totally
unusable on Windows, and it also means that this is dependent on
/bin/sh existing and having the behavior we expect and on all the
other executables in these scripts as well. But, at the very least,
there needs to be a clearer explanation of how the scripts are
intended to be used, which parts people are supposed to modify, what
arguments they're going to get called with, and things like that.
The comments in cipher.c and cipher_openssl.c could be improved to
explain that they are alternatives to each other. Perhaps the former
could be renamed to something like cipher_failure.c or cipher_noimpl.c
for clarity.
I believe that a StaticAssertStmt could be used to check the length of
the encryption_methods[] array, so that if someone changes
NUM_ENCRYPTION_METHODS without updating the array, compilation fails.
See UserAuthName[] for an example of how to do this.
You seem to have omitted to update the documentation with the names of
the new wait events that you added.
In process_postgres_switches(), when there's a multi-line comment
followed by a single line of actual code, I prefer to include braces
around the whole thing. There might be some disagreement on what is
best here, though.
What are the consequences of the placement of the code in
PostgresMain() for processes other than user backends and walsenders?
I think that the way you have it, background workers would not have
access to keys, nor auxiliary processes like the checkpointer ... at
least in the EXEC_BACKEND case. In the non-EXEC_BACKEND case you have
the postmaster doing it, so then I'm not sure why it has to be redone
for every backend. Won't they just inherit the data from the
postmaster? Has this code been meaningfully tested on Windows? How do
we know that it works? Maybe we need to think about adding some
asserts that guarantee that any process that attempts to access a
buffer has the key manager initialized; I bet such assertions would
fail at least on Windows as the code looks now.
I don't think it makes sense to think about committing this to v14. I
believe it only makes sense if we have a TDE patch that is relatively
close to committable that can be used with it. I also don't think that
this patch is in good enough shape to commit yet in terms of where
it's at in terms of quality; I think it needs more review first,
hopefully including review from people who can comment intelligently
specifically on the cryptography aspects of it. However, the
challenges don't seem insurmountable. There's also still some question
in my mind about whether the design choices here (one KEK, 2 DEKs, one
for data and one for WAL) have enough consensus. I don't have a
considered opinion on that, partly because I'm not quite sure what the
reasonable alternatives are, but it seems that other people had some
questions about it, IIUC.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-26 22:53 Bruce Momjian <bruce@momjian.us>
parent: Robert Haas <robertmhaas@gmail.com>
0 siblings, 2 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-26 22:53 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 26, 2021 at 03:24:30PM -0500, Robert Haas wrote:
> On Tue, Jan 26, 2021 at 11:15 AM Bruce Momjian <bruce@momjian.us> wrote:
> > This version fixes OpenSSL detection and improves docs for initdb
> > interactions.
>
> Hi,
>
> I'm wondering whether you've considered storing all the keys in one
> file instead of a file per key. The reason I ask is that it seems to
> me that the key rotation procedure would be a lot simpler if it were
> all in one file. You could just create a temporary file and atomically
> rename it over the existing file. If you see a temporary file you're
> always free to remove it. This is a lot simpler than what you have
> right now. The "repair" concept pretty much goes away completely,
> which seems nice. Granted I don't know exactly how to store multiple
> keys in one file, but I bet there's some way to do it.
We envisioned allowing heap/index key rotation by having a standby with
the same WAL key as the primary but different heap/index keys so that we
can failover to the standby to change the heap/index key and then change
the WAL key. This separation allows that. We also might need some
additional keys later and this allows that. I do like simplicity, but
the complexity here seems to serve a need.
> The way in which you are posting these patches is quite unlike what
> most people do when posting patches to this list. You seem to have
> generated a bunch of patches using 'git format-patch' but then
> concatenated them all together in a single file. It would be helpful
> if you could do this more like the way that is now standard on this
> list. Not only that, but the patches don't have meaningful commit
What is the standard? You want seven separate files? I can do that.
> messages in them, and don't seem to be meaningfully split for easy
> review. They just say things like 'crypto squash commit'. Compare this
Yes, the feature is at the backend, common, /bin, and test levels. I
was able to separate out the bin, pg_alterckey and test stuff, but the
backend interactions were hard to split.
> to for example what I did on the "cleaning up a few CLOG-related
> things" thread where the commits appear in a logical sequence, and
> each one has a meaningful commit message. Or here's an example from
> someone else --
> http://postgr.es/m/be72abfa-e62e-eb81-4e70-1b57fe6dc9e2@amazon.com --
> and note the inclusion of authorship information in the commit
> messages, so that the source of the code can be easily understood.
I see. I am not sure how to do that easily for all the pieces.
> The README in src/backend/crypto does not explain how the scripts in
> that directory are intended to be used. If I want to use AWS Secrets
> Manager with this feature, I can see that I should use
> ckey_aws.sh.sample as a basis for that integration, but I don't know
> what I should do with the script because the README says nothing about
> it. I am frankly pretty doubtful about the idea of shipping a bunch of
> /bin/sh scripts as a best practice; for one thing, that's totally
> unusable on Windows, and it also means that this is dependent on
> /bin/sh existing and having the behavior we expect and on all the
> other executables in these scripts as well. But, at the very least,
> there needs to be a clearer explanation of how the scripts are
> intended to be used, which parts people are supposed to modify, what
> arguments they're going to get called with, and things like that.
I added comments to most of the scripts. I don't know what more I can
do, or what other language would be appropriate.
> The comments in cipher.c and cipher_openssl.c could be improved to
> explain that they are alternatives to each other. Perhaps the former
> could be renamed to something like cipher_failure.c or cipher_noimpl.c
> for clarity.
This follows the way cryptohash.c and cryptohash_openssl.c are done. I
did just add comments to the top of cipher.c and cipher_openssl.c to be
just like cryptohash versions.
> I believe that a StaticAssertStmt could be used to check the length of
> the encryption_methods[] array, so that if someone changes
> NUM_ENCRYPTION_METHODS without updating the array, compilation fails.
> See UserAuthName[] for an example of how to do this.
Sure, good idea, done.
> You seem to have omitted to update the documentation with the names of
> the new wait events that you added.
OK, added.
> In process_postgres_switches(), when there's a multi-line comment
> followed by a single line of actual code, I prefer to include braces
> around the whole thing. There might be some disagreement on what is
> best here, though.
OK, done.
> What are the consequences of the placement of the code in
> PostgresMain() for processes other than user backends and walsenders?
> I think that the way you have it, background workers would not have
> access to keys, nor auxiliary processes like the checkpointer ... at
Well, there are three cases, --boot mode, postmaster mode, and postgres
single-user mode. I tried to have all those cases only unwrap the keys
once and store them in shared memory, or in boot mode, in local memory.
As far as I know, the startup does it once and everyone else uses shared
memory to access it.
> least in the EXEC_BACKEND case. In the non-EXEC_BACKEND case you have
> the postmaster doing it, so then I'm not sure why it has to be redone
> for every backend. Won't they just inherit the data from the
For postgres --single.
> postmaster? Has this code been meaningfully tested on Windows? How do
No, just by the cfbot Windows machine.
> we know that it works? Maybe we need to think about adding some
> asserts that guarantee that any process that attempts to access a
> buffer has the key manager initialized; I bet such assertions would
> fail at least on Windows as the code looks now.
Are you saying we should set a global variable and throw an error if it
is accessed without the array being initialized?
> I don't think it makes sense to think about committing this to v14. I
> believe it only makes sense if we have a TDE patch that is relatively
> close to committable that can be used with it. I also don't think that
> this patch is in good enough shape to commit yet in terms of where
> it's at in terms of quality; I think it needs more review first,
> hopefully including review from people who can comment intelligently
> specifically on the cryptography aspects of it. However, the
> challenges don't seem insurmountable. There's also still some question
> in my mind about whether the design choices here (one KEK, 2 DEKs, one
> for data and one for WAL) have enough consensus. I don't have a
> considered opinion on that, partly because I'm not quite sure what the
> reasonable alternatives are, but it seems that other people had some
> questions about it, IIUC.
While I am willing to make requested adjustments to the patch, I don't
plan to work on this feaure any further, assuming your analysis above is
correct. If after years we are still not sure this is the right
direction, I don't see any point in moving forward with the later
pieces, which are even more complicated. I will join the group of
people that feel there will never be consensus on implementing this
feature in the community, so it is not worth trying.
I would also like to add a "not wanted" entry for this feature on the
TODO list, baaed on the feature's limited usefulness, but I already
asked about that and no one seems to feel we don't want it.
I now better understand why the OpenSSL project has had such serious
problems in the past.
Updated patch attached as seven attachments.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
0001-crypto-squash-commit.patch.gz (application/gzip, 10.8K) download
0002-common-squash-commit.patch.gz (application/gzip, 7.3K) download
0003-backend-squash-commit.patch.gz (application/gzip, 6.8K) download
0004-pg_alterckey-squash-commit.patch.gz (application/gzip, 8.1K) download
0005-bin-squash-commit.patch.gz (application/gzip, 9.3K) download
0006-test-squash-commit.patch.gz (application/gzip, 82.5K) download
0007-key-squash-commit.patch.gz (application/gzip, 3.3K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-27 14:11 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-01-27 14:11 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; Tom Kincaid <tomjohnkincaid@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Jan 26, 2021 at 05:53:01PM -0500, Bruce Momjian wrote:
> On Tue, Jan 26, 2021 at 03:24:30PM -0500, Robert Haas wrote:
> > I'm wondering whether you've considered storing all the keys in one
> > file instead of a file per key. The reason I ask is that it seems to
> > me that the key rotation procedure would be a lot simpler if it were
> > all in one file. You could just create a temporary file and atomically
> > rename it over the existing file. If you see a temporary file you're
> > always free to remove it. This is a lot simpler than what you have
> > right now. The "repair" concept pretty much goes away completely,
> > which seems nice. Granted I don't know exactly how to store multiple
> > keys in one file, but I bet there's some way to do it.
>
> We envisioned allowing heap/index key rotation by having a standby with
> the same WAL key as the primary but different heap/index keys so that we
> can failover to the standby to change the heap/index key and then change
> the WAL key. This separation allows that. We also might need some
> additional keys later and this allows that. I do like simplicity, but
> the complexity here seems to serve a need.
Just to close this issue, several scripts, e,g., PIV, AWS, need to store
data to indicate the cluster encryption key used, and those need to be
kept synchronized with the wrapped data keys. Having separate
directories for each cluster key version allows that to work cleanly.
> > The README in src/backend/crypto does not explain how the scripts in
> > that directory are intended to be used. If I want to use AWS Secrets
> > Manager with this feature, I can see that I should use
> > ckey_aws.sh.sample as a basis for that integration, but I don't know
> > what I should do with the script because the README says nothing about
> > it. I am frankly pretty doubtful about the idea of shipping a bunch of
> > /bin/sh scripts as a best practice; for one thing, that's totally
> > unusable on Windows, and it also means that this is dependent on
> > /bin/sh existing and having the behavior we expect and on all the
> > other executables in these scripts as well. But, at the very least,
> > there needs to be a clearer explanation of how the scripts are
> > intended to be used, which parts people are supposed to modify, what
> > arguments they're going to get called with, and things like that.
>
> I added comments to most of the scripts. I don't know what more I can
> do, or what other language would be appropriate.
I think someone would need to write Windows versions of these scripts.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-28 19:41 Tom Kincaid <tomjohnkincaid@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 1 reply; 114+ messages in thread
From: Tom Kincaid @ 2021-01-28 19:41 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hello,
> > I don't think it makes sense to think about committing this to v14. I
> > believe it only makes sense if we have a TDE patch that is relatively
> > close to committable that can be used with it. I also don't think that
> > this patch is in good enough shape to commit yet in terms of where
> > it's at in terms of quality; I think it needs more review first,
> > hopefully including review from people who can comment intelligently
> > specifically on the cryptography aspects of it. However, the
> > challenges don't seem insurmountable. There's also still some question
> > in my mind about whether the design choices here (one KEK, 2 DEKs, one
> > for data and one for WAL) have enough consensus. I don't have a
> > considered opinion on that, partly because I'm not quite sure what the
> > reasonable alternatives are, but it seems that other people had some
> > questions about it, IIUC.
>
> While I am willing to make requested adjustments to the patch, I don't
> plan to work on this feaure any further, assuming your analysis above is
> correct. If after years we are still not sure this is the right
> direction, I don't see any point in moving forward with the later
> pieces, which are even more complicated. I will join the group of
> people that feel there will never be consensus on implementing this
> feature in the community, so it is not worth trying.
>
> I would also like to add a "not wanted" entry for this feature on the
> TODO list, baaed on the feature's limited usefulness, but I already
> asked about that and no one seems to feel we don't want it.
>
I want to avoid seeing this happen. As a result of a lot of customer and
user discussions, around their criteria for choosing a database, I believe
TDE is an important feature and having it appear with a "not-wanted" tag
will keep the version of PostgreSQL released by the community out of
certain (and possibly growing) number of deployment scenarios which I don't
think anybody wants to see.
I think the current situation to be as follows (if I missed something
please let me know):
1) We need to get the current patch for Key Management reviewed and tested
further.
I spoke to Bruce just now he will see if can get somebody to do this.
2) We need to start working on the actual TDE implementation and get it
pretty close to final before we start committing smaller portions of the
feature.
Unfortunately, on this front, the only things, I think I can offer are:
a) Ask for volunteers to work on the TDE implementation.
b) Facilitate the work between volunteers.
c) Prod folks along and cheer as we go.
So I will start with (a), do we have any volunteers who feel they can
contribute regularly for a while and would like to be part of a team that
moves this forward?
I now better understand why the OpenSSL project has had such serious
> problems in the past.
>
> Updated patch attached as seven attachments.
>
> --
> Bruce Momjian <bruce@momjian.us> https://momjian.us
> EDB https://enterprisedb.com
>
> The usefulness of a cup is in its emptiness, Bruce Lee
>
>
--
Thomas John Kincaid
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-28 20:22 Bruce Momjian <bruce@momjian.us>
parent: Tom Kincaid <tomjohnkincaid@gmail.com>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-01-28 20:22 UTC (permalink / raw)
To: Tom Kincaid <tomjohnkincaid@gmail.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote:
> I would also like to add a "not wanted" entry for this feature on the
> TODO list, baaed on the feature's limited usefulness, but I already
> asked about that and no one seems to feel we don't want it.
>
>
> I want to avoid seeing this happen. As a result of a lot of customer and user
> discussions, around their criteria for choosing a database, I believe TDE is an
> important feature and having it appear with a "not-wanted" tag will keep the
> version of PostgreSQL released by the community out of certain (and possibly
> growing) number of deployment scenarios which I don't think anybody wants to
> see.
With pg_upgrade, I could work on it out of the tree until it became
popular, with a small non-user-visible part in the backend. With the
Windows port, the port wasn't really visible to users until it we ready.
For the key management part of TDE, it can't be done outside the tree,
and it is user-visible before it is useful, so that restricts how much
incremental work can be committed to the tree for TDE. I highlighted
that concern emails months ago, but never got any feedback --- now it
seems people are realizing the ramifications of that.
> I think the current situation to be as follows (if I missed something please
> let me know):
>
> 1) We need to get the current patch for Key Management reviewed and tested
> further.
>
> I spoke to Bruce just now he will see if can get somebody to do this.
Well, if we don't get anyone committed to working on the data encryption
part of TDE, the key management part is useless, so why review/test it
further?
Although Sawada-san and Stephen Frost worked on the patch, they have not
commented much on my additions, and only a few others have commented on
the code, and there has been no discussion on who is working on the next
steps. This indicates to me that there is little interest in moving
this feature forward, which is why I started asking if it could be
labeled as "not wanted".
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-29 08:05 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 2 replies; 114+ messages in thread
From: Masahiko Sawada @ 2021-01-29 08:05 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 29, 2021 at 5:22 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote:
> > I would also like to add a "not wanted" entry for this feature on the
> > TODO list, baaed on the feature's limited usefulness, but I already
> > asked about that and no one seems to feel we don't want it.
> >
> >
> > I want to avoid seeing this happen. As a result of a lot of customer and user
> > discussions, around their criteria for choosing a database, I believe TDE is an
> > important feature and having it appear with a "not-wanted" tag will keep the
> > version of PostgreSQL released by the community out of certain (and possibly
> > growing) number of deployment scenarios which I don't think anybody wants to
> > see.
>
> With pg_upgrade, I could work on it out of the tree until it became
> popular, with a small non-user-visible part in the backend. With the
> Windows port, the port wasn't really visible to users until it we ready.
>
> For the key management part of TDE, it can't be done outside the tree,
> and it is user-visible before it is useful, so that restricts how much
> incremental work can be committed to the tree for TDE. I highlighted
> that concern emails months ago, but never got any feedback --- now it
> seems people are realizing the ramifications of that.
>
> > I think the current situation to be as follows (if I missed something please
> > let me know):
> >
> > 1) We need to get the current patch for Key Management reviewed and tested
> > further.
> >
> > I spoke to Bruce just now he will see if can get somebody to do this.
>
> Well, if we don't get anyone committed to working on the data encryption
> part of TDE, the key management part is useless, so why review/test it
> further?
>
> Although Sawada-san and Stephen Frost worked on the patch, they have not
> commented much on my additions, and only a few others have commented on
> the code, and there has been no discussion on who is working on the next
> steps. This indicates to me that there is little interest in moving
> this feature forward,
TBH I’m confused a bit about the recent situation of this patch, but I
can contribute to KMS work by discussing, writing, reviewing, and
testing the patch. Also, I can work on the data encryption part of TDE
(we need more discussion on that though). If the community concerns
about the high-level design and thinks the design reviews by
cryptography experts are still needed, we would need to do that first
since the data encryption part of TDE depends on KMS. As far as I
know, we have done that many times on pgsql-hackers, on offl-line and
including the discussion on the past proposal, etc but given that the
community still has a concern, it seems that we haven’t been able to
share the details of the discussion enough that led to the design
decision or the design is still not good. Honestly, I’m not sure how
this feature can get consensus. But maybe we would need to have a
break from refining the patch now and we need to marshal the
discussions so far and the point behind the design so that everyone
can understand why this feature is designed in that way. To do that,
it might be a good start to sort the wiki page since it has data
encryption part, KMS, and ToDo mixed.
Regards,
--
Masahiko Sawada
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-29 22:40 Stephen Frost <sfrost@snowman.net>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 2 replies; 114+ messages in thread
From: Stephen Frost @ 2021-01-29 22:40 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Bruce Momjian <bruce@momjian.us>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Masahiko Sawada (sawada.mshk@gmail.com) wrote:
> On Fri, Jan 29, 2021 at 5:22 AM Bruce Momjian <bruce@momjian.us> wrote:
> > On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote:
> > > I would also like to add a "not wanted" entry for this feature on the
> > > TODO list, baaed on the feature's limited usefulness, but I already
> > > asked about that and no one seems to feel we don't want it.
> > >
> > >
> > > I want to avoid seeing this happen. As a result of a lot of customer and user
> > > discussions, around their criteria for choosing a database, I believe TDE is an
> > > important feature and having it appear with a "not-wanted" tag will keep the
> > > version of PostgreSQL released by the community out of certain (and possibly
> > > growing) number of deployment scenarios which I don't think anybody wants to
> > > see.
> >
> > With pg_upgrade, I could work on it out of the tree until it became
> > popular, with a small non-user-visible part in the backend. With the
> > Windows port, the port wasn't really visible to users until it we ready.
> >
> > For the key management part of TDE, it can't be done outside the tree,
> > and it is user-visible before it is useful, so that restricts how much
> > incremental work can be committed to the tree for TDE. I highlighted
> > that concern emails months ago, but never got any feedback --- now it
> > seems people are realizing the ramifications of that.
> >
> > > I think the current situation to be as follows (if I missed something please
> > > let me know):
> > >
> > > 1) We need to get the current patch for Key Management reviewed and tested
> > > further.
> > >
> > > I spoke to Bruce just now he will see if can get somebody to do this.
> >
> > Well, if we don't get anyone committed to working on the data encryption
> > part of TDE, the key management part is useless, so why review/test it
> > further?
> >
> > Although Sawada-san and Stephen Frost worked on the patch, they have not
> > commented much on my additions, and only a few others have commented on
> > the code, and there has been no discussion on who is working on the next
> > steps. This indicates to me that there is little interest in moving
> > this feature forward,
>
> TBH I’m confused a bit about the recent situation of this patch, but I
> can contribute to KMS work by discussing, writing, reviewing, and
> testing the patch. Also, I can work on the data encryption part of TDE
> (we need more discussion on that though). If the community concerns
> about the high-level design and thinks the design reviews by
> cryptography experts are still needed, we would need to do that first
> since the data encryption part of TDE depends on KMS. As far as I
> know, we have done that many times on pgsql-hackers, on offl-line and
> including the discussion on the past proposal, etc but given that the
> community still has a concern, it seems that we haven’t been able to
> share the details of the discussion enough that led to the design
> decision or the design is still not good. Honestly, I’m not sure how
> this feature can get consensus. But maybe we would need to have a
> break from refining the patch now and we need to marshal the
> discussions so far and the point behind the design so that everyone
> can understand why this feature is designed in that way. To do that,
> it might be a good start to sort the wiki page since it has data
> encryption part, KMS, and ToDo mixed.
I hope it's pretty clear that I'm also very much in support of both this
effort with the KMS and of TDE in general- TDE is specifically,
repeatedly, called out as a capability whose lack is blocking PG from
being able to be used for certain use-cases that it would otherwise be
well suited for, and that's really unfortunate.
I appreciate the recent discussion and reviews of the KMS in particular,
and of the patches which have been sent enabling TDE based on the KMS
patches. Having them be relatively independent seems to be an ongoing
concern and perhaps we should figure out a way to more clearly put them
together. That is- the KMS patches have been posted on one thread, and
TDE PoC patches which use the KMS patches have been on another thread,
leading some to not realize that there's already been TDE PoC work done
based on the KMS patches. Seems like it might make sense to get one
patch set which goes all the way from the KMS and includes the TDE PoC,
even if they don't all go in at once.
I'm happy to go look over the KMS patches again if that'd be helpful and
to comment on the TDE PoC. I can also spend some time trying to improve
on each, as I've already done. A few of the larger concerns that I have
revolve around how to store integrity information (I've tried to find a
way to make room for such information in our existing page layout and,
perhaps unsuprisingly, it's far from trivial to do so in a way that will
avoid breaking the existing page layout, or where the same set of
binaries could work on both unencrypted pages and encrypted pages with
integrity validation information, and that's a problem that we really
should consider trying to solve...), and how to automate key rotation
(one of the nice things about Bruce's approach to storing the keys is
that we're leveraging the filesystem as an index- it's easy to see how
we might extend the key-per-file approach to allow us to, say, have a
different key for every 32GB of LSN, but if we tried to put all of the
keys into a single file then we'd have to figure out an indexing
solution for it which would allow us to find the key we need to decrypt
a given page...). I tend to agree with Bruce that we need to take
these things in steps, getting each piece implemented as we go. Maybe
we can do that in a separate repo for a time and then bring it all
together, as a few on this thread have voiced, but there's no doubt that
this is a large project and it's hard to see how we could possibly
commit all of it at once.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-01-30 13:23 Tom Kincaid <tomjohnkincaid@gmail.com>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 2 replies; 114+ messages in thread
From: Tom Kincaid @ 2021-01-30 13:23 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Thanks Stephen, Bruce and Masahiko,
> > discussions so far and the point behind the design so that everyone
> > can understand why this feature is designed in that way. To do that,
> > it might be a good start to sort the wiki page since it has data
> > encryption part, KMS, and ToDo mixed.
>
> I hope it's pretty clear that I'm also very much in support of both this
> effort with the KMS and of TDE in general- TDE is specifically,
> repeatedly, called out as a capability whose lack is blocking PG from
> being able to be used for certain use-cases that it would otherwise be
> well suited for, and that's really unfortunate.
>
It is clear you are supportive.
As you know, I share your point of view that PG adoption is suffering for
certain use cases because it does not have TDE.
I appreciate the recent discussion and reviews of the KMS in particular,
> and of the patches which have been sent enabling TDE based on the KMS
> patches. Having them be relatively independent seems to be an ongoing
> concern and perhaps we should figure out a way to more clearly put them
> together. That is- the KMS patches have been posted on one thread, and
> TDE PoC patches which use the KMS patches have been on another thread,
> leading some to not realize that there's already been TDE PoC work done
> based on the KMS patches. Seems like it might make sense to get one
> patch set which goes all the way from the KMS and includes the TDE PoC,
> even if they don't all go in at once.
>
Sounds good, thanks Masahiko, let's see if we can get consensus on the
approach for moving this forward see below.
>
> together, as a few on this thread have voiced, but there's no doubt that
> this is a large project and it's hard to see how we could possibly
> commit all of it at once.
>
I propose that we meet to discuss what approach we want to use to move TDE
forward. We then start a new thread with a proposal on the approach
and finalize it via community consensus. I will invite Bruce, Stephen and
Masahiko to this meeting. If anybody else would like to participate in this
discussion and subsequently in the effort to get TDE in PG1x, please let me
know. Assuming Bruce, Stephen and Masahiko are down for this, I (or a
volunteer from this meeting) will post the proposal for how we move this
patch forward in another thread. Hopefully, we can get consensus on that
and subsequently restart the execution of delivering this feature.
> Thanks!
>
> Stephen
>
--
Thomas John Kincaid
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 02:05 Moon, Insung <tsukiwamoon.pgsql@gmail.com>
parent: Tom Kincaid <tomjohnkincaid@gmail.com>
1 sibling, 0 replies; 114+ messages in thread
From: Moon, Insung @ 2021-02-01 02:05 UTC (permalink / raw)
To: Tom Kincaid <tomjohnkincaid@gmail.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <sawada.mshk@gmail.com>; Bruce Momjian <bruce@momjian.us>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Dear All.
Thank you for all opinions and discussions regarding the KMS/TDE function.
First of all, to get to the point of this email,
I want to participate in anything I can do (review or development)
when TDE related development is in progress.
If there is a meeting related to it, I can't communicate because of my
poor English skills, but I would like to attend if it is only possible
to listen.
I didn't understand KMS and didn't participate in the direct
development, so I didn't comment on anything so far. Still, when TDE
development starts, I wanted to join in the discussion and meeting if
there was anything I could do.
However, since I have a complicated and insufficient English ability
to communicate in English, maybe I will rarely say anything in
meetings (voice and video meetings).
But I would like to attend the discussion if it is only possible to listen.
Also, if the wiki page and other mail threads related to TDE start,
I'll join in discussions if there is anything I can do.
Best regards.
Moon.
On Sat, Jan 30, 2021 at 10:23 PM Tom Kincaid <tomjohnkincaid@gmail.com> wrote:
>
>
>
>
>
> Thanks Stephen, Bruce and Masahiko,
>
>>
>> > discussions so far and the point behind the design so that everyone
>> > can understand why this feature is designed in that way. To do that,
>> > it might be a good start to sort the wiki page since it has data
>> > encryption part, KMS, and ToDo mixed.
>>
>> I hope it's pretty clear that I'm also very much in support of both this
>> effort with the KMS and of TDE in general- TDE is specifically,
>> repeatedly, called out as a capability whose lack is blocking PG from
>> being able to be used for certain use-cases that it would otherwise be
>> well suited for, and that's really unfortunate.
>
>
> It is clear you are supportive.
>
> As you know, I share your point of view that PG adoption is suffering for certain use cases because it does not have TDE.
>
>> I appreciate the recent discussion and reviews of the KMS in particular,
>> and of the patches which have been sent enabling TDE based on the KMS
>> patches. Having them be relatively independent seems to be an ongoing
>> concern and perhaps we should figure out a way to more clearly put them
>> together. That is- the KMS patches have been posted on one thread, and
>> TDE PoC patches which use the KMS patches have been on another thread,
>> leading some to not realize that there's already been TDE PoC work done
>> based on the KMS patches. Seems like it might make sense to get one
>> patch set which goes all the way from the KMS and includes the TDE PoC,
>> even if they don't all go in at once.
>
>
> Sounds good, thanks Masahiko, let's see if we can get consensus on the approach for moving this forward see below.
>
>>
>>
>> together, as a few on this thread have voiced, but there's no doubt that
>> this is a large project and it's hard to see how we could possibly
>> commit all of it at once.
>
>
> I propose that we meet to discuss what approach we want to use to move TDE forward. We then start a new thread with a proposal on the approach and finalize it via community consensus. I will invite Bruce, Stephen and Masahiko to this meeting. If anybody else would like to participate in this discussion and subsequently in the effort to get TDE in PG1x, please let me know. Assuming Bruce, Stephen and Masahiko are down for this, I (or a volunteer from this meeting) will post the proposal for how we move this patch forward in another thread. Hopefully, we can get consensus on that and subsequently restart the execution of delivering this feature.
>
>
>
>
>>
>> Thanks!
>>
>> Stephen
>
>
>
> --
> Thomas John Kincaid
>
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 22:16 Bruce Momjian <bruce@momjian.us>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-02-01 22:16 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 29, 2021 at 05:05:06PM +0900, Masahiko Sawada wrote:
> TBH I’m confused a bit about the recent situation of this patch, but
> I
Yes, it is easy to get confused.
> can contribute to KMS work by discussing, writing, reviewing, and
> testing the patch. Also, I can work on the data encryption part of TDE
Great.
> (we need more discussion on that though). If the community concerns
> about the high-level design and thinks the design reviews by
> cryptography experts are still needed, we would need to do that first
> since the data encryption part of TDE depends on KMS. As far as I
I totally agree. While we don't need to commit the key management patch
to the tree before moving forward, we should have agreement on the key
management patch before doing more work on this. If we can't agree on
the key management part, there is no value in working further, as I
stated in an earlier email.
> know, we have done that many times on pgsql-hackers, on offl-line and
> including the discussion on the past proposal, etc but given that the
> community still has a concern, it seems that we haven’t been able
> to share the details of the discussion enough that led to the design
> decision or the design is still not good. Honestly, I’m not sure how
> this feature can get consensus. But maybe we would need to have a
Yes, I am also confused.
> break from refining the patch now and we need to marshal the
> discussions so far and the point behind the design so that everyone
> can understand why this feature is designed in that way. To do that,
> it might be a good start to sort the wiki page since it has data
> encryption part, KMS, and ToDo mixed.
What I ended up doing is to moving the majority of the
non-data-encryption part of the wiki into the patch, either in docs or
README files, since people asked for more of this in the patch, and
having the information in two places is confusing.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 22:43 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-01 22:43 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Jan 29, 2021 at 05:40:37PM -0500, Stephen Frost wrote:
> I hope it's pretty clear that I'm also very much in support of both this
> effort with the KMS and of TDE in general- TDE is specifically,
Yes, thanks. I know we have privately talked about this recently, but
it is nice to have it in public like this.
> repeatedly, called out as a capability whose lack is blocking PG from
> being able to be used for certain use-cases that it would otherwise be
> well suited for, and that's really unfortunate.
So, below, I am going to copy two doc paragraphs from the patch:
The purpose of cluster file encryption is to prevent users with read
access to the directories used to store database files and write-ahead
log files from being able to access the data stored in those files.
For example, when using cluster file encryption, users who have read
access to the cluster directories for backup purposes will not be able
to decrypt the data stored in these files. It also protects against
decrypted data access after media theft.
File system write access can allow for unauthorized file system data
decryption if the writes can be used to weaken the system's security
and this weakened system is later supplied with externally-stored keys.
This also does not protect from users who have read access to system
memory. This also does not detect or protect against users with write
access from removing or modifying database files.
Given what I said above, is the value of this feature for compliance, or
for actual additional security? If it just compliance, are we willing
to add all of this code just for that, even if it has limited security
value? We should answer this question now, and if we don't want it,
let's document that so users know and can consider alternatives.
FYI, I don't think we can detect or protect against writers modifying
the data files --- even if we could do it on a block level, they could
remove trailing pages (might cause index lookup failures) or copy
pages from other tables at the same offset. Therefore, I think we can
only offer viewing security, not modification detection/prevention.
> I appreciate the recent discussion and reviews of the KMS in particular,
> and of the patches which have been sent enabling TDE based on the KMS
> patches. Having them be relatively independent seems to be an ongoing
I was thinking some more and I have received productive feedback from at
least eight people on the key management patch, which is very good.
> concern and perhaps we should figure out a way to more clearly put them
> together. That is- the KMS patches have been posted on one thread, and
> TDE PoC patches which use the KMS patches have been on another thread,
> leading some to not realize that there's already been TDE PoC work done
> based on the KMS patches. Seems like it might make sense to get one
> patch set which goes all the way from the KMS and includes the TDE PoC,
> even if they don't all go in at once.
Uh, it is worse than that. Some people saw comments about the TDE PoC
patch (e.g., buffer pins) and thought they were related to the KMS
patch, so they thought the KMS patch wasn't ready. Now, I am not saying
the KMS patch is ready, but comments on the TDE PoC patch are unrelated
to the KMS patch being ready.
I think the TDE PoC was a big positive because it showed the KMS patch
being used for the actual use-case we are planning, so it was truly a
proof-of-concept.
> I'm happy to go look over the KMS patches again if that'd be helpful and
> to comment on the TDE PoC. I can also spend some time trying to improve
I think we eventually need a full review of the TDE PoC, combined with
the Cybertec patch, and the wiki, to get them all aligned. However, as
I said already, let's get the KMS patch approved, even if we don't apply
it now, so we know we are on an approved foundation.
> on each, as I've already done. A few of the larger concerns that I have
> revolve around how to store integrity information (I've tried to find a
> way to make room for such information in our existing page layout and,
> perhaps unsuprisingly, it's far from trivial to do so in a way that will
> avoid breaking the existing page layout, or where the same set of
> binaries could work on both unencrypted pages and encrypted pages with
> integrity validation information, and that's a problem that we really
As stated above, I think we only need a byte or two for the hint bit
counter (used in the IV), as I don't think the GCM verification bytes
will add any additional security, and I bet we can find a byte or two.
We do need a separate discussion on this, either here or privately.
> should consider trying to solve...), and how to automate key rotation
> (one of the nice things about Bruce's approach to storing the keys is
> that we're leveraging the filesystem as an index- it's easy to see how
> we might extend the key-per-file approach to allow us to, say, have a
> different key for every 32GB of LSN, but if we tried to put all of the
> keys into a single file then we'd have to figure out an indexing
> solution for it which would allow us to find the key we need to decrypt
> a given page...). I tend to agree with Bruce that we need to take
Yeah, yuck on that plan. I was very happy how the per-version directory
worked with scripts that needed to store matching state.
> these things in steps, getting each piece implemented as we go. Maybe
> we can do that in a separate repo for a time and then bring it all
> together, as a few on this thread have voiced, but there's no doubt that
> this is a large project and it's hard to see how we could possibly
> commit all of it at once.
I was putting stuff in a git tree/URL; you can see it here:
https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
https://github.com/postgres/postgres/compare/master...bmomjian:key.patch
https://github.com/postgres/postgres/compare/master...bmomjian:key
However, people wanted persistent patches attached, so I started doing that.
Attached is the current patch set.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments:
0001-crypto-squash-commit.patch.gz (application/gzip, 10.8K) download
0002-common-squash-commit.patch.gz (application/gzip, 7.3K) download
0003-backend-squash-commit.patch.gz (application/gzip, 6.8K) download
0004-pg_alterckey-squash-commit.patch.gz (application/gzip, 8.1K) download
0005-bin-squash-commit.patch.gz (application/gzip, 9.3K) download
0006-test-squash-commit.patch.gz (application/gzip, 82.5K) download
0007-key-squash-commit.patch.gz (application/gzip, 3.3K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 22:44 Bruce Momjian <bruce@momjian.us>
parent: Tom Kincaid <tomjohnkincaid@gmail.com>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-01 22:44 UTC (permalink / raw)
To: Tom Kincaid <tomjohnkincaid@gmail.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <sawada.mshk@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote:
> I propose that we meet to discuss what approach we want to use to move TDE
> forward. We then start a new thread with a proposal on the approach
> and finalize it via community consensus. I will invite Bruce, Stephen and
> Masahiko to this meeting. If anybody else would like to participate in this
> discussion and subsequently in the effort to get TDE in PG1x, please let me
> know. Assuming Bruce, Stephen and Masahiko are down for this, I (or a volunteer
> from this meeting) will post the proposal for how we move this patch forward in
> another thread. Hopefully, we can get consensus on that and subsequently
> restart the execution of delivering this feature.
We got complaints that decisions were not publicly discussed, or were
too long, so I am not sure this helps.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 23:31 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-02-01 23:31 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Fri, Jan 29, 2021 at 05:40:37PM -0500, Stephen Frost wrote:
> > I hope it's pretty clear that I'm also very much in support of both this
> > effort with the KMS and of TDE in general- TDE is specifically,
>
> Yes, thanks. I know we have privately talked about this recently, but
> it is nice to have it in public like this.
Certainly happy to lend my support and to spend some time working on
this to move it forward.
> > repeatedly, called out as a capability whose lack is blocking PG from
> > being able to be used for certain use-cases that it would otherwise be
> > well suited for, and that's really unfortunate.
>
> So, below, I am going to copy two doc paragraphs from the patch:
>
> The purpose of cluster file encryption is to prevent users with read
> access to the directories used to store database files and write-ahead
> log files from being able to access the data stored in those files.
> For example, when using cluster file encryption, users who have read
> access to the cluster directories for backup purposes will not be able
> to decrypt the data stored in these files. It also protects against
> decrypted data access after media theft.
That's one valid use-case and it particularly makes sense to consider,
now that we support group read-access to the data cluster. The last
line seems a bit unclear- I would update it to say:
Cluster file encryption also provides data-at-rest security, protecting
users from data loss should the physical media on which the cluster is
stored be stolen, improperly deprovisioned (not wiped or destroyed), or
otherwise ends up in the hands of an attacker.
> File system write access can allow for unauthorized file system data
> decryption if the writes can be used to weaken the system's security
> and this weakened system is later supplied with externally-stored keys.
This isn't very clear as to exactly what the concern is or how an
attacker would be able to thwart the system if they had write access to
it. An attacker with write access could possibly attempt to replace the
existing keys, but with the key wrapping that we're using, that should
result in just a decryption failure (unless, of course, the attacker has
the actual KEK that was used, but that's not terribly interesting to
worry about since then they could just go access the files directly).
Until and unless we solve the issue around storing the GCM tags for each
page, we will have the risk that an attacker could modify a page in a
manner that we wouldn't detect. This is the biggest concern that I have
currently with the existing TDE patch sets.
There's two options that I see around how to address that issue- either
we arrange to create space in the page for the tag, such as by making
the 'special' space on a page a bit bigger and making sure that
everything understands that, or we'll need to add another fork in which
we store the tags (and possibly other TDE/encryption related
information). If we go with a fork then it should be possible to do WAL
streaming from an unencrypted cluster to an encrypted one, which would
be pretty neat, but it means another fork and another page that has to
be read/written every time we modify a page. Getting some input into
the trade-offs here would be really helpful. I don't think it's really
reasonable to go out with TDE without having figured out the integrity
side. Certainly, when I review things like NIST 800-53, it's very clear
that the requirement is for both confidentiality *and* integrity.
> This also does not protect from users who have read access to system
> memory. This also does not detect or protect against users with write
> access from removing or modifying database files.
The last seems a bit obvious, but the first sentence quoted above is
important to make clear. I might even say:
All of the pages in memory and all of the keys which are used for the
encryption and decryption are stored in the clear in memory and
therefore an attacker who is able to read the memory allocated by
PostgreSQL would be able to decrypt the enitre cluster.
> Given what I said above, is the value of this feature for compliance, or
> for actual additional security? If it just compliance, are we willing
> to add all of this code just for that, even if it has limited security
> value? We should answer this question now, and if we don't want it,
> let's document that so users know and can consider alternatives.
The feature is for both compliance and additional security. While there
are other ways to achieve data-at-rest encryption, they are not always
available, for a variety of reasons.
> FYI, I don't think we can detect or protect against writers modifying
> the data files --- even if we could do it on a block level, they could
> remove trailing pages (might cause index lookup failures) or copy
> pages from other tables at the same offset. Therefore, I think we can
> only offer viewing security, not modification detection/prevention.
Protecting against file modification isn't about finding some way to
make it so that an attacker isn't able to modify the files, it's about
detecting the case where an unauthorized modification has happened.
Clearly if an attacker has gained write access to the system then we
can't protect against the attacker using the access they've gained, but
we can in most cases detect it and that's what we should be doing. It
would be really unfortunate to end up with a solution here that only
provides confidentiality and doesn't address integrity at all, and I
don't really think it's *that* hard to do both. That said, if we must
work at this in pieces and we can get agreement to handle
confidentiality initially and then add integrity later, that might be
reasonable.
> > I appreciate the recent discussion and reviews of the KMS in particular,
> > and of the patches which have been sent enabling TDE based on the KMS
> > patches. Having them be relatively independent seems to be an ongoing
>
> I was thinking some more and I have received productive feedback from at
> least eight people on the key management patch, which is very good.
Agreed.
> > concern and perhaps we should figure out a way to more clearly put them
> > together. That is- the KMS patches have been posted on one thread, and
> > TDE PoC patches which use the KMS patches have been on another thread,
> > leading some to not realize that there's already been TDE PoC work done
> > based on the KMS patches. Seems like it might make sense to get one
> > patch set which goes all the way from the KMS and includes the TDE PoC,
> > even if they don't all go in at once.
>
> Uh, it is worse than that. Some people saw comments about the TDE PoC
> patch (e.g., buffer pins) and thought they were related to the KMS
> patch, so they thought the KMS patch wasn't ready. Now, I am not saying
> the KMS patch is ready, but comments on the TDE PoC patch are unrelated
> to the KMS patch being ready.
I do agree with that and that it can lend to some confusion. I'm not
sure what the right solution there is except to continue to try and work
with those who are interested and to clarify the separation.
> I think the TDE PoC was a big positive because it showed the KMS patch
> being used for the actual use-case we are planning, so it was truly a
> proof-of-concept.
Agreed.
> > I'm happy to go look over the KMS patches again if that'd be helpful and
> > to comment on the TDE PoC. I can also spend some time trying to improve
>
> I think we eventually need a full review of the TDE PoC, combined with
> the Cybertec patch, and the wiki, to get them all aligned. However, as
> I said already, let's get the KMS patch approved, even if we don't apply
> it now, so we know we are on an approved foundation.
While the Cybertec patch is interesting, I'd really like to see
something that's a bit less invasive when it comes to how temporary
files are handled. In particular, I think it'd be possible to have an
API that's very similar to the existing one for serial reading and
writing of files which wouldn't require nearly as many changes to things
like reorderbuffer.c. I also believe there's some things we could do to
avoid having to modify quite as many places when it comes to LSN
assignment, so the base patch isn't as big.
> > on each, as I've already done. A few of the larger concerns that I have
> > revolve around how to store integrity information (I've tried to find a
> > way to make room for such information in our existing page layout and,
> > perhaps unsuprisingly, it's far from trivial to do so in a way that will
> > avoid breaking the existing page layout, or where the same set of
> > binaries could work on both unencrypted pages and encrypted pages with
> > integrity validation information, and that's a problem that we really
>
> As stated above, I think we only need a byte or two for the hint bit
> counter (used in the IV), as I don't think the GCM verification bytes
> will add any additional security, and I bet we can find a byte or two.
> We do need a separate discussion on this, either here or privately.
I have to disagree here- the GCM tag adds integrity which is really
quite important. Happy to chat about it independently, of course.
> > should consider trying to solve...), and how to automate key rotation
> > (one of the nice things about Bruce's approach to storing the keys is
> > that we're leveraging the filesystem as an index- it's easy to see how
> > we might extend the key-per-file approach to allow us to, say, have a
> > different key for every 32GB of LSN, but if we tried to put all of the
> > keys into a single file then we'd have to figure out an indexing
> > solution for it which would allow us to find the key we need to decrypt
> > a given page...). I tend to agree with Bruce that we need to take
>
> Yeah, yuck on that plan. I was very happy how the per-version directory
> worked with scripts that needed to store matching state.
I don't know that it's going to ultimately be the best answer, as we're
essentially using the filesystem as an index, as I mentioned above, but,
yeah, trying to do all of that ourselves during WAL replay doesn't seem
like it would be fun to try and figure out. This is an area that I
would think we'd be able to improve on in the future too- if someone
wants to spend the time coming up with a single-file format that is
indexed in some manner and still provides the guarantees that we need,
we could very likely teach pg_upgrade how to handle that and the data
set we're talking about here is quite small, even if we've got a bunch
of key rotation that's happened.
> > these things in steps, getting each piece implemented as we go. Maybe
> > we can do that in a separate repo for a time and then bring it all
> > together, as a few on this thread have voiced, but there's no doubt that
> > this is a large project and it's hard to see how we could possibly
> > commit all of it at once.
>
> I was putting stuff in a git tree/URL; you can see it here:
>
> https://github.com/postgres/postgres/compare/master...bmomjian:key.diff
> https://github.com/postgres/postgres/compare/master...bmomjian:key.patch
> https://github.com/postgres/postgres/compare/master...bmomjian:key
>
> However, people wanted persistent patches attached, so I started doing that.
> Attached is the current patch set.
Doing both seems likely to be the best option and hopefully will help
everyone see the complete picture.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 23:34 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-02-01 23:34 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote:
> > I propose that we meet to discuss what approach we want to use to move TDE
> > forward. We then start a new thread with a proposal on the approach
> > and finalize it via community consensus. I will invite Bruce, Stephen and
> > Masahiko to this meeting. If anybody else would like to participate in this
> > discussion and subsequently in the effort to get TDE in PG1x, please let me
> > know. Assuming Bruce, Stephen and Masahiko are down for this, I (or a volunteer
> > from this meeting) will post the proposal for how we move this patch forward in
> > another thread. Hopefully, we can get consensus on that and subsequently
> > restart the execution of delivering this feature.
>
> We got complaints that decisions were not publicly discussed, or were
> too long, so I am not sure this helps.
If the notes are published afterwords as an explanation of why certain
choices were made, I suspect it'd be reasonably well received. The
concern about back-room discussions is more that decisions are made
without explanation as to why, provided we avoid that, I believe they
can be helpful.
So, +1 for my part to have the conversation.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-01 23:52 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-02-01 23:52 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Tom Kincaid <tomjohnkincaid@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Feb 1, 2021 at 06:34:53PM -0500, Stephen Frost wrote:
> Greetings,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote:
> > > I propose that we meet to discuss what approach we want to use to move TDE
> > > forward. We then start a new thread with a proposal on the approach
> > > and finalize it via community consensus. I will invite Bruce, Stephen and
> > > Masahiko to this meeting. If anybody else would like to participate in this
> > > discussion and subsequently in the effort to get TDE in PG1x, please let me
> > > know. Assuming Bruce, Stephen and Masahiko are down for this, I (or a volunteer
> > > from this meeting) will post the proposal for how we move this patch forward in
> > > another thread. Hopefully, we can get consensus on that and subsequently
> > > restart the execution of delivering this feature.
> >
> > We got complaints that decisions were not publicly discussed, or were
> > too long, so I am not sure this helps.
>
> If the notes are published afterwords as an explanation of why certain
> choices were made, I suspect it'd be reasonably well received. The
> concern about back-room discussions is more that decisions are made
> without explanation as to why, provided we avoid that, I believe they
> can be helpful.
Well, I thought that was what the wiki was, but I guess not. I did
remove some of the decision logic recently since we had made a final
decision. However, most of the questions were not covered on the wiki,
since, as I said, everyone comes with a different need for details.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-02 00:47 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-02 00:47 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> > The purpose of cluster file encryption is to prevent users with read
> > access to the directories used to store database files and write-ahead
> > log files from being able to access the data stored in those files.
> > For example, when using cluster file encryption, users who have read
> > access to the cluster directories for backup purposes will not be able
> > to decrypt the data stored in these files. It also protects against
> > decrypted data access after media theft.
>
> That's one valid use-case and it particularly makes sense to consider,
> now that we support group read-access to the data cluster. The last
Do enough people use group read-access to be useful?
> line seems a bit unclear- I would update it to say:
> Cluster file encryption also provides data-at-rest security, protecting
> users from data loss should the physical media on which the cluster is
> stored be stolen, improperly deprovisioned (not wiped or destroyed), or
> otherwise ends up in the hands of an attacker.
I have split the section into three paragraphs, trimmed down some of the
suggested text, and added it. Full version below.
> > File system write access can allow for unauthorized file system data
> > decryption if the writes can be used to weaken the system's security
> > and this weakened system is later supplied with externally-stored keys.
>
> This isn't very clear as to exactly what the concern is or how an
> attacker would be able to thwart the system if they had write access to
> it. An attacker with write access could possibly attempt to replace the
> existing keys, but with the key wrapping that we're using, that should
> result in just a decryption failure (unless, of course, the attacker has
> the actual KEK that was used, but that's not terribly interesting to
> worry about since then they could just go access the files directly).
Uh, well, they could modify postgresql.conf to change the script to save
the secret returned by the script before returning it to the PG server.
We could require postgresql.conf to be somewhere secure, but then how do
we know that is secure? I just don't see a clean solution here, but the
idea that you write and then wait for the key to show up seems like a
very valid way of attack, and it took me a while to be able to
articulate it.
> Until and unless we solve the issue around storing the GCM tags for each
> page, we will have the risk that an attacker could modify a page in a
> manner that we wouldn't detect. This is the biggest concern that I have
> currently with the existing TDE patch sets.
Well, GCM certainly can detect page modification, but it can't detect
removing pages from the end of the table, or, since the nonce is
LSN/pageno, you could copy a page from another table that has the same
offset into another table, particularly with partitioning where the
tables have the same columns. We might be able to protect against the
later with some kind of table-id in the nonce, but I don't see how table
truncation can be detected without adding a whole lot of overhead and
complexity. And if we can't protect against those two, why bother with
detecting single-page modifications? We have to do a full job for it to
be useful.
> There's two options that I see around how to address that issue- either
> we arrange to create space in the page for the tag, such as by making
> the 'special' space on a page a bit bigger and making sure that
> everything understands that, or we'll need to add another fork in which
> we store the tags (and possibly other TDE/encryption related
> information). If we go with a fork then it should be possible to do WAL
> streaming from an unencrypted cluster to an encrypted one, which would
> be pretty neat, but it means another fork and another page that has to
> be read/written every time we modify a page. Getting some input into
> the trade-offs here would be really helpful. I don't think it's really
> reasonable to go out with TDE without having figured out the integrity
> side. Certainly, when I review things like NIST 800-53, it's very clear
> that the requirement is for both confidentiality *and* integrity.
Wow, well, if they are both required, and we can't do both, is it
valuable to do just one? Yes, we can do something later, but what if we
have no idea how to implement the second part? Your fork idea above
might need to store some table-id used for the nonce (to prevent copying
from another table) and the number of pages in the table, which fixes
the integrity check issue, but adds a lot of complexity and perhaps
overhead.
> > This also does not protect from users who have read access to system
> > memory. This also does not detect or protect against users with write
> > access from removing or modifying database files.
>
> The last seems a bit obvious, but the first sentence quoted above is
> important to make clear. I might even say:
>
> All of the pages in memory and all of the keys which are used for the
> encryption and decryption are stored in the clear in memory and
> therefore an attacker who is able to read the memory allocated by
> PostgreSQL would be able to decrypt the enitre cluster.
Same as above, full version below.
> > Given what I said above, is the value of this feature for compliance, or
> > for actual additional security? If it just compliance, are we willing
> > to add all of this code just for that, even if it has limited security
> > value? We should answer this question now, and if we don't want it,
> > let's document that so users know and can consider alternatives.
>
> The feature is for both compliance and additional security. While there
> are other ways to achieve data-at-rest encryption, they are not always
> available, for a variety of reasons.
True.
> > FYI, I don't think we can detect or protect against writers modifying
> > the data files --- even if we could do it on a block level, they could
> > remove trailing pages (might cause index lookup failures) or copy
> > pages from other tables at the same offset. Therefore, I think we can
> > only offer viewing security, not modification detection/prevention.
>
> Protecting against file modification isn't about finding some way to
> make it so that an attacker isn't able to modify the files, it's about
> detecting the case where an unauthorized modification has happened.
> Clearly if an attacker has gained write access to the system then we
> can't protect against the attacker using the access they've gained, but
> we can in most cases detect it and that's what we should be doing. It
> would be really unfortunate to end up with a solution here that only
> provides confidentiality and doesn't address integrity at all, and I
> don't really think it's *that* hard to do both. That said, if we must
> work at this in pieces and we can get agreement to handle
> confidentiality initially and then add integrity later, that might be
> reasonable.
See above.
> > > I'm happy to go look over the KMS patches again if that'd be helpful and
> > > to comment on the TDE PoC. I can also spend some time trying to improve
> >
> > I think we eventually need a full review of the TDE PoC, combined with
> > the Cybertec patch, and the wiki, to get them all aligned. However, as
> > I said already, let's get the KMS patch approved, even if we don't apply
> > it now, so we know we are on an approved foundation.
>
> While the Cybertec patch is interesting, I'd really like to see
> something that's a bit less invasive when it comes to how temporary
> files are handled. In particular, I think it'd be possible to have an
> API that's very similar to the existing one for serial reading and
> writing of files which wouldn't require nearly as many changes to things
> like reorderbuffer.c. I also believe there's some things we could do to
> avoid having to modify quite as many places when it comes to LSN
> assignment, so the base patch isn't as big.
Yes, I think we would get the best ideas from all patches.
> > > on each, as I've already done. A few of the larger concerns that I have
> > > revolve around how to store integrity information (I've tried to find a
> > > way to make room for such information in our existing page layout and,
> > > perhaps unsuprisingly, it's far from trivial to do so in a way that will
> > > avoid breaking the existing page layout, or where the same set of
> > > binaries could work on both unencrypted pages and encrypted pages with
> > > integrity validation information, and that's a problem that we really
> >
> > As stated above, I think we only need a byte or two for the hint bit
> > counter (used in the IV), as I don't think the GCM verification bytes
> > will add any additional security, and I bet we can find a byte or two.
> > We do need a separate discussion on this, either here or privately.
>
> I have to disagree here- the GCM tag adds integrity which is really
> quite important. Happy to chat about it independently, of course.
Yeah, see above.
> > > should consider trying to solve...), and how to automate key rotation
> > > (one of the nice things about Bruce's approach to storing the keys is
> > > that we're leveraging the filesystem as an index- it's easy to see how
> > > we might extend the key-per-file approach to allow us to, say, have a
> > > different key for every 32GB of LSN, but if we tried to put all of the
> > > keys into a single file then we'd have to figure out an indexing
> > > solution for it which would allow us to find the key we need to decrypt
> > > a given page...). I tend to agree with Bruce that we need to take
> >
> > Yeah, yuck on that plan. I was very happy how the per-version directory
> > worked with scripts that needed to store matching state.
>
> I don't know that it's going to ultimately be the best answer, as we're
> essentially using the filesystem as an index, as I mentioned above, but,
> yeah, trying to do all of that ourselves during WAL replay doesn't seem
> like it would be fun to try and figure out. This is an area that I
> would think we'd be able to improve on in the future too- if someone
> wants to spend the time coming up with a single-file format that is
> indexed in some manner and still provides the guarantees that we need,
> we could very likely teach pg_upgrade how to handle that and the data
> set we're talking about here is quite small, even if we've got a bunch
> of key rotation that's happened.
I thought we were going to use failover to a standby as our data key
rotation method.
Here is the full doc part you wanted improved:
The purpose of cluster file encryption is to prevent users with read
access to the directories used to store database files and write-ahead
log files from being able to access the data stored in those files.
For example, when using cluster file encryption, users who have read
access to the cluster directories for backup purposes will not be able
to decrypt the data stored in these files. It also provides data-at-rest
security, protecting users from data loss should the physical storage
media be stolen or improperly erased before disposal.
File system write access can allow for unauthorized file system data
decryption if the writes can be used to weaken the system's security
and this weakened system is later supplied with externally-stored keys.
This also does not always detect if users with write access remove or
modify database files.
This also does not protect from users who have read access to system
memory — all in-memory data pages and data encryption keys are
stored unencrypted in memory, so an attacker who is able to read the
PostgreSQL process's memory can decrypt the entire cluster.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-02 15:58 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-02 15:58 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote:
> On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote:
> > * Bruce Momjian (bruce@momjian.us) wrote:
> > > The purpose of cluster file encryption is to prevent users with read
> > > access to the directories used to store database files and write-ahead
> > > log files from being able to access the data stored in those files.
> > > For example, when using cluster file encryption, users who have read
> > > access to the cluster directories for backup purposes will not be able
> > > to decrypt the data stored in these files. It also protects against
> > > decrypted data access after media theft.
> >
> > That's one valid use-case and it particularly makes sense to consider,
> > now that we support group read-access to the data cluster. The last
>
> Do enough people use group read-access to be useful?
I am thinking group read-access might be a requirement for cluster file
encryption to be effective.
> > line seems a bit unclear- I would update it to say:
> > Cluster file encryption also provides data-at-rest security, protecting
> > users from data loss should the physical media on which the cluster is
> > stored be stolen, improperly deprovisioned (not wiped or destroyed), or
> > otherwise ends up in the hands of an attacker.
>
> I have split the section into three paragraphs, trimmed down some of the
> suggested text, and added it. Full version below.
Here is an updated doc description of memory reading:
This also does not protect against users who have read access to
database process memory — all in-memory data pages and data
encryption keys are stored unencrypted in memory, so an attacker who
--> is able to read memory can decrypt the entire cluster. The Postgres
--> operating system user and the operating system administrator, e.g.,
--> the <literal>root</literal> user, have such access.
> > > File system write access can allow for unauthorized file system data
> > > decryption if the writes can be used to weaken the system's security
> > > and this weakened system is later supplied with externally-stored keys.
> >
> > This isn't very clear as to exactly what the concern is or how an
> > attacker would be able to thwart the system if they had write access to
> > it. An attacker with write access could possibly attempt to replace the
> > existing keys, but with the key wrapping that we're using, that should
> > result in just a decryption failure (unless, of course, the attacker has
> > the actual KEK that was used, but that's not terribly interesting to
> > worry about since then they could just go access the files directly).
>
> Uh, well, they could modify postgresql.conf to change the script to save
> the secret returned by the script before returning it to the PG server.
> We could require postgresql.conf to be somewhere secure, but then how do
> we know that is secure? I just don't see a clean solution here, but the
> idea that you write and then wait for the key to show up seems like a
> very valid way of attack, and it took me a while to be able to
> articulate it.
Let's suppose you lock down your cluster --- the non-PGDATA files are
owned by root, postgresql.conf and pg_hba.conf are moved out of PGDATA
and are not writable by the database OS user, or we have the PGDATA
directory on another server, so the adversary can only write to the
remote PGDATA directory.
What can they do? Well, they can't modify pg_proc to add a shared
library since pg_proc is encrypted, so we have to focus on files needed
before encryption starts or files that can't be easily encrypted. They
could create postgresql.conf.auto in PGDATA, and modify
cluster_key_command to capture the key, or they could modify preload
libraries or archive command to call a command to read memory as the PG
OS user and write the key out somewhere, or use the key to rewrite the
database files --- those wouldn't even need a database restart, just a
reload.
They could also modify pg_xact files so that, even though the heap/index
files are encrypted, how the contents of those files are interpreted
would change.
In summary, to detect malicious user writes, you would need to protect
the files used before encryption starts (root owned or owned by another
user?), and encrypt all files after encryption starts --- any other
approach would probably leave open attack vectors, and I don't think
there is sufficient community desire to add such boundaries.
How do other database systems guarantee to detect malicious writes?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-03 15:33 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-02-03 15:33 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote:
> > On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote:
> > > * Bruce Momjian (bruce@momjian.us) wrote:
> > > > The purpose of cluster file encryption is to prevent users with read
> > > > access to the directories used to store database files and write-ahead
> > > > log files from being able to access the data stored in those files.
> > > > For example, when using cluster file encryption, users who have read
> > > > access to the cluster directories for backup purposes will not be able
> > > > to decrypt the data stored in these files. It also protects against
> > > > decrypted data access after media theft.
> > >
> > > That's one valid use-case and it particularly makes sense to consider,
> > > now that we support group read-access to the data cluster. The last
> >
> > Do enough people use group read-access to be useful?
>
> I am thinking group read-access might be a requirement for cluster file
> encryption to be effective.
People certainly do use group read-access, but I don't see that as being
a requirement for cluster file encryption to be effective, it's just one
thing TDE can address, among others, as discussed.
> > > line seems a bit unclear- I would update it to say:
> > > Cluster file encryption also provides data-at-rest security, protecting
> > > users from data loss should the physical media on which the cluster is
> > > stored be stolen, improperly deprovisioned (not wiped or destroyed), or
> > > otherwise ends up in the hands of an attacker.
> >
> > I have split the section into three paragraphs, trimmed down some of the
> > suggested text, and added it. Full version below.
>
> Here is an updated doc description of memory reading:
>
> This also does not protect against users who have read access to
> database process memory — all in-memory data pages and data
> encryption keys are stored unencrypted in memory, so an attacker who
> --> is able to read memory can decrypt the entire cluster. The Postgres
> --> operating system user and the operating system administrator, e.g.,
> --> the <literal>root</literal> user, have such access.
That's helpful, +1.
> > > > File system write access can allow for unauthorized file system data
> > > > decryption if the writes can be used to weaken the system's security
> > > > and this weakened system is later supplied with externally-stored keys.
> > >
> > > This isn't very clear as to exactly what the concern is or how an
> > > attacker would be able to thwart the system if they had write access to
> > > it. An attacker with write access could possibly attempt to replace the
> > > existing keys, but with the key wrapping that we're using, that should
> > > result in just a decryption failure (unless, of course, the attacker has
> > > the actual KEK that was used, but that's not terribly interesting to
> > > worry about since then they could just go access the files directly).
> >
> > Uh, well, they could modify postgresql.conf to change the script to save
> > the secret returned by the script before returning it to the PG server.
> > We could require postgresql.conf to be somewhere secure, but then how do
> > we know that is secure? I just don't see a clean solution here, but the
> > idea that you write and then wait for the key to show up seems like a
> > very valid way of attack, and it took me a while to be able to
> > articulate it.
postgresql.conf isn't always writable by the postgres user, though
postgresql.auto.conf is likely to always be. I'm not sure how much of a
concern that is, but it we wanted to take steps to explicitly address
this issue, we could have some kind of 'secure' postgresql.conf file
which we would encourage users to make owned by root and whose values
wouldn't be allowed to be overridden once set.
> Let's suppose you lock down your cluster --- the non-PGDATA files are
> owned by root, postgresql.conf and pg_hba.conf are moved out of PGDATA
> and are not writable by the database OS user, or we have the PGDATA
> directory on another server, so the adversary can only write to the
> remote PGDATA directory.
>
> What can they do? Well, they can't modify pg_proc to add a shared
> library since pg_proc is encrypted, so we have to focus on files needed
> before encryption starts or files that can't be easily encrypted.
This isn't accurate- just because it's encrypted doesn't mean they can't
modify it. That's exactly why integrity is important, because an
attacker absolutely could modify the files directly and potentially
exploit the system through those modifications.
> They could create postgresql.conf.auto in PGDATA, and modify
> cluster_key_command to capture the key, or they could modify preload
> libraries or archive command to call a command to read memory as the PG
> OS user and write the key out somewhere, or use the key to rewrite the
> database files --- those wouldn't even need a database restart, just a
> reload.
They would need to actually be able to effect that reload though. This
is where the question comes up as to just what attack vector we're
trying to address. It's certainly possible that an attacker has only
access to the stored data in an off-line fashion (eg: a hard drive that
was mistakenly thrown away without being properly wiped) and that's one
of the cases which is addressed by cluster encryption. An attacker
might have access to the LUN that PG is running on but not to the
running server itself, which it seems like is what you're contemplating
here. That's a much harder attack vector to fully protect against and
we might need to do more than we're currently contemplating to address
it- but I don't think we necessarily must solve for all cases in the
first pass at this.
> They could also modify pg_xact files so that, even though the heap/index
> files are encrypted, how the contents of those files are interpreted
> would change.
Yes, ideally, we'd encrypt/integrity check just about every part of the
running system and that's one area the patch doesn't address- things
like temporary files and other parts.
> In summary, to detect malicious user writes, you would need to protect
> the files used before encryption starts (root owned or owned by another
> user?), and encrypt all files after encryption starts --- any other
> approach would probably leave open attack vectors, and I don't think
> there is sufficient community desire to add such boundaries.
There's going to be some attack vectors that TDE doesn't address. We
should identify and document those where we're able to. We could offer
up some mitigations (eg: strongly suggest monitoring of key utilization
such that if the KEK is used without a reboot of the system or similar
happening that it is reported and someone goes to look into it). While
such mitigations aren't perfect, they can be enough to allow approval of
a system to go operational (ultimately it comes down to what the
relevant security officer is willing to accept).
> How do other database systems guarantee to detect malicious writes?
I doubt anyone would actually stipulate that they *guarantee* detection
of malicious writes, and I don't think we should either, but certainly
the other systems which provide TDE do so in a manner that provides both
confidentiality and integrity. The big O, at least, documents that they
use SHA-1 for their integrity checking, though they also provide an
option which disables it. If we used an additional fork to provide the
integrity then we could also give users the option of either having
integrity included or not.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-03 18:16 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-03 18:16 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote:
> > I am thinking group read-access might be a requirement for cluster file
> > encryption to be effective.
>
> People certainly do use group read-access, but I don't see that as being
> a requirement for cluster file encryption to be effective, it's just one
> thing TDE can address, among others, as discussed.
Agreed.
> > This also does not protect against users who have read access to
> > database process memory — all in-memory data pages and data
> > encryption keys are stored unencrypted in memory, so an attacker who
> > --> is able to read memory can decrypt the entire cluster. The Postgres
> > --> operating system user and the operating system administrator, e.g.,
> > --> the <literal>root</literal> user, have such access.
>
> That's helpful, +1.
Good.
> > > Uh, well, they could modify postgresql.conf to change the script to save
> > > the secret returned by the script before returning it to the PG server.
> > > We could require postgresql.conf to be somewhere secure, but then how do
> > > we know that is secure? I just don't see a clean solution here, but the
> > > idea that you write and then wait for the key to show up seems like a
> > > very valid way of attack, and it took me a while to be able to
> > > articulate it.
>
> postgresql.conf isn't always writable by the postgres user, though
> postgresql.auto.conf is likely to always be. I'm not sure how much of a
> concern that is, but it we wanted to take steps to explicitly address
> this issue, we could have some kind of 'secure' postgresql.conf file
> which we would encourage users to make owned by root and whose values
> wouldn't be allowed to be overridden once set.
Well, I think there is a lot more than postgresql.conf to worry about ---
see below.
> > Let's suppose you lock down your cluster --- the non-PGDATA files are
> > owned by root, postgresql.conf and pg_hba.conf are moved out of PGDATA
> > and are not writable by the database OS user, or we have the PGDATA
> > directory on another server, so the adversary can only write to the
> > remote PGDATA directory.
> >
> > What can they do? Well, they can't modify pg_proc to add a shared
> > library since pg_proc is encrypted, so we have to focus on files needed
> > before encryption starts or files that can't be easily encrypted.
>
> This isn't accurate- just because it's encrypted doesn't mean they can't
> modify it. That's exactly why integrity is important, because an
> attacker absolutely could modify the files directly and potentially
> exploit the system through those modifications.
They can't easily modify it to inject a shared object referenced into a
system column, was my point --- also see below.
> > They could create postgresql.conf.auto in PGDATA, and modify
> > cluster_key_command to capture the key, or they could modify preload
> > libraries or archive command to call a command to read memory as the PG
> > OS user and write the key out somewhere, or use the key to rewrite the
> > database files --- those wouldn't even need a database restart, just a
> > reload.
>
> They would need to actually be able to effect that reload though. This
> is where the question comes up as to just what attack vector we're
> trying to address. It's certainly possible that an attacker has only
> access to the stored data in an off-line fashion (eg: a hard drive that
> was mistakenly thrown away without being properly wiped) and that's one
> of the cases which is addressed by cluster encryption. An attacker
> might have access to the LUN that PG is running on but not to the
> running server itself, which it seems like is what you're contemplating
> here. That's a much harder attack vector to fully protect against and
> we might need to do more than we're currently contemplating to address
> it- but I don't think we necessarily must solve for all cases in the
> first pass at this.
See below.
> > They could also modify pg_xact files so that, even though the heap/index
> > files are encrypted, how the contents of those files are interpreted
> > would change.
>
> Yes, ideally, we'd encrypt/integrity check just about every part of the
> running system and that's one area the patch doesn't address- things
> like temporary files and other parts.
It is worse than that --- see below.
> > In summary, to detect malicious user writes, you would need to protect
> > the files used before encryption starts (root owned or owned by another
> > user?), and encrypt all files after encryption starts --- any other
> > approach would probably leave open attack vectors, and I don't think
> > there is sufficient community desire to add such boundaries.
>
> There's going to be some attack vectors that TDE doesn't address. We
> should identify and document those where we're able to. We could offer
> up some mitigations (eg: strongly suggest monitoring of key utilization
> such that if the KEK is used without a reboot of the system or similar
> happening that it is reported and someone goes to look into it). While
> such mitigations aren't perfect, they can be enough to allow approval of
> a system to go operational (ultimately it comes down to what the
> relevant security officer is willing to accept).
I ended up adding to the feature description in the docs to clearly
outline what this feature provides, and what it does not:
The purpose of cluster file encryption is to prevent users with read
access on the directories used to store database files and write-ahead
log files from being able to access the data stored in those files.
For example, when using cluster file encryption, users who have read
access to the cluster directories for backup purposes will not be able
to decrypt the data stored in these files. Read-only access for a group
of users can be enabled using the <application>initdb</application>
<option>--allow-group-access</option> option. Cluster file encryption
also provides data-at-rest security, protecting users from data loss
should the physical storage media be stolen or improperly erased before
disposal.
Cluster file encryption does not protect against unauthorized file
system writes. Such writes can allow data decryption if used to weaken
the system's security and the weakened system is later supplied with
the externally-stored cluster encryption key. This also does not always
detect if users with write access remove or modify database files.
This also does not protect against users who have read access to database
process memory because all in-memory data pages and data encryption keys
are stored unencrypted in memory. Therefore, an attacker who is able
to read memory can read the data encryption keys and decrypt the entire
cluster. The Postgres operating system user and the operating system
administrator, e.g., the <literal>root</literal> user, have such access.
> > How do other database systems guarantee to detect malicious writes?
>
> I doubt anyone would actually stipulate that they *guarantee* detection
> of malicious writes, and I don't think we should either, but certainly
> the other systems which provide TDE do so in a manner that provides both
> confidentiality and integrity. The big O, at least, documents that they
> use SHA-1 for their integrity checking, though they also provide an
> option which disables it. If we used an additional fork to provide the
> integrity then we could also give users the option of either having
> integrity included or not.
I thought more about this at an abstract level. If you are worried
about malicious users _reading_ data, you can encrypt the sensitive
parts, e.g., heap/index/WAL/temp, and leave some unencrypted, like
pg_xact. Reading pg_xact is pretty useless if you can't read the heap
pages. Reading postgresql.conf.auto, the external key retrieval
scripts, etc. are useless too.
However, when you are trying to protect against write access, you have
to really encrypt _everything_, because the system is very
interdependent, and changing one part where _reading_ is safe can affect
other parts that must remain secure. You can modify
postgresql.conf.auto to capture the cluster key, or maybe even change
something to dump out the data keys from memory. You can modify pg_xact
to affect how heap pages are interpreted.
My point is that being able to detect malicious heap/index writes really
doesn't gain us any security since there are much more serious writes
that can be made, and protecting against those more serious writes would
cause unacceptable Postgres source code changes which will probably
never be implemented.
My summary point is that we should clearly spell out exactly what
protections we are offering, and an estimate of the code impact, before
moving forward so the community can agree it is worthwhile to add this.
Also, looking at the PCI DSS 3.2.1 spec from May 2018 (click-through
required):
https://www.pcisecuritystandards.org/document_library?category=pcidss&document=pci_dss#agreement
or open PDF link here:
https://commerce.uwo.ca/pdf/PCI_DSS_v3-2-1.pdf
Page 41 covers what they expect from an encrypted file system, and from
key encryption key and data encryption keys. There is a v4.0 spec in
draft but I can't find a PDF available online.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-05 16:40 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-05 16:40 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote:
> On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote:
> > I doubt anyone would actually stipulate that they *guarantee* detection
> > of malicious writes, and I don't think we should either, but certainly
> > the other systems which provide TDE do so in a manner that provides both
> > confidentiality and integrity. The big O, at least, documents that they
> > use SHA-1 for their integrity checking, though they also provide an
> > option which disables it. If we used an additional fork to provide the
> > integrity then we could also give users the option of either having
> > integrity included or not.
>
> I thought more about this at an abstract level. If you are worried
> about malicious users _reading_ data, you can encrypt the sensitive
> parts, e.g., heap/index/WAL/temp, and leave some unencrypted, like
> pg_xact. Reading pg_xact is pretty useless if you can't read the heap
> pages. Reading postgresql.conf.auto, the external key retrieval
> scripts, etc. are useless too.
>
> However, when you are trying to protect against write access, you have
> to really encrypt _everything_, because the system is very
> interdependent, and changing one part where _reading_ is safe can affect
> other parts that must remain secure. You can modify
> postgresql.conf.auto to capture the cluster key, or maybe even change
> something to dump out the data keys from memory. You can modify pg_xact
> to affect how heap pages are interpreted.
>
> My point is that being able to detect malicious heap/index writes really
> doesn't gain us any security since there are much more serious writes
> that can be made, and protecting against those more serious writes would
> cause unacceptable Postgres source code changes which will probably
> never be implemented.
I looked further. First, I don't think we are going to be able to
protect at all against users who have _write_ access on the OS running
Postgres. It would be too easy to just read process memory, or modify
~/.profile.
I think the only possible option would be to try to give some protection
against users with write access to PGDATA, where PGDATA is on another
server, e.g., via NFS. We can't protect against all db modifications,
for reasons outlined above, but we might be able to protect against
write users being able to _read_ the keys and therefore decrypt data.
Looking at PGDATA, we have, at least:
postgresql.conf
pg_hba.conf
postmaster.opts
postgresql.conf.auto
which could be exploited to cause reading of the cluster key or process
memory. The first two can be located outside of PGDATA but the last two
currently cannot.
The problem is that this is a limited use-case, and there are probably
other problems I am not considering. It seems too error-prone to even
try protect against this, but it does limit the value of this feature.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-05 18:14 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-02-05 18:14 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote:
> > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote:
> > > I doubt anyone would actually stipulate that they *guarantee* detection
> > > of malicious writes, and I don't think we should either, but certainly
> > > the other systems which provide TDE do so in a manner that provides both
> > > confidentiality and integrity. The big O, at least, documents that they
> > > use SHA-1 for their integrity checking, though they also provide an
> > > option which disables it. If we used an additional fork to provide the
> > > integrity then we could also give users the option of either having
> > > integrity included or not.
> >
> > I thought more about this at an abstract level. If you are worried
> > about malicious users _reading_ data, you can encrypt the sensitive
> > parts, e.g., heap/index/WAL/temp, and leave some unencrypted, like
> > pg_xact. Reading pg_xact is pretty useless if you can't read the heap
> > pages. Reading postgresql.conf.auto, the external key retrieval
> > scripts, etc. are useless too.
> >
> > However, when you are trying to protect against write access, you have
> > to really encrypt _everything_, because the system is very
> > interdependent, and changing one part where _reading_ is safe can affect
> > other parts that must remain secure. You can modify
> > postgresql.conf.auto to capture the cluster key, or maybe even change
> > something to dump out the data keys from memory. You can modify pg_xact
> > to affect how heap pages are interpreted.
> >
> > My point is that being able to detect malicious heap/index writes really
> > doesn't gain us any security since there are much more serious writes
> > that can be made, and protecting against those more serious writes would
> > cause unacceptable Postgres source code changes which will probably
> > never be implemented.
>
> I looked further. First, I don't think we are going to be able to
> protect at all against users who have _write_ access on the OS running
> Postgres. It would be too easy to just read process memory, or modify
> ~/.profile.
I don't think anyone is really expecting that we'll be able to come up
with a way to protect against attackers who have fully compromised the
OS to the point where they can read/write OS memory, or even the PG unix
account. I'm certainly not suggesting that there is a way to do that or
that it's an attack vector we are trying to address here.
> I think the only possible option would be to try to give some protection
> against users with write access to PGDATA, where PGDATA is on another
> server, e.g., via NFS. We can't protect against all db modifications,
> for reasons outlined above, but we might be able to protect against
> write users being able to _read_ the keys and therefore decrypt data.
That certainly seems like a worthy goal. I also really want to stress
that I don't think anyone is expecting us to be able to "protect"
against users who have write access to the system- write access to files
is really an OS level issue and there's not much we can do once someone
has found a way to circumvent that (we can try to help the OS by doing
things like using SELinux, of course, but that's a different
discussion). At the point that an attacker has gotten write access, the
best we can do is complain loudly if we detect unexpected modifications.
Ideally, we would be able to do that for everything, but certainly doing
it for the principal data would go a long way and is far better than
nothing.
Now, that said, I don't know that we absolutely must have that in the
first release of TDE support for PG. In thinking about this, I would
say we have two basic options:
- Keep the same page layout, requiring that integrity data must be
stored elsewhere, eg: another fork
- Use a different page layout when TDE is enabled, making room for
integrity information to be included on each page
There's a set of pros and cons for these:
Same page layout pros:
- Simpler and less impactful on the overall system
- With integrity data stored elsewhere, could possibly be something
that's optional to enable/disable on a per-table basis
- Potential to do things like have an unencrypted primary and an
encrypted replica, providing an easier migration path
Same page layout cons:
- Integrity information must be stored elsewhere
- Increases the reads/memory that is needed, since we have to look up
the integrity information on every read.
- Increases the writes that have to be done since we'd be dirtying
multiple pages instead of just the main fork (though this isn't
exactly unusual- there's the vis map, and indexes, etc, but it'd be
yet another thing we're updating)
Different page layout pros:
- Avoids extra reads/writes for the integrity information
- Once done, this might provide us with a way to add other page level
information in the future while still being able to work with older
page formats
Different page layout cons:
- Wouldn't be able to have an encrypted replica follow an unencrypted
primary, migration would require logical replication or similar
- More core code changes, and extensions, to handle a different page
layout when cluster is initialized with TDE+integrity
While I've been thinking about this, I have to admit that either
approach could be done later and it's probably best to accept that and
push it off until we have the initial TDE work done. I had been
thinking that changing the page layout would be better to do in the same
release as TDE, but having been playing around with that approach for a
while it just seems like it's too much to try and include at the same
time. We should be sure to be clear and document that though.
> Looking at PGDATA, we have, at least:
>
> postgresql.conf
> pg_hba.conf
> postmaster.opts
> postgresql.conf.auto
>
> which could be exploited to cause reading of the cluster key or process
> memory. The first two can be located outside of PGDATA but the last two
> currently cannot.
There are certainly already users out there who intentionally make
postgresql.auto.conf owned by root/root, zero-sized, and monitor it to
make sure that it isn't updated. postgresql.conf actually is also often
monitored for changes by a change management system of some kind and may
also be owned by root/root already. I suspect that postmaster.opts is
not monitored as closely, but that's probably due more to the fact that
we don't really document it as a configuration system file and it can't
be put outside of PGDATA. Having a way to move it outside of PGDATA or
just not have it be used at all (do we really need it..?) would be
another way to address that risk though.
> The problem is that this is a limited use-case, and there are probably
> other problems I am not considering. It seems too error-prone to even
> try protect against this, but it does limit the value of this feature.
I don't think we need to consider it a failing of the capability every
time we think of something else that really should be addressed when
considering this attack vector. We aren't going to be releasing this
and saying "we guarantee that this protects against an attacker who has
write access to PGDATA". Instead, we would be documenting "XYZ, when
enabled, is used to validate the integrity of ABC data. Individuals
concerned with unexpected modifications to their system should consider
independently monitoring files D, E, F. Note that there is currently no
explicit protection against or detection of unexpected or malicious
modification of other parts of the system such as the transaction
record.", or something along those lines. Hardening guidelines would
also recommend things like having postgresql.conf moved out of PGDATA
and owned by root/root, etc. Users would then have the ability to
evaluate if what we're providing is sufficient for their requirements
or not, and to then provide us with feedback about what they feel is
still missing before they would be able to use PG for their use-case.
To that end, I would hope that we'd eventually develop a way to detect
unexpected modifications in other parts of the system, both as a way to
discover filesystem corruption earlier but also in the case of a
malicious attacker. The latter would involve more work, of course, but
it doesn't seem insurmountable. I don't think it's necessary to get
into that today though.
I am concerned when statements are made that we are just never going to
do something-or-other because we think it'd be a lot of source code
changes or won't be completely perfect against every attack we can think
of. There was a good bit of that with RLS which also made it a
particularly difficult feature to push forward, but, thanks to clearly
documenting what was and wasn't addressed, clearly admitting that there
are covert channel attacks that might be possible due to how it works,
it's been pretty well accepted and there hasn't been some huge number of
issues or CVEs that have been associated with it or mismatched
expectations that users of it have had regarding what it does and
doesn't protect against.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-05 20:37 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-05 20:37 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote:
> > I looked further. First, I don't think we are going to be able to
> > protect at all against users who have _write_ access on the OS running
> > Postgres. It would be too easy to just read process memory, or modify
> > ~/.profile.
>
> I don't think anyone is really expecting that we'll be able to come up
> with a way to protect against attackers who have fully compromised the
> OS to the point where they can read/write OS memory, or even the PG unix
> account. I'm certainly not suggesting that there is a way to do that or
> that it's an attack vector we are trying to address here.
OK, that's good.
> > I think the only possible option would be to try to give some protection
> > against users with write access to PGDATA, where PGDATA is on another
> > server, e.g., via NFS. We can't protect against all db modifications,
> > for reasons outlined above, but we might be able to protect against
> > write users being able to _read_ the keys and therefore decrypt data.
>
> That certainly seems like a worthy goal. I also really want to stress
> that I don't think anyone is expecting us to be able to "protect"
> against users who have write access to the system- write access to files
> is really an OS level issue and there's not much we can do once someone
> has found a way to circumvent that (we can try to help the OS by doing
> things like using SELinux, of course, but that's a different
> discussion). At the point that an attacker has gotten write access, the
Agreed.
> best we can do is complain loudly if we detect unexpected modifications.
> Ideally, we would be able to do that for everything, but certainly doing
> it for the principal data would go a long way and is far better than
> nothing.
I disagree. If we only warn about some parts, attackers will just
attack other parts. It will also give users a false sense of security.
If you can get the keys, it doesn't matter if there is one or ten ways
of getting them, if they are all of equal difficulty. Same with
modifying the system files.
> Now, that said, I don't know that we absolutely must have that in the
> first release of TDE support for PG. In thinking about this, I would
> say we have two basic options:
I skipped this part since I think we need a fully secure plan before
considering page format changes. We don't need it for our currently
outlined feature-set.
> > Looking at PGDATA, we have, at least:
> >
> > postgresql.conf
> > pg_hba.conf
> > postmaster.opts
> > postgresql.conf.auto
> >
> > which could be exploited to cause reading of the cluster key or process
> > memory. The first two can be located outside of PGDATA but the last two
> > currently cannot.
>
> There are certainly already users out there who intentionally make
> postgresql.auto.conf owned by root/root, zero-sized, and monitor it to
> make sure that it isn't updated. postgresql.conf actually is also often
> monitored for changes by a change management system of some kind and may
> also be owned by root/root already. I suspect that postmaster.opts is
> not monitored as closely, but that's probably due more to the fact that
> we don't really document it as a configuration system file and it can't
> be put outside of PGDATA. Having a way to move it outside of PGDATA or
> just not have it be used at all (do we really need it..?) would be
> another way to address that risk though.
I think postmaster.opts is used for pg_ctl reload. I think the question
is whether the value of maliciously writable PGDATA being able to read
the keys, while not protecting or detecting all malicious
writes/db-modifications, is worth it. And, while I listed the files
above, there are probably many more ways to break the system.
> > The problem is that this is a limited use-case, and there are probably
> > other problems I am not considering. It seems too error-prone to even
> > try protect against this, but it does limit the value of this feature.
>
> I don't think we need to consider it a failing of the capability every
> time we think of something else that really should be addressed when
> considering this attack vector. We aren't going to be releasing this
> and saying "we guarantee that this protects against an attacker who has
> write access to PGDATA". Instead, we would be documenting "XYZ, when
> enabled, is used to validate the integrity of ABC data. Individuals
> concerned with unexpected modifications to their system should consider
> independently monitoring files D, E, F. Note that there is currently no
> explicit protection against or detection of unexpected or malicious
> modification of other parts of the system such as the transaction
> record.", or something along those lines. Hardening guidelines would
> also recommend things like having postgresql.conf moved out of PGDATA
> and owned by root/root, etc. Users would then have the ability to
> evaluate if what we're providing is sufficient for their requirements
> or not, and to then provide us with feedback about what they feel is
> still missing before they would be able to use PG for their use-case.
See above --- I think we can't just say we close _most_ of the doors
here, and I am afraid there will be more and more cases we miss. It
feels too open-ended. For example, imagine modifying a PGDATA file so
it is a symbolic link to another file that is not in PGDATA? Seems that
would break all sorts of security restrictions, and that's just a new
idea I came up with today.
What I don't want to do is to add a lot of complexity to the system, and
not really gain any meaningful security.
> To that end, I would hope that we'd eventually develop a way to detect
> unexpected modifications in other parts of the system, both as a way to
> discover filesystem corruption earlier but also in the case of a
> malicious attacker. The latter would involve more work, of course, but
> it doesn't seem insurmountable. I don't think it's necessary to get
> into that today though.
>
> I am concerned when statements are made that we are just never going to
> do something-or-other because we think it'd be a lot of source code
> changes or won't be completely perfect against every attack we can think
> of. There was a good bit of that with RLS which also made it a
> particularly difficult feature to push forward, but, thanks to clearly
> documenting what was and wasn't addressed, clearly admitting that there
> are covert channel attacks that might be possible due to how it works,
> it's been pretty well accepted and there hasn't been some huge number of
> issues or CVEs that have been associated with it or mismatched
> expectations that users of it have had regarding what it does and
> doesn't protect against.
Oh, that is a very meaningful lesson. I do think that for cluster file
encryption, if we have a vulnerability, someone will write a script for
it, and it could be widely exploited. I think RLS gets a little more
flexibility since someone is already in the database when using it.
I am not against adding more security features, but I need agreement
that the existing features/protections, with the planned source code
impact, is acceptable. I don't want to go down the road of getting the
feature with the _hope_ that later changes will make the feature
acceptable --- for me, either what we are planning now is acceptable
given its code impact, or it is not. If the feature is not sufficient,
then I would not move forward until we had a reasonable plan of when the
feature would have acceptable usefulness, and acceptable source code
impact.
The big problem, as you outlined above, is that adding to the
protections, like malicious write detection for a remote PGDATA, greatly
increases the code impact, and ultimately, might be unsolvable.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-05 22:21 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-02-05 22:21 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote:
> > > I looked further. First, I don't think we are going to be able to
> > > protect at all against users who have _write_ access on the OS running
> > > Postgres. It would be too easy to just read process memory, or modify
> > > ~/.profile.
> >
> > I don't think anyone is really expecting that we'll be able to come up
> > with a way to protect against attackers who have fully compromised the
> > OS to the point where they can read/write OS memory, or even the PG unix
> > account. I'm certainly not suggesting that there is a way to do that or
> > that it's an attack vector we are trying to address here.
>
> OK, that's good.
>
> > > I think the only possible option would be to try to give some protection
> > > against users with write access to PGDATA, where PGDATA is on another
> > > server, e.g., via NFS. We can't protect against all db modifications,
> > > for reasons outlined above, but we might be able to protect against
> > > write users being able to _read_ the keys and therefore decrypt data.
> >
> > That certainly seems like a worthy goal. I also really want to stress
> > that I don't think anyone is expecting us to be able to "protect"
> > against users who have write access to the system- write access to files
> > is really an OS level issue and there's not much we can do once someone
> > has found a way to circumvent that (we can try to help the OS by doing
> > things like using SELinux, of course, but that's a different
> > discussion). At the point that an attacker has gotten write access, the
>
> Agreed.
>
> > best we can do is complain loudly if we detect unexpected modifications.
> > Ideally, we would be able to do that for everything, but certainly doing
> > it for the principal data would go a long way and is far better than
> > nothing.
>
> I disagree. If we only warn about some parts, attackers will just
> attack other parts. It will also give users a false sense of security.
> If you can get the keys, it doesn't matter if there is one or ten ways
> of getting them, if they are all of equal difficulty. Same with
> modifying the system files.
I agree that there's an additional concern around the keys and that we
would want to have a solid way to avoid having them be compromised. We
might not be able to guarantee that attackers who can write to PGDATA
can't gain access to the keys in the first implementation, but I don't
see that as a problem- the TDE capability would still provide protection
against improper disposal and some other use-cases, which is useful. I
do think it'd be useful to consider how we could provide protection
against an attacker who has write access from being able to acquire the
keys, but that seems like a tractable problem. Following that, we could
look at how to provide integrity checking for principal data, using one
of the outlined approaches or maybe something else entirely. Lastly,
perhaps we can find a way to provide confidentiality and integrity for
other parts of the system.
Each of these steps is a useful improvement in its own right and will
open up more opportunities for PG to be used. It wasn't my intent to
suggest otherwise, but rather to see if there was an opportunity to get
a few things done at once if it wasn't too impactful. I agree now that
it makes sense to focus on the first step, so we can hopefully get that
accomplished.
> > There are certainly already users out there who intentionally make
> > postgresql.auto.conf owned by root/root, zero-sized, and monitor it to
> > make sure that it isn't updated. postgresql.conf actually is also often
> > monitored for changes by a change management system of some kind and may
> > also be owned by root/root already. I suspect that postmaster.opts is
> > not monitored as closely, but that's probably due more to the fact that
> > we don't really document it as a configuration system file and it can't
> > be put outside of PGDATA. Having a way to move it outside of PGDATA or
> > just not have it be used at all (do we really need it..?) would be
> > another way to address that risk though.
>
> I think postmaster.opts is used for pg_ctl reload. I think the question
> is whether the value of maliciously writable PGDATA being able to read
> the keys, while not protecting or detecting all malicious
> writes/db-modifications, is worth it. And, while I listed the files
> above, there are probably many more ways to break the system.
postmaster.opts is used for pg_ctl restart, just to be clear.
As I try to state above- I don't think we need to provide any specific
protections against a malicious writer for plain encryption to be
useful for some important use-cases. Providing protections against a
malicious writer being able to access the keys is certainly important
as, if they acquire the keys, they would be able to trivially both
decrypt the data and modify any other data they wished to, so it seems
likely that solving that would be the first step towards protecting
against a malicious writer, after which it's useful to think about what
else we could provide integrity checking of, and principal data strikes
me as the next sensible step, followed by what's essentially metadata.
> > > The problem is that this is a limited use-case, and there are probably
> > > other problems I am not considering. It seems too error-prone to even
> > > try protect against this, but it does limit the value of this feature.
> >
> > I don't think we need to consider it a failing of the capability every
> > time we think of something else that really should be addressed when
> > considering this attack vector. We aren't going to be releasing this
> > and saying "we guarantee that this protects against an attacker who has
> > write access to PGDATA". Instead, we would be documenting "XYZ, when
> > enabled, is used to validate the integrity of ABC data. Individuals
> > concerned with unexpected modifications to their system should consider
> > independently monitoring files D, E, F. Note that there is currently no
> > explicit protection against or detection of unexpected or malicious
> > modification of other parts of the system such as the transaction
> > record.", or something along those lines. Hardening guidelines would
> > also recommend things like having postgresql.conf moved out of PGDATA
> > and owned by root/root, etc. Users would then have the ability to
> > evaluate if what we're providing is sufficient for their requirements
> > or not, and to then provide us with feedback about what they feel is
> > still missing before they would be able to use PG for their use-case.
>
> See above --- I think we can't just say we close _most_ of the doors
> here, and I am afraid there will be more and more cases we miss. It
> feels too open-ended. For example, imagine modifying a PGDATA file so
> it is a symbolic link to another file that is not in PGDATA? Seems that
> would break all sorts of security restrictions, and that's just a new
> idea I came up with today.
It's not clear how that would provide the attacker with much, if
anything.
> What I don't want to do is to add a lot of complexity to the system, and
> not really gain any meaningful security.
Integrity is very meaningful to security, but key management would
certainly come first because if an attacker is able to acquire the keys
then they can circumvent any integrity check being done by simply using
the key. I appreciate that protecting the keys is non-trivial but it's
absolutely critical as everything else falls apart if the key is
compromised. I don't think we should be thinking that we're going to be
done with key management or with providing ways to acquire keys even if
the currently proposed patches go in- we'll undoubtably need to provide
other options in the future. There's an interesting point in this
regarding how the flexibility of the shell-script based approach also
introduces this risk that an attacker could modify it and write the key
out to somewhere that they could get at pretty easily. Having support
for directly fetching the key from the Linux kernel or the various
vaulting systems would avoid this risk, I would think. Maybe there's a
way to get PG to dump the key out of system memory by modifying other
files in PGDATA but that's surely quite a bit more difficult.
Ultimately, I don't think this voids the proposed approach but I do
think it means we'll want to improve on this in the future.
> > To that end, I would hope that we'd eventually develop a way to detect
> > unexpected modifications in other parts of the system, both as a way to
> > discover filesystem corruption earlier but also in the case of a
> > malicious attacker. The latter would involve more work, of course, but
> > it doesn't seem insurmountable. I don't think it's necessary to get
> > into that today though.
> >
> > I am concerned when statements are made that we are just never going to
> > do something-or-other because we think it'd be a lot of source code
> > changes or won't be completely perfect against every attack we can think
> > of. There was a good bit of that with RLS which also made it a
> > particularly difficult feature to push forward, but, thanks to clearly
> > documenting what was and wasn't addressed, clearly admitting that there
> > are covert channel attacks that might be possible due to how it works,
> > it's been pretty well accepted and there hasn't been some huge number of
> > issues or CVEs that have been associated with it or mismatched
> > expectations that users of it have had regarding what it does and
> > doesn't protect against.
>
> Oh, that is a very meaningful lesson. I do think that for cluster file
> encryption, if we have a vulnerability, someone will write a script for
> it, and it could be widely exploited. I think RLS gets a little more
> flexibility since someone is already in the database when using it.
In the current attack we're contemplating, the attacker's got write
access to the filesystem and if that's happening then they've managed to
get through a few layers already, I would think, so it seems unlikely
that it would be widely exploited. Of course, we'd like to avoid having
vulnerabilities where we can, but a particular behavior is only a
vulnerabiliy if there's an expectation that we protect against that kind
of attack, which is why documentation is extremely important, which is
what I was trying to get at with the RLS example.
> I am not against adding more security features, but I need agreement
> that the existing features/protections, with the planned source code
> impact, is acceptable. I don't want to go down the road of getting the
> feature with the _hope_ that later changes will make the feature
> acceptable --- for me, either what we are planning now is acceptable
> given its code impact, or it is not. If the feature is not sufficient,
> then I would not move forward until we had a reasonable plan of when the
> feature would have acceptable usefulness, and acceptable source code
> impact.
See above. I do think that the proposed approach is a valuable
capability and improvement in its own right. It seems likely that this
first step, as proposed, would allow us to support use-cases such as the
PCI one you mentioned previously. Taking it further and adding
integrity validation would move us into even more use-cases as it would
address NIST requirements which explicitly call for confidentiality and
integrity.
> The big problem, as you outlined above, is that adding to the
> protections, like malicious write detection for a remote PGDATA, greatly
> increases the code impact, and ultimately, might be unsolvable.
I don't think we really know that it increases the code impact hugely or
is unsolveable, but ultimately those are really debates for another day
at this point.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-06 00:53 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-06 00:53 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote:
> > I disagree. If we only warn about some parts, attackers will just
> > attack other parts. It will also give users a false sense of security.
> > If you can get the keys, it doesn't matter if there is one or ten ways
> > of getting them, if they are all of equal difficulty. Same with
> > modifying the system files.
>
> I agree that there's an additional concern around the keys and that we
> would want to have a solid way to avoid having them be compromised. We
> might not be able to guarantee that attackers who can write to PGDATA
> can't gain access to the keys in the first implementation, but I don't
> see that as a problem- the TDE capability would still provide protection
> against improper disposal and some other use-cases, which is useful. I
Agreed.
> do think it'd be useful to consider how we could provide protection
> against an attacker who has write access from being able to acquire the
> keys, but that seems like a tractable problem. Following that, we could
> look at how to provide integrity checking for principal data, using one
> of the outlined approaches or maybe something else entirely. Lastly,
> perhaps we can find a way to provide confidentiality and integrity for
> other parts of the system.
Yes, we should consider it, and I want to have this discussion. Ideally
we could implement that now, because it might be harder later. However,
I don't see how we can add additional security protections without
adding a lot more complexity. You are right we might have better ideas
later.
> Each of these steps is a useful improvement in its own right and will
> open up more opportunities for PG to be used. It wasn't my intent to
> suggest otherwise, but rather to see if there was an opportunity to get
> a few things done at once if it wasn't too impactful. I agree now that
> it makes sense to focus on the first step, so we can hopefully get that
> accomplished.
OK, good.
> > I think postmaster.opts is used for pg_ctl reload. I think the question
> > is whether the value of maliciously writable PGDATA being able to read
> > the keys, while not protecting or detecting all malicious
> > writes/db-modifications, is worth it. And, while I listed the files
> > above, there are probably many more ways to break the system.
>
> postmaster.opts is used for pg_ctl restart, just to be clear.
Yes, sorry, "restart".
> As I try to state above- I don't think we need to provide any specific
> protections against a malicious writer for plain encryption to be
> useful for some important use-cases. Providing protections against a
> malicious writer being able to access the keys is certainly important
> as, if they acquire the keys, they would be able to trivially both
> decrypt the data and modify any other data they wished to, so it seems
> likely that solving that would be the first step towards protecting
> against a malicious writer, after which it's useful to think about what
> else we could provide integrity checking of, and principal data strikes
> me as the next sensible step, followed by what's essentially metadata.
Agreed.
> > See above --- I think we can't just say we close _most_ of the doors
> > here, and I am afraid there will be more and more cases we miss. It
> > feels too open-ended. For example, imagine modifying a PGDATA file so
> > it is a symbolic link to another file that is not in PGDATA? Seems that
> > would break all sorts of security restrictions, and that's just a new
> > idea I came up with today.
>
> It's not clear how that would provide the attacker with much, if
> anything.
Not sure myself either.
> > What I don't want to do is to add a lot of complexity to the system, and
> > not really gain any meaningful security.
>
> Integrity is very meaningful to security, but key management would
> certainly come first because if an attacker is able to acquire the keys
> then they can circumvent any integrity check being done by simply using
> the key. I appreciate that protecting the keys is non-trivial but it's
> absolutely critical as everything else falls apart if the key is
> compromised. I don't think we should be thinking that we're going to be
Agreed,
> done with key management or with providing ways to acquire keys even if
> the currently proposed patches go in- we'll undoubtably need to provide
> other options in the future. There's an interesting point in this
> regarding how the flexibility of the shell-script based approach also
> introduces this risk that an attacker could modify it and write the key
> out to somewhere that they could get at pretty easily. Having support
> for directly fetching the key from the Linux kernel or the various
> vaulting systems would avoid this risk, I would think. Maybe there's a
Agreed.
> way to get PG to dump the key out of system memory by modifying other
> files in PGDATA but that's surely quite a bit more difficult.
> Ultimately, I don't think this voids the proposed approach but I do
> think it means we'll want to improve on this in the future.
OK. I was just saying we can't be sure we can improve it.
> > Oh, that is a very meaningful lesson. I do think that for cluster file
> > encryption, if we have a vulnerability, someone will write a script for
> > it, and it could be widely exploited. I think RLS gets a little more
> > flexibility since someone is already in the database when using it.
>
> In the current attack we're contemplating, the attacker's got write
> access to the filesystem and if that's happening then they've managed to
> get through a few layers already, I would think, so it seems unlikely
> that it would be widely exploited. Of course, we'd like to avoid having
Agreed.
> vulnerabilities where we can, but a particular behavior is only a
> vulnerabiliy if there's an expectation that we protect against that kind
> of attack, which is why documentation is extremely important, which is
> what I was trying to get at with the RLS example.
True.
> > I am not against adding more security features, but I need agreement
> > that the existing features/protections, with the planned source code
> > impact, is acceptable. I don't want to go down the road of getting the
> > feature with the _hope_ that later changes will make the feature
> > acceptable --- for me, either what we are planning now is acceptable
> > given its code impact, or it is not. If the feature is not sufficient,
> > then I would not move forward until we had a reasonable plan of when the
> > feature would have acceptable usefulness, and acceptable source code
> > impact.
>
> See above. I do think that the proposed approach is a valuable
> capability and improvement in its own right. It seems likely that this
> first step, as proposed, would allow us to support use-cases such as the
> PCI one you mentioned previously. Taking it further and adding
> integrity validation would move us into even more use-cases as it would
> address NIST requirements which explicitly call for confidentiality and
> integrity.
Good. I wanted to express this so everyone is clear on what we are
doing, and what we are not doing but might be able to do in the future.
> > The big problem, as you outlined above, is that adding to the
> > protections, like malicious write detection for a remote PGDATA, greatly
> > increases the code impact, and ultimately, might be unsolvable.
>
> I don't think we really know that it increases the code impact hugely or
> is unsolveable, but ultimately those are really debates for another day
> at this point.
True.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-02-07 19:00 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-02-07 19:00 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Fri, Feb 5, 2021 at 07:53:18PM -0500, Bruce Momjian wrote:
> On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote:
> > > I disagree. If we only warn about some parts, attackers will just
> > > attack other parts. It will also give users a false sense of security.
> > > If you can get the keys, it doesn't matter if there is one or ten ways
> > > of getting them, if they are all of equal difficulty. Same with
> > > modifying the system files.
> >
> > I agree that there's an additional concern around the keys and that we
> > would want to have a solid way to avoid having them be compromised. We
> > might not be able to guarantee that attackers who can write to PGDATA
> > can't gain access to the keys in the first implementation, but I don't
> > see that as a problem- the TDE capability would still provide protection
> > against improper disposal and some other use-cases, which is useful. I
>
> Agreed.
>
> > do think it'd be useful to consider how we could provide protection
> > against an attacker who has write access from being able to acquire the
> > keys, but that seems like a tractable problem. Following that, we could
> > look at how to provide integrity checking for principal data, using one
> > of the outlined approaches or maybe something else entirely. Lastly,
> > perhaps we can find a way to provide confidentiality and integrity for
> > other parts of the system.
>
> Yes, we should consider it, and I want to have this discussion. Ideally
> we could implement that now, because it might be harder later. However,
> I don't see how we can add additional security protections without
> adding a lot more complexity. You are right we might have better ideas
> later.
I added a Limitations section so we can consider future improvements:
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Limitations
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-12 03:31 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-03-12 03:31 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
I have made significant progress on the cluster file encryption feature so
it is time for me to post a new set of patches.
I went backward and forward on this patch. I went backward and created
significant user and developer documentation for this feature. I went
forward and added heap/index file encryption, so it actually does
something useful.
Patch
-----
I have broken the patch into 13 parts, attached, which can be reviewed
separately. However, to get a running system, you have to apply all the
patches in order. The GitHub patch link has more details, has a combined
patch link, and is updated regularly:
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Patches
I have 13 questions about the patch's approach, marked with XXX
comments. I hope to get these 13 patches into committable shape, but
not commit them, and then continue on to WAL and temporary file
encryption, and pg_basebackup changes. My hope is to have this full
feature committed together early in the PG 15 development cycle. This
progress would not be possible without critical work by Masahiko Sawada
and Stephen Frost.
Performance
-----------
I have used pgbench to test the performance overhead of this feature.
AES128 shows 2.5% overhead, and AES256 shows 3.5% overhead. My testing
was done on this 16-core, 24MB, single SSD server:
https://momjian.us/main/blogs/pgblog/2012.html#January_20_2012
My CPU supports AES acceleration. I made these configuration changes:
shared_buffers = 6GB
effective_io_concurrency = 256
effective_cache_size = 12GB
checkpoint_completion_target = 0.9
max_wal_size = 3GB
wal_compression = on
I used a scale factor of 1000 (15GB database), with this invocation:
pgbench --no-vacuum --protocol prepared --client 32 --jobs 16 --time 3600
I prewarmed the server via pg_prewarm and ran pgbench for an hour before
using the results from another one-hour run. TPS was around 6.5k. For
the default shared_buffers size of 128MB, I see 7% overhead.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
The usefulness of a cup is in its emptiness, Bruce Lee
Attachments: download all (13 files, .tar.gz)
cfe-01-doc_over_master.diff.gz (application/gzip, 3.6K) download
cfe-02-internaldoc_over_cfe-01-doc.diff.gz (application/gzip, 4.7K) download
cfe-03-scripts_over_cfe-02-internaldoc.diff.gz (application/gzip, 3.1K) download
cfe-04-common_over_cfe-03-scripts.diff.gz (application/gzip, 8.2K) download
cfe-05-crypto_over_cfe-04-common.diff.gz (application/gzip, 6.5K) download
cfe-06-backend_over_cfe-05-crypto.diff.gz (application/gzip, 7.0K) download
cfe-07-bin_over_cfe-06-backend.diff.gz (application/gzip, 9.1K) download
cfe-08-pg_alterckey_over_cfe-07-bin.diff.gz (application/gzip, 8.2K) download
cfe-09-test_over_cfe-08-pg_alterckey.diff.gz (application/gzip, 54.6K) download
cfe-10-hint_over_cfe-09-test.diff.gz (application/gzip, 4.7K) download
cfe-11-persistent_over_cfe-10-hint.diff (text/x-diff, 7.2K) download
From 5581d25039fda613e6c2328fd21d362d866de6b4 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Thu, 11 Mar 2021 20:02:54 -0500
Subject: [PATCH] cfe-11-persistent_over_cfe-10-hint squash commit
---
src/backend/access/gist/gistutil.c | 2 +-
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/catalog/pg_publication.c | 2 +-
src/backend/commands/tablecmds.c | 10 +++++-----
src/backend/optimizer/util/plancat.c | 3 +--
src/backend/utils/cache/relcache.c | 2 +-
src/include/utils/rel.h | 10 ++++++++--
src/include/utils/snapmgr.h | 3 +--
8 files changed, 19 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/gist/gistutil.c b/src/backend/access/gist/gistutil.c
index a3ec9f2cfe..1ff1bf816f 100644
--- a/src/backend/access/gist/gistutil.c
+++ b/src/backend/access/gist/gistutil.c
@@ -1036,7 +1036,7 @@ gistGetFakeLSN(Relation rel)
return counter++;
}
- else if (rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ else if (RelationIsPermanent(rel))
{
/*
* WAL-logging on this relation will start after commit, so its LSNs
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index bd5faf0c1f..0da8f00443 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -660,7 +660,7 @@ heapam_relation_copy_data(Relation rel, const RelFileNode *newrnode)
* WAL log creation if the relation is persistent, or this is the
* init fork of an unlogged relation.
*/
- if (rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT ||
+ if (RelationIsPermanent(rel) ||
(rel->rd_rel->relpersistence == RELPERSISTENCE_UNLOGGED &&
forkNum == INIT_FORKNUM))
log_smgrcreate(newrnode, forkNum);
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 84d2efcfd2..86e415af89 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel)
errdetail("System tables cannot be added to publications.")));
/* UNLOGGED and TEMP relations cannot be part of publication. */
- if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT)
+ if (!RelationIsPermanent(targetrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("table \"%s\" cannot be replicated",
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 559fa1d2e5..3dcd797cae 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8553,13 +8553,13 @@ ATAddForeignKeyConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,
switch (rel->rd_rel->relpersistence)
{
case RELPERSISTENCE_PERMANENT:
- if (pkrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT)
+ if (!RelationIsPermanent(pkrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("constraints on permanent tables may reference only permanent tables")));
break;
case RELPERSISTENCE_UNLOGGED:
- if (pkrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT
+ if (!RelationIsPermanent(pkrel)
&& pkrel->rd_rel->relpersistence != RELPERSISTENCE_UNLOGGED)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
@@ -13598,7 +13598,7 @@ index_copy_data(Relation rel, RelFileNode newrnode)
* WAL log creation if the relation is persistent, or this is the
* init fork of an unlogged relation.
*/
- if (rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT ||
+ if (RelationIsPermanent(rel) ||
(rel->rd_rel->relpersistence == RELPERSISTENCE_UNLOGGED &&
forkNum == INIT_FORKNUM))
log_smgrcreate(&newrnode, forkNum);
@@ -15035,7 +15035,7 @@ ATPrepChangePersistence(Relation rel, bool toLogged)
if (toLogged)
{
- if (foreignrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT)
+ if (!RelationIsPermanent(foreignrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("could not change table \"%s\" to logged because it references unlogged table \"%s\"",
@@ -15045,7 +15045,7 @@ ATPrepChangePersistence(Relation rel, bool toLogged)
}
else
{
- if (foreignrel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ if (RelationIsPermanent(foreignrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("could not change table \"%s\" to unlogged because it references logged table \"%s\"",
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index c5947fa418..7f2e40ae39 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -126,8 +126,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
relation = table_open(relationObjectId, NoLock);
/* Temporary and unlogged relations are inaccessible during recovery. */
- if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT &&
- RecoveryInProgress())
+ if (!RelationIsPermanent(relation) && RecoveryInProgress())
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("cannot access temporary or unlogged relations during recovery")));
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..a4dbc286cb 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -2989,7 +2989,7 @@ static void
AssertPendingSyncConsistency(Relation relation)
{
bool relcache_verdict =
- relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT &&
+ RelationIsPermanent(relation) &&
((relation->rd_createSubid != InvalidSubTransactionId &&
RELKIND_HAS_STORAGE(relation->rd_rel->relkind)) ||
relation->rd_firstRelfilenodeSubid != InvalidSubTransactionId);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 10b63982c0..9a3a03e520 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -552,6 +552,13 @@ typedef struct ViewOptions
(relation)->rd_smgr->smgr_targblock = (targblock); \
} while (0)
+/*
+ * RelationIsPermanent
+ * True if relation is permanent.
+ */
+#define RelationIsPermanent(relation) \
+ ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+
/*
* RelationNeedsWAL
* True if relation needs WAL.
@@ -561,8 +568,7 @@ typedef struct ViewOptions
* RelFileNode" in src/backend/access/transam/README.
*/
#define RelationNeedsWAL(relation) \
- ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
- (XLogIsNeeded() || \
+ (RelationIsPermanent(relation) && (XLogIsNeeded() || \
(relation->rd_createSubid == InvalidSubTransactionId && \
relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId)))
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 2c8b881a09..f66ac58188 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -37,8 +37,7 @@
*/
#define RelationAllowsEarlyPruning(rel) \
( \
- (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \
- && !IsCatalogRelation(rel) \
+ RelationIsPermanent(rel) && !IsCatalogRelation(rel) \
&& !RelationIsAccessibleInLogicalDecoding(rel) \
)
--
2.20.1
cfe-12-gist_over_cfe-11-persistent.diff (text/x-diff, 6.6K) download
From ad92ec577aeacb91b7722e7de8f7d6ddc014ffed Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Thu, 11 Mar 2021 20:02:57 -0500
Subject: [PATCH] cfe-12-gist_over_cfe-11-persistent squash commit
---
src/backend/access/gist/README | 10 ++++++++++
src/backend/access/gist/gist.c | 21 +++++++++++++++------
src/backend/access/gist/gistbuild.c | 9 ++++++---
src/backend/access/gist/gistvacuum.c | 18 ++++++++++++------
4 files changed, 43 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/gist/README b/src/backend/access/gist/README
index 25cab0047b..6b1f84634f 100644
--- a/src/backend/access/gist/README
+++ b/src/backend/access/gist/README
@@ -461,6 +461,16 @@ value. The page is not recycled, until that XID is no longer visible to
anyone. That's much more conservative than necessary, but let's keep it
simple.
+GiST and Encryption
+-------------------
+
+GiST uses LSNs and NSNs for concurrency. This means that unlogged and
+temporary relations also need LSNs. GiST has a mechanism to assign LSNs
+to such relations, but sometimes uses fixed LSNs and or calls
+gistGetFakeLSN(), which can cause duplicate LSNs to be used. Therefore,
+when encryption is enabled and fake LSNs are needed, the GiST code calls
+LSNForEncryption(). See src/backend/crypto/README for more details.
+
Authors:
Teodor Sigaev <teodor@sigaev.ru>
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..5ca5f17db3 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -501,13 +501,16 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
* we don't need to be able to detect concurrent splits yet.)
*/
if (is_build)
- recptr = GistBuildLSN;
+ recptr = !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(rel));
else
{
if (RelationNeedsWAL(rel))
recptr = gistXLogSplit(is_leaf,
dist, oldrlink, oldnsn, leftchildbuf,
markfollowright);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
recptr = gistGetFakeLSN(rel);
}
@@ -568,7 +571,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
MarkBufferDirty(leftchildbuf);
if (is_build)
- recptr = GistBuildLSN;
+ recptr = !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(rel));
else
{
if (RelationNeedsWAL(rel))
@@ -586,6 +590,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
deloffs, ndeloffs, itup, ntup,
leftchildbuf);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
recptr = gistGetFakeLSN(rel);
}
@@ -1665,6 +1671,8 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel)
if (ndeletable > 0)
{
+ XLogRecPtr recptr;
+
TransactionId latestRemovedXid = InvalidTransactionId;
if (XLogStandbyInfoActive() && RelationNeedsWAL(rel))
@@ -1690,16 +1698,17 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel)
/* XLOG stuff */
if (RelationNeedsWAL(rel))
{
- XLogRecPtr recptr;
-
recptr = gistXLogDelete(buffer,
deletable, ndeletable,
latestRemovedXid);
- PageSetLSN(page, recptr);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- PageSetLSN(page, gistGetFakeLSN(rel));
+ recptr = gistGetFakeLSN(rel);
+
+ PageSetLSN(page, recptr);
END_CRIT_SECTION();
}
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 1054f6f1f2..a8b82483e8 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -299,7 +299,8 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
GISTInitBuffer(buffer, F_LEAF);
MarkBufferDirty(buffer);
- PageSetLSN(page, GistBuildLSN);
+ PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(index)));
UnlockReleaseBuffer(buffer);
@@ -451,7 +452,8 @@ gist_indexsortbuild(GISTBuildState *state)
/* Write out the root */
RelationOpenSmgr(state->indexrel);
- PageSetLSN(pagestate->page, GistBuildLSN);
+ PageSetLSN(pagestate->page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(state->indexrel)));
PageSetChecksumInplace(pagestate->page, GIST_ROOT_BLKNO);
smgrwrite(state->indexrel->rd_smgr, MAIN_FORKNUM, GIST_ROOT_BLKNO,
pagestate->page, true);
@@ -573,7 +575,8 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state)
if (blkno != state->pages_written)
elog(ERROR, "unexpected block number to flush GiST sorting build");
- PageSetLSN(page, GistBuildLSN);
+ PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(state->indexrel)));
PageSetChecksumInplace(page, blkno);
smgrextend(state->indexrel->rd_smgr, MAIN_FORKNUM, blkno, page, true);
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0663193531..f783edd627 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -174,6 +174,8 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
vstate.callback_state = callback_state;
if (RelationNeedsWAL(rel))
vstate.startNSN = GetInsertRecPtr();
+ else if (FileEncryptionEnabled)
+ vstate.startNSN = LSNForEncryption(RelationIsPermanent(rel));
else
vstate.startNSN = gistGetFakeLSN(rel);
@@ -353,6 +355,8 @@ restart:
*/
if (ntodelete > 0)
{
+ XLogRecPtr recptr;
+
START_CRIT_SECTION();
MarkBufferDirty(buffer);
@@ -361,16 +365,15 @@ restart:
GistMarkTuplesDeleted(page);
if (RelationNeedsWAL(rel))
- {
- XLogRecPtr recptr;
-
recptr = gistXLogUpdate(buffer,
todelete, ntodelete,
NULL, 0, InvalidBuffer);
- PageSetLSN(page, recptr);
- }
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- PageSetLSN(page, gistGetFakeLSN(rel));
+ recptr = gistGetFakeLSN(rel);
+
+ PageSetLSN(page, recptr);
END_CRIT_SECTION();
@@ -657,8 +660,11 @@ gistdeletepage(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (RelationNeedsWAL(info->index))
recptr = gistXLogPageDelete(leafBuffer, txid, parentBuffer, downlink);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(info->index));
else
recptr = gistGetFakeLSN(info->index);
+
PageSetLSN(parentPage, recptr);
PageSetLSN(leafPage, recptr);
--
2.20.1
* Re: Key management with tests
@ 2021-03-15 16:50 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 3 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-03-15 16:50 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote:
> I have made significant progress on the cluster file encryption feature so
> it is time for me to post a new set of patches.
Here is a rebase, to keep the cfbot green.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Attachments: download all (13 files, .tar.gz)
cfe-01-doc_over_master.diff.gz (application/gzip, 3.6K) download
cfe-02-internaldoc_over_cfe-01-doc.diff.gz (application/gzip, 4.7K) download
cfe-03-scripts_over_cfe-02-internaldoc.diff.gz (application/gzip, 3.1K) download
cfe-04-common_over_cfe-03-scripts.diff.gz (application/gzip, 8.2K) download
cfe-05-crypto_over_cfe-04-common.diff.gz (application/gzip, 6.5K) download
cfe-06-backend_over_cfe-05-crypto.diff.gz (application/gzip, 7.0K) download
cfe-07-bin_over_cfe-06-backend.diff.gz (application/gzip, 9.1K) download
cfe-08-pg_alterckey_over_cfe-07-bin.diff.gz (application/gzip, 8.2K) download
cfe-09-test_over_cfe-08-pg_alterckey.diff.gz (application/gzip, 54.6K) download
cfe-10-hint_over_cfe-09-test.diff.gz (application/gzip, 4.7K) download
cfe-11-persistent_over_cfe-10-hint.diff (text/x-diff, 7.2K) download
From 110358c9ce8764f0c41c12dd37dabde57a92cf1f Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Mon, 15 Mar 2021 10:20:32 -0400
Subject: [PATCH] cfe-11-persistent_over_cfe-10-hint squash commit
---
src/backend/access/gist/gistutil.c | 2 +-
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/catalog/pg_publication.c | 2 +-
src/backend/commands/tablecmds.c | 10 +++++-----
src/backend/optimizer/util/plancat.c | 3 +--
src/backend/utils/cache/relcache.c | 2 +-
src/include/utils/rel.h | 10 ++++++++--
src/include/utils/snapmgr.h | 3 +--
8 files changed, 19 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/gist/gistutil.c b/src/backend/access/gist/gistutil.c
index a3ec9f2cfe..1ff1bf816f 100644
--- a/src/backend/access/gist/gistutil.c
+++ b/src/backend/access/gist/gistutil.c
@@ -1036,7 +1036,7 @@ gistGetFakeLSN(Relation rel)
return counter++;
}
- else if (rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ else if (RelationIsPermanent(rel))
{
/*
* WAL-logging on this relation will start after commit, so its LSNs
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index bd5faf0c1f..0da8f00443 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -660,7 +660,7 @@ heapam_relation_copy_data(Relation rel, const RelFileNode *newrnode)
* WAL log creation if the relation is persistent, or this is the
* init fork of an unlogged relation.
*/
- if (rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT ||
+ if (RelationIsPermanent(rel) ||
(rel->rd_rel->relpersistence == RELPERSISTENCE_UNLOGGED &&
forkNum == INIT_FORKNUM))
log_smgrcreate(newrnode, forkNum);
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 84d2efcfd2..86e415af89 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -67,7 +67,7 @@ check_publication_add_relation(Relation targetrel)
errdetail("System tables cannot be added to publications.")));
/* UNLOGGED and TEMP relations cannot be part of publication. */
- if (targetrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT)
+ if (!RelationIsPermanent(targetrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("table \"%s\" cannot be replicated",
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 559fa1d2e5..3dcd797cae 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -8553,13 +8553,13 @@ ATAddForeignKeyConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,
switch (rel->rd_rel->relpersistence)
{
case RELPERSISTENCE_PERMANENT:
- if (pkrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT)
+ if (!RelationIsPermanent(pkrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("constraints on permanent tables may reference only permanent tables")));
break;
case RELPERSISTENCE_UNLOGGED:
- if (pkrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT
+ if (!RelationIsPermanent(pkrel)
&& pkrel->rd_rel->relpersistence != RELPERSISTENCE_UNLOGGED)
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
@@ -13598,7 +13598,7 @@ index_copy_data(Relation rel, RelFileNode newrnode)
* WAL log creation if the relation is persistent, or this is the
* init fork of an unlogged relation.
*/
- if (rel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT ||
+ if (RelationIsPermanent(rel) ||
(rel->rd_rel->relpersistence == RELPERSISTENCE_UNLOGGED &&
forkNum == INIT_FORKNUM))
log_smgrcreate(&newrnode, forkNum);
@@ -15035,7 +15035,7 @@ ATPrepChangePersistence(Relation rel, bool toLogged)
if (toLogged)
{
- if (foreignrel->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT)
+ if (!RelationIsPermanent(foreignrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("could not change table \"%s\" to logged because it references unlogged table \"%s\"",
@@ -15045,7 +15045,7 @@ ATPrepChangePersistence(Relation rel, bool toLogged)
}
else
{
- if (foreignrel->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+ if (RelationIsPermanent(foreignrel))
ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),
errmsg("could not change table \"%s\" to unlogged because it references logged table \"%s\"",
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index c5947fa418..7f2e40ae39 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -126,8 +126,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
relation = table_open(relationObjectId, NoLock);
/* Temporary and unlogged relations are inaccessible during recovery. */
- if (relation->rd_rel->relpersistence != RELPERSISTENCE_PERMANENT &&
- RecoveryInProgress())
+ if (!RelationIsPermanent(relation) && RecoveryInProgress())
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("cannot access temporary or unlogged relations during recovery")));
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..a4dbc286cb 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -2989,7 +2989,7 @@ static void
AssertPendingSyncConsistency(Relation relation)
{
bool relcache_verdict =
- relation->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT &&
+ RelationIsPermanent(relation) &&
((relation->rd_createSubid != InvalidSubTransactionId &&
RELKIND_HAS_STORAGE(relation->rd_rel->relkind)) ||
relation->rd_firstRelfilenodeSubid != InvalidSubTransactionId);
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 10b63982c0..9a3a03e520 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -552,6 +552,13 @@ typedef struct ViewOptions
(relation)->rd_smgr->smgr_targblock = (targblock); \
} while (0)
+/*
+ * RelationIsPermanent
+ * True if relation is permanent.
+ */
+#define RelationIsPermanent(relation) \
+ ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT)
+
/*
* RelationNeedsWAL
* True if relation needs WAL.
@@ -561,8 +568,7 @@ typedef struct ViewOptions
* RelFileNode" in src/backend/access/transam/README.
*/
#define RelationNeedsWAL(relation) \
- ((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
- (XLogIsNeeded() || \
+ (RelationIsPermanent(relation) && (XLogIsNeeded() || \
(relation->rd_createSubid == InvalidSubTransactionId && \
relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId)))
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 2c8b881a09..f66ac58188 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -37,8 +37,7 @@
*/
#define RelationAllowsEarlyPruning(rel) \
( \
- (rel)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT \
- && !IsCatalogRelation(rel) \
+ RelationIsPermanent(rel) && !IsCatalogRelation(rel) \
&& !RelationIsAccessibleInLogicalDecoding(rel) \
)
--
2.20.1
cfe-12-gist_over_cfe-11-persistent.diff (text/x-diff, 6.6K) download
From 235b1763416875c11ee3ba288d8b2eb38c2cbbfd Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Mon, 15 Mar 2021 10:20:35 -0400
Subject: [PATCH] cfe-12-gist_over_cfe-11-persistent squash commit
---
src/backend/access/gist/README | 10 ++++++++++
src/backend/access/gist/gist.c | 21 +++++++++++++++------
src/backend/access/gist/gistbuild.c | 9 ++++++---
src/backend/access/gist/gistvacuum.c | 18 ++++++++++++------
4 files changed, 43 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/gist/README b/src/backend/access/gist/README
index 25cab0047b..6b1f84634f 100644
--- a/src/backend/access/gist/README
+++ b/src/backend/access/gist/README
@@ -461,6 +461,16 @@ value. The page is not recycled, until that XID is no longer visible to
anyone. That's much more conservative than necessary, but let's keep it
simple.
+GiST and Encryption
+-------------------
+
+GiST uses LSNs and NSNs for concurrency. This means that unlogged and
+temporary relations also need LSNs. GiST has a mechanism to assign LSNs
+to such relations, but sometimes uses fixed LSNs and or calls
+gistGetFakeLSN(), which can cause duplicate LSNs to be used. Therefore,
+when encryption is enabled and fake LSNs are needed, the GiST code calls
+LSNForEncryption(). See src/backend/crypto/README for more details.
+
Authors:
Teodor Sigaev <teodor@sigaev.ru>
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..5ca5f17db3 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -501,13 +501,16 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
* we don't need to be able to detect concurrent splits yet.)
*/
if (is_build)
- recptr = GistBuildLSN;
+ recptr = !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(rel));
else
{
if (RelationNeedsWAL(rel))
recptr = gistXLogSplit(is_leaf,
dist, oldrlink, oldnsn, leftchildbuf,
markfollowright);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
recptr = gistGetFakeLSN(rel);
}
@@ -568,7 +571,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
MarkBufferDirty(leftchildbuf);
if (is_build)
- recptr = GistBuildLSN;
+ recptr = !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(rel));
else
{
if (RelationNeedsWAL(rel))
@@ -586,6 +590,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
deloffs, ndeloffs, itup, ntup,
leftchildbuf);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
recptr = gistGetFakeLSN(rel);
}
@@ -1665,6 +1671,8 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel)
if (ndeletable > 0)
{
+ XLogRecPtr recptr;
+
TransactionId latestRemovedXid = InvalidTransactionId;
if (XLogStandbyInfoActive() && RelationNeedsWAL(rel))
@@ -1690,16 +1698,17 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel)
/* XLOG stuff */
if (RelationNeedsWAL(rel))
{
- XLogRecPtr recptr;
-
recptr = gistXLogDelete(buffer,
deletable, ndeletable,
latestRemovedXid);
- PageSetLSN(page, recptr);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- PageSetLSN(page, gistGetFakeLSN(rel));
+ recptr = gistGetFakeLSN(rel);
+
+ PageSetLSN(page, recptr);
END_CRIT_SECTION();
}
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 1054f6f1f2..a8b82483e8 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -299,7 +299,8 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
GISTInitBuffer(buffer, F_LEAF);
MarkBufferDirty(buffer);
- PageSetLSN(page, GistBuildLSN);
+ PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(index)));
UnlockReleaseBuffer(buffer);
@@ -451,7 +452,8 @@ gist_indexsortbuild(GISTBuildState *state)
/* Write out the root */
RelationOpenSmgr(state->indexrel);
- PageSetLSN(pagestate->page, GistBuildLSN);
+ PageSetLSN(pagestate->page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(state->indexrel)));
PageSetChecksumInplace(pagestate->page, GIST_ROOT_BLKNO);
smgrwrite(state->indexrel->rd_smgr, MAIN_FORKNUM, GIST_ROOT_BLKNO,
pagestate->page, true);
@@ -573,7 +575,8 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state)
if (blkno != state->pages_written)
elog(ERROR, "unexpected block number to flush GiST sorting build");
- PageSetLSN(page, GistBuildLSN);
+ PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(state->indexrel)));
PageSetChecksumInplace(page, blkno);
smgrextend(state->indexrel->rd_smgr, MAIN_FORKNUM, blkno, page, true);
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0663193531..f783edd627 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -174,6 +174,8 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
vstate.callback_state = callback_state;
if (RelationNeedsWAL(rel))
vstate.startNSN = GetInsertRecPtr();
+ else if (FileEncryptionEnabled)
+ vstate.startNSN = LSNForEncryption(RelationIsPermanent(rel));
else
vstate.startNSN = gistGetFakeLSN(rel);
@@ -353,6 +355,8 @@ restart:
*/
if (ntodelete > 0)
{
+ XLogRecPtr recptr;
+
START_CRIT_SECTION();
MarkBufferDirty(buffer);
@@ -361,16 +365,15 @@ restart:
GistMarkTuplesDeleted(page);
if (RelationNeedsWAL(rel))
- {
- XLogRecPtr recptr;
-
recptr = gistXLogUpdate(buffer,
todelete, ntodelete,
NULL, 0, InvalidBuffer);
- PageSetLSN(page, recptr);
- }
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- PageSetLSN(page, gistGetFakeLSN(rel));
+ recptr = gistGetFakeLSN(rel);
+
+ PageSetLSN(page, recptr);
END_CRIT_SECTION();
@@ -657,8 +660,11 @@ gistdeletepage(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (RelationNeedsWAL(info->index))
recptr = gistXLogPageDelete(leafBuffer, txid, parentBuffer, downlink);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(info->index));
else
recptr = gistGetFakeLSN(info->index);
+
PageSetLSN(parentPage, recptr);
PageSetLSN(leafPage, recptr);
--
2.20.1
* Re: Key management with tests
@ 2021-03-18 15:31 Stephen Frost <sfrost@snowman.net>
parent: Bruce Momjian <bruce@momjian.us>
2 siblings, 2 replies; 114+ messages in thread
From: Stephen Frost @ 2021-03-18 15:31 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Bruce Momjian (bruce@momjian.us) wrote:
> On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote:
> > I have made significant progress on the cluster file encryption feature so
> > it is time for me to post a new set of patches.
>
> Here is a rebase, to keep the cfbot green.
Good stuff.
> >From 110358c9ce8764f0c41c12dd37dabde57a92cf1f Mon Sep 17 00:00:00 2001
> From: Bruce Momjian <bruce@momjian.us>
> Date: Mon, 15 Mar 2021 10:20:32 -0400
> Subject: [PATCH] cfe-11-persistent_over_cfe-10-hint squash commit
>
> ---
> src/backend/access/gist/gistutil.c | 2 +-
> src/backend/access/heap/heapam_handler.c | 2 +-
> src/backend/catalog/pg_publication.c | 2 +-
> src/backend/commands/tablecmds.c | 10 +++++-----
> src/backend/optimizer/util/plancat.c | 3 +--
> src/backend/utils/cache/relcache.c | 2 +-
> src/include/utils/rel.h | 10 ++++++++--
> src/include/utils/snapmgr.h | 3 +--
> 8 files changed, 19 insertions(+), 15 deletions(-)
This particular patch (introducing the RelationIsPermanent() macro)
seems like it'd be a nice thing to commit independently of the rest,
reducing the size of this patch set..?
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 16:55 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-03-18 16:55 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote:
> Greetings,
>
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote:
> > > I have made significant progress on the cluster file encryption feature so
> > > it is time for me to post a new set of patches.
> >
> > Here is a rebase, to keep the cfbot green.
>
> Good stuff.
Yes, I was happy I got to a stage where the encryption actually did
something useful.
> > >From 110358c9ce8764f0c41c12dd37dabde57a92cf1f Mon Sep 17 00:00:00 2001
> > From: Bruce Momjian <bruce@momjian.us>
> > Date: Mon, 15 Mar 2021 10:20:32 -0400
> > Subject: [PATCH] cfe-11-persistent_over_cfe-10-hint squash commit
> >
> > ---
> > src/backend/access/gist/gistutil.c | 2 +-
> > src/backend/access/heap/heapam_handler.c | 2 +-
> > src/backend/catalog/pg_publication.c | 2 +-
> > src/backend/commands/tablecmds.c | 10 +++++-----
> > src/backend/optimizer/util/plancat.c | 3 +--
> > src/backend/utils/cache/relcache.c | 2 +-
> > src/include/utils/rel.h | 10 ++++++++--
> > src/include/utils/snapmgr.h | 3 +--
> > 8 files changed, 19 insertions(+), 15 deletions(-)
>
> This particular patch (introducing the RelationIsPermanent() macro)
> seems like it'd be a nice thing to commit independently of the rest,
> reducing the size of this patch set..?
OK, if no one objects I will apply it in the next few days. The macro is
used more in my later patches, which I will not apply now.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 17:13 Alvaro Herrera <alvherre@alvh.no-ip.org>
parent: Bruce Momjian <bruce@momjian.us>
2 siblings, 1 reply; 114+ messages in thread
From: Alvaro Herrera @ 2021-03-18 17:13 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Patch 10 uses the term "WAL-skip relations". What does that mean? Is
it "relations that are not WAL-logged"? I suppose we already have a
term for this; I'm not sure it's a good idea to invent a different term
that is only used in this new place.
--
Álvaro Herrera 39°49'30"S 73°17'W
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 17:21 Stephen Frost <sfrost@snowman.net>
parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
0 siblings, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-03-18 17:21 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Bruce Momjian <bruce@momjian.us>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Alvaro Herrera (alvherre@alvh.no-ip.org) wrote:
> Patch 10 uses the term "WAL-skip relations". What does that mean? Is
> it "relations that are not WAL-logged"? I suppose we already have a
> term for this; I'm not sure it's a good idea to invent a different term
> that is only used in this new place.
This is discussed in src/backend/access/transam/README, specifically the
section that talks about Skipping WAL for New RelFileNode. Basically,
it's the 'wal_level=minimal' optimization which allows WAL to be
skipped.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 17:37 Alvaro Herrera <alvherre@alvh.no-ip.org>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 114+ messages in thread
From: Alvaro Herrera @ 2021-03-18 17:37 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Bruce Momjian <bruce@momjian.us>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On 2021-Mar-18, Stephen Frost wrote:
> * Alvaro Herrera (alvherre@alvh.no-ip.org) wrote:
> > Patch 10 uses the term "WAL-skip relations". What does that mean? Is
> > it "relations that are not WAL-logged"? I suppose we already have a
> > term for this; I'm not sure it's a good idea to invent a different term
> > that is only used in this new place.
>
> This is discussed in src/backend/access/transam/README, specifically the
> section that talks about Skipping WAL for New RelFileNode. Basically,
> it's the 'wal_level=minimal' optimization which allows WAL to be
> skipped.
Hmm ... that talks about WAL-skipping *changes*, not WAL-skipping
*relations*. I thought WAL-skipping meant unlogged relations, but
I understand now that that's unrelated. In the transam/README, WAL-skip
means a change in a transaction in a relfilenode that, if rolled back,
would disappear; and I'm not sure I understand how the code is handling
the case that a relation is under that condition.
This caught my attention because a comment says "encryption does not
support WAL-skipped relations", but there's no direct change to the
definition of RelFileNodeSkippingWAL() to account for that. Perhaps I
am just overlooking something, since I'm just skimming anyway.
--
Álvaro Herrera Valdivia, Chile
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 17:46 Stephen Frost <sfrost@snowman.net>
parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 1 reply; 114+ messages in thread
From: Stephen Frost @ 2021-03-18 17:46 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Bruce Momjian <bruce@momjian.us>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Greetings,
* Alvaro Herrera (alvherre@alvh.no-ip.org) wrote:
> On 2021-Mar-18, Stephen Frost wrote:
>
> > * Alvaro Herrera (alvherre@alvh.no-ip.org) wrote:
> > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is
> > > it "relations that are not WAL-logged"? I suppose we already have a
> > > term for this; I'm not sure it's a good idea to invent a different term
> > > that is only used in this new place.
> >
> > This is discussed in src/backend/access/transam/README, specifically the
> > section that talks about Skipping WAL for New RelFileNode. Basically,
> > it's the 'wal_level=minimal' optimization which allows WAL to be
> > skipped.
>
> Hmm ... that talks about WAL-skipping *changes*, not WAL-skipping
> *relations*. I thought WAL-skipping meant unlogged relations, but
> I understand now that that's unrelated. In the transam/README, WAL-skip
> means a change in a transaction in a relfilenode that, if rolled back,
> would disappear; and I'm not sure I understand how the code is handling
> the case that a relation is under that condition.
>
> This caught my attention because a comment says "encryption does not
> support WAL-skipped relations", but there's no direct change to the
> definition of RelFileNodeSkippingWAL() to account for that. Perhaps I
> am just overlooking something, since I'm just skimming anyway.
This is relatively current activity and so it's entirely possible
comments and perhaps code need further updating in this area, but to
explain what's going on in a bit more detail-
Ultimately, we need to make sure that LSNs aren't re-used. There's two
sources of LSNs today: those for relations which are being written into
the WAL and those for relations which are not (UNLOGGED relations,
specifically). The 'minimal' WAL level introduces complications with
this requirement because tables created (or truncated) inside a
transaction are considered permanent once they're committed, but the
data pages in those relations don't go into the WAL and the LSNs on the
pages of those relations isn't guaranteed to be either unique or even
necessarily set, and if we were to generate LSNs for those it would be
required to be done by actually advancing the WAL LSN, which would
require writing into the WAL and therefore wouldn't be quite the
optimization that's expected.
I'm not sure if it's been explicitly done yet but I believe the idea is,
based on my last discussion with Bruce, at least initially, simply
disallow encrypted clusters from running with wal_level=minimal to avoid
this issue.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 18:49 Bruce Momjian <bruce@momjian.us>
parent: Alvaro Herrera <alvherre@alvh.no-ip.org>
1 sibling, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-03-18 18:49 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@alvh.no-ip.org>; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Mar 18, 2021 at 02:37:43PM -0300, Álvaro Herrera wrote:
> On 2021-Mar-18, Stephen Frost wrote:
> > This is discussed in src/backend/access/transam/README, specifically the
> > section that talks about Skipping WAL for New RelFileNode. Basically,
> > it's the 'wal_level=minimal' optimization which allows WAL to be
> > skipped.
>
> Hmm ... that talks about WAL-skipping *changes*, not WAL-skipping
> *relations*. I thought WAL-skipping meant unlogged relations, but
> I understand now that that's unrelated. In the transam/README, WAL-skip
> means a change in a transaction in a relfilenode that, if rolled back,
> would disappear; and I'm not sure I understand how the code is handling
> the case that a relation is under that condition.
>
> This caught my attention because a comment says "encryption does not
> support WAL-skipped relations", but there's no direct change to the
> definition of RelFileNodeSkippingWAL() to account for that. Perhaps I
> am just overlooking something, since I'm just skimming anyway.
First, thanks for looking at these patches --- I know it isn't easy.
Second, you are right that I equated WAL-skipping relfilenodes with
relations, and this was wrong. I have updated the attached patch to use
the term WAL-skipping "relfilenodes", and checked the rest of the
patches for any incorrect 'skipping' term, but didn't find any.
If "WAL-skipping relfilenodes" is not clear enough, we should probably
rename RelFileNodeSkippingWAL().
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Attachments:
cfe-10-hint_over_cfe-09-test.diff.gz (application/gzip, 4.7K) download
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-18 18:59 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-03-18 18:59 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Mar 18, 2021 at 01:46:28PM -0400, Stephen Frost wrote:
> * Alvaro Herrera (alvherre@alvh.no-ip.org) wrote:
> > This caught my attention because a comment says "encryption does not
> > support WAL-skipped relations", but there's no direct change to the
> > definition of RelFileNodeSkippingWAL() to account for that. Perhaps I
> > am just overlooking something, since I'm just skimming anyway.
>
> This is relatively current activity and so it's entirely possible
> comments and perhaps code need further updating in this area, but to
> explain what's going on in a bit more detail-
>
> Ultimately, we need to make sure that LSNs aren't re-used. There's two
> sources of LSNs today: those for relations which are being written into
> the WAL and those for relations which are not (UNLOGGED relations,
> specifically). The 'minimal' WAL level introduces complications with
Well, the story is a little more complex than that --- we currently have
four LSN uses:
1. real LSNs for WAL-logged relfilenodes
2. real LSNs for GiST indexes for non-WAL-logged relfilenodes of permanenet relations
3. fake LSNs for GiST indexes for relfilenodes of non-permanenet relations
4. zero LSNs for non-GiST non-permanenet relations
This patch changes it so #4 gets fake LSNs, and slightly adjusts #2 & #3
so the LSNs are always unique.
> I'm not sure if it's been explicitly done yet but I believe the idea is,
> based on my last discussion with Bruce, at least initially, simply
> disallow encrypted clusters from running with wal_level=minimal to avoid
> this issue.
I adjusted the hint bit code so it potentially could work with wal_level
minimal (just for safety), but the code disallows wal_level minimal, and
is documented as such.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-23 00:38 Bruce Momjian <bruce@momjian.us>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 114+ messages in thread
From: Bruce Momjian @ 2021-03-23 00:38 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote:
> > src/backend/access/gist/gistutil.c | 2 +-
> > src/backend/access/heap/heapam_handler.c | 2 +-
> > src/backend/catalog/pg_publication.c | 2 +-
> > src/backend/commands/tablecmds.c | 10 +++++-----
> > src/backend/optimizer/util/plancat.c | 3 +--
> > src/backend/utils/cache/relcache.c | 2 +-
> > src/include/utils/rel.h | 10 ++++++++--
> > src/include/utils/snapmgr.h | 3 +--
> > 8 files changed, 19 insertions(+), 15 deletions(-)
>
> This particular patch (introducing the RelationIsPermanent() macro)
> seems like it'd be a nice thing to commit independently of the rest,
> reducing the size of this patch set..?
Committed as suggested.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-03-23 02:19 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-03-23 02:19 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Mon, Mar 22, 2021 at 08:38:37PM -0400, Bruce Momjian wrote:
> > This particular patch (introducing the RelationIsPermanent() macro)
> > seems like it'd be a nice thing to commit independently of the rest,
> > reducing the size of this patch set..?
>
> Committed as suggested.
Also, I have written a short presentation on where I think we are with
cluster file encryption:
https://momjian.us/main/writings/pgsql/cfe.pdf
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-04-06 08:56 Neil Chen <carpenter.nail.cz@gmail.com>
parent: Bruce Momjian <bruce@momjian.us>
2 siblings, 1 reply; 114+ messages in thread
From: Neil Chen @ 2021-04-06 08:56 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
Hi Bruce,
I went through these patches and executed the test script you added for the
KMS section, which looks all good.
This is a point that looks like a bug - in patch 10, you changed the
location and use of *RelFileNodeSkippingWAL()*, but the modified code logic
seems different from the original when encryption is not enabled. After
applying this patch, it still will execute the set LSN code flow when
RelFileNodeSkippingWAL returns true, and encryption not enabled.
On Thu, Apr 1, 2021 at 2:47 PM Bruce Momjian <bruce@momjian.us> wrote:
> On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote:
> > I have made significant progress on the cluster file encryption feature
> so
> > it is time for me to post a new set of patches.
>
> Here is a rebase, to keep the cfbot green.
>
> --
> Bruce Momjian <bruce@momjian.us> https://momjian.us
> EDB https://enterprisedb.com
>
> If only the physical world exists, free will is an illusion.
>
>
--
There is no royal road to learning.
HighGo Software Co.
^ permalink raw reply [nested|flat] 114+ messages in thread
* Re: Key management with tests
@ 2021-04-06 18:42 Bruce Momjian <bruce@momjian.us>
parent: Neil Chen <carpenter.nail.cz@gmail.com>
0 siblings, 0 replies; 114+ messages in thread
From: Bruce Momjian @ 2021-04-06 18:42 UTC (permalink / raw)
To: Neil Chen <carpenter.nail.cz@gmail.com>; +Cc: Stephen Frost <sfrost@snowman.net>; Masahiko Sawada <sawada.mshk@gmail.com>; Tom Kincaid <tomjohnkincaid@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Alvaro Herrera <alvherre@alvh.no-ip.org>; Andres Freund <andres@anarazel.de>; Amit Kapila <amit.kapila16@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
On Tue, Apr 6, 2021 at 04:56:36PM +0800, Neil Chen wrote:
> Hi Bruce,
>
> I went through these patches and executed the test script you added for the KMS
> section, which looks all good.
Thank you for checking it. The src/test/crypto/t/003_clusterkey.pl test
is one of the craziest tests I have ever written, so I am glad it worked
for you.
> This is a point that looks like a bug - in patch 10, you changed the location
> and use of *RelFileNodeSkippingWAL()*, but the modified code logic seems
> different from the original when encryption is not enabled. After applying this
> patch, it still will execute the set LSN code flow when RelFileNodeSkippingWAL
> returns true, and encryption not enabled.
You are very correct. That 'return' inside the 'if' statement gave me
trouble, and MarkBufferDirtyHint() was the hardest function I had to
deal with. Attached is an updated version of patches with a rebase; the
GitHub links listed on the wiki are updated too.
Thanks for your help.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Attachments: download all (12 files, .tar.gz)
cfe-01-doc_over_master.diff (text/x-diff, 10.1K) download
From 9e997a5f1cdd630a95dd2bd2dc7d742e9d95ef36 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:34 -0400
Subject: [PATCH] cfe-01-doc_over_master squash commit
---
doc/src/sgml/database-encryption.sgml (new) | 149 ++++++++++++++++++++
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/installation.sgml | 2 +-
doc/src/sgml/monitoring.sgml | 13 ++
doc/src/sgml/postgres.sgml | 1 +
5 files changed, 165 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/database-encryption.sgml b/doc/src/sgml/database-encryption.sgml
new file mode 100644
index 0000000000..c4377ada64
--- /dev/null
+++ b/doc/src/sgml/database-encryption.sgml
@@ -0,0 +1,149 @@
+<!-- doc/src/sgml/database-encryption.sgml -->
+
+<chapter id="cluster-file-encryption">
+ <title>Cluster File Encryption</title>
+
+ <indexterm zone="cluster-file-encryption">
+ <primary>Cluster File Encryption</primary>
+ </indexterm>
+
+ <para>
+ The purpose of cluster file encryption is to prevent users with read
+ access on the directories used to store database files and write-ahead
+ log files from being able to access the data stored in those files.
+ For example, when using cluster file encryption, users who have read
+ access to the cluster directories for backup purposes will not be able
+ to decrypt the data stored in these files. Read-only access for a group
+ of users can be enabled using the <application>initdb</application>
+ <option>--allow-group-access</option> option. Cluster file encryption
+ also provides data-at-rest security, protecting users from data loss
+ should the physical storage media be stolen or improperly erased before
+ disposal.
+ </para>
+
+ <para>
+ Cluster file encryption does not protect against unauthorized file
+ system writes. Such writes can allow data decryption if used to weaken
+ the system's security and the weakened system is later supplied with
+ the externally-stored cluster encryption key. This also does not always
+ detect if users with write access remove or modify database files.
+ </para>
+
+ <para>
+ This also does not protect against users who have read access to database
+ process memory because all in-memory data pages and data encryption keys
+ are stored unencrypted in memory. Therefore, an attacker who is able
+ to read memory can read the data encryption keys and decrypt the entire
+ cluster. The Postgres operating system user and the operating system
+ administrator, e.g., the <literal>root</literal> user, have such access.
+ </para>
+
+ <sect1 id="cluster-encryption-keys">
+ <title>Keys</title>
+
+ <para>
+ Cluster file encryption uses two levels of encryption — an upper
+ key which encrypts lower-level keys. The upper-level key is often
+ referred to as a Key Encryption Key (<acronym>KEK</acronym>). This key
+ is <emphasis>not</emphasis> stored in the file system, but provided at
+ <command>initdb</command> time and each time the server is started. This
+ key can be easily changed via <command>pg_alterckey</command> without
+ requiring any changes to the the data files or <command>WAL</command>
+ files.
+ </para>
+
+ <para>
+ The lower level keys are data encryption keys, specifically for relations
+ and <acronym>WAL</acronym>. The relation key is used to encrypt database
+ heap and index files. The WAL key is used to encrypt write-ahead log
+ (WAL) files. Two different keys are used so that primary and standby
+ servers can use different relation keys, but the same WAL key, so that
+ these keys can (in a future release) be rotated by switching the
+ primary to the standby and then changing the WAL key. Eventually,
+ encryption will be able to added to non-encrypted clusters by creating
+ encrypted replicas and switching over to them.
+ </para>
+
+ <para>
+ Postgres stores the data encryption (lower-level) keys in the data
+ directory encrypted (wrapped) by key encryption (upper-level) key.
+ Though the data encryption keys technically exist in the file system,
+ the key encryption key does not, so the data encryption keys are
+ securely stored. Data encryption keys are used to security encrypt
+ other database files.
+ </para>
+ </sect1>
+
+ <sect1 id="cluster-encryption-initialization">
+ <title>Initialization</title>
+
+ <para>
+ Cluster file encryption is enabled when
+ <productname>PostgreSQL</productname> is built
+ with <literal>--with-openssl</literal> and <xref
+ linkend="app-initdb-cluster-key-command"/> is specified
+ during <command>initdb</command>. The cluster key
+ provided by the <option>--cluster-key-command</option>
+ option during <command>initdb</command> and the one generated
+ by <xref linkend="guc-cluster-key-command"/> in the
+ <filename>postgresql.conf</filename> must match for the database
+ cluster to start. Note that the cluster key command
+ passed to <command>initdb</command> must return a key of
+ 64 hexadecimal characters. For example:
+<programlisting>
+initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
+</programlisting>
+ Cluster file encryption does not support a <varname>wal_level</varname>
+ of <literal>minimal</literal>.
+ </para>
+ </sect1>
+
+ <sect1 id="cluster-encryption-operation">
+ <title>Operation</title>
+
+ <para>
+ During the <command>initdb</command> process, if
+ <option>--cluster-key-command</option> is specified, two data-level
+ encryption keys are created. These two keys are then encrypted with
+ the key encryption key (KEK) supplied by the cluster key command before
+ being stored in the database directory. The key or passphrase that
+ derives the key must be supplied from the terminal or stored in a
+ trusted key store, such as key vault software or a hardware security
+ module.
+ </para>
+
+ <para>
+ If the <productname>PostgreSQL</productname> server has
+ been initialized to require a cluster key, each time the
+ server starts the <filename>postgresql.conf</filename>
+ <varname>cluster_key_command</varname> command will be executed
+ and the cluster key retrieved. The data encryption keys in the
+ <filename>pg_cryptokeys</filename> directory will then be decrypted
+ using the supplied key and integrity-checked to ensure it matches the
+ initdb-supplied key. (If this check fails, the server will refuse
+ to start.) The cluster encryption key will then be removed from
+ system memory. The decrypted data encryption keys will remain in
+ shared memory until the server is stopped.
+ </para>
+
+ <para>
+ The data encryption keys are randomly generated and can be 128, 192,
+ or 256-bits in length, depending on whether <literal>AES128</literal>,
+ <literal>AES192</literal>, or <literal>AES256</literal> is specified.
+ They are encrypted by the key encryption key (KEK) using Advanced
+ Encryption Standard (<acronym>AES256</acronym>) encryption in Key
+ Wrap Padded Mode, which also provides KEK authentication; see <ulink
+ url="https://tools.ietf.org/html/rfc5649">RFC 5649</ulink>. While
+ 128-bit encryption is sufficient for most sites, 256-bit encryption
+ is thought to be more immune to future quantum cryptographic attacks.
+ </para>
+
+ <para>.
+ If you prefer to create the random keys on your own, you can create
+ a empty directory with a <filename>pg_cryptokeys/live</filename>
+ subdirectory, generate the keys there using your tools. and use the
+ <command>initdb</command> <option>--copy-encryption-keys</option>
+ to copy those keys into the newly-created cluster.
+ </para>
+ </sect1>
+</chapter>
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 45b701426b..baf65fa24c 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -49,6 +49,7 @@
<!ENTITY wal SYSTEM "wal.sgml">
<!ENTITY logical-replication SYSTEM "logical-replication.sgml">
<!ENTITY jit SYSTEM "jit.sgml">
+<!ENTITY database-encryption SYSTEM "database-encryption.sgml">
<!-- programmer's guide -->
<!ENTITY bgworker SYSTEM "bgworker.sgml">
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 66ad4ba938..405cdc2fac 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -976,7 +976,7 @@ build-postgresql:
<listitem>
<para>
Build with support for <acronym>SSL</acronym> (encrypted)
- connections. The only <replaceable>LIBRARY</replaceable>
+ connections and cluster file encryption. The only <replaceable>LIBRARY</replaceable>
supported is <option>openssl</option>. This requires the
<productname>OpenSSL</productname> package to be installed.
<filename>configure</filename> will check for the required
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 56018745c8..418e4e2644 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1302,6 +1302,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry><literal>DataFileWrite</literal></entry>
<entry>Waiting for a write to a relation data file.</entry>
</row>
+ <row>
+ <entry><literal>KeyFileRead</literal></entry>
+ <entry>Waiting for a read of the wrapped data encryption keys.</entry>
+ </row>
+ <row>
+ <entry><literal>KeyFileWrite</literal></entry>
+ <entry>Waiting for a write of the wrapped data encryption keys.</entry>
+ </row>
+ <row>
+ <entry><literal>KeyFileSync</literal></entry>
+ <entry>Waiting for changes to the wrapped data encryption keys to reach
+ durable storage.</entry>
+ </row>
<row>
<entry><literal>LockFileAddToDataDirRead</literal></entry>
<entry>Waiting for a read while adding a line to the data directory lock
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index d453be3909..628cb86098 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -171,6 +171,7 @@ break is not needed in a wider output rendering.
&wal;
&logical-replication;
&jit;
+ &database-encryption;
®ress;
</part>
--
2.20.1
cfe-02-internaldoc_over_cfe-01-doc.diff (text/x-diff, 11.4K) download
From a7bbf41909d7d968f143b781fd26ae54817778ce Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:35 -0400
Subject: [PATCH] cfe-02-internaldoc_over_cfe-01-doc squash commit
---
src/backend/crypto/README (new) | 231 ++++++++++++++++++++++++++++++++
1 file changed, 231 insertions(+)
diff --git a/src/backend/crypto/README b/src/backend/crypto/README
new file mode 100644
index 0000000000..be5e5557ba
--- /dev/null
+++ b/src/backend/crypto/README
@@ -0,0 +1,231 @@
+Cluster File Encryption
+=======================
+
+This directory contains support functions and sample scripts to be used
+for cluster file encryption.
+
+Architecture
+------------
+
+Fundamentally, cluster file encryption must store data in a file system
+in such a way that the keys required to decrypt the file system data can
+only be accessed using somewhere outside of the file system itself. The
+external requirement can be someone typing in a passphrase, getting a
+key from a key management server (KMS), or decrypting a key stored in
+the file system using a hardware security module (HSM). The current
+architecture supports all of these methods, and includes sample scripts
+for them.
+
+The simplest method for accessing data keys using some external
+requirement would be to retrieve all data encryption keys from a KMS.
+However, retrieved keys would still need to be verified as valid. This
+method also introduces unacceptable complexity for simpler use-cases,
+like user-supplied passphrases or HSM usage. External key rotation
+would also be very hard since it would require re-encrypting all the
+file system data with the new externally-stored keys.
+
+For these reason, a two-tiered architecture is used, which uses two
+types of encryption keys: a key encryption key (KEK) and data encryption
+keys (DEK). The KEK should not be present unencrypted in the file system
+--- it should be supplied the user, stored externally (e.g., in a KMS)
+or stored in the file system encrypted with a HSM (e.g., PIV device).
+The DEK is used to encrypt database files and is stored in the same file
+system as the database but is encrypted using the KEK. Because the DEK
+is encrypted, its storage in the file system is no more of a security
+weakness and the storage of the encrypted database files in the same
+file system.
+
+Implementation
+--------------
+
+To enable cluster file encryption, the initdb option
+--cluster-key-command must be used, which specifies a command to
+retrieve the KEK. initdb records the cluster_key_command in
+postgresql.conf. Every time the KEK is needed, the command is run and
+must return 64 hex characters which are decoded into the KEK. The
+command is called twice during initdb, and every time the server starts.
+initdb also sets the encryption method in controldata during server
+bootstrap.
+
+initdb runs "postgres --boot", which calls function
+kmgr.c::BootStrapKmgr(), which calls the cluster key command. The
+cluster key command returns a KEK which is used to encrypt random bytes
+for each DEK and writes them to the file system by
+kmgr.c::KmgrWriteCryptoKeys() (unless --copy-encryption-keys is used).
+Currently the DEK files are 0 and 1 and are stored in
+$PGDATA/pg_cryptokeys/live. The wrapped DEK files use Key Wrapping with
+Padding which verifies the validity of the KEK.
+
+initdb also does a non-boot backend start which calls
+kmgr.c::InitializeKmgr(), which calls the cluster key command a second
+time. This decrypts/unwraps the DEK keys and stores them in the shared
+memory structure KmgrShmem. This step also happens every time the server
+starts. Later patches will use the keys stored in KmgrShmem to
+encrypt/decrypt database files. KmgrShmem is erased via
+explicit_bzero() on server shutdown.
+
+Limitations
+-----------
+
+There doesn't seem to be a reasonable way to detect all malicious data
+modification or key extraction if a user has write permission on the
+files in PGDATA. It might be possible to limit the key extraction risk
+if postgresql.auto.conf were able to be moved to a directory outside of
+PGDATA, and if postmaster.opts could be moved or ignored when cluster
+file encryption is used. (This file is used by pg_ctl restart.)
+
+It doesn't appear possible to detect all malicious writes --- even if
+you add message authentication code (MAC) checks to encrypted files,
+modifying non-encrypted files could still affect encrypted ones, e.g.,
+modifying files in pg_xact could affect how heap rows are interpreted.
+Basically you would need to encrypt all files, and at that point you
+might as well just use an encrypted file system. There also doesn't seem
+to be a way to prevent key extraction if someone has read permission on
+postgres process memory.
+
+Initialization Vector
+---------------------
+
+Nonce means "number used once". An Initialization Vector (IV) is a
+specific type of nonce. That is, unique but not necessarily random or
+secret, as specified by the NIST
+(https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf).
+To generate unique IVs, the NIST recommends two methods:
+
+ The first method is to apply the forward cipher function, under
+ the same key that is used for the encryption of the plaintext,
+ to a nonce. The nonce must be a data block that is unique to
+ each execution of the encryption operation. For example, the
+ nonce may be a counter, as described in Appendix B, or a message
+ number. The second method is to generate a random data block
+ using a FIPS-approved random number generator.
+
+We will use the first method to generate IVs. That is, select nonce
+carefully and use a cipher with the key to make it unique enough to use
+as an IV. The nonce selection for buffer encryption and WAL encryption
+are described below.
+
+If the IV was used more than once with the same key (and we only use one
+data encryption key), changes in the unencrypted data would be visible
+in the encrypted data.
+
+IV for Heap/Index Encryption
+- - - - - - - - - - - - - -
+
+To create the 16-byte IV needed by AES for each page version, we will
+use the page LSN (8 bytes) and page number (4 bytes). In the remaining
+four bytes, one bit will be used to indicate if the LSN is WAL (real) or
+fake (see below). The LSN is ideal for use in the IV because it is
+always increasing, and is changed every time a page is updated. The
+same LSN is never used for two relations with different page contents.
+
+However, the same LSN can be used in multiple pages in the same relation
+--- this can happen when a heap update expires an old tuple and adds a
+new tuple to another page. By adding the page number to the IV, we keep
+the IV unique.
+
+By not using the database id in the IV, CREATE DATABASE can copy the
+heap/index files from the old database to a new one without
+decryption/encryption. Both page copies are valid. Once a database
+changes its pages, it gets new LSNs, and hence new IV. Using only the
+LSN and page number also avoids requiring pg_upgrade to preserve
+database oids, tablespace oids, and relfilenodes.
+
+As part of WAL logging, every change of a WAL-logged page gets a new
+LSN, and therefore a new IV automatically.
+
+However, the LSN must then be visible on encrypted pages, so we will not
+encrypt the LSN on the page. We will also not encrypt the CRC so
+pg_checksums can still check pages offline without access to the keys.
+
+Non-Permanent Relations
+- - - - - - - - - - - -
+
+To avoid the overhead of generating WAL for non-permanent (unlogged and
+temporary) relations, we assign fake LSNs that are derived from a
+counter via xlog.c::GetFakeLSNForUnloggedRel(). (GiST also uses this
+counter for LSNs.) We also set a bit in the IV so the use of the same
+value for WAL (real) and fake LSNs will still generate unique IVs. Only
+main forks are encrypted, not init, vm, or fsm files.
+
+In the code, we need to identify if a page uses WAL or fake LSNs in
+four places, when:
+
+1. Reading a page from the file system and decrypting
+2. Setting the WAL or fake LSN on a page
+3. Hint bits changes requiring new LSNs for the encryption IV
+4. Encrypting and writing a page to the file system
+
+For all these case, we have access to the fork number and either the
+relation's persistence state or the buffer state. If it is a "main"
+fork and the relation persistence state is RELPERSISTENCE_PERMANENT, or
+if it is an "init" fork, we use a real LSN. If it is a main fork and
+RELPERSISTENCE_PERMANENT is false, we use a fake LSN. The buffer state
+BM_PERMANENT is true if the relation is PERMANENT or is an init fork.
+
+Init Forks
+- - - - -
+
+Init forks for unlogged relations get permanent LSNs because unlogged
+relation creation is WAL logged/crash safe, even though the relation's
+contents are not. When the init fork is copied to represent an empty
+relation during crash recovery, it becomes a non-permanent page and must
+be successfully decrypted as such. Therefore, when it is copied, its
+LSN is changed to e fake LSN and then encrypted. This prevents a real
+LSN from being encrypted with the fake nonce bit.
+
+LSN Assignment, GiST, & Non-Permanent Relations
+- - - - - - - - - - - - - - - - - - - - - - - -
+
+LSN assignment has to be slightly modified for encryption. In normal,
+non-encryption mode, LSNs are assigned to pages following these rules:
+
+1. During GiST builds, some pages are assigned fixed LSNs (GistBuildLSN)
+
+2. During GiST builds, non-permanent pages not assigned fixed LSNs in
+#1 are assigned fake LSNs, via gistutil.c::gistGetFakeLSN().
+
+3. All other permanent pages are assigned WAL-based LSNs based on the
+WAL position of their WAL records.
+
+4. All other non-permanent pages have LSNs of zero.
+
+When encryption is enabled:
+
+1. During GiST builds, permanent pages are assigned WAL-based LSNs
+generated by xloginsert.c::LSNForEncryption().
+
+2. During GiST builds, non-permanent pages are assigned fake LSNs.
+(No constant LSNs are used in #1 or #2.)
+
+3. same as #3 above
+
+4. All other non-permanent pages are assigned fake LSNs before page
+encryption.
+
+When switching to an encrypted replica from a non-encrypted primary,
+GiST indexes will be using fixed LSNs for permanent tables, so it is
+recommended to rebuild GiST indexes. Non-permanent relations are not
+replicated, so they are not an issue.
+
+Hint Bits
+- - - - -
+
+For hint bit changes, the LSN normally doesn't change, which is a
+problem. By enabling wal_log_hints, you get full page writes to the WAL
+after the first hint bit change of the checkpoint. This is useful for
+two reasons. First, it generates a new LSN, which is needed for the IV
+to be secure. Second, full page images protect against torn pages,
+which is an even bigger requirement for encryption because the new LSN
+is re-encrypting the entire page, not just the hint bit changes. You
+can safely lose the hint bit changes, but you need to use the same LSN
+to decrypt the entire page, so a torn page with an LSN change cannot be
+decrypted. To prevent this, wal_log_hints guarantees that the
+pre-hint-bit version (and previous LSN version) of the page is restored.
+
+However, if a hint-bit-modified page is written to the file system
+during a checkpoint, and there is a later hint bit change switching the
+same page from clean to dirty during the same checkpoint, we need a new
+LSN, and wal_log_hints doesn't give us a new LSN here. The fix for this
+is to update the page LSN by writing a dummy WAL record via
+xloginsert.c::LSNForEncryption() in such cases.
--
2.20.1
cfe-03-scripts_over_cfe-02-internaldoc.diff (text/x-diff, 13.4K) download
From 53d415b70f06dd6cc015852acd1c799c0b9a841f Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:36 -0400
Subject: [PATCH] cfe-03-scripts_over_cfe-02-internaldoc squash commit
---
src/backend/Makefile | 15 +++-
src/backend/crypto/ckey_aws.sh.sample (new) | 53 ++++++++++++
.../crypto/ckey_direct.sh.sample (new) | 39 +++++++++
.../crypto/ckey_passphrase.sh.sample (new) | 35 ++++++++
.../crypto/ckey_piv_nopin.sh.sample (new) | 68 ++++++++++++++++
.../crypto/ckey_piv_pin.sh.sample (new) | 81 +++++++++++++++++++
.../crypto/ssl_passphrase.sh.sample (new) | 35 ++++++++
7 files changed, 325 insertions(+), 1 deletion(-)
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 0da848b1fd..76eb44f632 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -211,6 +211,12 @@ endif
$(INSTALL_DATA) $(srcdir)/libpq/pg_hba.conf.sample '$(DESTDIR)$(datadir)/pg_hba.conf.sample'
$(INSTALL_DATA) $(srcdir)/libpq/pg_ident.conf.sample '$(DESTDIR)$(datadir)/pg_ident.conf.sample'
$(INSTALL_DATA) $(srcdir)/utils/misc/postgresql.conf.sample '$(DESTDIR)$(datadir)/postgresql.conf.sample'
+ $(INSTALL_DATA) $(srcdir)/crypto/ckey_aws.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_aws.sh.sample'
+ $(INSTALL_DATA) $(srcdir)/crypto/ckey_direct.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_direct.sh.sample'
+ $(INSTALL_DATA) $(srcdir)/crypto/ckey_passphrase.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_passphrase.sh.sample'
+ $(INSTALL_DATA) $(srcdir)/crypto/ckey_piv_nopin.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_nopin.sh.sample'
+ $(INSTALL_DATA) $(srcdir)/crypto/ckey_piv_pin.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_pin.sh.sample'
+ $(INSTALL_DATA) $(srcdir)/crypto/ssl_passphrase.sh.sample '$(DESTDIR)$(datadir)/auth_commands/ssl_passphrase.sh.sample'
ifeq ($(with_llvm), yes)
install-bin: install-postgres-bitcode
@@ -236,6 +242,7 @@ endif
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)' '$(DESTDIR)$(datadir)'
+ $(MKDIR_P) '$(DESTDIR)$(datadir)' '$(DESTDIR)$(datadir)/auth_commands'
ifeq ($(PORTNAME), cygwin)
ifeq ($(MAKE_DLL), true)
$(MKDIR_P) '$(DESTDIR)$(libdir)'
@@ -275,7 +282,13 @@ endif
$(MAKE) -C utils uninstall-data
rm -f '$(DESTDIR)$(datadir)/pg_hba.conf.sample' \
'$(DESTDIR)$(datadir)/pg_ident.conf.sample' \
- '$(DESTDIR)$(datadir)/postgresql.conf.sample'
+ '$(DESTDIR)$(datadir)/postgresql.conf.sample' \
+ '$(DESTDIR)$(datadir)/auth_commands/ckey_aws.sh.sample' \
+ '$(DESTDIR)$(datadir)/auth_commands/ckey_direct.sh.sample' \
+ '$(DESTDIR)$(datadir)/auth_commands/ckey_passphrase.sh.sample' \
+ '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_nopin.sh.sample' \
+ '$(DESTDIR)$(datadir)/auth_commands/ckey_piv_pin.sh.sample' \
+ '$(DESTDIR)$(datadir)/auth_commands/ssl_passphrase.sh.sample'
ifeq ($(with_llvm), yes)
$(call uninstall_llvm_module,postgres)
endif
diff --git a/src/backend/crypto/ckey_aws.sh.sample b/src/backend/crypto/ckey_aws.sh.sample
new file mode 100644
index 0000000000..d9bee53132
--- /dev/null
+++ b/src/backend/crypto/ckey_aws.sh.sample
@@ -0,0 +1,53 @@
+#!/bin/sh
+
+# This uses the AWS Secrets Manager using the AWS CLI and OpenSSL.
+# This stores the AWS secret Id in $DIR.
+# Do not create any file with extension "wkey" in $DIR; these are
+# reserved for wrapped data key files.
+
+[ "$#" -ne 1 ] && echo "cluster_key_command usage: $0 \"%d\"" 1>&2 && exit 1
+# No need for %R or -R since we are not prompting
+
+DIR="$1"
+[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1
+[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1
+
+# File containing the id of the AWS secret
+AWS_ID_FILE="$DIR/aws-secret.id"
+
+
+# ----------------------------------------------------------------------
+
+
+# Create an AWS Secrets Manager secret?
+if [ ! -e "$AWS_ID_FILE" ]
+then # The 'postgres' operating system user must have permission to
+ # access the AWS CLI
+
+ # The epoch-time/directory/hostname combination is unique
+ HASH=$(echo -n "$(date '+%s')$DIR$(hostname)" | sha1sum | cut -d' ' -f1)
+ AWS_SECRET_ID="Postgres-cluster-key-$HASH"
+
+ # Use stdin to avoid passing the secret on the command line
+ openssl rand -hex 32 |
+ aws secretsmanager create-secret \
+ --name "$AWS_SECRET_ID" \
+ --description "Postgres cluster file encryption on $(hostname)" \
+ --secret-string 'file:///dev/stdin' \
+ --output text > /dev/null
+ if [ "$?" -ne 0 ]
+ then echo 'cluster key generation failed' 1>&2
+ exit 1
+ fi
+
+ echo "$AWS_SECRET_ID" > "$AWS_ID_FILE"
+fi
+
+if ! aws secretsmanager get-secret-value \
+ --secret-id "$(cat "$AWS_ID_FILE")" \
+ --output text
+then echo 'cluster key retrieval failed' 1>&2
+ exit 1
+fi | awk -F'\t' 'NR == 1 {print $4}'
+
+exit 0
diff --git a/src/backend/crypto/ckey_direct.sh.sample b/src/backend/crypto/ckey_direct.sh.sample
new file mode 100644
index 0000000000..492defcffe
--- /dev/null
+++ b/src/backend/crypto/ckey_direct.sh.sample
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+# This uses a 64-character hex key supplied by the user.
+# If OpenSSL is installed, you can generate a pseudo-random key by running:
+# openssl rand -hex 32
+# To get a true random key, run:
+# wget -q -O - 'https://www.random.org/cgi-bin/randbyte?nbytes=32&format=h' | tr -d ' \n'; echo
+# Do not create any fie with extension "wkey" in $DIR; these are
+# reserved for wrapped data key files.
+
+[ "$#" -lt 1 ] && echo "cluster_key_command usage: $0 %R [%p]" 1>&2 && exit 1
+# Supports environment variable PROMPT
+
+FD="$1"
+[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1
+
+[ "$2" ] && PROMPT="$2"
+
+
+# ----------------------------------------------------------------------
+
+[ ! "$PROMPT" ] && PROMPT='Enter cluster key as 64 hexadecimal characters: '
+
+stty -echo <&"$FD"
+
+echo 1>&"$FD"
+echo -n "$PROMPT" 1>&"$FD"
+read KEY <&"$FD"
+
+stty echo <&"$FD"
+
+if [ "$(expr "$KEY" : '[0-9a-fA-F]*$')" -ne 64 ]
+then echo 'invalid; must be 64 hexadecimal characters' 1>&2
+ exit 1
+fi
+
+echo "$KEY"
+
+exit 0
diff --git a/src/backend/crypto/ckey_passphrase.sh.sample b/src/backend/crypto/ckey_passphrase.sh.sample
new file mode 100644
index 0000000000..a5d837b45e
--- /dev/null
+++ b/src/backend/crypto/ckey_passphrase.sh.sample
@@ -0,0 +1,35 @@
+#!/bin/sh
+
+# This uses a passphrase supplied by the user.
+# Do not create any fie with extension "wkey" in $DIR; these are
+# reserved for wrapped data key files.
+
+[ "$#" -lt 1 ] && echo "cluster_key_command usage: $0 %R [\"%p\"]" 1>&2 && exit 1
+
+FD="$1"
+[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1
+# Supports environment variable PROMPT
+
+[ "$2" ] && PROMPT="$2"
+
+
+# ----------------------------------------------------------------------
+
+[ ! "$PROMPT" ] && PROMPT='Enter cluster passphrase: '
+
+stty -echo <&"$FD"
+
+echo 1>&"$FD"
+echo -n "$PROMPT" 1>&"$FD"
+read PASS <&"$FD"
+
+stty echo <&"$FD"
+
+if [ ! "$PASS" ]
+then echo 'invalid: empty passphrase' 1>&2
+ exit 1
+fi
+
+echo "$PASS" | sha256sum | cut -d' ' -f1
+
+exit 0
diff --git a/src/backend/crypto/ckey_piv_nopin.sh.sample b/src/backend/crypto/ckey_piv_nopin.sh.sample
new file mode 100644
index 0000000000..e90a579dea
--- /dev/null
+++ b/src/backend/crypto/ckey_piv_nopin.sh.sample
@@ -0,0 +1,68 @@
+#!/bin/sh
+
+# This uses the public/private keys on a PIV device, like a CAC or Yubikey.
+# It uses a PIN stored in a file.
+# It uses OpenSSL with PKCS11 enabled via OpenSC.
+# This stores the cluster encryption key encrypted with the PIV public
+# key in $DIR. This is technically a three-level encryption
+# architecture, with the third level requiring the PIV and PIN.
+# Do not create any fie with extension "wkey" in $DIR; these are
+# reserved for wrapped data key files.
+
+[ "$#" -ne 1 ] && echo "cluster_key_command usage: $0 \"%d\"" 1>&2 && exit 1
+# Supports environment variable PIV_PIN_FILE
+# No need for %R or -R since we are not prompting for a PIN
+
+DIR="$1"
+[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1
+[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1
+
+# Set these here or pass in as environment variables.
+# File that stores the PIN to unlock the PIV
+#PIV_PIN_FILE=''
+# PIV slot 3 is the "Key Management" slot, so we use '0:3'
+PIV_SLOT='0:3'
+
+# File containing the cluster key encrypted with the PIV_SLOT's public key
+KEY_FILE="$DIR/pivpass.key"
+
+
+# ----------------------------------------------------------------------
+
+[ ! "$PIV_PIN_FILE" ] && echo 'PIV_PIN_FILE undefined' 1>&2 && exit 1
+[ ! -e "$PIV_PIN_FILE" ] && echo "$PIV_PIN_FILE does not exist" 1>&2 && exit 1
+[ -d "$PIV_PIN_FILE" ] && echo "$PIV_PIN_FILE is a directory" 1>&2 && exit 1
+
+[ ! "$KEY_FILE" ] && echo 'KEY_FILE undefined' 1>&2 && exit 1
+[ -d "$KEY_FILE" ] && echo "$KEY_FILE is a directory" 1>&2 && exit 1
+
+# Create a cluster key encrypted with the PIV_SLOT's public key?
+if [ ! -e "$KEY_FILE" ]
+then # The 'postgres' operating system user must have permission to
+ # access the PIV device.
+
+ openssl rand -hex 32 |
+ if ! openssl rsautl -engine pkcs11 -keyform engine -encrypt \
+ -inkey "$PIV_SLOT" -passin file:"$PIV_PIN_FILE" -out "$KEY_FILE"
+ then echo 'cluster key generation failed' 1>&2
+ exit 1
+ fi
+
+ # Warn the user to save the cluster key in a safe place
+ cat 1>&2 <<END
+
+WARNING: The PIV device can be locked and require a reset if too many PIN
+attempts fail. It is recommended to run this command manually and save
+the cluster key in a secure location for possible recovery.
+END
+
+fi
+
+# Decrypt the cluster key encrypted with the PIV_SLOT's public key
+if ! openssl rsautl -engine pkcs11 -keyform engine -decrypt \
+ -inkey "$PIV_SLOT" -passin file:"$PIV_PIN_FILE" -in "$KEY_FILE"
+then echo 'cluster key decryption failed' 1>&2
+ exit 1
+fi
+
+exit 0
diff --git a/src/backend/crypto/ckey_piv_pin.sh.sample b/src/backend/crypto/ckey_piv_pin.sh.sample
new file mode 100644
index 0000000000..e693ac31ba
--- /dev/null
+++ b/src/backend/crypto/ckey_piv_pin.sh.sample
@@ -0,0 +1,81 @@
+#!/bin/sh
+
+# This uses the public/private keys on a PIV device, like a CAC or Yubikey.
+# It requires a user-entered PIN.
+# It uses OpenSSL with PKCS11 enabled via OpenSC.
+# This stores the cluster encryption key encrypted with the PIV public
+# key in $DIR. This is technically a three-level encryption
+# architecture, with the third level requiring the PIV and PIN.
+# Do not create any fie with extension "wkey" in $DIR; these are
+# reserved for wrapped data key files.
+
+[ "$#" -lt 2 ] && echo "cluster_key_command usage: $0 \"%d\" %R [\"%p\"]" 1>&2 && exit 1
+# Supports environment variable PROMPT
+
+DIR="$1"
+[ ! -e "$DIR" ] && echo "$DIR does not exist" 1>&2 && exit 1
+[ ! -d "$DIR" ] && echo "$DIR is not a directory" 1>&2 && exit 1
+
+FD="$2"
+[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1
+
+[ "$3" ] && PROMPT="$3"
+
+# PIV slot 3 is the "Key Management" slot, so we use '0:3'
+PIV_SLOT='0:3'
+
+# File containing the cluster key encrypted with the PIV_SLOT's public key
+KEY_FILE="$DIR/pivpass.key"
+
+
+# ----------------------------------------------------------------------
+
+[ ! "$PROMPT" ] && PROMPT='Enter PIV PIN: '
+
+stty -echo <&"$FD"
+
+# Create a cluster key encrypted with the PIV_SLOT's public key?
+if [ ! -e "$KEY_FILE" ]
+then echo 1>&"$FD"
+ echo -n "$PROMPT" 1>&"$FD"
+
+ # The 'postgres' operating system user must have permission to
+ # access the PIV device.
+
+ openssl rand -hex 32 |
+ # 'engine "pkcs11" set.' message confuses prompting
+ if ! openssl rsautl -engine pkcs11 -keyform engine -encrypt \
+ -inkey "$PIV_SLOT" -passin fd:"$FD" -out "$KEY_FILE" 2>&1
+ then stty echo <&"$FD"
+ echo 'cluster key generation failed' 1>&2
+ exit 1
+ fi | grep -v 'engine "pkcs11" set\.'
+
+ echo 1>&"$FD"
+
+ # Warn the user to save the cluster key in a safe place
+ cat 1>&"$FD" <<END
+
+WARNING: The PIV can be locked and require a reset if too many PIN
+attempts fail. It is recommended to run this command manually and save
+the cluster key in a secure location for possible recovery.
+END
+
+fi
+
+echo 1>&"$FD"
+echo -n "$PROMPT" 1>&"$FD"
+
+# Decrypt the cluster key encrypted with the PIV_SLOT's public key
+if ! openssl rsautl -engine pkcs11 -keyform engine -decrypt \
+ -inkey "$PIV_SLOT" -passin fd:"$FD" -in "$KEY_FILE" 2>&1
+then stty echo <&"$FD"
+ echo 'cluster key retrieval failed' 1>&2
+ exit 1
+fi | grep -v 'engine "pkcs11" set\.'
+
+echo 1>&"$FD"
+
+stty echo <&"$FD"
+
+exit 0
diff --git a/src/backend/crypto/ssl_passphrase.sh.sample b/src/backend/crypto/ssl_passphrase.sh.sample
new file mode 100644
index 0000000000..efbf5c0720
--- /dev/null
+++ b/src/backend/crypto/ssl_passphrase.sh.sample
@@ -0,0 +1,35 @@
+#!/bin/sh
+
+# This uses a passphrase supplied by the user.
+# Do not create any fie with extension "wkey" in $DIR; these are
+# reserved for wrapped data key files.
+
+[ "$#" -lt 1 ] && echo "ssl_passphrase_command usage: $0 %R [\"%p\"]" 1>&2 && exit 1
+
+FD="$1"
+[ ! -t "$FD" ] && echo "file descriptor $FD does not refer to a terminal" 1>&2 && exit 1
+# Supports environment variable PROMPT
+
+[ "$2" ] && PROMPT="$2"
+
+
+# ----------------------------------------------------------------------
+
+[ ! "$PROMPT" ] && PROMPT='Enter cluster passphrase: '
+
+stty -echo <&"$FD"
+
+echo 1>&"$FD"
+echo -n "$PROMPT" 1>&"$FD"
+read PASS <&"$FD"
+
+stty echo <&"$FD"
+
+if [ ! "$PASS" ]
+then echo 'invalid: empty passphrase' 1>&2
+ exit 1
+fi
+
+echo "$PASS"
+
+exit 0
--
2.20.1
cfe-04-common_over_cfe-03-scripts.diff (text/x-diff, 32.8K) download
From 57e9242bebe197bcf42012485e22e44530660f86 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:37 -0400
Subject: [PATCH] cfe-04-common_over_cfe-03-scripts squash commit
---
src/common/Makefile | 3 +
src/common/cipher.c (new) | 98 ++++++
src/common/cipher_openssl.c (new) | 419 +++++++++++++++++++++++
src/common/kmgr_utils.c (new) | 470 ++++++++++++++++++++++++++
src/include/common/cipher.h (new) | 74 ++++
src/include/common/kmgr_utils.h (new) | 94 ++++++
src/include/utils/wait_event.h | 3 +
src/tools/msvc/Mkvcbuild.pm | 4 +-
8 files changed, 1164 insertions(+), 1 deletion(-)
diff --git a/src/common/Makefile b/src/common/Makefile
index 38a8599337..5f14491c25 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -62,6 +62,7 @@ OBJS_COMMON = \
ip.o \
jsonapi.o \
keywords.o \
+ kmgr_utils.o \
kwlookup.o \
link-canary.o \
md5_common.o \
@@ -82,11 +83,13 @@ OBJS_COMMON = \
ifeq ($(with_ssl),openssl)
OBJS_COMMON += \
+ cipher_openssl.o \
protocol_openssl.o \
cryptohash_openssl.o \
hmac_openssl.o
else
OBJS_COMMON += \
+ cipher.o \
cryptohash.o \
hmac.o \
md5.o \
diff --git a/src/common/cipher.c b/src/common/cipher.c
new file mode 100644
index 0000000000..7aa3ea6138
--- /dev/null
+++ b/src/common/cipher.c
@@ -0,0 +1,98 @@
+/*-------------------------------------------------------------------------
+ *
+ * cipher.c
+ * Shared frontend/backend for cryptographic functions
+ *
+ * This is the set of in-core functions used when there are no other
+ * alternative options like OpenSSL.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/cipher.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/cipher.h"
+
+static void cipher_failure(void) pg_attribute_noreturn();
+
+
+PgCipherCtx *
+pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc)
+{
+ cipher_failure();
+ return NULL; /* keep compiler quiet */
+}
+
+void
+pg_cipher_ctx_free(PgCipherCtx *ctx)
+{
+ cipher_failure();
+}
+
+int
+pg_cipher_blocksize(PgCipherCtx *ctx)
+{
+ cipher_failure();
+ return -1; /* keep compiler quiet */
+}
+
+bool
+pg_cipher_encrypt(PgCipherCtx *ctx, const int cipher,
+ const unsigned char *plaintext,
+ const int inlen, unsigned char *ciphertext, int *outlen,
+ const unsigned char *iv, const int ivlen,
+ unsigned char *outtag, const int taglen)
+{
+ cipher_failure();
+ return false; /* keep compiler quiet */
+}
+
+bool
+pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher,
+ const unsigned char *ciphertext,
+ const int inlen, unsigned char *plaintext, int *outlen,
+ const unsigned char *iv, const int ivlen,
+ unsigned char *intag, const int taglen)
+{
+ cipher_failure();
+ return false; /* keep compiler quiet */
+}
+
+bool
+pg_cipher_keywrap(PgCipherCtx *ctx, const unsigned char *plaintext,
+ const int inlen, unsigned char *ciphertext, int *outlen)
+{
+ cipher_failure();
+ return false; /* keep compiler quiet */
+}
+
+bool
+pg_cipher_keyunwrap(PgCipherCtx *ctx, const unsigned char *ciphertext,
+ const int inlen, unsigned char *plaintext, int *outlen)
+{
+ cipher_failure();
+ return false; /* keep compiler quiet */
+}
+
+static void
+cipher_failure(void)
+{
+#ifndef FRONTEND
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"),
+ errhint("Compile with --with-openssl to use this feature."))));
+#else
+ fprintf(stderr, _("cluster file encryption is not supported because OpenSSL is not supported by this build"));
+ exit(1);
+#endif
+}
diff --git a/src/common/cipher_openssl.c b/src/common/cipher_openssl.c
new file mode 100644
index 0000000000..a03fbe435f
--- /dev/null
+++ b/src/common/cipher_openssl.c
@@ -0,0 +1,419 @@
+/*-------------------------------------------------------------------------
+ * cipher_openssl.c
+ * Cryptographic function using OpenSSL
+ *
+ * This contains the common low-level functions needed in both frontend and
+ * backend, for implement the database encryption.
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/cipher_openssl.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/cipher.h"
+#include <openssl/conf.h>
+#include <openssl/err.h>
+#include <openssl/evp.h>
+#include <openssl/ssl.h>
+
+/*
+ * prototype for the EVP functions that return an algorithm, e.g.
+ * EVP_aes_128_gcm().
+ */
+typedef const EVP_CIPHER *(*ossl_EVP_cipher_func) (void);
+
+static ossl_EVP_cipher_func get_evp_aes_gcm(int klen);
+static EVP_CIPHER_CTX *ossl_cipher_ctx_create(int cipher, unsigned char *key,
+ int klen, bool enc);
+
+/*
+ * Return a newly created cipher context. 'cipher' specifies cipher algorithm
+ * by identifier like PG_CIPHER_XXX.
+ */
+PgCipherCtx *
+pg_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc)
+{
+ PgCipherCtx *ctx = NULL;
+
+ if (cipher > PG_MAX_CIPHER_ID)
+ return NULL;
+
+ ctx = ossl_cipher_ctx_create(cipher, key, klen, enc);
+
+ return ctx;
+}
+
+void
+pg_cipher_ctx_free(PgCipherCtx *ctx)
+{
+ EVP_CIPHER_CTX_free(ctx);
+}
+
+int
+pg_cipher_blocksize(PgCipherCtx *ctx)
+{
+ Assert(ctx);
+
+ return EVP_CIPHER_CTX_block_size(ctx);
+}
+
+/*
+ * Encryption routine to encrypt data provided.
+ *
+ * ctx is the encryption context which must have been created previously.
+ *
+ * plaintext is the data we are going to encrypt
+ * inlen is the length of the data to encrypt
+ *
+ * ciphertext is the encrypted result
+ * outlen is the encrypted length
+ *
+ * iv is the IV to use.
+ * ivlen is the IV length to use.
+ *
+ * outtag is the resulting tag.
+ * taglen is the length of the tag.
+ */
+bool
+pg_cipher_encrypt(PgCipherCtx *ctx, int cipher,
+ const unsigned char *plaintext, const int inlen,
+ unsigned char *ciphertext, int *outlen,
+ const unsigned char *iv, const int ivlen,
+ unsigned char *outtag, const int taglen)
+{
+ int len;
+ int enclen;
+
+ Assert(ctx != NULL);
+
+ /*
+ * Here we are setting the IV for the context which was passed in. Note
+ * that we signal to OpenSSL that we are configuring a new value for the
+ * context by passing in 'NULL' for the 2nd ('type') parameter.
+ */
+
+ /*
+ * We don't use GCM mode, but it has a MAC, so we support it and test it
+ * in case we need it later. XXX is this correct for GCM and CTR?
+ */
+ /* Set the GCM IV length first */
+ if (cipher == PG_CIPHER_AES_GCM &&
+ !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, ivlen, NULL))
+ return false;
+
+ /* Set the IV for this encryption. */
+ if (!EVP_EncryptInit_ex(ctx, NULL, NULL, NULL, iv))
+ return false;
+
+ /*
+ * This is the function which is actually performing the encryption for
+ * us.
+ */
+ if (!EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, inlen))
+ return false;
+
+ enclen = len;
+
+ /* Finalize the encryption, which could add more to output. */
+ if (!EVP_EncryptFinal_ex(ctx, ciphertext + enclen, &len))
+ return false;
+
+ *outlen = enclen + len;
+
+ /*
+ * Once all of the encryption has been completed we grab the tag.
+ */
+ if (cipher == PG_CIPHER_AES_GCM &&
+ !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, taglen, outtag))
+ return false;
+
+ return true;
+}
+
+/*
+ * Decryption routine
+ *
+ * ctx is the encryption context which must have been created previously.
+ *
+ * ciphertext is the data we are going to decrypt
+ * inlen is the length of the data to decrypt
+ *
+ * plaintext is the decrypted result
+ * outlen is the decrypted length
+ *
+ * iv is the IV to use.
+ * ivlen is the length of the IV.
+ *
+ * intag is the tag to use to verify.
+ * taglen is the length of the tag.
+ */
+bool
+pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher,
+ const unsigned char *ciphertext, const int inlen,
+ unsigned char *plaintext, int *outlen,
+ const unsigned char *iv, const int ivlen,
+ unsigned char *intag, const int taglen)
+{
+ int declen;
+ int len;
+
+ /*
+ * Here we are setting the IV for the context which was passed in. Note
+ * that we signal to OpenSSL that we are configuring a new value for the
+ * context by passing in 'NULL' for the 2nd ('type') parameter.
+ */
+
+ /* XXX is this correct for GCM and CTR? */
+ /* Set the GCM IV length first */
+ if (cipher == PG_CIPHER_AES_GCM &&
+ !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, ivlen, NULL))
+ return false;
+
+ /* Set the IV for this decryption. */
+ if (!EVP_DecryptInit_ex(ctx, NULL, NULL, NULL, iv))
+ return false;
+
+ /*
+ * This is the function which is actually performing the decryption for
+ * us.
+ */
+ if (!EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, inlen))
+ return false;
+
+ declen = len;
+
+ /* Set the expected tag value. */
+ if (cipher == PG_CIPHER_AES_GCM &&
+ !EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, taglen, intag))
+ return false;
+
+ /*
+ * Finalize the decryption, which could add more to output, this is also
+ * the step which checks the tag and we MUST fail if this indicates an
+ * invalid tag!
+ */
+ if (!EVP_DecryptFinal_ex(ctx, plaintext + declen, &len))
+ return false;
+
+ *outlen = declen + len;
+
+ return true;
+}
+
+/*
+ * Routine to perform key wrapping for data provided.
+ *
+ * ctx is the encryption context which must have been created previously.
+ *
+ * plaintext is the data/key we are going to encrypt/wrap
+ * inlen is the length of the data
+ *
+ * ciphertext is the wrapped result
+ * outlen is the encrypted length (will be larger than input!)
+ */
+bool
+pg_cipher_keywrap(PgCipherCtx *ctx,
+ const unsigned char *plaintext, const int inlen,
+ unsigned char *ciphertext, int *outlen)
+{
+ int len;
+ int enclen;
+
+ Assert(ctx != NULL);
+
+ /*
+ * This is the function which is actually performing the encryption for
+ * us.
+ */
+ if (!EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, inlen))
+ return false;
+
+ enclen = len;
+
+ /* Finalize the encryption, which could add more to output. */
+ if (!EVP_EncryptFinal_ex(ctx, ciphertext + enclen, &len))
+ return false;
+
+ *outlen = enclen + len;
+
+ return true;
+}
+
+/*
+ * Routine to perform key unwrapping of the data provided.
+ *
+ * ctx is the encryption context which must have been created previously.
+ *
+ * ciphertext is the wrapped key we are going to unwrap
+ * inlen is the length of the data to decrypt/unwrap
+ *
+ * plaintext is the decrypted result
+ * outlen is the decrypted length (will be smaller than input!)
+ */
+bool
+pg_cipher_keyunwrap(PgCipherCtx *ctx,
+ const unsigned char *ciphertext, const int inlen,
+ unsigned char *plaintext, int *outlen)
+{
+ int declen;
+ int len;
+
+ /*
+ * This is the function which is actually performing the decryption for
+ * us.
+ */
+ if (!EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, inlen))
+ return false;
+
+ declen = len;
+
+ /*
+ * Finalize the decryption, which could add more to output, this is also
+ * the step which checks the tag and we MUST fail if this indicates an
+ * invalid result!
+ */
+ if (!EVP_DecryptFinal_ex(ctx, plaintext + declen, &len))
+ return false;
+
+ *outlen = declen + len;
+
+ return true;
+}
+
+/*
+ * Returns the correct GCM cipher functions for OpenSSL based
+ * on the key length requested.
+ */
+static ossl_EVP_cipher_func
+get_evp_aes_gcm(int klen)
+{
+ switch (klen)
+ {
+ case PG_AES128_KEY_LEN:
+ return EVP_aes_128_gcm;
+ case PG_AES192_KEY_LEN:
+ return EVP_aes_192_gcm;
+ case PG_AES256_KEY_LEN:
+ return EVP_aes_256_gcm;
+ default:
+ return NULL;
+ }
+}
+
+/*
+ * Returns the correct KWP cipher functions for OpenSSL based
+ * on the key length requested.
+ */
+static ossl_EVP_cipher_func
+get_evp_aes_kwp(int klen)
+{
+ switch (klen)
+ {
+ case PG_AES128_KEY_LEN:
+ return EVP_aes_128_wrap_pad;
+ case PG_AES192_KEY_LEN:
+ return EVP_aes_192_wrap_pad;
+ case PG_AES256_KEY_LEN:
+ return EVP_aes_256_wrap_pad;
+ default:
+ return NULL;
+ }
+}
+
+/*
+ * Returns the correct CTR cipher functions for OpenSSL based
+ * on the key length requested.
+ */
+static ossl_EVP_cipher_func
+get_evp_aes_ctr(int klen)
+{
+ switch (klen)
+ {
+ case PG_AES128_KEY_LEN:
+ return EVP_aes_128_ctr;
+ case PG_AES192_KEY_LEN:
+ return EVP_aes_192_ctr;
+ case PG_AES256_KEY_LEN:
+ return EVP_aes_256_ctr;
+ default:
+ return NULL;
+ }
+}
+
+/*
+ * Initialize and return an EVP_CIPHER_CTX. Returns NULL if the given
+ * cipher algorithm is not supported or on failure.
+ */
+static EVP_CIPHER_CTX *
+ossl_cipher_ctx_create(int cipher, unsigned char *key, int klen, bool enc)
+{
+ EVP_CIPHER_CTX *ctx;
+ ossl_EVP_cipher_func func;
+ int ret;
+
+ ctx = EVP_CIPHER_CTX_new();
+
+ /*
+ * We currently only support AES GCM but others could be added in the
+ * future.
+ */
+ switch (cipher)
+ {
+ case PG_CIPHER_AES_GCM:
+ func = get_evp_aes_gcm(klen);
+ if (!func)
+ goto failed;
+ break;
+ case PG_CIPHER_AES_KWP:
+ func = get_evp_aes_kwp(klen);
+
+ /*
+ * Since wrapping will produce more output then input, and we have
+ * to be ready for that, OpenSSL requires that we explicitly
+ * enable wrapping for the context.
+ */
+ EVP_CIPHER_CTX_set_flags(ctx, EVP_CIPHER_CTX_FLAG_WRAP_ALLOW);
+ if (!func)
+ goto failed;
+ break;
+ case PG_CIPHER_AES_CTR:
+ func = get_evp_aes_ctr(klen);
+ if (!func)
+ goto failed;
+ break;
+ default:
+ goto failed;
+ }
+
+ /*
+ * We create the context here based on the cipher requested and the
+ * provided key. Note that the IV will be provided in the actual
+ * encryption call through another EVP_EncryptInit_ex call- this is fine
+ * as long as 'type' is passed in as NULL!
+ */
+ if (enc)
+ ret = EVP_EncryptInit_ex(ctx, (const EVP_CIPHER *) func(), NULL, key, NULL);
+ else
+ ret = EVP_DecryptInit_ex(ctx, (const EVP_CIPHER *) func(), NULL, key, NULL);
+
+ if (!ret)
+ goto failed;
+
+ /* Set the key length based on the key length requested. */
+ if (!EVP_CIPHER_CTX_set_key_length(ctx, klen))
+ goto failed;
+
+ return ctx;
+
+failed:
+ EVP_CIPHER_CTX_free(ctx);
+ return NULL;
+}
diff --git a/src/common/kmgr_utils.c b/src/common/kmgr_utils.c
new file mode 100644
index 0000000000..30adfd3069
--- /dev/null
+++ b/src/common/kmgr_utils.c
@@ -0,0 +1,470 @@
+/*-------------------------------------------------------------------------
+ *
+ * kmgr_utils.c
+ * Shared frontend/backend for cluster file encryption
+ *
+ * These functions handle reading the wrapped DEK files from the
+ * file system, and wrapping and unwrapping them. It also handles
+ * running the cluster_key_command.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/common/kmgr_utils.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include <unistd.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#ifdef FRONTEND
+#include "common/logging.h"
+#endif
+#include "common/cryptohash.h"
+#include "common/file_perm.h"
+#include "common/hex.h"
+#include "common/string.h"
+#include "crypto/kmgr.h"
+#include "lib/stringinfo.h"
+#include "storage/fd.h"
+
+#ifndef FRONTEND
+#include "pgstat.h"
+#include "storage/fd.h"
+#endif
+
+#define KMGR_PROMPT_MSG "Enter authentication needed to generate the cluster key: "
+
+#ifdef FRONTEND
+static FILE *open_pipe_stream(const char *command);
+static int close_pipe_stream(FILE *file);
+#endif
+
+static void read_wrapped_data_key(const char *cryptoKeyDir, uint32 id, unsigned char **key_p, int *key_len);
+
+/*
+ * We need this array in frontend and backend code, so define it here.
+ * You can only add to the end of this array since the array index is
+ * stored in pg_control.
+ */
+encryption_method encryption_methods[NUM_ENCRYPTION_METHODS] = {
+ {"", 0},
+ {"AES128", 128},
+ {"AES192", 192},
+ {"AES256", 256}
+};
+
+/* This maps wrapped key filesnames to their array slots */
+char *wkey_filenames[KMGR_NUM_DATA_KEYS] = {
+ "relation",
+ "wal"
+};
+
+
+/*
+ * Wrap the given CryptoKey.
+ *
+ * Returns true and writes encrypted/wrapped/padded data to 'out', and the length
+ * of the result to outlen, if successful.
+ *
+ * Otherwise returns false. The caller must allocate sufficient space
+ * for cipher data calculated by using KmgrSizeOfCipherText(). Please note that
+ * this function modifies 'out' data even on failure.
+ */
+bool
+kmgr_wrap_data_key(PgCipherCtx *ctx, CryptoKey *in, unsigned char *out, int *outlen)
+{
+ Assert(ctx && in && out);
+
+ if (!pg_cipher_keywrap(ctx, (unsigned char *) in, sizeof(CryptoKey), out, outlen))
+ return false;
+
+ return true;
+}
+
+/*
+ * Decrypt the given data. Return true and set plain text data to `out` if
+ * successful. Otherwise return false. The caller must allocate sufficient
+ * space for cipher data calculated by using KmgrSizeOfPlainText(). Please
+ * note that this function modifies 'out' data even on failure.
+ */
+bool
+kmgr_unwrap_data_key(PgCipherCtx *ctx, unsigned char *in, int inlen, CryptoKey *out)
+{
+ int outlen;
+
+ Assert(ctx && in && out);
+
+ if (!pg_cipher_keyunwrap(ctx, in, inlen, (unsigned char *) out, &outlen))
+ return false;
+
+ Assert(outlen == sizeof(CryptoKey));
+
+ return true;
+}
+
+/*
+ * Verify the correctness of the given cluster key by unwrapping the given keys.
+ * If the given cluster key is correct we set unwrapped keys to out_keys and return
+ * true. Otherwise return false. Please note that this function changes the
+ * contents of out_keys even on failure. Both in_keys and out_keys must be the
+ * same length.
+ */
+bool
+kmgr_verify_cluster_key(unsigned char *cluster_key,
+ unsigned char **in_keys, int *key_lens, CryptoKey *out_keys)
+{
+ PgCipherCtx *ctx;
+
+ /* Create decryption context with the KEK. */
+ ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, cluster_key,
+ KMGR_CLUSTER_KEY_LEN, false);
+
+ /* unwrap each DEK */
+ for (int i = 0; i < KMGR_NUM_DATA_KEYS; i++)
+ {
+ if (!kmgr_unwrap_data_key(ctx, in_keys[i], key_lens[i], &(out_keys[i])))
+ {
+ /* The cluster key is not correct */
+ pg_cipher_ctx_free(ctx);
+ return false;
+ }
+ explicit_bzero(in_keys[i], key_lens[i]);
+ }
+
+ /* The cluster key is correct, free the cipher context */
+ pg_cipher_ctx_free(ctx);
+
+ return true;
+}
+
+/*
+ * Run cluster key command.
+ *
+ * Substitute %d for directory, %p for prompt, %R for file descriptor.
+ *
+ * The result will be put in buffer buf, which is of size "size".
+ * The return value is the length of the actual result.
+ */
+int
+kmgr_run_cluster_key_command(char *cluster_key_command, char *buf,
+ int size, char *dir, int terminal_fd)
+{
+ StringInfoData command;
+ const char *sp;
+ FILE *fh;
+ int pclose_rc;
+ size_t len = 0;
+
+ buf[0] = '\0';
+
+ Assert(size > 0);
+
+ /*
+ * Build the command to be executed.
+ */
+ initStringInfo(&command);
+
+ for (sp = cluster_key_command; *sp; sp++)
+ {
+ if (*sp == '%')
+ {
+ switch (sp[1])
+ {
+ /* directory */
+ case 'd':
+ {
+ char *nativePath;
+
+ sp++;
+
+ /*
+ * This needs to use a placeholder to not modify the
+ * input with the conversion done via
+ * make_native_path().
+ */
+ nativePath = pstrdup(dir);
+ make_native_path(nativePath);
+ appendStringInfoString(&command, nativePath);
+ pfree(nativePath);
+ break;
+ }
+ /* prompt string */
+ case 'p':
+ sp++;
+ appendStringInfoString(&command, KMGR_PROMPT_MSG);
+ break;
+ /* file descriptor number */
+ case 'R':
+ {
+ char fd_str[20];
+
+ if (terminal_fd == -1)
+ {
+#ifdef FRONTEND
+ pg_log_fatal("cluster key command referenced %%R, but --authprompt not specified");
+ exit(EXIT_FAILURE);
+#else
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("cluster key command referenced %%R, but --authprompt not specified")));
+#endif
+ }
+
+ sp++;
+ snprintf(fd_str, sizeof(fd_str), "%d", terminal_fd);
+ appendStringInfoString(&command, fd_str);
+ break;
+ }
+ /* literal "%" */
+ case '%':
+ /* convert %% to a single % */
+ sp++;
+ appendStringInfoChar(&command, *sp);
+ break;
+ default:
+ /* otherwise treat the % as not special */
+ appendStringInfoChar(&command, *sp);
+ break;
+ }
+ }
+ else
+ {
+ appendStringInfoChar(&command, *sp);
+ }
+ }
+
+#ifdef FRONTEND
+ fh = open_pipe_stream(command.data);
+ if (fh == NULL)
+ {
+ pg_log_fatal("could not execute command \"%s\": %m",
+ command.data);
+ exit(EXIT_FAILURE);
+ }
+#else
+ fh = OpenPipeStream(command.data, "r");
+ if (fh == NULL)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not execute command \"%s\": %m",
+ command.data)));
+#endif
+
+ if (!fgets(buf, size, fh))
+ {
+ if (ferror(fh))
+ {
+#ifdef FRONTEND
+ pg_log_fatal("could not read from command \"%s\": %m",
+ command.data);
+ exit(EXIT_FAILURE);
+#else
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read from command \"%s\": %m",
+ command.data)));
+#endif
+ }
+ }
+
+#ifdef FRONTEND
+ pclose_rc = close_pipe_stream(fh);
+#else
+ pclose_rc = ClosePipeStream(fh);
+#endif
+
+ if (pclose_rc == -1)
+ {
+#ifdef FRONTEND
+ pg_log_fatal("could not close pipe to external command: %m");
+ exit(EXIT_FAILURE);
+#else
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close pipe to external command: %m")));
+#endif
+ }
+ else if (pclose_rc != 0)
+ {
+#ifdef FRONTEND
+ pg_log_fatal("command \"%s\" failed", command.data);
+ exit(EXIT_FAILURE);
+#else
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("command \"%s\" failed",
+ command.data),
+ errdetail_internal("%s", wait_result_to_str(pclose_rc))));
+#endif
+ }
+
+ /* strip trailing newline and carriage returns */
+ len = pg_strip_crlf(buf);
+
+ pfree(command.data);
+
+ return len;
+}
+
+#ifdef FRONTEND
+static FILE *
+open_pipe_stream(const char *command)
+{
+ FILE *res;
+
+#ifdef WIN32
+ size_t cmdlen = strlen(command);
+ char *buf;
+ int save_errno;
+
+ buf = malloc(cmdlen + 2 + 1);
+ if (buf == NULL)
+ {
+ errno = ENOMEM;
+ return NULL;
+ }
+ buf[0] = '"';
+ memcpy(&buf[1], command, cmdlen);
+ buf[cmdlen + 1] = '"';
+ buf[cmdlen + 2] = '\0';
+
+ res = _popen(buf, "r");
+
+ save_errno = errno;
+ free(buf);
+ errno = save_errno;
+#else
+ res = popen(command, "r");
+#endif /* WIN32 */
+ return res;
+}
+
+static int
+close_pipe_stream(FILE *file)
+{
+#ifdef WIN32
+ return _pclose(file);
+#else
+ return pclose(file);
+#endif /* WIN32 */
+}
+#endif /* FRONTEND */
+
+/*
+ * Reads the keys at path.
+ *
+ * This routine simply reads in the raw encrypted/wrapped keys;
+ * it does not handle any decryption, see kmgr_key_unwrap().
+ *
+ * For each key returned, the key and key length are returned
+ * in the keys and key_lens arrays respectfully.
+ *
+ * Note that keys and key_lens must be allocated before calling
+ * this function as arrays of at least KMGR_NUM_DATA_KEYS length.
+ */
+void
+kmgr_read_wrapped_data_keys(const char *path, unsigned char **keys, int *key_lens)
+{
+/* StaticAssertStmt(lengthof(wkey_filenames) == KMGR_NUM_DATA_KEYS,
+ "wkey_filenames[] must match KMGR_NUM_DATA_KEYS");
+*/
+ StaticAssertStmt(1 == 1,
+ "wkey_filenames[] must match KMGR_NUM_DATA_KEYS");
+
+ for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++)
+ read_wrapped_data_key(path, id, &(keys[id]), &(key_lens[id]));
+
+ return;
+}
+
+/* Read a wrapped DEK file */
+static void
+read_wrapped_data_key(const char *cryptoKeyDir, uint32 id, unsigned char **key_p, int *key_len)
+{
+ char path[MAXPGPATH];
+ int fd;
+ int r;
+ struct stat st;
+
+ CryptoKeyFilePath(path, cryptoKeyDir, id);
+
+#ifndef FRONTEND
+ if ((fd = OpenTransientFile(path, O_RDONLY | PG_BINARY)) == -1)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\" for reading: %m",
+ path)));
+ else if (fstat(fd, &st))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ path)));
+#else
+ if ((fd = open(path, O_RDONLY | PG_BINARY, 0)) == -1)
+ pg_log_fatal("could not open file \"%s\" for reading: %m",
+ path);
+ else if (fstat(fd, &st))
+ pg_log_fatal("could not stat file \"%s\": %m",
+ path);
+#endif
+
+ *key_len = st.st_size;
+
+#ifndef FRONTEND
+ pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_READ);
+#endif
+
+ *key_p = (unsigned char *) palloc0(*key_len);
+
+ /* Get key bytes */
+ r = read(fd, *key_p, *key_len);
+ if (r != *key_len)
+ {
+ if (r < 0)
+ {
+#ifndef FRONTEND
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m", path)));
+#else
+ pg_log_fatal("could not read file \"%s\": %m", path);
+#endif
+ }
+ else
+ {
+#ifndef FRONTEND
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("could not read file \"%s\": read %d of %u",
+ path, r, *key_len)));
+#else
+ pg_log_fatal("could not read file \"%s\": read %d of %u",
+ path, r, *key_len);
+#endif
+ }
+ }
+
+#ifndef FRONTEND
+ pgstat_report_wait_end();
+#endif
+
+#ifndef FRONTEND
+ if (CloseTransientFile(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"%s\": %m",
+ path)));
+#else
+ if (close(fd) != 0)
+ pg_log_fatal("could not close file \"%s\": %m", path);
+#endif
+}
diff --git a/src/include/common/cipher.h b/src/include/common/cipher.h
new file mode 100644
index 0000000000..da97ca3776
--- /dev/null
+++ b/src/include/common/cipher.h
@@ -0,0 +1,74 @@
+/*-------------------------------------------------------------------------
+ *
+ * cipher.h
+ * Declarations for cryptographic functions
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/common/cipher.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_CIPHER_H
+#define PG_CIPHER_H
+
+#ifdef USE_OPENSSL
+#include <openssl/evp.h>
+#include <openssl/conf.h>
+#include <openssl/err.h>
+#endif
+
+/*
+ * Supported symmetric encryption algorithm. These identifiers are passed
+ * to pg_cipher_ctx_create() function, and then actual encryption
+ * implementations need to initialize their context of the given encryption
+ * algorithm.
+ */
+#define PG_CIPHER_AES_GCM 0
+#define PG_CIPHER_AES_CTR 1
+#define PG_CIPHER_AES_KWP 2
+#define PG_MAX_CIPHER_ID 2
+
+/* AES128/192/256 various length definitions */
+#define PG_AES128_KEY_LEN (128 / 8)
+#define PG_AES192_KEY_LEN (192 / 8)
+#define PG_AES256_KEY_LEN (256 / 8)
+
+/*
+ * The encrypted data is a series of blocks of size. Initialization
+ * vector(IV) is the same size of cipher block.
+ */
+#define PG_AES_BLOCK_SIZE 16
+#define PG_AES_IV_SIZE (PG_AES_BLOCK_SIZE)
+
+#ifdef USE_OPENSSL
+typedef EVP_CIPHER_CTX PgCipherCtx;
+#else
+typedef void PgCipherCtx;
+#endif
+
+extern PgCipherCtx *pg_cipher_ctx_create(int cipher, unsigned char *key, int klen,
+ bool enc);
+
+extern void pg_cipher_ctx_free(PgCipherCtx *ctx);
+extern bool pg_cipher_encrypt(PgCipherCtx *ctx, int cipher,
+ const unsigned char *plaintext, const int inlen,
+ unsigned char *ciphertext, int *outlen,
+ const unsigned char *iv, const int ivlen,
+ unsigned char *tag, const int taglen);
+extern bool pg_cipher_decrypt(PgCipherCtx *ctx, const int cipher,
+ const unsigned char *ciphertext, const int inlen,
+ unsigned char *plaintext, int *outlen,
+ const unsigned char *iv, const int ivlen,
+ unsigned char *intag, const int taglen);
+
+extern bool pg_cipher_keywrap(PgCipherCtx *ctx,
+ const unsigned char *plaintext, const int inlen,
+ unsigned char *ciphertext, int *outlen);
+extern bool pg_cipher_keyunwrap(PgCipherCtx *ctx,
+ const unsigned char *ciphertext, const int inlen,
+ unsigned char *plaintext, int *outlen);
+
+extern int pg_cipher_blocksize(PgCipherCtx *ctx);
+
+#endif /* PG_CIPHER_H */
diff --git a/src/include/common/kmgr_utils.h b/src/include/common/kmgr_utils.h
new file mode 100644
index 0000000000..ca264db728
--- /dev/null
+++ b/src/include/common/kmgr_utils.h
@@ -0,0 +1,94 @@
+/*-------------------------------------------------------------------------
+ *
+ * kmgr_utils.h
+ * Declarations for utility function for file encryption key
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/common/kmgr_utils.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef KMGR_UTILS_H
+#define KMGR_UTILS_H
+
+#include "common/cipher.h"
+
+/* Current version number */
+#define KMGR_VERSION 1
+
+/*
+ * Directories where cluster file encryption keys reside within PGDATA.
+ */
+#define KMGR_DIR "pg_cryptokeys"
+#define KMGR_DIR_PID KMGR_DIR"/pg_alterckey.pid"
+#define LIVE_KMGR_DIR KMGR_DIR"/live"
+/* used during cluster key rotation */
+#define NEW_KMGR_DIR KMGR_DIR"/new"
+#define OLD_KMGR_DIR KMGR_DIR"/old"
+
+/* CryptoKey file name is keys id */
+#define CryptoKeyFilePath(path, dir, id) \
+ snprintf((path), MAXPGPATH, "%s/%s.wkey", (dir), (wkey_filenames[id]))
+
+/*
+ * Identifiers of internal keys.
+ */
+#define KMGR_KEY_ID_REL 0
+#define KMGR_KEY_ID_WAL 1
+#define KMGR_NUM_DATA_KEYS 2
+
+/* We always, today, use a 256-bit AES key. */
+#define KMGR_CLUSTER_KEY_LEN PG_AES256_KEY_LEN
+
+/* double for hex format, plus some for spaces, \r,\n, and null byte */
+#define ALLOC_KMGR_CLUSTER_KEY_LEN (KMGR_CLUSTER_KEY_LEN * 2 + 10 + 2 + 1)
+
+/* Maximum length of key the key manager can store */
+#define KMGR_MAX_KEY_LEN 256
+#define KMGR_MAX_KEY_LEN_BYTES KMGR_MAX_KEY_LEN / 8
+
+
+/*
+ * Cryptographic key data structure.
+ *
+ * This is the structure we use to write out the encrypted keys and
+ * which we use to store the keys in shared memory.
+ *
+ * Note that wrapping this structure results in an encrypted byte
+ * string which is what we actually write and then read back in.
+ *
+ * klen is the key length in bytes
+ * key is the encryption key of klen length
+ */
+typedef struct CryptoKey
+{
+ int klen; /* key length in bytes */
+ unsigned char key[KMGR_MAX_KEY_LEN_BYTES];
+} CryptoKey;
+
+/* Encryption method array */
+typedef struct encryption_method
+{
+ const char *name;
+ const int bit_length;
+} encryption_method;
+
+#define NUM_ENCRYPTION_METHODS 4
+#define DISABLED_ENCRYPTION_METHOD 0
+#define DEFAULT_ENABLED_ENCRYPTION_METHOD 1
+
+extern encryption_method encryption_methods[NUM_ENCRYPTION_METHODS];
+extern char *wkey_filenames[KMGR_NUM_DATA_KEYS];
+
+extern bool kmgr_wrap_data_key(PgCipherCtx *ctx, CryptoKey *in, unsigned char *out, int *outlen);
+extern bool kmgr_unwrap_data_key(PgCipherCtx *ctx, unsigned char *in, int inlen, CryptoKey *out);
+extern bool kmgr_verify_cluster_key(unsigned char *cluster_key,
+ unsigned char **in_keys, int *klens, CryptoKey *out_keys);
+extern int kmgr_run_cluster_key_command(char *cluster_key_command,
+ char *buf, int size, char *dir,
+ int terminal_fd);
+extern void kmgr_read_wrapped_data_keys(const char *path, unsigned char **keys,
+ int *key_lens);
+
+#endif /* KMGR_UTILS_H */
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index 44448b48ec..c872fc4746 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -170,6 +170,9 @@ typedef enum
WAIT_EVENT_DATA_FILE_TRUNCATE,
WAIT_EVENT_DATA_FILE_WRITE,
WAIT_EVENT_DSM_FILL_ZERO_WRITE,
+ WAIT_EVENT_KEY_FILE_READ,
+ WAIT_EVENT_KEY_FILE_WRITE,
+ WAIT_EVENT_KEY_FILE_SYNC,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_SYNC,
WAIT_EVENT_LOCK_FILE_ADDTODATADIR_WRITE,
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 58a99e4f10..893f73af7d 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -124,19 +124,21 @@ sub mkvcbuild
archive.c base64.c checksum_helper.c
config_info.c controldata_utils.c d2s.c encnames.c exec.c
f2s.c file_perm.c file_utils.c hashfn.c hex.c ip.c jsonapi.c
- keywords.c kwlookup.c link-canary.c md5_common.c
+ keywords.c kmgr_utils.c kwlookup.c link-canary.c md5_common.c
pg_get_line.c pg_lzcompress.c pgfnames.c psprintf.c relpath.c rmtree.c
saslprep.c scram-common.c string.c stringinfo.c unicode_norm.c username.c
wait_error.c wchar.c);
if ($solution->{options}->{openssl})
{
+ push(@pgcommonallfiles, 'cipher_openssl.c');
push(@pgcommonallfiles, 'cryptohash_openssl.c');
push(@pgcommonallfiles, 'hmac_openssl.c');
push(@pgcommonallfiles, 'protocol_openssl.c');
}
else
{
+ push(@pgcommonallfiles, 'cipher.c');
push(@pgcommonallfiles, 'cryptohash.c');
push(@pgcommonallfiles, 'hmac.c');
push(@pgcommonallfiles, 'md5.c');
--
2.20.1
cfe-05-crypto_over_cfe-04-common.diff (text/x-diff, 23.0K) download
From 3d106557aa708d74efb742d225a8362f1085ae89 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:38 -0400
Subject: [PATCH] cfe-05-crypto_over_cfe-04-common squash commit
---
doc/src/sgml/config.sgml | 118 +++++++-
src/backend/crypto/Makefile (new) | 18 ++
src/backend/crypto/kmgr.c (new) | 439 ++++++++++++++++++++++++++++++
src/include/crypto/kmgr.h (new) | 25 ++
4 files changed, 586 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index effc60c07b..d7b6e368d0 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1531,19 +1531,31 @@ include_dir 'conf.d'
mechanism is used.
</para>
<para>
- The command must print the passphrase to the standard output and exit
- with code 0. In the parameter value, <literal>%p</literal> is
- replaced by a prompt string. (Write <literal>%%</literal> for a
- literal <literal>%</literal>.) Note that the prompt string will
- probably contain whitespace, so be sure to quote adequately. A single
- newline is stripped from the end of the output if present.
+ The command must print the passphrase to the standard output
+ and exit with code 0. It can prompt from the terminal if
+ <option>--authprompt</option> is used. In the parameter
+ value, <literal>%R</literal> is replaced by a file descriptor
+ number opened to the terminal that started the server. A file
+ descriptor is only available if enabled at server start via
+ <option>-R</option>. If <literal>%R</literal> is specified and
+ no file descriptor is available, the server will not start. Value
+ <literal>%p</literal> is replaced by a pre-defined prompt string.
+ (Write <literal>%%</literal> for a literal <literal>%</literal>.)
+ Note that the prompt string will probably contain whitespace,
+ so be sure to quote its use adequately. Newlines are stripped
+ from the end of the output if present.
</para>
+
<para>
- The command does not actually have to prompt the user for a
- passphrase. It can read it from a file, obtain it from a keychain
- facility, or similar. It is up to the user to make sure the chosen
- mechanism is adequately secure.
+ Sample scripts can be found in
+ <filename>$SHAREDIR/auth_commands</filename>,
+ where <literal>$SHAREDIR</literal> means the
+ <productname>PostgreSQL</productname> installation's shared-data
+ directory, often <filename>/usr/local/share/postgresql</filename>
+ (use <command>pg_config --sharedir</command> to determine it if
+ you're not sure).
</para>
+
<para>
This parameter can only be set in the <filename>postgresql.conf</filename>
file or on the server command line.
@@ -1565,10 +1577,12 @@ include_dir 'conf.d'
parameter is off (the default), then
<varname>ssl_passphrase_command</varname> will be ignored during a
reload and the SSL configuration will not be reloaded if a passphrase
- is needed. That setting is appropriate for a command that requires a
- TTY for prompting, which might not be available when the server is
- running. Setting this parameter to on might be appropriate if the
- passphrase is obtained from a file, for example.
+ is needed. This setting is appropriate for a command that requires a
+ terminal for prompting, which will likely not be available when the server is
+ running. (<option>--authprompt</option> closes the terminal file
+ descriptor soon after server start.) Setting this parameter on
+ might be appropriate, for example, if the passphrase is obtained
+ from a file.
</para>
<para>
This parameter can only be set in the <filename>postgresql.conf</filename>
@@ -2724,6 +2738,8 @@ include_dir 'conf.d'
<literal>minimal</literal> makes any base backups taken before
unavailable for archive recovery and standby server, which may
lead to database loss.
+ Cluster file encryption also does not support
+ <varname>wal_level</varname> <literal>minimal</literal>.
</para>
<para>
In <literal>logical</literal> level, the same information is logged as
@@ -7998,6 +8014,64 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</variablelist>
</sect1>
+ <sect1 id="runtime-config-encryption">
+ <title>Cluster File Encryption</title>
+
+ <variablelist>
+ <varlistentry id="guc-cluster-key-command" xreflabel="cluster_key_command">
+ <term><varname>cluster_key_command</varname> (<type>string</type>)
+ <indexterm>
+ <primary><varname>cluster_key_command</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This option specifies an external command to obtain the cluster-level
+ key for cluster file encryption during server initialization and
+ server start.
+ </para>
+ <para>
+ The command must print the cluster key to the standard
+ output as 64 hexadecimal characters, and exit with code 0.
+ The command can prompt for the passphrase or PIN from the
+ terminal if <option>--authprompt</option> is used. In the
+ parameter value, <literal>%R</literal> is replaced by a file
+ descriptor number opened to the terminal that started the server.
+ A file descriptor is only available if enabled at server start
+ via <option>-R</option>. If <literal>%R</literal> is specified
+ and no file descriptor is available, the server will not start.
+ Value <literal>%p</literal> is replaced by a pre-defined
+ prompt string. Value <literal>%d</literal> is replaced by the
+ directory containing the keys; this is useful if the command
+ must create files with the keys, e.g., to store a cluster-level
+ key encrypted by a key stored in a hardware security module.
+ (Write <literal>%%</literal> for a literal <literal>%</literal>.)
+ Note that the prompt string will probably contain whitespace,
+ so be sure to quote its use adequately. Newlines are stripped
+ from the end of the output if present.
+ </para>
+
+ <para>
+ Sample script can be found in
+ <filename>$SHAREDIR/auth_commands</filename>,
+ where <literal>$SHAREDIR</literal> means the
+ <productname>PostgreSQL</productname> installation's shared-data
+ directory, often <filename>/usr/local/share/postgresql</filename>
+ (use <command>pg_config --sharedir</command> to determine it if
+ you're not sure).
+ </para>
+
+ <para>
+ This parameter can only be set by
+ <application>initdb</application>, in the
+ <filename>postgresql.conf</filename> file, or on the server
+ command line.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect1>
+
<sect1 id="runtime-config-client">
<title>Client Connection Defaults</title>
@@ -9890,6 +9964,22 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir'
</listitem>
</varlistentry>
+ <varlistentry id="guc-file-encryption-method" xreflabel="file_encryption_method">
+ <term><varname>file_encryption_method</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary>Cluster file encryption method</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Reports the cluster file
+ encryption method. See <xref
+ linkend="app-initdb-cluster-key-command"/> for more
+ information.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-data-directory-mode" xreflabel="data_directory_mode">
<term><varname>data_directory_mode</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile
new file mode 100644
index 0000000000..c27362029d
--- /dev/null
+++ b/src/backend/crypto/Makefile
@@ -0,0 +1,18 @@
+#-------------------------------------------------------------------------
+#
+# Makefile
+# Makefile for src/backend/crypto
+#
+# IDENTIFICATION
+# src/backend/crypto/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/crypto
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ kmgr.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c
new file mode 100644
index 0000000000..f4d97879ce
--- /dev/null
+++ b/src/backend/crypto/kmgr.c
@@ -0,0 +1,439 @@
+/*-------------------------------------------------------------------------
+ *
+ * kmgr.c
+ * Cluster file encryption routines
+ *
+ * Cluster file encryption is enabled if user requests it during initdb.
+ * During bootstrap, we generate data encryption keys, wrap them with the
+ * cluster-level key, and store them into each file located at KMGR_DIR.
+ * During startup, we decrypt all internal keys and load them to the shared
+ * memory. Internal keys in the shared memory are read-only. The wrapping
+ * and unwrapping key routines require the OpenSSL library.
+ *
+ * Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/crypto/kmgr.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "funcapi.h"
+#include "miscadmin.h"
+#include "pgstat.h"
+
+#include "access/xlog.h"
+#include "common/file_perm.h"
+#include "common/hex.h"
+#include "common/kmgr_utils.h"
+#include "common/sha2.h"
+#include "access/xlog.h"
+#include "common/controldata_utils.h"
+#include "crypto/kmgr.h"
+#include "postmaster/postmaster.h"
+#include "storage/copydir.h"
+#include "storage/fd.h"
+#include "storage/ipc.h"
+#include "storage/shmem.h"
+#include "utils/builtins.h"
+#include "utils/memutils.h"
+/* Struct stores file encryption keys in plaintext format */
+typedef struct KmgrShmemData
+{
+ CryptoKey intlKeys[KMGR_NUM_DATA_KEYS];
+} KmgrShmemData;
+static KmgrShmemData *KmgrShmem;
+
+/* GUC variables */
+char *cluster_key_command = NULL;
+
+CryptoKey bootstrap_keys[KMGR_NUM_DATA_KEYS];
+
+extern char *bootstrap_old_key_datadir;
+extern int bootstrap_file_encryption_method;
+
+static void bzeroKmgrKeys(int status, Datum arg);
+static void KmgrWriteCryptoKeys(const char *dir, unsigned char **keys, int *key_lens);
+static CryptoKey *generate_crypto_key(int len);
+
+/*
+ * This function must be called ONCE during initdb. It creates the DEK
+ * files wrapped with the KEK supplied by kmgr_run_cluster_key_command().
+ * There is also an option for the keys to be copied from another cluster.
+ */
+void
+BootStrapKmgr(void)
+{
+ char live_path[MAXPGPATH];
+ unsigned char *keys_wrap[KMGR_NUM_DATA_KEYS];
+ int key_lens[KMGR_NUM_DATA_KEYS];
+ char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN];
+ int cluster_key_hex_len;
+ unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN];
+
+ if (!FileEncryptionEnabled)
+ return;
+
+#ifndef USE_OPENSSL
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"),
+ errhint("Compile with --with-openssl to use this feature."))));
+#endif
+
+ /*
+ * There are too many optimizations for wal_level=minimal that don't set
+ * LSNs for permanent tables, so just disallow it.
+ */
+ if (!XLogIsNeeded())
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ (errmsg("cluster file encryption is not supported with a wal_level of \"minimal\""))));
+
+ snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR);
+
+ /*
+ * Copy cluster file encryption keys from an old cluster? This is useful
+ * for pg_upgrade upgrades where the copied database files are already
+ * encrypted using the old cluster's DEK keys.
+ */
+ if (bootstrap_old_key_datadir != NULL)
+ {
+ char old_key_dir[MAXPGPATH];
+
+ snprintf(old_key_dir, sizeof(old_key_dir), "%s/%s",
+ bootstrap_old_key_datadir, LIVE_KMGR_DIR);
+ copydir(old_key_dir, LIVE_KMGR_DIR, true);
+ }
+ /* create an empty directory */
+ else
+ {
+ if (mkdir(LIVE_KMGR_DIR, pg_dir_create_mode) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create cluster file encryption directory \"%s\": %m",
+ LIVE_KMGR_DIR)));
+ }
+
+ /*
+ * Get key encryption key (KEK) from the cluster_key command. The cluster
+ * key command might need to check for the existence of files in the live
+ * directory, e.g., PIV, so run this _after_ copying the directory in
+ * place.
+ */
+ cluster_key_hex_len = kmgr_run_cluster_key_command(cluster_key_command,
+ cluster_key_hex,
+ ALLOC_KMGR_CLUSTER_KEY_LEN,
+ live_path, terminal_fd);
+
+ /* decode supplied hex */
+ if (pg_hex_decode(cluster_key_hex, cluster_key_hex_len,
+ (char *) cluster_key, KMGR_CLUSTER_KEY_LEN) !=
+ KMGR_CLUSTER_KEY_LEN)
+ ereport(ERROR,
+ (errmsg("cluster key must be %d hexadecimal characters",
+ KMGR_CLUSTER_KEY_LEN * 2)));
+
+ /* We are not in copy mode? Generate new cluster file encryption keys. */
+ if (bootstrap_old_key_datadir == NULL)
+ {
+ unsigned char *bootstrap_keys_wrap[KMGR_NUM_DATA_KEYS];
+ int key_lens[KMGR_NUM_DATA_KEYS];
+ PgCipherCtx *cluster_key_ctx;
+
+ /* Create KEK encryption context */
+ cluster_key_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP, cluster_key,
+ KMGR_CLUSTER_KEY_LEN, true);
+ if (!cluster_key_ctx)
+ elog(ERROR, "could not initialize encryption context");
+
+ /* Wrap data encryption keys (DEK) using the key encryption key (KEK) */
+ for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++)
+ {
+ CryptoKey *key;
+
+ /* generate a DEK */
+ key = generate_crypto_key(
+ encryption_methods[bootstrap_file_encryption_method].bit_length / 8);
+
+ /* output generated random string as hex, for testing */
+ {
+ char str[MAXPGPATH];
+ int out_len;
+
+ out_len = pg_hex_encode((char *) (key->key), key->klen,
+ str, MAXPGPATH - 1);
+ str[out_len] = '\0';
+ }
+
+ bootstrap_keys_wrap[id] = palloc0(KMGR_MAX_KEY_LEN_BYTES +
+ pg_cipher_blocksize(cluster_key_ctx));
+
+ /* wrap DEK with KEK */
+ if (!kmgr_wrap_data_key(cluster_key_ctx, key, bootstrap_keys_wrap[id], &(key_lens[id])))
+ {
+ pg_cipher_ctx_free(cluster_key_ctx);
+ elog(ERROR, "failed to wrap data encryption key");
+ }
+
+ /* remove DEK from memory */
+ explicit_bzero(key, sizeof(CryptoKey));
+ }
+
+ /* Write data encryption keys to the disk */
+ KmgrWriteCryptoKeys(LIVE_KMGR_DIR, bootstrap_keys_wrap, key_lens);
+
+ pg_cipher_ctx_free(cluster_key_ctx);
+ }
+
+ /*
+ * We are either decrypting keys we copied from an old cluster, or
+ * decrypting keys we just wrote above --- either way, we decrypt them
+ * here and store them in a file-scoped variable for use in later
+ * encrypting during bootstrap mode.
+ */
+
+ /* Get the crypto keys from the live directory */
+ kmgr_read_wrapped_data_keys(LIVE_KMGR_DIR, keys_wrap, key_lens);
+
+ if (!kmgr_verify_cluster_key(cluster_key, keys_wrap, key_lens, bootstrap_keys))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("supplied cluster key does not match expected cluster_key")));
+
+ /* bzero DEK on exit */
+ on_proc_exit(bzeroKmgrKeys, 0);
+
+ /* bzero KEK */
+ explicit_bzero(cluster_key_hex, cluster_key_hex_len);
+ explicit_bzero(cluster_key, KMGR_CLUSTER_KEY_LEN);
+}
+
+/* Report shared-memory space needed by KmgrShmem */
+Size
+KmgrShmemSize(void)
+{
+ if (!FileEncryptionEnabled)
+ return 0;
+
+ return MAXALIGN(sizeof(KmgrShmemData));
+}
+
+/* Allocate and initialize key manager memory */
+void
+KmgrShmemInit(void)
+{
+ bool found;
+
+ if (!FileEncryptionEnabled)
+ return;
+
+ KmgrShmem = (KmgrShmemData *) ShmemInitStruct("File encryption key manager",
+ KmgrShmemSize(), &found);
+
+ /* bzero DEK on exit */
+ on_shmem_exit(bzeroKmgrKeys, 0);
+}
+
+/*
+ * Get cluster key and verify it, then get the data encryption keys.
+ * This function is called by postmaster at startup time.
+ */
+void
+InitializeKmgr(void)
+{
+ unsigned char *keys_wrap[KMGR_NUM_DATA_KEYS];
+ int key_lens[KMGR_NUM_DATA_KEYS];
+ char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN];
+ int cluster_key_hex_len;
+ struct stat buffer;
+ char live_path[MAXPGPATH];
+ unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN];
+
+#ifndef USE_OPENSSL
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ (errmsg("cluster file encryption is not supported because OpenSSL is not supported by this build"),
+ errhint("Compile with --with-openssl to use this feature."))));
+#endif
+
+ /*
+ * There are too many optimizations for wal_level=minimal that don't set
+ * LSNs for permanent tables, so just disallow it.
+ */
+ if (!XLogIsNeeded())
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ (errmsg("cluster file encryption is not supported with a wal_level of \"minimal\""))));
+
+ elog(DEBUG1, "starting up cluster file encryption manager");
+
+ if (stat(KMGR_DIR, &buffer) != 0 || !S_ISDIR(buffer.st_mode))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ (errmsg("cluster file encryption directory %s is missing", KMGR_DIR))));
+
+ if (stat(KMGR_DIR_PID, &buffer) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ (errmsg("cluster had a pg_alterckey failure that needs repair or pg_alterckey is running"),
+ errhint("Run pg_alterckey --repair or wait for it to complete."))));
+
+ /*
+ * We want OLD deleted since it allows access to the data encryption keys
+ * using the old cluster key. If NEW exists, it means either the new
+ * directory is partly written, or NEW wasn't renamed to LIVE --- in
+ * either case, it needs to be repaired. See src/bin/pg_alterckey/README
+ * for more details.
+ */
+ if (stat(OLD_KMGR_DIR, &buffer) == 0 || stat(NEW_KMGR_DIR, &buffer) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ (errmsg("cluster had a pg_alterckey failure that needs repair"),
+ errhint("Run pg_alterckey --repair."))));
+
+ /* If OLD, NEW, and LIVE do not exist, there is a serious problem. */
+ if (stat(LIVE_KMGR_DIR, &buffer) != 0 || !S_ISDIR(buffer.st_mode))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ (errmsg("cluster has no data encryption keys"))));
+
+ /* Get the cluster key (KEK) */
+ snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR);
+ cluster_key_hex_len = kmgr_run_cluster_key_command(cluster_key_command,
+ cluster_key_hex,
+ ALLOC_KMGR_CLUSTER_KEY_LEN,
+ live_path, terminal_fd);
+
+ /* decode supplied hex */
+ if (pg_hex_decode(cluster_key_hex, cluster_key_hex_len,
+ (char *) cluster_key, KMGR_CLUSTER_KEY_LEN) !=
+ KMGR_CLUSTER_KEY_LEN)
+ ereport(ERROR,
+ (errmsg("cluster key must be %d hexadecimal characters",
+ KMGR_CLUSTER_KEY_LEN * 2)));
+
+ /* Load wrapped DEKs from their files into an array */
+ kmgr_read_wrapped_data_keys(LIVE_KMGR_DIR, keys_wrap, key_lens);
+
+ /*
+ * Verify cluster key and store the unwrapped data encryption keys in
+ * shared memory.
+ */
+ if (!kmgr_verify_cluster_key(cluster_key, keys_wrap, key_lens, KmgrShmem->intlKeys))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("supplied cluster key does not match expected cluster key")));
+
+ /* Check that retrieved key lengths match controldata length. */
+ for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++)
+ if (KmgrShmem->intlKeys[id].klen * 8 !=
+ encryption_methods[GetFileEncryptionMethod()].bit_length)
+ {
+ char path[MAXPGPATH];
+
+ CryptoKeyFilePath(path, DataDir, id);
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("data encryption key %s of length %d does not match controldata key length %d",
+ path, KmgrShmem->intlKeys[id].klen * 8,
+ encryption_methods[GetFileEncryptionMethod()].bit_length)));
+ }
+
+ /* bzero KEK */
+ explicit_bzero(cluster_key_hex, cluster_key_hex_len);
+ explicit_bzero(cluster_key, KMGR_CLUSTER_KEY_LEN);
+}
+
+static void
+bzeroKmgrKeys(int status, Datum arg)
+{
+ if (IsBootstrapProcessingMode())
+ explicit_bzero(bootstrap_keys, sizeof(bootstrap_keys));
+ else
+ explicit_bzero(KmgrShmem->intlKeys, sizeof(KmgrShmem->intlKeys));
+}
+
+/* return requested DEK */
+const CryptoKey *
+KmgrGetKey(int id)
+{
+ Assert(id < KMGR_NUM_DATA_KEYS);
+
+ return (const CryptoKey *) (IsBootstrapProcessingMode() ?
+ &(bootstrap_keys[id]) : &(KmgrShmem->intlKeys[id]));
+}
+
+/* Generate a DEK inside a CryptoKey */
+static CryptoKey *
+generate_crypto_key(int len)
+{
+ CryptoKey *newkey;
+
+ Assert(len <= KMGR_MAX_KEY_LEN);
+ newkey = (CryptoKey *) palloc0(sizeof(CryptoKey));
+
+ newkey->klen = len;
+
+ if (!pg_strong_random(newkey->key, len))
+ elog(ERROR, "failed to generate new file encryption key");
+
+ return newkey;
+}
+
+/*
+ * Write the DEKs to the disk.
+ */
+static void
+KmgrWriteCryptoKeys(const char *dir, unsigned char **keys, int *key_lens)
+{
+ elog(DEBUG2, "writing data encryption keys wrapped using the cluster key");
+
+ for (int i = 0; i < KMGR_NUM_DATA_KEYS; i++)
+ {
+ int fd;
+ char path[MAXPGPATH];
+
+ CryptoKeyFilePath(path, dir, i);
+
+ if ((fd = BasicOpenFile(path, O_RDWR | O_CREAT | O_EXCL | PG_BINARY)) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+
+ errno = 0;
+ pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_WRITE);
+ if (write(fd, keys[i], key_lens[i]) != key_lens[i])
+ {
+ /* if write didn't set errno, assume problem is no disk space */
+ if (errno == 0)
+ errno = ENOSPC;
+
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write file \"%s\": %m",
+ path)));
+ }
+ pgstat_report_wait_end();
+
+ pgstat_report_wait_start(WAIT_EVENT_KEY_FILE_SYNC);
+ if (pg_fsync(fd) != 0)
+ ereport(PANIC,
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ pgstat_report_wait_end();
+
+ if (close(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"%s\": %m",
+ path)));
+ }
+}
diff --git a/src/include/crypto/kmgr.h b/src/include/crypto/kmgr.h
new file mode 100644
index 0000000000..ec341c70b9
--- /dev/null
+++ b/src/include/crypto/kmgr.h
@@ -0,0 +1,25 @@
+/*-------------------------------------------------------------------------
+ *
+ * kmgr.h
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/crypto/kmgr.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef KMGR_H
+#define KMGR_H
+
+#include "common/kmgr_utils.h"
+
+/* GUC parameters */
+extern char *cluster_key_command;
+
+extern Size KmgrShmemSize(void);
+extern void KmgrShmemInit(void);
+extern void BootStrapKmgr(void);
+extern void InitializeKmgr(void);
+extern const CryptoKey *KmgrGetKey(int id);
+
+#endif /* KMGR_H */
--
2.20.1
cfe-06-backend_over_cfe-05-crypto.diff (text/x-diff, 21.2K) download
From 0a3850da7744313de481b4ee50141738d50e163c Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:40 -0400
Subject: [PATCH] cfe-06-backend_over_cfe-05-crypto squash commit
---
doc/src/sgml/storage.sgml | 5 +++
src/backend/Makefile | 2 +-
src/backend/access/transam/xlog.c | 33 +++++++++++++++++++
src/backend/bootstrap/bootstrap.c | 29 +++++++++++++++-
src/backend/crypto/kmgr.c | 3 ++
src/backend/libpq/be-secure-common.c | 14 ++++++++
src/backend/main/main.c | 3 ++
src/backend/postmaster/postmaster.c | 12 ++++++-
src/backend/replication/basebackup.c | 5 +++
src/backend/storage/ipc/ipci.c | 3 ++
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/tcop/postgres.c | 27 ++++++++++++++-
src/backend/utils/activity/wait_event.c | 9 +++++
src/backend/utils/misc/guc.c | 25 ++++++++++++++
src/backend/utils/misc/pg_controldata.c | 11 +++++--
src/backend/utils/misc/postgresql.conf.sample | 5 +++
src/include/access/xlog.h | 5 +++
src/include/catalog/pg_control.h | 5 ++-
src/include/postmaster/postmaster.h | 2 ++
src/include/utils/guc_tables.h | 1 +
20 files changed, 192 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 3234adb639..cdbc214a51 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -77,6 +77,11 @@ Item
<entry>Subdirectory containing transaction commit timestamp data</entry>
</row>
+<row>
+ <entry><filename>pg_cryptokeys</filename></entry>
+ <entry>Subdirectory containing file encryption keys</entry>
+</row>
+
<row>
<entry><filename>pg_dynshmem</filename></entry>
<entry>Subdirectory containing files used by the dynamic shared memory
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 76eb44f632..7d30845a94 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -21,7 +21,7 @@ SUBDIRS = access bootstrap catalog parser commands executor foreign lib libpq \
main nodes optimizer partitioning port postmaster \
regex replication rewrite \
statistics storage tcop tsearch utils $(top_builddir)/src/timezone \
- jit
+ jit crypto
include $(srcdir)/common.mk
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index c1d4415a43..51165f6593 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -43,6 +43,7 @@
#include "commands/progress.h"
#include "commands/tablespace.h"
#include "common/controldata_utils.h"
+#include "crypto/kmgr.h"
#include "executor/instrument.h"
#include "miscadmin.h"
#include "pg_trace.h"
@@ -50,6 +51,7 @@
#include "port/atomics.h"
#include "port/pg_iovec.h"
#include "postmaster/bgwriter.h"
+#include "postmaster/postmaster.h"
#include "postmaster/startup.h"
#include "postmaster/walwriter.h"
#include "replication/basebackup.h"
@@ -82,6 +84,7 @@
#include "utils/timestamp.h"
extern uint32 bootstrap_data_checksum_version;
+extern int bootstrap_file_encryption_method;
/* Unsupported old recovery command file names (relative to $PGDATA) */
#define RECOVERY_COMMAND_FILE "recovery.conf"
@@ -4650,6 +4653,7 @@ InitControlFile(uint64 sysidentifier)
ControlFile->wal_log_hints = wal_log_hints;
ControlFile->track_commit_timestamp = track_commit_timestamp;
ControlFile->data_checksum_version = bootstrap_data_checksum_version;
+ ControlFile->file_encryption_method = bootstrap_file_encryption_method;
}
static void
@@ -4937,6 +4941,12 @@ ReadControlFile(void)
/* Make the initdb settings visible as GUC variables, too */
SetConfigOption("data_checksums", DataChecksumsEnabled() ? "yes" : "no",
PGC_INTERNAL, PGC_S_OVERRIDE);
+
+ StaticAssertStmt(lengthof(encryption_methods) == NUM_ENCRYPTION_METHODS,
+ "encryption_methods[] must match NUM_ENCRYPTION_METHODS");
+ SetConfigOption("file_encryption_method",
+ encryption_methods[ControlFile->file_encryption_method].name,
+ PGC_INTERNAL, PGC_S_OVERRIDE);
}
/*
@@ -4979,6 +4989,21 @@ DataChecksumsEnabled(void)
return (ControlFile->data_checksum_version > 0);
}
+/*
+ * Is cluster file encryption enabled?
+ */
+int
+GetFileEncryptionMethod(void)
+{
+ if (IsBootstrapProcessingMode())
+ return bootstrap_file_encryption_method;
+ else
+ {
+ Assert(ControlFile != NULL);
+ return ControlFile->file_encryption_method;
+ }
+}
+
/*
* Returns a fake LSN for unlogged relations.
*
@@ -5387,6 +5412,14 @@ BootStrapXLOG(void)
/* some additional ControlFile fields are set in WriteControlFile() */
WriteControlFile();
+ BootStrapKmgr();
+
+ if (terminal_fd != -1)
+ {
+ close(terminal_fd);
+ terminal_fd = -1;
+ }
+
/* Bootstrap the commit log, too */
BootStrapCLOG();
BootStrapCommitTs();
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index 174727b501..ce6419ddd6 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -35,6 +35,7 @@
#include "pg_getopt.h"
#include "pgstat.h"
#include "postmaster/bgwriter.h"
+#include "postmaster/postmaster.h"
#include "postmaster/startup.h"
#include "postmaster/walwriter.h"
#include "replication/walreceiver.h"
@@ -52,6 +53,8 @@
#include "utils/relmapper.h"
uint32 bootstrap_data_checksum_version = 0; /* No checksum */
+int bootstrap_file_encryption_method = DISABLED_ENCRYPTION_METHOD;
+char *bootstrap_old_key_datadir = NULL; /* disabled */
static void CheckerModeMain(void);
@@ -225,7 +228,7 @@ AuxiliaryProcessMain(int argc, char *argv[])
/* If no -x argument, we are a CheckerProcess */
MyAuxProcType = CheckerProcess;
- while ((flag = getopt(argc, argv, "B:c:d:D:Fkr:x:X:-:")) != -1)
+ while ((flag = getopt(argc, argv, "B:c:d:D:FkK:r:R:u:x:X:-:")) != -1)
{
switch (flag)
{
@@ -254,9 +257,33 @@ AuxiliaryProcessMain(int argc, char *argv[])
case 'k':
bootstrap_data_checksum_version = PG_DATA_CHECKSUM_VERSION;
break;
+ case 'K':
+ {
+ int i;
+
+ /* method 0/disabled cannot be specified */
+ for (i = DISABLED_ENCRYPTION_METHOD + 1;
+ i < NUM_ENCRYPTION_METHODS; i++)
+ if (pg_strcasecmp(optarg, encryption_methods[i].name) == 0)
+ {
+ bootstrap_file_encryption_method = i;
+ break;
+ }
+ if (i == NUM_ENCRYPTION_METHODS)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid encryption method specified")));
+ }
+ break;
+ case 'u':
+ bootstrap_old_key_datadir = pstrdup(optarg);
+ break;
case 'r':
strlcpy(OutputFileName, optarg, MAXPGPATH);
break;
+ case 'R':
+ terminal_fd = atoi(optarg);
+ break;
case 'x':
MyAuxProcType = atoi(optarg);
break;
diff --git a/src/backend/crypto/kmgr.c b/src/backend/crypto/kmgr.c
index f4d97879ce..fc98670950 100644
--- a/src/backend/crypto/kmgr.c
+++ b/src/backend/crypto/kmgr.c
@@ -254,6 +254,9 @@ InitializeKmgr(void)
char live_path[MAXPGPATH];
unsigned char cluster_key[KMGR_CLUSTER_KEY_LEN];
+ if (!FileEncryptionEnabled)
+ return;
+
#ifndef USE_OPENSSL
ereport(ERROR,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
diff --git a/src/backend/libpq/be-secure-common.c b/src/backend/libpq/be-secure-common.c
index a212308666..66468f4647 100644
--- a/src/backend/libpq/be-secure-common.c
+++ b/src/backend/libpq/be-secure-common.c
@@ -22,6 +22,7 @@
#include <sys/stat.h>
#include <unistd.h>
+#include "postmaster/postmaster.h"
#include "common/string.h"
#include "libpq/libpq.h"
#include "storage/fd.h"
@@ -61,6 +62,19 @@ run_ssl_passphrase_command(const char *prompt, bool is_server_start, char *buf,
appendStringInfoString(&command, prompt);
p++;
break;
+ case 'R':
+ {
+ char fd_str[20];
+
+ if (terminal_fd == -1)
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("ssl_passphrase_command referenced %%R, but -R not specified")));
+ p++;
+ snprintf(fd_str, sizeof(fd_str), "%d", terminal_fd);
+ appendStringInfoString(&command, fd_str);
+ break;
+ }
case '%':
appendStringInfoChar(&command, '%');
p++;
diff --git a/src/backend/main/main.c b/src/backend/main/main.c
index e58e24a646..96a332ecfe 100644
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
@@ -324,6 +324,7 @@ help(const char *progname)
#endif
printf(_(" -N MAX-CONNECT maximum number of allowed connections\n"));
printf(_(" -p PORT port number to listen on\n"));
+ printf(_(" -R fd prompt for the cluster key\n"));
printf(_(" -s show statistics after each query\n"));
printf(_(" -S WORK-MEM set amount of memory for sorts (in kB)\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -351,7 +352,9 @@ help(const char *progname)
printf(_("\nOptions for bootstrapping mode:\n"));
printf(_(" --boot selects bootstrapping mode (must be first argument)\n"));
printf(_(" DBNAME database name (mandatory argument in bootstrapping mode)\n"));
+ printf(_(" -K LEN enable cluster file encryption with specified key bit length\n"));
printf(_(" -r FILENAME send stdout and stderr to given file\n"));
+ printf(_(" -u DATADIR copy encryption keys from datadir\n"));
printf(_(" -x NUM internal use\n"));
printf(_("\nPlease read the documentation for the complete list of run-time\n"
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 4a3ca78c1b..11dd3eadff 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -231,6 +231,7 @@ static int SendStop = false;
/* still more option variables */
bool EnableSSL = false;
+int terminal_fd = -1;
int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
@@ -694,7 +695,7 @@ PostmasterMain(int argc, char *argv[])
* tcop/postgres.c (the option sets should not conflict) and with the
* common help() function in main/main.c.
*/
- while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:W:-:")) != -1)
+ while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:R:S:sTt:W:-:")) != -1)
{
switch (opt)
{
@@ -785,6 +786,10 @@ PostmasterMain(int argc, char *argv[])
/* only used by single-user backend */
break;
+ case 'R':
+ terminal_fd = atoi(optarg);
+ break;
+
case 'S':
SetConfigOption("work_mem", optarg, PGC_POSTMASTER, PGC_S_ARGV);
break;
@@ -1333,6 +1338,11 @@ PostmasterMain(int argc, char *argv[])
*/
RemovePgTempFiles();
+ InitializeKmgr();
+
+ if (terminal_fd != -1)
+ close(terminal_fd);
+
/*
* Initialize stats collection subsystem (this does NOT start the
* collector process!)
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 56cd473f9f..5864420ec3 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -18,6 +18,7 @@
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
#include "catalog/pg_type.h"
+#include "common/kmgr_utils.h"
#include "common/file_perm.h"
#include "commands/progress.h"
#include "lib/stringinfo.h"
@@ -152,6 +153,10 @@ struct exclude_list_item
*/
static const char *const excludeDirContents[] =
{
+ /* Skip temporary crypto key directories */
+ NEW_KMGR_DIR,
+ OLD_KMGR_DIR,
+
/*
* Skip temporary statistics files. PG_STAT_TMP_DIR must be skipped even
* when stats_temp_directory is set because PGSS_TEXT_FILE is always
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 3e4ec53a97..b90c5fcd0c 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -23,6 +23,7 @@
#include "access/syncscan.h"
#include "access/twophase.h"
#include "commands/async.h"
+#include "crypto/kmgr.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "postmaster/autovacuum.h"
@@ -150,6 +151,7 @@ CreateSharedMemoryAndSemaphores(void)
size = add_size(size, BTreeShmemSize());
size = add_size(size, SyncScanShmemSize());
size = add_size(size, AsyncShmemSize());
+ size = add_size(size, KmgrShmemSize());
#ifdef EXEC_BACKEND
size = add_size(size, ShmemBackendArraySize());
#endif
@@ -269,6 +271,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ KmgrShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index 6c7cf6c295..56ac32407f 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+KmgrFileLock 48
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 330ec5b028..d36ac60da3 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3675,7 +3675,7 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx,
* postmaster/postmaster.c (the option sets should not conflict) and with
* the common help() function in main/main.c.
*/
- while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:v:W:-:")) != -1)
+ while ((flag = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:R:S:sTt:v:W:-:")) != -1)
{
switch (flag)
{
@@ -3767,6 +3767,19 @@ process_postgres_switches(int argc, char *argv[], GucContext ctx,
strlcpy(OutputFileName, optarg, MAXPGPATH);
break;
+ case 'R':
+ terminal_fd = atoi(optarg);
+ if (terminal_fd == -1)
+ {
+ /*
+ * Allow file descriptor closing to be bypassed via -1.
+ * We just duplicate sterr. This is useful for
+ * single-user mode.
+ */
+ terminal_fd = dup(2);
+ }
+ break;
+
case 'S':
SetConfigOption("work_mem", optarg, ctx, gucsource);
break;
@@ -4019,6 +4032,18 @@ PostgresMain(int argc, char *argv[],
/* Early initialization */
BaseInit();
+ /*
+ * Initialize kmgr for cluster encryption. Since kmgr needs to attach to
+ * shared memory the initialization must be called after BaseInit().
+ */
+ if (!IsUnderPostmaster)
+ {
+ InitializeKmgr();
+
+ if (terminal_fd != -1)
+ close(terminal_fd);
+ }
+
/*
* Create a per-backend PGPROC struct in shared memory, except in the
* EXEC_BACKEND case where this was done in SubPostmasterMain. We must do
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index accc1eb577..4ff89f9b3f 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -561,6 +561,15 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_DSM_FILL_ZERO_WRITE:
event_name = "DSMFillZeroWrite";
break;
+ case WAIT_EVENT_KEY_FILE_READ:
+ event_name = "KeyFileRead";
+ break;
+ case WAIT_EVENT_KEY_FILE_WRITE:
+ event_name = "KeyFileWrite";
+ break;
+ case WAIT_EVENT_KEY_FILE_SYNC:
+ event_name = "KeyFileSync";
+ break;
case WAIT_EVENT_LOCK_FILE_ADDTODATADIR_READ:
event_name = "LockFileAddToDataDirRead";
break;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index c9c9da85f3..c429361bf9 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -51,6 +51,7 @@
#include "commands/vacuum.h"
#include "commands/variable.h"
#include "common/string.h"
+#include "crypto/kmgr.h"
#include "funcapi.h"
#include "jit/jit.h"
#include "libpq/auth.h"
@@ -640,6 +641,7 @@ static char *recovery_target_string;
static char *recovery_target_xid_string;
static char *recovery_target_name_string;
static char *recovery_target_lsn_string;
+static char *file_encryption_method_str;
/* should be static, but commands/variable.c needs to get at this */
@@ -770,6 +772,8 @@ const char *const config_group_names[] =
gettext_noop("Statistics / Monitoring"),
/* STATS_COLLECTOR */
gettext_noop("Statistics / Query and Index Statistics Collector"),
+ /* ENCRYPTION */
+ gettext_noop("Encryption"),
/* AUTOVACUUM */
gettext_noop("Autovacuum"),
/* CLIENT_CONN */
@@ -4511,6 +4515,27 @@ static struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"cluster_key_command", PGC_SIGHUP, ENCRYPTION,
+ gettext_noop("Command to obtain cluster key for cluster file encryption."),
+ NULL
+ },
+ &cluster_key_command,
+ "",
+ NULL, NULL, NULL
+ },
+
+ {
+ {"file_encryption_method", PGC_INTERNAL, PRESET_OPTIONS,
+ gettext_noop("Shows the cluster file encryption method."),
+ NULL,
+ GUC_NOT_IN_SAMPLE | GUC_DISALLOW_IN_FILE
+ },
+ &file_encryption_method_str,
+ "",
+ NULL, NULL, NULL
+ },
+
{
{"application_name", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Sets the application name to be reported in statistics and logs."),
diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c
index 209a20a882..63f064462a 100644
--- a/src/backend/utils/misc/pg_controldata.c
+++ b/src/backend/utils/misc/pg_controldata.c
@@ -263,8 +263,8 @@ pg_control_recovery(PG_FUNCTION_ARGS)
Datum
pg_control_init(PG_FUNCTION_ARGS)
{
- Datum values[11];
- bool nulls[11];
+ Datum values[12];
+ bool nulls[12];
TupleDesc tupdesc;
HeapTuple htup;
ControlFileData *ControlFile;
@@ -274,7 +274,7 @@ pg_control_init(PG_FUNCTION_ARGS)
* Construct a tuple descriptor for the result row. This must match this
* function's pg_proc entry!
*/
- tupdesc = CreateTemplateTupleDesc(11);
+ tupdesc = CreateTemplateTupleDesc(12);
TupleDescInitEntry(tupdesc, (AttrNumber) 1, "max_data_alignment",
INT4OID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 2, "database_block_size",
@@ -297,6 +297,8 @@ pg_control_init(PG_FUNCTION_ARGS)
BOOLOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 11, "data_page_checksum_version",
INT4OID, -1, 0);
+ TupleDescInitEntry(tupdesc, (AttrNumber) 12, "file_encryption_method",
+ INT4OID, -1, 0);
tupdesc = BlessTupleDesc(tupdesc);
/* read the control file */
@@ -338,6 +340,9 @@ pg_control_init(PG_FUNCTION_ARGS)
values[10] = Int32GetDatum(ControlFile->data_checksum_version);
nulls[10] = false;
+ values[11] = Int32GetDatum(ControlFile->file_encryption_method);
+ nulls[11] = false;
+
htup = heap_form_tuple(tupdesc, values, nulls);
PG_RETURN_DATUM(HeapTupleGetDatum(htup));
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 39da7cc942..8c26994911 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -638,6 +638,11 @@
# autovacuum, -1 means use
# vacuum_cost_limit
+#------------------------------------------------------------------------------
+# ENCRYPTION
+#------------------------------------------------------------------------------
+
+#cluster_key_command = ''
#------------------------------------------------------------------------------
# CLIENT CONNECTION DEFAULTS
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 77187c12be..ab66a660a5 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -15,6 +15,7 @@
#include "access/xlogdefs.h"
#include "access/xloginsert.h"
#include "access/xlogreader.h"
+#include "common/kmgr_utils.h"
#include "datatype/timestamp.h"
#include "lib/stringinfo.h"
#include "nodes/pg_list.h"
@@ -328,6 +329,10 @@ extern void UpdateControlFile(void);
extern uint64 GetSystemIdentifier(void);
extern char *GetMockAuthenticationNonce(void);
extern bool DataChecksumsEnabled(void);
+
+#define FileEncryptionEnabled (GetFileEncryptionMethod() != DISABLED_ENCRYPTION_METHOD)
+extern int GetFileEncryptionMethod(void);
+
extern XLogRecPtr GetFakeLSNForUnloggedRel(void);
extern Size XLOGShmemSize(void);
extern void XLOGShmemInit(void);
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index e3f48158ce..6216d1faf5 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -22,7 +22,7 @@
/* Version identifier for this pg_control format */
-#define PG_CONTROL_VERSION 1300
+#define PG_CONTROL_VERSION 1400
/* Nonce key length, see below */
#define MOCK_AUTH_NONCE_LEN 32
@@ -226,6 +226,9 @@ typedef struct ControlFileData
*/
char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN];
+ /* File encryption method; index into encryption_methods[]. */
+ int file_encryption_method;
+
/* CRC of all above ... MUST BE LAST! */
pg_crc32c crc;
} ControlFileData;
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index 0efdd7c232..9f88240ed6 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -31,6 +31,8 @@ extern char *bonjour_name;
extern bool restart_after_crash;
extern bool remove_temp_files_after_crash;
+extern int terminal_fd;
+
#ifdef WIN32
extern HANDLE PostmasterHandle;
#else
diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h
index b9b5c1adda..c0fadeb1c1 100644
--- a/src/include/utils/guc_tables.h
+++ b/src/include/utils/guc_tables.h
@@ -89,6 +89,7 @@ enum config_group
STATS,
STATS_MONITORING,
STATS_COLLECTOR,
+ ENCRYPTION,
AUTOVACUUM,
CLIENT_CONN,
CLIENT_CONN_STATEMENT,
--
2.20.1
cfe-07-bin_over_cfe-06-backend.diff (text/x-diff, 33.4K) download
From 15ea2196d7d9457e0cefe626c410fe88a835bbba Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:41 -0400
Subject: [PATCH] cfe-07-bin_over_cfe-06-backend squash commit
---
doc/src/sgml/ref/initdb.sgml | 47 +++++++++
doc/src/sgml/ref/pg_ctl-ref.sgml | 14 +++
doc/src/sgml/ref/pgupgrade.sgml | 20 +++-
doc/src/sgml/ref/postgres-ref.sgml | 13 +++
src/bin/initdb/initdb.c | 126 ++++++++++++++++++++++--
src/bin/pg_controldata/pg_controldata.c | 3 +
src/bin/pg_ctl/pg_ctl.c | 59 +++++++++--
src/bin/pg_resetwal/pg_resetwal.c | 3 +
src/bin/pg_rewind/filemap.c | 8 ++
src/bin/pg_upgrade/check.c | 34 +++++++
src/bin/pg_upgrade/controldata.c | 41 +++++++-
src/bin/pg_upgrade/file.c | 2 +
src/bin/pg_upgrade/option.c | 7 +-
src/bin/pg_upgrade/pg_upgrade.h | 3 +
src/bin/pg_upgrade/server.c | 5 +-
15 files changed, 362 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml
index afd344b4c0..236d213f14 100644
--- a/doc/src/sgml/ref/initdb.sgml
+++ b/doc/src/sgml/ref/initdb.sgml
@@ -163,6 +163,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry id="app-initdb-cluster-key-command" xreflabel="cluster key command">
+ <term><option>-c <replaceable class="parameter">command</replaceable></option></term>
+ <term><option>--cluster-key-command=<replaceable class="parameter">command</replaceable></option></term>
+ <listitem>
+ <para>
+ This option specifies an external command to obtain the cluster-level
+ key for cluster file encryption during server initialization and
+ server start; see <xref linkend="guc-cluster-key-command"/> for details.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
<term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
@@ -224,6 +236,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry id="app-initdb-file-encryption-method"
+ xreflabel="file encryption">
+ <term><option>-K <replaceable class="parameter">encryption_method</replaceable></option></term>
+ <term><option>--file-encryption-method=<replaceable class="parameter">encryption_method</replaceable></option></term>
+ <listitem>
+ <para>
+ Specifies the cluster file encryption method. The
+ value values are <literal>AES128</literal> (the default), <literal>AES192</literal>, and <literal>AES256</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--locale=<replaceable>locale</replaceable></option></term>
<listitem>
@@ -299,6 +323,17 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--authprompt</option></term>
+ <listitem>
+ <para>
+ Allows the <option>--cluster-key-command</option> command
+ to prompt for a passphrase or PIN.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-S</option></term>
<term><option>--sync-only</option></term>
@@ -321,6 +356,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-u <replaceable>datadir</replaceable></option></term>
+ <term><option>--copy-encryption-keys=<replaceable>datadir</replaceable></option></term>
+ <listitem>
+ <para>
+ Copies cluster file encryption keys from another cluster; required
+ when using <application>pg_upgrade</application> on a cluster
+ with cluster file encryption enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-U <replaceable class="parameter">username</replaceable></option></term>
<term><option>--username=<replaceable class="parameter">username</replaceable></option></term>
diff --git a/doc/src/sgml/ref/pg_ctl-ref.sgml b/doc/src/sgml/ref/pg_ctl-ref.sgml
index 3946fa52ea..0662ae051a 100644
--- a/doc/src/sgml/ref/pg_ctl-ref.sgml
+++ b/doc/src/sgml/ref/pg_ctl-ref.sgml
@@ -38,6 +38,7 @@ PostgreSQL documentation
<arg choice="opt"><option>-s</option></arg>
<arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg>
<arg choice="opt"><option>-p</option> <replaceable>path</replaceable></arg>
+ <arg choice="opt"><option>-R</option></arg>
<arg choice="opt"><option>-c</option></arg>
</cmdsynopsis>
@@ -72,6 +73,7 @@ PostgreSQL documentation
<arg choice="opt"><option>-t</option> <replaceable>seconds</replaceable></arg>
<arg choice="opt"><option>-s</option></arg>
<arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg>
+ <arg choice="opt"><option>-R</option></arg>
<arg choice="opt"><option>-c</option></arg>
</cmdsynopsis>
@@ -373,6 +375,18 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--authprompt</option></term>
+ <listitem>
+ <para>
+ Allows <option>ssl_passphrase_command</option> or
+ <option>cluster_key_command</option> to prompt for a passphrase
+ or PIN.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-s</option></term>
<term><option>--silent</option></term>
diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml
index 92e1d09a55..9b49e8da59 100644
--- a/doc/src/sgml/ref/pgupgrade.sgml
+++ b/doc/src/sgml/ref/pgupgrade.sgml
@@ -167,6 +167,15 @@ PostgreSQL documentation
</para></listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--authprompt</option></term>
+ <listitem><para>allows <option>ssl_passphrase_command</option> or
+ <option>cluster_key_command</option> to prompt for a passphrase
+ or PIN.
+ </para></listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-s</option> <replaceable>dir</replaceable></term>
<term><option>--socketdir=</option><replaceable>dir</replaceable></term>
@@ -309,7 +318,9 @@ make prefix=/usr/local/pgsql.new install
Again, use compatible <command>initdb</command>
flags that match the old cluster. Many
prebuilt installers do this step automatically. There is no need to
- start the new cluster.
+ start the new cluster. If upgrading a cluster that uses
+ cluster file encryption, the <command>initdb</command> option
+ <option>--copy-encryption-keys</option> must be specified.
</para>
</step>
@@ -838,6 +849,13 @@ psql --username=postgres --file=script.sql postgres
is down.
</para>
+ <para>
+ If the old cluster uses file encryption, the new cluster must use
+ the same keys, so <command>pg_upgrade</command> copies them to the
+ new cluster. It is necessary to initialize the new cluster with
+ the same <varname>cluster_key_command</varname> and the same
+ file encryption method.
+ </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/postgres-ref.sgml b/doc/src/sgml/ref/postgres-ref.sgml
index 4aaa7abe1a..805da81e07 100644
--- a/doc/src/sgml/ref/postgres-ref.sgml
+++ b/doc/src/sgml/ref/postgres-ref.sgml
@@ -297,6 +297,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-R <replaceable class="parameter">file-descriptor</replaceable></option></term>
+ <listitem>
+ <para>
+ Makes <command>postgres</command> prompt for a passphrase or PIN
+ from the specified open numeric file descriptor. The descriptor
+ is closed after the key is read. The file descriptor number
+ <literal>-1</literal> duplicates standard error for the terminal;
+ this is useful for single-user mode.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-s</option></term>
<listitem>
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index 4500df6fc8..cb8a843f2b 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -65,6 +65,7 @@
#include "catalog/pg_collation_d.h"
#include "common/file_perm.h"
#include "common/file_utils.h"
+#include "common/kmgr_utils.h"
#include "common/logging.h"
#include "common/restricted_token.h"
#include "common/string.h"
@@ -142,11 +143,16 @@ static bool noclean = false;
static bool noinstructions = false;
static bool do_sync = true;
static bool sync_only = false;
+static bool pass_terminal_fd = false;
+static char *term_fd_opt = NULL;
+static int file_encryption_method = DISABLED_ENCRYPTION_METHOD;
static bool show_setting = false;
static bool data_checksums = false;
static char *xlog_dir = NULL;
static char *str_wal_segment_size_mb = NULL;
static int wal_segment_size_mb;
+static char *cluster_key_cmd = NULL;
+static char *old_key_datadir = NULL;
/* internal vars */
@@ -205,6 +211,7 @@ static const char *const subdirs[] = {
"global",
"pg_wal/archive_status",
"pg_commit_ts",
+ "pg_cryptokeys",
"pg_dynshmem",
"pg_notify",
"pg_serial",
@@ -961,12 +968,13 @@ test_config_settings(void)
test_buffs = MIN_BUFS_FOR_CONNS(test_conns);
snprintf(cmd, sizeof(cmd),
- "\"%s\" --boot -x0 %s "
+ "\"%s\" --boot -x0 %s %s "
"-c max_connections=%d "
"-c shared_buffers=%d "
"-c dynamic_shared_memory_type=%s "
"< \"%s\" > \"%s\" 2>&1",
backend_exec, boot_options,
+ term_fd_opt ? term_fd_opt : "",
test_conns, test_buffs,
dynamic_shared_memory_type,
DEVNULL, DEVNULL);
@@ -997,12 +1005,13 @@ test_config_settings(void)
}
snprintf(cmd, sizeof(cmd),
- "\"%s\" --boot -x0 %s "
+ "\"%s\" --boot -x0 %s %s "
"-c max_connections=%d "
"-c shared_buffers=%d "
"-c dynamic_shared_memory_type=%s "
"< \"%s\" > \"%s\" 2>&1",
backend_exec, boot_options,
+ term_fd_opt ? term_fd_opt : "",
n_connections, test_buffs,
dynamic_shared_memory_type,
DEVNULL, DEVNULL);
@@ -1192,6 +1201,13 @@ setup_config(void)
"password_encryption = md5");
}
+ if (cluster_key_cmd)
+ {
+ snprintf(repltok, sizeof(repltok), "cluster_key_command = '%s'",
+ escape_quotes(cluster_key_cmd));
+ conflines = replace_token(conflines, "#cluster_key_command = ''", repltok);
+ }
+
/*
* If group access has been enabled for the cluster then it makes sense to
* ensure that the log files also allow group access. Otherwise a backup
@@ -1402,12 +1418,17 @@ bootstrap_template1(void)
unsetenv("PGCLIENTENCODING");
snprintf(cmd, sizeof(cmd),
- "\"%s\" --boot -x1 -X %u %s %s %s",
+ "\"%s\" --boot -x1 -X %u %s %s %s %s %s %s %s %s",
backend_exec,
wal_segment_size_mb * (1024 * 1024),
data_checksums ? "-k" : "",
+ cluster_key_cmd ? "-K" : "",
+ cluster_key_cmd ? encryption_methods[file_encryption_method].name : "",
+ old_key_datadir ? "-u" : "",
+ old_key_datadir ? old_key_datadir : "",
boot_options,
- debug ? "-d 5" : "");
+ debug ? "-d 5" : "",
+ term_fd_opt ? term_fd_opt : "");
PG_CMD_OPEN;
@@ -2254,21 +2275,29 @@ usage(const char *progname)
" set default locale in the respective category for\n"
" new databases (default taken from environment)\n"));
printf(_(" --no-locale equivalent to --locale=C\n"));
- printf(_(" --pwfile=FILE read password for the new superuser from file\n"));
+ printf(_(" --pwfile=FILE read the new superuser password from file\n"));
printf(_(" -T, --text-search-config=CFG\n"
" default text search configuration\n"));
printf(_(" -U, --username=NAME database superuser name\n"));
- printf(_(" -W, --pwprompt prompt for a password for the new superuser\n"));
+ printf(_(" -W, --pwprompt prompt for the new superuser password\n"));
printf(_(" -X, --waldir=WALDIR location for the write-ahead log directory\n"));
printf(_(" --wal-segsize=SIZE size of WAL segments, in megabytes\n"));
printf(_("\nLess commonly used options:\n"));
+ printf(_(" -c, --cluster-key-command=COMMAND\n"
+ " enable cluster file encryption and set command\n"
+ " to obtain the cluster key\n"));
printf(_(" -d, --debug generate lots of debugging output\n"));
+ printf(_(" -K, --file-encryption-method=METHOD\n"
+ " cluster file encryption method\n"));
printf(_(" -L DIRECTORY where to find the input files\n"));
printf(_(" -n, --no-clean do not clean up after errors\n"));
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
+ printf(_(" -R, --authprompt prompt for a passphrase or PIN\n"));
printf(_(" --no-instructions do not print instructions for next steps\n"));
printf(_(" -s, --show show internal settings\n"));
printf(_(" -S, --sync-only only sync data directory\n"));
+ printf(_(" -u, --copy-encryption-keys=DATADIR\n"
+ " copy the file encryption key from another cluster\n"));
printf(_("\nOther options:\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -?, --help show this help, then exit\n"));
@@ -2836,6 +2865,23 @@ initialize_data_directory(void)
/* Top level PG_VERSION is checked by bootstrapper, so make it first */
write_version_file(NULL);
+ if (pass_terminal_fd)
+ {
+#ifndef WIN32
+ int terminal_fd = open("/dev/tty", O_RDWR, 0);
+#else
+ int terminal_fd = open("CONOUT$", O_RDWR, 0);
+#endif
+
+ if (terminal_fd < 0)
+ {
+ pg_log_error(_("%s: could not open terminal: %s\n"),
+ progname, strerror(errno));
+ exit(1);
+ }
+ term_fd_opt = psprintf("-R %d", terminal_fd);
+ }
+
/* Select suitable configuration settings */
set_null_conf();
test_config_settings();
@@ -2859,8 +2905,9 @@ initialize_data_directory(void)
fflush(stdout);
snprintf(cmd, sizeof(cmd),
- "\"%s\" %s template1 >%s",
+ "\"%s\" %s %s template1 >%s",
backend_exec, backend_options,
+ term_fd_opt ? term_fd_opt : "",
DEVNULL);
PG_CMD_OPEN;
@@ -2936,7 +2983,11 @@ main(int argc, char *argv[])
{"waldir", required_argument, NULL, 'X'},
{"wal-segsize", required_argument, NULL, 12},
{"data-checksums", no_argument, NULL, 'k'},
+ {"authprompt", no_argument, NULL, 'R'},
+ {"file-encryption-method", required_argument, NULL, 'K'},
{"allow-group-access", no_argument, NULL, 'g'},
+ {"cluster-key-command", required_argument, NULL, 'c'},
+ {"copy-encryption-keys", required_argument, NULL, 'u'},
{NULL, 0, NULL, 0}
};
@@ -2978,7 +3029,7 @@ main(int argc, char *argv[])
/* process command-line options */
- while ((c = getopt_long(argc, argv, "A:dD:E:gkL:nNsST:U:WX:", long_options, &option_index)) != -1)
+ while ((c = getopt_long(argc, argv, "A:c:dD:E:gkK:L:nNRsST:u:U:WX:", long_options, &option_index)) != -1)
{
switch (c)
{
@@ -3024,6 +3075,28 @@ main(int argc, char *argv[])
case 'N':
do_sync = false;
break;
+ case 'R':
+ pass_terminal_fd = true;
+ break;
+ case 'K':
+ {
+ int i;
+
+ /* method 0/disabled cannot be specified */
+ for (i = DISABLED_ENCRYPTION_METHOD + 1;
+ i < NUM_ENCRYPTION_METHODS; i++)
+ if (pg_strcasecmp(optarg, encryption_methods[i].name) == 0)
+ {
+ file_encryption_method = i;
+ break;
+ }
+ if (i == NUM_ENCRYPTION_METHODS)
+ {
+ fprintf(stderr, _("invalid cluster encryption method\n"));
+ exit(1);
+ }
+ }
+ break;
case 'S':
sync_only = true;
break;
@@ -3060,6 +3133,12 @@ main(int argc, char *argv[])
case 9:
pwfilename = pg_strdup(optarg);
break;
+ case 'c':
+ cluster_key_cmd = pg_strdup(optarg);
+ break;
+ case 'u':
+ old_key_datadir = pg_strdup(optarg);
+ break;
case 's':
show_setting = true;
break;
@@ -3133,6 +3212,32 @@ main(int argc, char *argv[])
exit(1);
}
+#ifndef USE_OPENSSL
+ if (cluster_key_cmd)
+ {
+ pg_log_error("cluster file encryption is not supported because OpenSSL is not supported by this build");
+ exit(1);
+ }
+#endif
+
+ if (old_key_datadir != NULL && cluster_key_cmd == NULL)
+ {
+ pg_log_error("copying encryption keys requires the cluster key command to be specified");
+ exit(1);
+ }
+
+ if (file_encryption_method != DISABLED_ENCRYPTION_METHOD &&
+ cluster_key_cmd == NULL)
+ {
+ pg_log_error("a file encryption method requires the cluster key command to be specified");
+ exit(1);
+ }
+
+ /* set the default */
+ if (file_encryption_method == DISABLED_ENCRYPTION_METHOD &&
+ cluster_key_cmd != NULL)
+ file_encryption_method = DEFAULT_ENABLED_ENCRYPTION_METHOD;
+
check_authmethod_unspecified(&authmethodlocal);
check_authmethod_unspecified(&authmethodhost);
@@ -3200,6 +3305,11 @@ main(int argc, char *argv[])
else
printf(_("Data page checksums are disabled.\n"));
+ if (cluster_key_cmd)
+ printf(_("Cluster file encryption is enabled.\n"));
+ else
+ printf(_("Cluster file encryption is disabled.\n"));
+
if (pwprompt || pwfilename)
get_su_pwd();
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index f911f98d94..e95eaae5e5 100644
--- a/src/bin/pg_controldata/pg_controldata.c
+++ b/src/bin/pg_controldata/pg_controldata.c
@@ -25,6 +25,7 @@
#include "access/xlog_internal.h"
#include "catalog/pg_control.h"
#include "common/controldata_utils.h"
+#include "common/kmgr_utils.h"
#include "common/logging.h"
#include "getopt_long.h"
#include "pg_getopt.h"
@@ -328,5 +329,7 @@ main(int argc, char *argv[])
ControlFile->data_checksum_version);
printf(_("Mock authentication nonce: %s\n"),
mock_auth_nonce_str);
+ printf(_("File encryption method: %s\n"),
+ encryption_methods[ControlFile->file_encryption_method].name);
return 0;
}
diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c
index 7985da0a94..8fd60cc78c 100644
--- a/src/bin/pg_ctl/pg_ctl.c
+++ b/src/bin/pg_ctl/pg_ctl.c
@@ -79,6 +79,7 @@ typedef enum
static bool do_wait = true;
static int wait_seconds = DEFAULT_WAIT;
static bool wait_seconds_arg = false;
+static bool pass_terminal_fd = false;
static bool silent_mode = false;
static ShutdownMode shutdown_mode = FAST_MODE;
static int sig = SIGINT; /* default */
@@ -442,7 +443,7 @@ free_readfile(char **optlines)
static pgpid_t
start_postmaster(void)
{
- char cmd[MAXPGPATH];
+ char cmd[MAXPGPATH], *term_fd_opt = NULL;
#ifndef WIN32
pgpid_t pm_pid;
@@ -467,6 +468,19 @@ start_postmaster(void)
/* fork succeeded, in child */
+ if (pass_terminal_fd)
+ {
+ int terminal_fd = open("/dev/tty", O_RDWR, 0);
+
+ if (terminal_fd < 0)
+ {
+ write_stderr(_("%s: could not open terminal: %s\n"),
+ progname, strerror(errno));
+ exit(1);
+ }
+ term_fd_opt = psprintf(" -R %d", terminal_fd);
+ }
+
/*
* If possible, detach the postmaster process from the launching process
* group and make it a group leader, so that it doesn't get signaled along
@@ -487,12 +501,14 @@ start_postmaster(void)
* has the same PID as the current child process.
*/
if (log_file != NULL)
- snprintf(cmd, MAXPGPATH, "exec \"%s\" %s%s < \"%s\" >> \"%s\" 2>&1",
+ snprintf(cmd, MAXPGPATH, "exec \"%s\" %s%s%s < \"%s\" >> \"%s\" 2>&1",
exec_path, pgdata_opt, post_opts,
+ term_fd_opt ? term_fd_opt : "",
DEVNULL, log_file);
else
- snprintf(cmd, MAXPGPATH, "exec \"%s\" %s%s < \"%s\" 2>&1",
- exec_path, pgdata_opt, post_opts, DEVNULL);
+ snprintf(cmd, MAXPGPATH, "exec \"%s\" %s%s%s < \"%s\" 2>&1",
+ exec_path, pgdata_opt, post_opts,
+ term_fd_opt ? term_fd_opt : "", DEVNULL);
(void) execl("/bin/sh", "/bin/sh", "-c", cmd, (char *) NULL);
@@ -513,6 +529,21 @@ start_postmaster(void)
PROCESS_INFORMATION pi;
const char *comspec;
+ if (pass_terminal_fd)
+ {
+ /* Hopefully we can read and write CONOUT, see simple_prompt() XXX */
+ /* Do CreateRestrictedProcess() children even inherit open file descriptors? XXX */
+ int terminal_fd = open("CONOUT$", O_RDWR, 0);
+
+ if (terminal_fd < 0)
+ {
+ write_stderr(_("%s: could not open terminal: %s\n"),
+ progname, strerror(errno));
+ exit(1);
+ }
+ term_fd_opt = psprintf(" -R %d", terminal_fd);
+ }
+
/* Find CMD.EXE location using COMSPEC, if it's set */
comspec = getenv("COMSPEC");
if (comspec == NULL)
@@ -553,12 +584,14 @@ start_postmaster(void)
else
close(fd);
- snprintf(cmd, MAXPGPATH, "\"%s\" /C \"\"%s\" %s%s < \"%s\" >> \"%s\" 2>&1\"",
- comspec, exec_path, pgdata_opt, post_opts, DEVNULL, log_file);
+ snprintf(cmd, MAXPGPATH, "\"%s\" /C \"\"%s\" %s%s%s < \"%s\" >> \"%s\" 2>&1\"",
+ comspec, exec_path, pgdata_opt, post_opts,
+ term_fd_opt ? term_fd_opt : "", DEVNULL, log_file);
}
else
- snprintf(cmd, MAXPGPATH, "\"%s\" /C \"\"%s\" %s%s < \"%s\" 2>&1\"",
- comspec, exec_path, pgdata_opt, post_opts, DEVNULL);
+ snprintf(cmd, MAXPGPATH, "\"%s\" /C \"\"%s\" %s%s%s < \"%s\" 2>&1\"",
+ comspec, exec_path, pgdata_opt, post_opts,
+ term_fd_opt ? term_fd_opt : "", DEVNULL);
if (!CreateRestrictedProcess(cmd, &pi, false))
{
@@ -689,7 +722,8 @@ wait_for_postmaster(pgpid_t pm_pid, bool do_checkpoint)
}
else
#endif
- print_msg(".");
+ if (!pass_terminal_fd)
+ print_msg(".");
}
pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
@@ -2065,6 +2099,7 @@ do_help(void)
printf(_(" -o, --options=OPTIONS command line options to pass to postgres\n"
" (PostgreSQL server executable) or initdb\n"));
printf(_(" -p PATH-TO-POSTGRES normally not necessary\n"));
+ printf(_(" -R, --authprompt prompt for a passphrase or PIN\n"));
printf(_("\nOptions for stop or restart:\n"));
printf(_(" -m, --mode=MODE MODE can be \"smart\", \"fast\", or \"immediate\"\n"));
@@ -2259,6 +2294,7 @@ main(int argc, char **argv)
{"mode", required_argument, NULL, 'm'},
{"pgdata", required_argument, NULL, 'D'},
{"options", required_argument, NULL, 'o'},
+ {"authprompt", no_argument, NULL, 'R'},
{"silent", no_argument, NULL, 's'},
{"timeout", required_argument, NULL, 't'},
{"core-files", no_argument, NULL, 'c'},
@@ -2331,7 +2367,7 @@ main(int argc, char **argv)
/* process command-line options */
while (optind < argc)
{
- while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:sS:t:U:wW",
+ while ((c = getopt_long(argc, argv, "cD:e:l:m:N:o:p:P:RsS:t:U:wW",
long_options, &option_index)) != -1)
{
switch (c)
@@ -2383,6 +2419,9 @@ main(int argc, char **argv)
case 'P':
register_password = pg_strdup(optarg);
break;
+ case 'R':
+ pass_terminal_fd = true;
+ break;
case 's':
silent_mode = true;
break;
diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c
index 805dafef07..682c0e26c3 100644
--- a/src/bin/pg_resetwal/pg_resetwal.c
+++ b/src/bin/pg_resetwal/pg_resetwal.c
@@ -52,6 +52,7 @@
#include "common/controldata_utils.h"
#include "common/fe_memutils.h"
#include "common/file_perm.h"
+#include "common/kmgr_utils.h"
#include "common/logging.h"
#include "common/restricted_token.h"
#include "common/string.h"
@@ -804,6 +805,8 @@ PrintControlValues(bool guessed)
(ControlFile.float8ByVal ? _("by value") : _("by reference")));
printf(_("Data page checksum version: %u\n"),
ControlFile.data_checksum_version);
+ printf(_("File encryption method: %s\n"),
+ encryption_methods[ControlFile.file_encryption_method].name);
}
diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c
index 2618b4c957..e52a914361 100644
--- a/src/bin/pg_rewind/filemap.c
+++ b/src/bin/pg_rewind/filemap.c
@@ -28,6 +28,7 @@
#include "catalog/pg_tablespace_d.h"
#include "common/hashfn.h"
+#include "common/kmgr_utils.h"
#include "common/string.h"
#include "datapagemap.h"
#include "filemap.h"
@@ -107,6 +108,13 @@ static const char *excludeDirContents[] =
/* Contents removed on startup, see AsyncShmemInit(). */
"pg_notify",
+ /*
+ * Skip cryptographic keys. It's generally not a good idea to copy the
+ * cryptographic keys from source database because these might use
+ * different cluster key.
+ */
+ KMGR_DIR,
+
/*
* Old contents are loaded for possible debugging but are not required for
* normal operation, see SerialInit().
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index d77183b8d1..b816fb952f 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -10,6 +10,7 @@
#include "postgres_fe.h"
#include "catalog/pg_authid_d.h"
+#include "common/kmgr_utils.h"
#include "fe_utils/string_utils.h"
#include "mb/pg_wchar.h"
#include "pg_upgrade.h"
@@ -27,6 +28,7 @@ static void check_for_tables_with_oids(ClusterInfo *cluster);
static void check_for_reg_data_type_usage(ClusterInfo *cluster);
static void check_for_jsonb_9_4_usage(ClusterInfo *cluster);
static void check_for_pg_role_prefix(ClusterInfo *cluster);
+static void check_for_cluster_key_failure(ClusterInfo *cluster);
static void check_for_new_tablespace_dir(ClusterInfo *new_cluster);
static void check_for_user_defined_encoding_conversions(ClusterInfo *cluster);
static char *get_canonical_locale_name(int category, const char *locale);
@@ -149,6 +151,9 @@ check_and_dump_old_cluster(bool live_check)
if (GET_MAJOR_VERSION(old_cluster.major_version) <= 905)
check_for_pg_role_prefix(&old_cluster);
+ if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1400)
+ check_for_cluster_key_failure(&old_cluster);
+
if (GET_MAJOR_VERSION(old_cluster.major_version) == 904 &&
old_cluster.controldata.cat_ver < JSONB_FORMAT_CHANGE_CAT_VER)
check_for_jsonb_9_4_usage(&old_cluster);
@@ -183,6 +188,9 @@ check_new_cluster(void)
check_loadable_libraries();
+ if (GET_MAJOR_VERSION(old_cluster.major_version) >= 1400)
+ check_for_cluster_key_failure(&new_cluster);
+
switch (user_opts.transfer_mode)
{
case TRANSFER_MODE_CLONE:
@@ -1364,6 +1372,32 @@ check_for_user_defined_encoding_conversions(ClusterInfo *cluster)
}
+/*
+ * check_for_cluster_key_failure()
+ *
+ * Make sure there was no unrepaired pg_alterckey failure
+ */
+static void
+check_for_cluster_key_failure(ClusterInfo *cluster)
+{
+ struct stat buffer;
+
+ if (stat (KMGR_DIR_PID, &buffer) == 0)
+ {
+ if (cluster == &old_cluster)
+ pg_fatal("The source cluster had a pg_alterckey failure that needs repair or\n"
+ "pg_alterckey is running. Run pg_alterckey --repair or wait for it\n"
+ "to complete.\n");
+ else
+ pg_fatal("The target cluster had a pg_alterckey failure that needs repair or\n"
+ "pg_alterckey is running. Run pg_alterckey --repair or wait for it\n"
+ "to complete.\n");
+ }
+
+ check_ok();
+}
+
+
/*
* get_canonical_locale_name
*
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 4f647cdf33..7deb040777 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -9,10 +9,16 @@
#include "postgres_fe.h"
+#include <dirent.h>
#include <ctype.h>
#include "pg_upgrade.h"
+#include "access/xlog_internal.h"
+#include "common/controldata_utils.h"
+#include "common/file_utils.h"
+#include "common/kmgr_utils.h"
+
/*
* get_control_data()
*
@@ -59,6 +65,7 @@ get_control_data(ClusterInfo *cluster, bool live_check)
bool got_date_is_int = false;
bool got_data_checksum_version = false;
bool got_cluster_state = false;
+ int got_file_encryption_method = false;
char *lc_collate = NULL;
char *lc_ctype = NULL;
char *lc_monetary = NULL;
@@ -202,6 +209,13 @@ get_control_data(ClusterInfo *cluster, bool live_check)
got_data_checksum_version = true;
}
+ /* Only in <= 14 */
+ if (GET_MAJOR_VERSION(cluster->major_version) <= 1400)
+ {
+ cluster->controldata.file_encryption_method = DISABLED_ENCRYPTION_METHOD;
+ got_file_encryption_method = true;
+ }
+
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -485,6 +499,18 @@ get_control_data(ClusterInfo *cluster, bool live_check)
cluster->controldata.data_checksum_version = str2uint(p);
got_data_checksum_version = true;
}
+ else if ((p = strstr(bufin, "Cluster file encryption method:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem\n", __LINE__);
+
+ p++; /* remove ':' char */
+ /* used later for contrib check */
+ cluster->controldata.file_encryption_method = atoi(p);
+ got_file_encryption_method = true;
+ }
}
pclose(output);
@@ -553,7 +579,8 @@ get_control_data(ClusterInfo *cluster, bool live_check)
!got_index || !got_toast ||
(!got_large_object &&
cluster->controldata.ctrl_ver >= LARGE_OBJECT_SIZE_PG_CONTROL_VER) ||
- !got_date_is_int || !got_data_checksum_version)
+ !got_date_is_int || !got_data_checksum_version ||
+ !got_file_encryption_method)
{
if (cluster == &old_cluster)
pg_log(PG_REPORT,
@@ -619,6 +646,10 @@ get_control_data(ClusterInfo *cluster, bool live_check)
if (!got_data_checksum_version)
pg_log(PG_REPORT, " data checksum version\n");
+ /* value added in Postgres 14 */
+ if (!got_file_encryption_method)
+ pg_log(PG_REPORT, " file encryption method\n");
+
pg_fatal("Cannot continue without required control information, terminating\n");
}
}
@@ -683,6 +714,14 @@ check_control_data(ControlData *oldctrl,
pg_fatal("old cluster uses data checksums but the new one does not\n");
else if (oldctrl->data_checksum_version != newctrl->data_checksum_version)
pg_fatal("old and new cluster pg_controldata checksum versions do not match\n");
+
+ /*
+ * We cannot upgrade if the old cluster file encryption method
+ * doesn't match the new one.
+ */
+ if (oldctrl->file_encryption_method != newctrl->file_encryption_method)
+ pg_fatal("old and new clusters use different file encryption methods or\n"
+ "one cluster uses encryption and the other does not");
}
diff --git a/src/bin/pg_upgrade/file.c b/src/bin/pg_upgrade/file.c
index 9b0cc16e45..ffa3017ad3 100644
--- a/src/bin/pg_upgrade/file.c
+++ b/src/bin/pg_upgrade/file.c
@@ -11,6 +11,7 @@
#include <sys/stat.h>
#include <fcntl.h>
+#include <dirent.h>
#ifdef HAVE_COPYFILE_H
#include <copyfile.h>
#endif
@@ -21,6 +22,7 @@
#include "access/visibilitymap.h"
#include "common/file_perm.h"
+#include "common/file_utils.h"
#include "pg_upgrade.h"
#include "storage/bufpage.h"
#include "storage/checksum.h"
diff --git a/src/bin/pg_upgrade/option.c b/src/bin/pg_upgrade/option.c
index 9c9b313e0c..db76534e68 100644
--- a/src/bin/pg_upgrade/option.c
+++ b/src/bin/pg_upgrade/option.c
@@ -52,6 +52,7 @@ parseCommandLine(int argc, char *argv[])
{"check", no_argument, NULL, 'c'},
{"link", no_argument, NULL, 'k'},
{"retain", no_argument, NULL, 'r'},
+ {"authprompt", no_argument, NULL, 'R'},
{"jobs", required_argument, NULL, 'j'},
{"socketdir", required_argument, NULL, 's'},
{"verbose", no_argument, NULL, 'v'},
@@ -102,7 +103,7 @@ parseCommandLine(int argc, char *argv[])
if (os_user_effective_id == 0)
pg_fatal("%s: cannot be run as root\n", os_info.progname);
- while ((option = getopt_long(argc, argv, "d:D:b:B:cj:ko:O:p:P:rs:U:v",
+ while ((option = getopt_long(argc, argv, "d:D:b:B:cj:ko:O:p:P:rRs:U:v",
long_options, &optindex)) != -1)
{
switch (option)
@@ -180,6 +181,10 @@ parseCommandLine(int argc, char *argv[])
log_opts.retain = true;
break;
+ case 'R':
+ user_opts.pass_terminal_fd = true;
+ break;
+
case 's':
user_opts.socketdir = pg_strdup(optarg);
break;
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 919a7849fd..3fc8e299ad 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -11,6 +11,7 @@
#include <sys/time.h>
#include "libpq-fe.h"
+#include "common/kmgr_utils.h"
/* Use port in the private/dynamic port number range */
#define DEF_PGUPORT 50432
@@ -219,6 +220,7 @@ typedef struct
bool date_is_int;
bool float8_pass_by_value;
bool data_checksum_version;
+ int file_encryption_method;
} ControlData;
/*
@@ -293,6 +295,7 @@ typedef struct
int jobs; /* number of processes/threads to use */
char *socketdir; /* directory to use for Unix sockets */
bool ind_coll_unknown; /* mark unknown index collation versions */
+ bool pass_terminal_fd; /* pass -R to pg_ctl? */
} UserOpts;
typedef struct
diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c
index 7fed0ae108..f77ad24db2 100644
--- a/src/bin/pg_upgrade/server.c
+++ b/src/bin/pg_upgrade/server.c
@@ -245,8 +245,9 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error)
* vacuumdb --freeze actually freezes the tuples.
*/
snprintf(cmd, sizeof(cmd),
- "\"%s/pg_ctl\" -w -l \"%s\" -D \"%s\" -o \"-p %d%s%s %s%s\" start",
- cluster->bindir, SERVER_LOG_FILE, cluster->pgconfig, cluster->port,
+ "\"%s/pg_ctl\" -w%s -l \"%s\" -D \"%s\" -o \"-p %d%s%s %s%s\" start",
+ cluster->bindir, user_opts.pass_terminal_fd ? " -R" : "",
+ SERVER_LOG_FILE, cluster->pgconfig, cluster->port,
(cluster->controldata.cat_ver >=
BINARY_UPGRADE_SERVER_FLAG_CAT_VER) ? " -b" :
" -c autovacuum=off -c autovacuum_freeze_max_age=2000000000",
--
2.20.1
cfe-08-pg_alterckey_over_cfe-07-bin.diff (text/x-diff, 30.5K) download
From a91b429ec9b26999fbfac6536e879ba419201f1d Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:43 -0400
Subject: [PATCH] cfe-08-pg_alterckey_over_cfe-07-bin squash commit
---
doc/src/sgml/ref/allfiles.sgml | 1 +
doc/src/sgml/ref/pg_alterckey.sgml (new) | 210 +++++++
doc/src/sgml/reference.sgml | 1 +
src/bin/Makefile | 1 +
src/bin/pg_alterckey/.gitignore (new) | 1 +
src/bin/pg_alterckey/Makefile (new) | 38 ++
src/bin/pg_alterckey/README (new) | 24 +
src/bin/pg_alterckey/pg_alterckey.c (new) | 724 ++++++++++++++++++++++
8 files changed, 1000 insertions(+)
diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml
index d67270ccc3..59fe707895 100644
--- a/doc/src/sgml/ref/allfiles.sgml
+++ b/doc/src/sgml/ref/allfiles.sgml
@@ -196,6 +196,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY dropuser SYSTEM "dropuser.sgml">
<!ENTITY ecpgRef SYSTEM "ecpg-ref.sgml">
<!ENTITY initdb SYSTEM "initdb.sgml">
+<!ENTITY pgalterckey SYSTEM "pg_alterckey.sgml">
<!ENTITY pgamcheck SYSTEM "pg_amcheck.sgml">
<!ENTITY pgarchivecleanup SYSTEM "pgarchivecleanup.sgml">
<!ENTITY pgBasebackup SYSTEM "pg_basebackup.sgml">
diff --git a/doc/src/sgml/ref/pg_alterckey.sgml b/doc/src/sgml/ref/pg_alterckey.sgml
new file mode 100644
index 0000000000..867c439b8e
--- /dev/null
+++ b/doc/src/sgml/ref/pg_alterckey.sgml
@@ -0,0 +1,210 @@
+<!--
+doc/src/sgml/ref/pg_alterckey.sgml
+PostgreSQL documentation
+-->
+
+<refentry id="app-pg_alterckey">
+ <indexterm zone="app-pg_alterckey">
+ <primary>pg_alterckey</primary>
+ </indexterm>
+
+ <refmeta>
+ <refentrytitle><application>pg_alterckey</application></refentrytitle>
+ <manvolnum>1</manvolnum>
+ <refmiscinfo>Application</refmiscinfo>
+ </refmeta>
+
+ <refnamediv>
+ <refname>pg_alterckey</refname>
+ <refpurpose>alter the <productname>PostgreSQL</productname> cluster key</refpurpose>
+ </refnamediv>
+
+ <refsynopsisdiv>
+ <cmdsynopsis>
+ <command>pg_alterckey</command>
+
+ <group choice="plain">
+ <arg choice="plain"><option>-R</option></arg>
+ <arg choice="plain"><option>--authprompt</option></arg>
+ </group>
+
+ <arg choice="plain">
+ <replaceable class="parameter">old_cluster_key_command</replaceable>
+ <replaceable class="parameter">new_cluster_key_command</replaceable>
+ </arg>
+
+ <arg choice="opt">
+ <group choice="plain">
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><option>--pgdata</option></arg>
+ </group>
+ <replaceable class="parameter">datadir</replaceable>
+ </arg>
+
+ </cmdsynopsis>
+
+ <cmdsynopsis>
+ <command>pg_alterckey</command>
+
+ <group choice="opt">
+ <arg choice="plain"><option>-R</option></arg>
+ <arg choice="plain"><option>--authprompt</option></arg>
+ </group>
+
+ <group choice="plain">
+ <arg choice="plain"><option>-r</option></arg>
+ <arg choice="plain"><option>--repair</option></arg>
+ </group>
+
+ <arg choice="opt">
+ <group choice="opt">
+ <arg choice="plain"><option>-D</option></arg>
+ <arg choice="plain"><option>--pgdata</option></arg>
+ </group>
+ <replaceable class="parameter">datadir</replaceable>
+ </arg>
+
+ </cmdsynopsis>
+ </refsynopsisdiv>
+
+ <refsect1 id="r1-app-pg_alterckey-1">
+ <title>Description</title>
+ <para>
+ <command>pg_alterckey</command> alters the cluster key used
+ for cluster file encryption. The cluster key is initially set
+ during <xref linkend="app-initdb"/>. The command can be run while the
+ server is running or stopped. The new password must be used the next
+ time the server is started.
+ </para>
+
+ <para>
+ <command>pg_alterckey</command> changes the key encryption key
+ (<acronym>KEK</acronym>) which encrypts the data encryption keys;
+ it does not change the data encryption keys. It does this by
+ decrypting each data encryption key using the <replaceable
+ class="parameter">old_cluster_key_command</replaceable>,
+ re-encrypting it using the <replaceable
+ class="parameter">new_cluster_key_command</replaceable>, and then
+ writes the result back to the cluster directory.
+ </para>
+
+ <para>
+ See the <xref linkend="app-initdb"/> documentation for how to define
+ the old and new passphrase commands. You can use different executables
+ for these commands, or you can use the same executable with different
+ arguments to specify retrieval of the old or new key.
+ </para>
+
+ <para>
+ <command>pg_alterckey</command> manages data encryption keys,
+ which are critical to allowing Postgres to access its decrypted
+ data. For this reason, it is very careful to preserve these
+ keys in most possible failure conditions, e.g., operating system
+ failure during cluster encryption key rotation.
+ </para>
+
+ <para>
+ When started, <command>pg_alterckey</command> repairs any files that
+ remain from previous failures before altering the cluster encryption
+ key. During this repair phase, <command>pg_alterckey</command> will
+ either roll back the cluster key or roll forward the changes that
+ were previously requested. The server will not start if repair is
+ needed, though a running server is unaffected by an unrepaired cluster
+ key configuration. Therefore, if <command>pg_alterckey</command>
+ fails for any reason, it is recommended you run the command with
+ <option>--repair</option> to simply roll back or forward any previous
+ changes. This will report if it rolled the cluster key back or forward,
+ and then run the command again to change the cluster key if needed.
+ </para>
+
+ <para>
+ You can specify the data directory on the command line, or use
+ the environment variable <envar>PGDATA</envar>.
+ </para>
+ </refsect1>
+
+ <refsect1>
+ <title>Options</title>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--authprompt</option></term>
+ <listitem>
+ <para>
+ Allows the <option>old_cluster_key_command</option> and
+ <option>new_cluster_key_command</option> commands
+ to prompt for a passphrase or PIN.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Other options:
+
+ <variablelist>
+ <varlistentry>
+ <term><option>-V</option></term>
+ <term><option>--version</option></term>
+ <listitem>
+ <para>
+ Print the <application>pg_alterckey</application> version and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-?</option></term>
+ <term><option>--help</option></term>
+ <listitem>
+ <para>
+ Show help about <application>pg_alterckey</application> command line
+ arguments, and exit.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </refsect1>
+
+ <refsect1>
+ <title>Environment</title>
+
+ <variablelist>
+ <varlistentry>
+ <term><envar>PGDATA</envar></term>
+
+ <listitem>
+ <para>
+ Default data directory location
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><envar>PG_COLOR</envar></term>
+ <listitem>
+ <para>
+ Specifies whether to use color in diagnostic messages. Possible values
+ are <literal>always</literal>, <literal>auto</literal> and
+ <literal>never</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </refsect1>
+
+ <refsect1>
+ <title>See Also</title>
+
+ <simplelist type="inline">
+ <member><xref linkend="app-initdb"/></member>
+ </simplelist>
+ </refsect1>
+
+</refentry>
diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml
index da421ff24e..dff7a42645 100644
--- a/doc/src/sgml/reference.sgml
+++ b/doc/src/sgml/reference.sgml
@@ -240,6 +240,7 @@
</para>
</partintro>
+ &pgalterckey;
&clusterdb;
&createdb;
&createuser;
diff --git a/src/bin/Makefile b/src/bin/Makefile
index 2fe0ae6652..6db542c1bf 100644
--- a/src/bin/Makefile
+++ b/src/bin/Makefile
@@ -17,6 +17,7 @@ SUBDIRS = \
initdb \
pg_amcheck \
pg_archivecleanup \
+ pg_alterckey \
pg_basebackup \
pg_checksums \
pg_config \
diff --git a/src/bin/pg_alterckey/.gitignore b/src/bin/pg_alterckey/.gitignore
new file mode 100644
index 0000000000..4c4f39f2cc
--- /dev/null
+++ b/src/bin/pg_alterckey/.gitignore
@@ -0,0 +1 @@
+/pg_alterckey
diff --git a/src/bin/pg_alterckey/Makefile b/src/bin/pg_alterckey/Makefile
new file mode 100644
index 0000000000..2133a4ba06
--- /dev/null
+++ b/src/bin/pg_alterckey/Makefile
@@ -0,0 +1,38 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/bin/pg_alterckey
+#
+# Copyright (c) 1998-2021, PostgreSQL Global Development Group
+#
+# src/bin/pg_alterckey/Makefile
+#
+#-------------------------------------------------------------------------
+
+PGFILEDESC = "pg_alterckey - alter the cluster key"
+PGAPPICON=win32
+
+subdir = src/bin/pg_alterckey
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = \
+ $(WIN32RES) \
+ pg_alterckey.o
+
+all: pg_alterckey
+
+pg_alterckey: $(OBJS) | submake-libpgport
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+ $(INSTALL_PROGRAM) pg_alterckey$(X) '$(DESTDIR)$(bindir)/pg_alterckey$(X)'
+
+installdirs:
+ $(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+ rm -f '$(DESTDIR)$(bindir)/pg_alterckey$(X)'
+
+clean distclean maintainer-clean:
+ rm -f pg_alterckey$(X) $(OBJS)
+ rm -rf tmp_check
diff --git a/src/bin/pg_alterckey/README b/src/bin/pg_alterckey/README
new file mode 100644
index 0000000000..6db3739271
--- /dev/null
+++ b/src/bin/pg_alterckey/README
@@ -0,0 +1,24 @@
+pg_alterckey
+============
+
+This directory contains the code to generate the pg_alterckey binary.
+
+Architecture
+------------
+
+pg_alterckey allows altering of the cluster encryption key (key
+encryption key or KEK) which is stored outside of the file system; see
+src/backend/crypto/README for more details. This must be done in a
+crash-safe manner since the keys are critical to reading an encrypted
+cluster. The active data encryption keys (DEK) are encrypted/wrapped by
+the KEK and stored in PGDATA/pg_cryptokeys/live as separate files,
+currently files 0 and 1.
+
+This process can be interrupted at anytime; the new execution of
+pg_alterckey will repair any previously interrupted execution of
+pg_alterckey.
+
+pg_alterckey should never be run concurrently. A lock file prevents
+almost all concurrent execution. pg_alterckey can be run if the
+database server is running or stopped, so it can't use database locking
+that is only available when the server is running.
diff --git a/src/bin/pg_alterckey/pg_alterckey.c b/src/bin/pg_alterckey/pg_alterckey.c
new file mode 100644
index 0000000000..7e3d4ad915
--- /dev/null
+++ b/src/bin/pg_alterckey/pg_alterckey.c
@@ -0,0 +1,724 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_alterckey.c
+ * A utility to change the cluster key (key encryption key, KEK)
+ * used for cluster file encryption. The KEK wrap data encryption
+ * keys (DEK).
+ *
+ * The theory of operation is fairly simple:
+ * 1. Create lock file
+ * 2. Retrieve current and new cluster key using the supplied
+ * commands.
+ * 3. Revert any failed alter operation.
+ * 4. Create a "new" directory
+ * 5. Unwrap each DEK in "live" using the old KEK
+ * 6. Wrap each DEK using the new KEK and write it to "new"
+ * 7. Rename "live" to "old"
+ * 8. Rename "new" to "live"
+ * 9. Remove "old"
+ * 10. Remove lock file
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_alterckey/pg_alterckey.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+
+#define FRONTEND 1
+
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#include "common/file_perm.h"
+#include "common/file_utils.h"
+#include "common/hex.h"
+#include "common/restricted_token.h"
+#include "crypto/kmgr.h"
+#include "common/logging.h"
+#include "getopt_long.h"
+#include "pg_getopt.h"
+
+typedef enum
+{
+ SUCCESS_EXIT = 0,
+ ERROR_EXIT,
+ RMDIR_EXIT,
+ REPAIR_EXIT
+} exit_action;
+
+#define MAX_WRAPPED_KEY_LENGTH 64
+
+static int lock_fd = -1;
+static bool pass_terminal_fd = false;
+int terminal_fd = -1;
+static bool repair_mode = false;
+static char *old_cluster_key_cmd = NULL,
+ *new_cluster_key_cmd = NULL;
+static char old_cluster_key[KMGR_CLUSTER_KEY_LEN],
+ new_cluster_key[KMGR_CLUSTER_KEY_LEN];
+static CryptoKey data_key;
+unsigned char in_key[MAX_WRAPPED_KEY_LENGTH],
+ out_key[MAX_WRAPPED_KEY_LENGTH];
+int in_klen,
+ out_klen;
+static char top_path[MAXPGPATH],
+ pid_path[MAXPGPATH],
+ live_path[MAXPGPATH],
+ new_path[MAXPGPATH],
+ old_path[MAXPGPATH];
+
+static char *DataDir = NULL;
+static const char *progname;
+
+static void create_lockfile(void);
+static void recover_failure(void);
+static void retrieve_cluster_keys(void);
+static void bzero_keys_and_exit(exit_action action);
+static void reencrypt_data_keys(void);
+static void install_new_keys(void);
+
+static void
+usage(const char *progname)
+{
+ printf(_("%s changes the cluster key of a PostgreSQL database cluster.\n\n"), progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION] old_cluster_key_command new_cluster_key_command [DATADIR]\n"), progname);
+ printf(_(" %s [repair_option] [DATADIR]\n"), progname);
+ printf(_("\nOptions:\n"));
+ printf(_(" -R, --authprompt prompt for a passphrase or PIN\n"));
+ printf(_(" [-D, --pgdata=]DATADIR data directory\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf(_("\nRepair options:\n"));
+ printf(_(" -r, --repair repair previous failure\n"));
+ printf(_("\nIf no data directory (DATADIR) is specified, "
+ "the environment variable PGDATA\nis used.\n\n"));
+ printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+ printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
+}
+
+
+int
+main(int argc, char *argv[])
+{
+ static struct option long_options[] = {
+ {"authprompt", required_argument, NULL, 'R'},
+ {"repair", required_argument, NULL, 'r'},
+ {"pgdata", required_argument, NULL, 'D'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+
+ pg_logging_init(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_alterckey"));
+ progname = get_progname(argv[0]);
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage(progname);
+ exit(0);
+ }
+ if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+ {
+ puts("pg_alterckey (PostgreSQL) " PG_VERSION);
+ exit(0);
+ }
+ }
+
+ /* check for -r/-R */
+ while ((c = getopt_long(argc, argv, "D:rR", long_options, NULL)) != -1)
+ {
+ switch (c)
+ {
+ case 'D':
+ DataDir = optarg;
+ break;
+ case 'r':
+ repair_mode = true;
+ break;
+ case 'R':
+ pass_terminal_fd = true;
+ break;
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+ }
+
+ if (!repair_mode)
+ {
+ /* get cluster key commands */
+ if (optind < argc)
+ old_cluster_key_cmd = argv[optind++];
+ else
+ {
+ pg_log_error("missing old_cluster_key_command");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
+ if (optind < argc)
+ new_cluster_key_cmd = argv[optind++];
+ else
+ {
+ pg_log_error("missing new_cluster_key_command");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+ }
+
+ if (DataDir == NULL)
+ {
+ if (optind < argc)
+ DataDir = argv[optind++];
+ else
+ DataDir = getenv("PGDATA");
+
+ /* If no DataDir was specified, and none could be found, error out */
+ if (DataDir == NULL)
+ {
+ pg_log_error("no data directory specified");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+ }
+
+ /* Complain if any arguments remain */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
+ /*
+ * Disallow running as root because we create directories in PGDATA
+ */
+#ifndef WIN32
+ if (geteuid() == 0)
+ {
+ pg_log_error("%s: cannot be run as root\n"
+ "Please log in (using, e.g., \"su\") as the "
+ "(unprivileged) user that will\n"
+ "own the server process.\n",
+ progname);
+ exit(1);
+ }
+#endif
+
+ get_restricted_token();
+
+ /* Set mask based on PGDATA permissions */
+ if (!GetDataDirectoryCreatePerm(DataDir))
+ {
+ pg_log_error("could not read permissions of directory \"%s\": %m",
+ DataDir);
+ exit(1);
+ }
+
+ umask(pg_mode_mask);
+
+ snprintf(top_path, sizeof(top_path), "%s/%s", DataDir, KMGR_DIR);
+ snprintf(pid_path, sizeof(pid_path), "%s/%s", DataDir, KMGR_DIR_PID);
+ snprintf(live_path, sizeof(live_path), "%s/%s", DataDir, LIVE_KMGR_DIR);
+ snprintf(new_path, sizeof(new_path), "%s/%s", DataDir, NEW_KMGR_DIR);
+ snprintf(old_path, sizeof(old_path), "%s/%s", DataDir, OLD_KMGR_DIR);
+
+ /* Complain if any arguments remain */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(1);
+ }
+
+ if (DataDir == NULL)
+ {
+ pg_log_error("no data directory specified");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(1);
+ }
+
+ create_lockfile();
+
+ recover_failure();
+
+ if (!repair_mode)
+ {
+ retrieve_cluster_keys();
+ reencrypt_data_keys();
+ install_new_keys();
+ }
+
+#ifndef WIN32
+ /* remove file system reference to file */
+ if (unlink(pid_path) < 0)
+ {
+ pg_log_error("could not delete lock file \"%s\": %m", KMGR_DIR_PID);
+ exit(1);
+ }
+#endif
+
+ close(lock_fd);
+
+ bzero_keys_and_exit(SUCCESS_EXIT);
+}
+
+/* Create a lock file; this prevents almost all cases of concurrent access */
+void
+create_lockfile(void)
+{
+ struct stat buffer;
+ char lock_pid_str[20];
+
+ if (stat(top_path, &buffer) != 0 || !S_ISDIR(buffer.st_mode))
+ {
+ pg_log_error("cluster file encryption directory \"%s\" is missing; is it enabled?", KMGR_DIR_PID);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+
+ /* Does a lockfile exist? */
+ if ((lock_fd = open(pid_path, O_RDONLY, 0)) != -1)
+ {
+ int lock_pid;
+ int len;
+
+ /* read the PID */
+ if ((len = read(lock_fd, lock_pid_str, sizeof(lock_pid_str) - 1)) == 0)
+ {
+ pg_log_error("cannot read pid from lock file \"%s\": %m", KMGR_DIR_PID);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+ lock_pid_str[len] = '\0';
+
+ if ((lock_pid = atoi(lock_pid_str)) == 0)
+ {
+ pg_log_error("invalid pid in lock file \"%s\": %m", KMGR_DIR_PID);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+
+ /* Is the PID running? */
+ if (kill(lock_pid, 0) == 0)
+ {
+ pg_log_error("active process %d currently holds a lock on this operation, recorded in \"%s\"",
+ lock_pid, KMGR_DIR_PID);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+
+ close(lock_fd);
+
+ if (repair_mode)
+ printf("old lock file removed\n");
+
+ /* ----------
+ * pid is no longer running, so remove the lock file.
+ * This is not 100% safe from concurrent access, e.g.:
+ *
+ * process 1 exits and leaves stale lock file
+ * process 2 checks stale lock file of process 1
+ * process 3 checks stale lock file of process 1
+ * process 2 remove the lock file of process 1
+ * process 4 creates a lock file
+ * process 3 remove the lock file of process 4
+ * process 5 creates a lock file
+ *
+ * The sleep(2) helps with this since it reduces the likelihood
+ * a process that did an unlock will interfere with another unlock
+ * process. We could ask users to remove the lock, but that seems
+ * even more error-prone, especially since this might happen
+ * on server start. Many PG tools seem to have problems with
+ * concurrent access.
+ * ----------
+ */
+ unlink(pid_path);
+
+ /* Sleep to reduce the likelihood of concurrent unlink */
+ pg_usleep(2000000L); /* 2 seconds */
+ }
+
+ /* Create our own lockfile? */
+#ifndef WIN32
+ lock_fd = open(pid_path, O_RDWR | O_CREAT | O_EXCL, pg_file_create_mode);
+#else
+ /* delete on close */
+ lock_fd = open(pid_path, O_RDWR | O_CREAT | O_EXCL | O_TEMPORARY,
+ pg_file_create_mode);
+#endif
+
+ if (lock_fd == -1)
+ {
+ if (errno == EEXIST)
+ pg_log_error("an active process currently holds a lock on this operation, recorded in \"%s\"",
+ KMGR_DIR_PID);
+ else
+ pg_log_error("unable to create lock file \"%s\": %m", KMGR_DIR_PID);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+
+ snprintf(lock_pid_str, sizeof(lock_pid_str), "%d\n", getpid());
+ if (write(lock_fd, lock_pid_str, strlen(lock_pid_str)) != strlen(lock_pid_str))
+ {
+ pg_log_error("could not write pid to lock file \"%s\": %m", KMGR_DIR_PID);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+}
+
+/*
+ * ----------
+ * recover_failure
+ *
+ * A previous pg_alterckey might have failed, so it might need recovery.
+ * The normal operation is:
+ * 1. reencrypt LIVE_KMGR_DIR -> NEW_KMGR_DIR
+ * 2. rename KMGR_DIR -> OLD_KMGR_DIR
+ * 3. rename NEW_KMGR_DIR -> LIVE_KMGR_DIR
+ * remove OLD_KMGR_DIR
+ *
+ * There are eight possible directory configurations:
+ *
+ * LIVE_KMGR_DIR NEW_KMGR_DIR OLD_KMGR_DIR
+ *
+ * Normal:
+ * 0. normal X
+ * 1. remove new X X
+ * 2. install new X X
+ * 3. remove old X X
+ *
+ * Abnormal:
+ * fatal
+ * restore old X
+ * install new X
+ * remove old and new X X X
+ *
+ * We don't handle the abnormal cases, just report an error.
+ * ----------
+ */
+static void
+recover_failure(void)
+{
+ struct stat buffer;
+ bool is_live,
+ is_new,
+ is_old;
+
+ is_live = !stat(live_path, &buffer);
+ is_new = !stat(new_path, &buffer);
+ is_old = !stat(old_path, &buffer);
+
+ /* normal #0 */
+ if (is_live && !is_new && !is_old)
+ {
+ if (repair_mode)
+ printf("repair unnecessary\n");
+ return;
+ }
+ /* remove new #1 */
+ else if (is_live && is_new && !is_old)
+ {
+ if (!rmtree(new_path, true))
+ {
+ pg_log_error("unable to remove new directory \"%s\": %m", NEW_KMGR_DIR);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+ printf(_("removed files created during previously aborted alter operation\n"));
+ return;
+ }
+ /* install new #2 */
+ else if (!is_live && is_new && is_old)
+ {
+ if (rename(new_path, live_path) != 0)
+ {
+ pg_log_error("unable to rename directory \"%s\" to \"%s\": %m",
+ NEW_KMGR_DIR, LIVE_KMGR_DIR);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+ printf(_("Installed new cluster password supplied in previous alter operation\n"));
+ return;
+ }
+ /* remove old #3 */
+ else if (is_live && !is_new && is_old)
+ {
+ if (!rmtree(old_path, true))
+ {
+ pg_log_error("unable to remove old directory \"%s\": %m", OLD_KMGR_DIR);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+ printf(_("Removed old files invalidated during previous alter operation\n"));
+ return;
+ }
+ else
+ {
+ pg_log_error("cluster file encryption directory \"%s\" is in an abnormal state and cannot be processed",
+ KMGR_DIR);
+ fprintf(stderr, _("Exiting with no changes made.\n"));
+ exit(1);
+ }
+}
+
+/* Retrieve old and new cluster keys */
+void
+retrieve_cluster_keys(void)
+{
+ int cluster_key_len;
+ char cluster_key_hex[ALLOC_KMGR_CLUSTER_KEY_LEN];
+
+ /*
+ * If we have been asked to pass an open file descriptor to the user
+ * terminal to the commands, set one up.
+ */
+ if (pass_terminal_fd)
+ {
+#ifndef WIN32
+ terminal_fd = open("/dev/tty", O_RDWR, 0);
+#else
+ terminal_fd = open("CONOUT$", O_RDWR, 0);
+#endif
+ if (terminal_fd < 0)
+ {
+ pg_log_error(_("%s: could not open terminal: %s\n"),
+ progname, strerror(errno));
+ exit(1);
+ }
+ }
+
+ /* Get old key encryption key from the cluster key command */
+ cluster_key_len = kmgr_run_cluster_key_command(old_cluster_key_cmd,
+ (char *) cluster_key_hex,
+ ALLOC_KMGR_CLUSTER_KEY_LEN,
+ live_path, terminal_fd);
+ if (pg_hex_decode(cluster_key_hex, cluster_key_len,
+ (char *) old_cluster_key, KMGR_CLUSTER_KEY_LEN) !=
+ KMGR_CLUSTER_KEY_LEN)
+ {
+ pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN);
+ bzero_keys_and_exit(ERROR_EXIT);
+ }
+
+ /*
+ * Create new key directory here in case the new cluster key command needs
+ * it to exist.
+ */
+ if (mkdir(new_path, pg_dir_create_mode) != 0)
+ {
+ pg_log_error("unable to create new cluster key directory \"%s\": %m", NEW_KMGR_DIR);
+ bzero_keys_and_exit(ERROR_EXIT);
+ }
+
+ /* Get new key */
+ cluster_key_len = kmgr_run_cluster_key_command(new_cluster_key_cmd,
+ (char *) cluster_key_hex,
+ ALLOC_KMGR_CLUSTER_KEY_LEN,
+ new_path, terminal_fd);
+ if (pg_hex_decode(cluster_key_hex, cluster_key_len,
+ (char *) new_cluster_key, KMGR_CLUSTER_KEY_LEN) !=
+ KMGR_CLUSTER_KEY_LEN)
+ {
+ pg_log_error("cluster key must be at %d hex bytes", KMGR_CLUSTER_KEY_LEN);
+ bzero_keys_and_exit(ERROR_EXIT);
+ }
+
+ if (pass_terminal_fd)
+ close(terminal_fd);
+
+ /* output newline */
+ puts("");
+
+ if (memcmp(old_cluster_key, new_cluster_key, KMGR_CLUSTER_KEY_LEN) == 0)
+ {
+ pg_log_error("cluster keys are identical, exiting\n");
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+}
+
+/* Decrypt old keys encrypted with old pass phrase and reencrypt with new one */
+void
+reencrypt_data_keys(void)
+{
+ PgCipherCtx *old_ctx,
+ *new_ctx;
+
+ old_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP,
+ (unsigned char *) old_cluster_key,
+ KMGR_CLUSTER_KEY_LEN, false);
+ if (!old_ctx)
+ pg_log_error("could not initialize encryption context");
+
+ new_ctx = pg_cipher_ctx_create(PG_CIPHER_AES_KWP,
+ (unsigned char *) new_cluster_key,
+ KMGR_CLUSTER_KEY_LEN, true);
+ if (!new_ctx)
+ pg_log_error("could not initialize encryption context");
+
+ for (int id = 0; id < KMGR_NUM_DATA_KEYS; id++)
+ {
+ char src_path[MAXPGPATH],
+ dst_path[MAXPGPATH];
+ int src_fd,
+ dst_fd;
+ int len;
+ struct stat st;
+
+ CryptoKeyFilePath(src_path, live_path, id);
+ CryptoKeyFilePath(dst_path, new_path, id);
+
+ if ((src_fd = open(src_path, O_RDONLY | PG_BINARY, 0)) < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", src_path);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ if ((dst_fd = open(dst_path, O_RDWR | O_CREAT | O_TRUNC | PG_BINARY,
+ pg_file_create_mode)) < 0)
+ {
+ pg_log_error("could not open file \"%s\": %m", dst_path);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ if (fstat(src_fd, &st))
+ {
+ pg_log_error("could not stat file \"%s\": %m", src_path);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ in_klen = st.st_size;
+
+ if (in_klen > MAX_WRAPPED_KEY_LENGTH)
+ {
+ pg_log_error("invalid wrapped key length (%d) for file \"%s\"", in_klen, src_path);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ /* Read the source key */
+ len = read(src_fd, in_key, in_klen);
+ if (len != in_klen)
+ {
+ if (len < 0)
+ pg_log_error("could read file \"%s\": %m", src_path);
+ else
+ pg_log_error("could read file \"%s\": read %d of %u",
+ src_path, len, in_klen);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ /* decrypt with old key */
+ if (!kmgr_unwrap_data_key(old_ctx, in_key, in_klen, &data_key))
+ {
+ pg_log_error("incorrect old key specified");
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ if (KMGR_MAX_KEY_LEN_BYTES + pg_cipher_blocksize(new_ctx) > MAX_WRAPPED_KEY_LENGTH)
+ {
+ pg_log_error("invalid max wrapped key length");
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ /* encrypt with new key */
+ if (!kmgr_wrap_data_key(new_ctx, &data_key, out_key, &out_klen))
+ {
+ pg_log_error("could not encrypt new key");
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ /* Write to the dest key */
+ len = write(dst_fd, out_key, out_klen);
+ if (len != out_klen)
+ {
+ pg_log_error("could not write fie \"%s\"", dst_path);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+
+ close(src_fd);
+ close(dst_fd);
+ }
+
+ /* The cluster key is correct, free the cipher context */
+ pg_cipher_ctx_free(old_ctx);
+ pg_cipher_ctx_free(new_ctx);
+}
+
+/* Install new keys */
+void
+install_new_keys(void)
+{
+ /*
+ * Issue fsync's so key rotation is less likely to be left in an
+ * inconsistent state in case of a crash during this operation.
+ */
+
+ if (rename(live_path, old_path) != 0)
+ {
+ pg_log_error("unable to rename directory \"%s\" to \"%s\": %m",
+ LIVE_KMGR_DIR, OLD_KMGR_DIR);
+ bzero_keys_and_exit(RMDIR_EXIT);
+ }
+ fsync_dir_recurse(top_path);
+
+ if (rename(new_path, live_path) != 0)
+ {
+ pg_log_error("unable to rename directory \"%s\" to \"%s\": %m",
+ NEW_KMGR_DIR, LIVE_KMGR_DIR);
+ bzero_keys_and_exit(REPAIR_EXIT);
+ }
+ fsync_dir_recurse(top_path);
+
+ if (!rmtree(old_path, true))
+ {
+ pg_log_error("unable to remove old directory \"%s\": %m", OLD_KMGR_DIR);
+ bzero_keys_and_exit(REPAIR_EXIT);
+ }
+ fsync_dir_recurse(top_path);
+}
+
+/* Erase memory and exit */
+void
+bzero_keys_and_exit(exit_action action)
+{
+ explicit_bzero(old_cluster_key, sizeof(old_cluster_key));
+ explicit_bzero(new_cluster_key, sizeof(new_cluster_key));
+
+ explicit_bzero(in_key, sizeof(in_key));
+ explicit_bzero(&data_key, sizeof(data_key));
+ explicit_bzero(out_key, sizeof(out_key));
+
+ if (action == RMDIR_EXIT)
+ {
+ if (!rmtree(new_path, true))
+ pg_log_error("unable to remove new directory \"%s\": %m", NEW_KMGR_DIR);
+ printf("Re-running pg_alterckey to repair might be needed before the next server start\n");
+ exit(1);
+ }
+ else if (action == REPAIR_EXIT)
+ {
+ unlink(pid_path);
+ printf("Re-running pg_alterckey to repair might be needed before the next server start\n");
+ }
+
+ /* return 0 or 1 */
+ exit(action != SUCCESS_EXIT);
+}
--
2.20.1
cfe-09-test_over_cfe-08-pg_alterckey.diff (text/x-diff, 113.4K) download
From 7d171de2a1a247a619f6c6b197b45e17d223cfc6 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:44 -0400
Subject: [PATCH] cfe-09-test_over_cfe-08-pg_alterckey squash commit
---
src/test/Makefile | 4 +
src/test/README | 3 +
src/test/crypto/.gitignore (new) | 4 +
src/test/crypto/KWP_AD_128.txt (new) | 35 ++
src/test/crypto/KWP_AD_256.txt (new) | 35 ++
src/test/crypto/KWP_AE_128.txt (new) | 35 ++
src/test/crypto/KWP_AE_256.txt (new) | 35 ++
src/test/crypto/Makefile (new) | 39 ++
src/test/crypto/README (new) | 33 ++
src/test/crypto/gcmDecrypt128.rsp (new) | 129 ++++++
src/test/crypto/gcmDecrypt256.rsp (new) | 129 ++++++
src/test/crypto/gcmEncryptExtIV128.rsp (new) | 129 ++++++
src/test/crypto/gcmEncryptExtIV256.rsp (new) | 129 ++++++
src/test/crypto/t/001_testcrypto.pl (new) | 137 ++++++
src/test/crypto/t/002_testkwp.pl (new) | 126 +++++
src/test/crypto/t/003_clusterkey.pl (new) | 93 ++++
src/test/crypto/t/004_buffers.pl (new) | 157 +++++++
src/test/crypto/testcrypto.c (new) | 458 +++++++++++++++++++
18 files changed, 1710 insertions(+)
diff --git a/src/test/Makefile b/src/test/Makefile
index f7859c2fd5..590846ecd2 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -15,6 +15,10 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = perl regress isolation modules authentication recovery subscription \
locale
+ifeq ($(with_openssl),yes)
+SUBDIRS += crypto
+endif
+
# Test suites that are not safe by default but can be run if selected
# by the user via the whitespace-separated list in variable
# PG_TEST_EXTRA:
diff --git a/src/test/README b/src/test/README
index afdc767651..0955a764e3 100644
--- a/src/test/README
+++ b/src/test/README
@@ -11,6 +11,9 @@ which tests get run automatically.
authentication/
Tests for authentication (but see also below)
+crypto/
+ Tests for cluster file encryption
+
examples/
Demonstration programs for libpq that double as regression tests via
"make check"
diff --git a/src/test/crypto/.gitignore b/src/test/crypto/.gitignore
new file mode 100644
index 0000000000..7b92218ad2
--- /dev/null
+++ b/src/test/crypto/.gitignore
@@ -0,0 +1,4 @@
+# Generated by test suite
+/tmp_check/
+/log/
+/testcrypto
diff --git a/src/test/crypto/KWP_AD_128.txt b/src/test/crypto/KWP_AD_128.txt
new file mode 100644
index 0000000000..d77aeab719
--- /dev/null
+++ b/src/test/crypto/KWP_AD_128.txt
@@ -0,0 +1,35 @@
+# CAVS 21.4
+# 'NIST SP 800-38F KWP-AD with AES-128 cipher function' information for test-files
+# Seed = 0f0c6b6602d959d22ff2305478ea477a85d196d9695abf5b445010b45ce046c501bc04e6ca4faf46137adf6eac452d97314cd91137e33fc30cc6df316117fed3
+# Generated on Fri Apr 6 14:46:45 2018
+[PLAINTEXT LENGTH = 4096]
+
+COUNT = 0
+K = 1dd51f0d3a0a784174ba81b2c9f89005
+C = e1bde6d2df3b8e48ca127f97b56b5dc2672b3736cc3157c7b80a0316ef1efbdbbce19fea23da831836ccd2e002b2c1dfad206b5cec358446b8434d7f4c39e65b0e0b50897642ffc34bfb3cb3e233aa9c1058ff0d4fd48e98bc8cc3d214c06d514dd97db2278093a308f91f4ae92626d85771fb1447b36a3467fff02ac7e81ddbd0fdbcd02d1acd4f053c989ef3dcc2c01e23bc2f6090f3e8c0ba5f0082341200b1c37b99daa9cb6fec78bce3429aec5badb9fd28fdbdbdc5d53570675a9e39535b4594095658ef950ecd79a162223b60d2eb91765e022dc6e1bbdd86f1bcc280ed9df350da08a801fa16a1bf2701947acfb08f19fdfcaa1d76f466a5de2458a78fb82f6af3e1be68f405a4289f25896f4c9830005c9e895c86e67eceab0ad544856071b8d9585835b5e85a07ab01515f7ab54f98dffb4ca49a15068eefc6a01f7f52fd1adbe3631c59f6f43f79d2b4f2a691e2b30bb1d43a848dc3ee39c7f2e50f0c9deb7ab51e33bf40903ac255bb1510fd61676a6c13c3c776b8aacc6cefb95e24973ebb11192e2692dd0c6a085b58f86e11cc28ee2194988c123e3666da7339c0a4ac6afbacc83f1f100fbb39efff7cc605c9213828224a17c476395aeb9bb0a3150fb8889a8c2a494c8c526203f261642bfa69a94b86de9e6d3d932fe20fffe4bd76d502c0d437a3e1d0d8727b7a8dc0e361967109e93566326b6c517663731c4c9bdd0295d8
+P = 1a4eed4bf5b8d2e2a58f1f1277f164cc32cdadaed848f76fe634034082ff9aa1711870bf3936d01a2aa48de30de5143b9148cf56f4490f9d480dda0b672e8e17a012cd26cec3c68837bd5b2f9beb13e0110f21c6c36343e09e027f39557d1596d4ca406e3e7aa113e9bb8623106bae25f0ea23d46bc29970ba2596f83fe4f73a6f978a4d949fa7c271570a2ae5d2b50792d5ab5c43d455f359fb83c35ca3da37cd73cd66b6adce94d78ecdeabf667daa47ea70799af299e1d898ccf3fca6c42c6fff8cf2ec992f596fed4a0cdb502a00f9b5689302931d15cba691e2f8079a0411332438b714ace5234b91e4aebee8f8dda0e1968c2016fed350430a65d8d206c9436f40b79ce03083b8dc207d6960be1ce97007ed22a388ebb7b3d8f7d2b7d9f8f49731fbcb21e21db0cdd15674c795d5af2b2cd727f83e634e8c47157ed0c6873a5c9419e683f16f4a7827b444967812f9d1adb9201b89a0e66bbcf0591465f5d7036a21cdda0e10099feb819dfc37fdd3105120044dab716882d3971f312e3f4459006fd5a1eab08ff63edf6718f47ddaa37f7f40c9c372995f3aec97bc45e287b64fc8cf5559ab04a4d4d3ed482f5d61d3abd99cc87ee406da3ab9c9cd22ba3b8d191b26754aa94a2412f39e332d77fe72210adb0cbb5c96adebdbde036f1f1aaafad74a7ac2594f81efa734054e2e16dc931d49b970b81756862705fcd4
+
+COUNT = 1
+K = b3fa008b5947ce58dfbd354dd01f2d43
+C = 55cd8e45138f477ce0a84f07bd28a93d7d628bb4860207a2f6dc4256bd79843e32c856a4fa831d1603699d49e6c36291b60aa80635900cc6c78cf0a2ddc457beb41782de0de03f08a064df90b41f2e98ce61185d735380403fe56b68f8343a801a14afb8a7ba79684dc2a585110da83e9a836cae1fd9e1a220dd6dc922b4f02b15ca88d43ab61e1da24a9b3cb99c4e5024ce5667f4841ca2a305b1f4c1ae9fb63d1d4dcb83870755a1a646b16c088e612d82ba2bf0e7e2fa0e8035c3baeb595f1ac9bb49b01f6f71392e217c049c0e9bd794b9aa2383cf59ee0a90f965610c65ecd629a17cba2bdf2458e3a8e1a9d219cb66eb9ec8e5226b34f95003064952523920a0b4e94ec8ecd1bdca8a65fe46ed25fd4d076e46fa62a8cde6eabc593045d17cef996ebbeca4b537f65c4f683a10baeb4c42b9867bbb49ca7ea1c5437bc114948c542cffced9bb1ebe3c946eb24ff55be89be004596ba648b264167217d267b881020b905f508e4f0e1a58eca051d56ff30d91891838c574c3de54e3feafcdf514740ddc94ba92cb85fe86033e67f14d90be7a0222e4bd1624cea8894df66a36a8e848dfe9168d8024b7ba5636afbcf6b945a53e6b2778f229af7dc2e59bebbf8bdbdfde1e21465f6b6344b13afa0e5ceac212b3b88932f21b1ae04268476597c92e64ff7c14b9ef678f10a35b56cd70ba03063f94aed97b0a6cf883d1f07facfa37b6e5b070
+P = a067ab39cede4ac6c6cb7630cba48c52a794ac8ebec037125bcd97d1a3c52a8ed64764899f9035a6944d0605a5d977172a55bbf86cd81aef5d6bafb1ac86bfa65da2b3c39bf5da94a98f7b6dbc5df16a7b38061e0665ad16b20fb6aedc9ce7f6d3497c3c55cea92e6343f21251092ef2ea307b35f999683298098bedaea847d1ccbf8bda18dc477e8d49fee4e357273396ad2245703485b97b5a7d97057bad875a3e76b67ad5adbc6ef3b8ba9a1786aa93149f0f8dd166535acbf93f1b9839754d537da3fae1ab02973427c3f353fe9aa6c5a100bf0e6ccb08dc1fdb0fc363a95c77c5758d440db0a70f0340a4c488de51e1ecb932ce2fcb2c95ea28c9f55695d97ba1765c8f11e523ae3e4e1efceb69000a192c047ab197f4840c664c035064ecc12926fd3bca0527a160b5b5a2bbaf5db11437f2c38a1c7535e87f552b9f04f2fdd309a826e4ec7708217022fb075cdfc6cc23e9301e33068caa69ef746f357b09ccc098443a3a2979a225e70be1e722e8d6fbb57d0dded2456c1d47eeb0af2241f769836026fec8fc51d97c4abbe9710a4aa5b95aaac83bee57e1333fa244ccc971b6260a9be16e31cc2fd283fec1b247a7340d149fe5309acb47c9cdb955b7bcc4df277eaf611e8af281ff0bcd64b4534309282d1b5cb14efa93141869d67ce7e418f06bb4c2feebcb7a1151aea2eb8bc2fc4dcee53de9b2fb1803490caf
+
+COUNT = 2
+K = 4b4c43c9de4fb4a2a7a7adafeabe2dbd
+C = 6e4d08b8124f7d3e23303fac1a842014f95e3d71c438f8f1990307842796dc5e404ad81802e35c183fe000390a12c81ee684c5cf26c1d90e414cfffe6931b0f352936fcf0b31429eb5c7612cc359a15371390e518cf5c6a6bff1bb0348d14e2c39b98c9f30672ed2af1d96296df8b5567db25b9510a2083461810e119735490058ed1b46b7fdfa885041d8749f90a072b43ba49f2f51fbcda0dbf3cf99fca1d8f46330e5f6fe079d6679cfa26214c8831b782aaa023a2e0ea91050d277dab876aa6865f2bb3fc1a4a77db52f6179d5e5325993280948b6b7002b572829641d35ed3d735d8423e5b24673c4570ca25064fc2c2ad4840632536bcfaf2a7a814f3eaed92b4d501bc51c1719a0d8d8f420b66db845682bb41c88038cfedf13417143a3a701b521a9bf0bb639875a728c3b5ce6ca7e7a45bc75285c193902e6b5e7a4c6e720493d3937bf485e587bff894f70fd6165a1d0129cc673a992e0a4f5489d228a066b1df60002ec0521924f8d672cd1452fec927e58e75807b2a390256f920743fa4d0fc8f59f2469a595ef65095ca0c80adfc843e9e69b6d4a3f824af47b2bfbf2a7a6c1b650378f096f6f0bfabc752c8f279d4f45d56d09dce97962c119de3a64d83b93ea55066f24d4238a229ae86e6a7857af1d8aba823370a72fe358046049a84a70213ef31d9e77a722def8e21480e79b71299438070946bd459a7251707446c911e381
+FAIL
+
+COUNT = 3
+K = 96ab719a3d08df2393ebc330e151dab1
+C = d50ae797f6c3418f388a7513d693c6dd665e858767531fbccd3eb1aabe796690ec8fbb757d88b169adf5c136de50ff0f2cfdd8389f812382578aee0b0b61e13c6a2bc500640fe1585f068eee0d1fa3420220e23090e24e3248fe16f4e0c7c0e996a21b4947ddd08fd3ccc1f036651be4f48ee1ffb486cdc05911244480548221d8da1f2bc37dece080e51b2cdd1ddebf37213a4dfa1b252e567243d9cec8c89eb8db544e7c389a2e13f1b91d860df3cbcec3e85c93276c2a9a5fa080efc85e9bad3bfe2d9bb06498dd8b3720456bfabd3c69b345f6954872baa1d43b9f7ceb92ae9ad77b270d0b94c79275a48874dafb136105f5553529687b6aeeaa521790b9376c9f88ace94049235cd52c4387ad210442513dd5e07171519d58b1294fb8ac1f60ac68b8f07b418e1bb0598601ec38b9a9b137dd87d0c8a41089d17ca1c720fd0e7e3b81b85a373753bed0f5e29586f84cb29e1d88c379c965c50f6a803ddfac2e1555beb9c208a3821f53bead8f120f4ef4a1490b730a0b8a2f1869c6b985520d709bdc0e5fce44316b8aa2448a2743761bf77bdbbfdab6a721a8ec79f38f7e7321a80a2cd3a35a912eaac5eace85c4cad3c6685b88be4517cd1c20971b85bd9e8eb6e52869e014831dff7585a163f5a4dbf1d59160104da90a9cfcc8d6a0324942b40fde4319a32442d83ebbf5d7a36e9495be2ffd0e7faec1b66c96f71843750b8a051b7170
+P = 3a3b9e6de537458875e59204ef7565b6dde796e5ab11c83f7a361b8143f0f7a7eadb5b53c6efa6d199f759cad5c029004024eabbaff717bafb95646dc31a8f6063b9f8faaea650dfa8803bfa0c79091f299a55f78611c2e0d015021d6c6d3abf3d85cac306740acc144201516b787421a77c78a566c6eadc88ecdeea4ff861b6db73f7b00f0a8f62faedefa58866fb368424d7267afdf5ff1279916d2f177408d780697e1c45e58a524bb0365858d2b5a42ee2bd9e8904134d04cf071e84db8a31804aa8bebc0b28dd621360385117764178fe74b29da3ac390ac4812fdc7eedf91fce6eaae3d03163435001ce42f55982daeda5cec5deb960b35df231463cbc26267746be628c53b55f4f21ef003816eb7bfc6c710efa03d0994a1b3c8595fc9293a2c101483798034d4ee7e3d5e07bbd897c9de4b8315e53cbd1f81bdecbd59d093c844a0ed1e3e9d238707a7b893ca453745223c67756d9062152b239ceec44c436e0896a59ea9ea8cf79a93b8b759389bb5e73c5f5330e26580d9777817400166d826008be5e8c7184ae2ecf8fb9dba92af3c747c74e1534c05395f9204b5e8481fdcf4dab5ea6224a8e0ee52576d467d930c0899d31a4e288e3eecb8cb7a3be3a66c79ae93033de5d0d422a6d54ab002d1a82f3f60db97834d9fa3782dd64cbec8ddcac2216a393dc263cea2705fd072ec82dfa1ddef9c588c49f17c275
+
+COUNT = 4
+K = d8c221e426109cb5911d7d6f0836f4cd
+C = d853d57eaacac8096346564eccd33281ee864fb290ce91eb717fa153ca00064e033635178c59860a567215b7320fab4a72ccf716501dbc9a44d5b3d501729674987d2413cad79dece055a9b0d47ec980331f4a236b31984f5d62f9d7f58c0f3afb81fe60f266652da65d06874334be065f56096e98536bd1f2120313b0905ffe2f2c3b6de265ab7800c42be810bed18548c08f9193b02a3981a922b32b618fd9a978439ea382bf2890ad1f30d115b2319276289cf4f7a9917b0c064180e79c8644f9ac880a793b4a8ee424dff32cf2b6ca46f52ff8bd8359ed18ea8aac23e63ae337f5baea9e2ff845a5fdc0b79d5767d47c2a1a536d889f553c52696cbe91ccd2ec671a0644689bdb0f4db7e5d58c854eb539b6b4cd9214e361a216d315b1b124b43c76c703c01d3bc3142f760a399ba4887a6e326a58ecaf56fd49ae128a86cda485eedc3da80b75b171e77cade00c903c1f216eefa845dfaef660fc5ecb6791ed53765683f44da6c4ed8a9ad9e995f7d920cdee8463e79b18e8874b0a2f573b1825f8a480b1ed1245c81f4ec097bf0a0504aab9bdaef27b67d98805a7ec687c8cbcbc92ad3ce58651162a1f57f8af427ea0a111dbd6e3c7f240eb6b2360650a72b9c1c4417b1d541dfcc2a8d6ce3e8c160d8d417e4efbdce809bfe30802696bfd52a0f40be4db9be247dfd867179d82390b55180ebc6ceba0a990e3f6d32eef9dfdb946706371
+P = 92814e18dbe6e83714c4a82ba3ac3baf682a8054eb36666c9546db040d40b8613fc560d97b05265fb19ade180efeb55dfce2cc5981ca222e66b547b78a42401710535b1181674fdd426cf2b0b55e5b7f0505f11307120d495cfb197a3de00569b3d39f93c27270df4725243d314a026549692b0e2b4079c60a8053f0f36e83aaa3494307b175fd40643c1bc264eed1c00f8b565f2a3aeeb78bd94970bd9267d21f5a9a1b07df09ed44a3bd4255a139a328235b921833f92904a74ec202b0eac65df1caed05dc84e52b06c1ecf0f7914324ac4d828b7de7189705308959be42401948e3bf4bdd50ce24101c67ae745a73a67d7e366b6b432ce67b05cefd149a17247010f42dae4de1b2ca42a8e71824cd32c5cb2e2055443ec3ff24339c774dc9207744b84e9203fda1f85595f961987d847ed33867f1ddfce0795e3f2d78c5d749a488a4997392b8c9022c810197c93186894faa55cb0b6775b57a7ba2729c617c9430a44098d5081e3c5c4908ddd1a475cf9211408a8ddbe19ee527ddb2596456e1b1481a09b04e091b1c14b7b2e41bb4434a906736e115cb25ada0950ac5d2845b4a9f1e95f4d80f64440e983324c3aa9f3ec8964f9da0d26aa47e86355aa80ad99d0e573fa9932da70bd65cb1a06d8cb77e455fe7cada4561e027ca1608132c2605b6d0489bba6b29f293951883c451f37bd545f6605364ddc75918df097e
+
+COUNT = 5
+K = 704eb91dd5ba3d85279cf47c01eca2a5
+C = 51c71fd7778aa3648f3e31e1db0c73cb1479372f2e35f65f00188f08f794993a2ee2bb7e91cd1a2b86e92b8ccca7277207fb525ab17600173fa28844ae27f093e0e5ae00585cc714dac90cbe9b6332cbe4cb689b2cd141c102c6881f5b71ec477c5f4a91f7bdcb5871aadd478f1a9ccc6e069b7283f4d70b26e8748eda6d443ab13804c543a44fe2fb366f90de35d83fbf6354a9a9ab4a93ff7d61cbc0bfe05d6102c9c393273e7d3a04d61eba771f05cee29e5dacb7abf34ec9159e121841e2c39848f604c8f743313cbdca828bfa4635a81136e7a37f230c0d3c814d35c2eaabdd94183312909ab3a09b87cce0c719408f837bf24bfb2dad87630aabc9eab35bdb9cc536198389aceec68e8779f9e1eee84392189823a68195b75bbb6d33addf580564e696a362928e2ac506b79480600bc2f9eaa3e96f323390d1d92cf3c6d4bd4147ada5634cff2bf2d97b259904a335eaf11ec3fc84dcd8e27f7538e0fac1dbe7cb4533f4fa58913535d957b90678fac58aa96694a8047ac774afe488ab429c6807e709351f8159dcfbf83b865aeeb26722ef64a537ce932b2cfa6d53ed6cc1ca8ab58748c06a753515fffc56e294f51ab257585b610d261c6fe12def38a1b5dceaa4681569124c679b20984ed2967740419b342e9010eabd291de026f6e829e4dba5300cb668191358ab58e178c29a0194a639233f9c28c50a609bc42f8fa6cd17bc58eccd
+FAIL
diff --git a/src/test/crypto/KWP_AD_256.txt b/src/test/crypto/KWP_AD_256.txt
new file mode 100644
index 0000000000..6d28f39705
--- /dev/null
+++ b/src/test/crypto/KWP_AD_256.txt
@@ -0,0 +1,35 @@
+# CAVS 21.4
+# 'NIST SP 800-38F KWP-AD with AES-256 cipher function' information for test-files
+# Seed = bf1ba8f321ce0abadb40026686c9d9e7f0c4a55388f2ea7cefc81aeb0e054d40c94f48093f2739580010d6ad6e6ce734f21e7338100b750ec9c7bb06bf46f7f4
+# Generated on Fri Apr 6 14:47:06 2018
+[PLAINTEXT LENGTH = 4096]
+
+COUNT = 0
+K = 08f5c088acec18e6cf1f03a8f85d772e327e7fb07f8c2939eb554e84c42ab93d
+C = dff30fd43647d4be54cf2dfd6187e2ddffb55267313f980fb09c833a9c2bfa558a95861711f0acb2a5c7e731ba22f24a9c4dfdd9e9b0216e9088f817a175b9835b0e17615687a20f68c067205626494cd04fbabc0b3eea7c0a4cd6236bc8b3e52e721dfc357fb8a3722bfcc4c690d8f63dbb864bb6e3a15805aea7270f8eb748deebaa2d066fcda11c2e67221f9a91d2c29a6c79ffae76aa80a2590b4f9e35f623fbf2f8ceb2a205493077556a186e25e5bd52dcff7bcc6909b37a66c1d1431be1b363bb40da25386eaaf5fcabc7be6422a04434a21d1d3105328e7c56770b9f59b03395e4138f5f06fc7e6b80dab87b08caa7bfffc45a095c15263efd3f06c651ded6f58074efc20620d704997fc84721a0a8e9e5b9f5cd330bbb156b31d9d1b1c260e4a24535f30404dc5b2dd6b35d916a1391b25a7d8790be09d85483ed1522074a2785812005bda10dd55acb245b3bd3d9bb777dd23f9b02538ba1a114ba53386d7ca4d9524b2f8a18e0ffb21580b560540bb2146f08f04974b90eb324547d56222df95f44bc6e5f183bef283e4816fb1b2933f9c7c6726a245a495e304d8318d0008c51b0be8090f8f668fbc3f31e073be4b9e97468f4dd8c798e9d682868df493db8a85738b58cfd005190f365849072577772672c6f82555c65046eb34e86fe61103327a063bacbbe33cea7eaa3d1de45471b7269e1b6b38608626e323447a3d5fe0599a6
+P = 8b68f66a3d2f59d419851b94d9a6f2f0e667f8125e11d463a6bc2cea46b12dcc40ce8018b204972c735fdd6d2d05b628f4905c6690f5ac5b1b51e12f3af2dc3ae9b9dab616f0a2a66a1ac197592fd5b15900547f32f54110b58d51a0340aa80e9eeb7b2e0eb97e80aa22ba918f2fe1c678c730ed5c3d8d24774f17d8ab6e01a06243d36e764df1dbb8af1faadbc55281f0242abd7a162c984fd0b05ab8b0bcaedffb2962024f009a8d7c9e71281c09f52ec0707ee3bbeb1ecb918be6ae3e9c1fabbcd3512af928db3ba6c109ff9e9839a616b2a53f092160a48222b84d53cd52490515ef93e1ebb33897263492ab8ec6fad2e633276ae367f76d7f926309478c0205d4f22506a451795dc98f5410d8f5d3e049cbedf381620861e7b4ae08f2d8a71abc1f230248cb636a2d7b4e7717ab2b7b5f2dc6e5b5a18e8043254208b50fd6f8929eaf974c48551233661ad67321b64d69245d536d9a8ca2a6a10966dddb9d2ce36641c9281c460ae524b077867258f638e6ac872cb5f5c6fb216b1ae60a9d0c5ea0dbcd060f255da26111175af4e9935df59ddade6a2a70cddff8cae6a98e4f3843c2dd59d09053b07b648a46f5de0eb21ebb192828279a386ea3eedf2cdc355d73d51111e8c1d522e059752bc56226a4225bcab713bfaaaec78167d7cfd33e913b26fda93ca7524aa8a8b17977c88ff9bc23ea810b4de59eac18d1523b
+
+COUNT = 1
+K = 94c4d5d70f881e58e10e7246cf812d40e2be258adb2b6c13c6603fc7daf7e85a
+C = 6c07b5ffd1b9be182413ef8eae4a6eac657108a46008a0d898727f2711e6fa0ca60fd1d51fad683b57d4202fa2b0eb88b856e08b07155439bdb03890cbb7e0f228172bf297a4e0917dadaa5e89a287bb9ba6441c852c5b0cff5084e6c425aaf866815b3fc45f5f7fb5d14b270343e6a30f402e11d62e433a0d84f65684b2df78d4e7758bc0bf81783316905cdc3c1150ec47f225c966f7f339b2538970eb3b8a2c13f95df1310d6e3b2a1f8aed19105846557d8f0018fc0f17146bf836b654dec98e9ad639c7e4b2f922b4396e82c690cdecb65f5e0ea282dd6262f34346ff9adbc8b2f361ddd4356f0feadf7c750fc0580c4e12c00ee049d06eed2242b14727ef4d58386dc8df279a7bf8131c3befaea2f059ab757826e5e381d49a2f11b8cbc2b0021af4da7a779e5df0083edeb54348cc36ce96a19a3d7ff5bd2f19d05fef6b200e76399a02a991111832173353bff4ce1859ff534ae13290dd176ba8e1384ed24d9702dbff127e15e5c66618f94680271732d19f64552ed03df76dc9d46c3cfdd53a1b253992fbcbea6db006f16e8dd92406f0090ad9100856c6b71f7767fcb895136416b374285efe1c6506941911a380e2bf74ffd0f67e853f9ac7b5df6666b177a2908fda9add0eb798f8ccc52801535b2bdf9507f3fb3b46915aa889d62ac5909040a1a28856105dfe2e10d5cfbb569c380551fc8bbe7d83dc87ef7a92faa3fff4b1e2c2
+P = 85693a16ae69d751cfa6799b95a6396de2eabe7e4da74d734691d992cba353a39f3b9615c1325db5b0563ce1a846bb0f0534a86130ce6657736b9a9b35b0f8d89dd1b3a295131d2f3f57f94deef9606dad76a377d0b24e632b3680e4d3338f3e4484609e8063e9ec621297f55802d7c347e8085ba6e514884b8fc1ae109409c5c3a5bddf4daf034d300e31eccba07a9380f5325666c4a3aa12d60b30ca272fe03534aabe78ba0452a7e4648ebfd4645675629676be6f122a54b6b810cf9cc0c68b7c61470a537a5a664ec24dbb3eb4f9fa8355cc7ae8fef27a0146df5ccc585d8c106a1eeb64ad4c701fd5a54ef18295b07e9e47f7f7dd2f67d38ed776a5f0b28843cc4bb5d7fdbea9cb0088dee849ae232e4e016d8cf3681971e8a45d6b25451538212b91f30e17580a8107a7a95587a06d22d615f5475a5f616fdf2fab79152f2643054d96ba88f50888eb0f2f1f154c6fff53dd44c3613269751dca4fa86f45d6b1af9ad0159685223889529609e7003c8f3cab491fd6c1a020305da8f94ec833d721d9fac7e575c2a1bc26eb4fb5010c35ffbd39b98d857f12584f4ab7de92aa6d7e7148a0120cc6b3f7ae47a291ba1cf55a28d38d3a30dfc3917d663458cf840385ca81cf70acce45a5cd509f8387d450bbdd6fa51830cf9a7387887c620b86809c55a3eb322ca784a51693f1054759804314ae86048f0d9c99650a5a12
+
+COUNT = 2
+K = d65338fd3771fd58c07b6b689577378939d439628529b92cd5625edd18afac76
+C = 1c429bf25c144a2cc649fbd60de5c26c31a0c352de99b34b86101c551994f082feffe1db8853de59b3e8593785eca100a71c5392f0c71eca9f411cbd87fc77ea1a96376dc13f6ad460a11e9cd5a829875a7b7dcd2ba4eaac08c5bb48ab5d4c338a6f8bc5e760739edcff2db116b5b1802e35f936d473db168edd12532a992bcc418a759cc9fd3f97f561623078af29d7ab489b7ec564ba981c188f11240dd9354c324f8d0cdf1c74252f0fc75e390e837b8be90a670f5803ee53eb75c3ce95b2853b2342e54f86dd9aeb308eb82ceb2bae7b3e0b364d17105eb61b3843f7206bdb6abb818efe0f0d3b1004e370191e8218cca14947aa8070f7c66fd0422b02ab4a1d94fa46197acd24e272c765667353e819588402feb85d7f00243521d0e7a9d9e70753d8b51d374ea9c8355536594bf05a6960ca7176a4b66086b055b099e315a23e042a7e0807316d7a11a657a6dc9043806e248a9af06570f710af65267d436a5fcb001104fe8a7c564afe075d85bc0a2ce3d33d8d93d5ab1e923f51d4ef26cbb6fd4a935a97cb115aed678e75d5d67fbfcd2362cb3d74ed6b9b9fb0cf82569a474a25e5aa39d22fe5cd301045203d9f93cf5c9e9e9451f1bf3566eec75fbd995cf8c640aa68fb04f5419344057fd1c0e655d750a68c523b0fab24cab03d7393ee3a5735039daed52895dfe7937f55d7ae9a8c0256e9d638a8598452f5329353a20c4bd9958c
+P = 58b20979cba48a9dc95a8857f5bce433087ff93470fc62546e86e72dfaaf7b233ffe428802390c1db7cba00b1f23678aace4a16a237b41d26bcd83d471030929a34e8467f85eaef070b9b74a57f13e91b4e95a3c0b8dfea87d026196a10168c152c4ac42718989003b7e688ca43207034b674d3cbab6f57db6513f8883d27f2280c742896a62e7d0f3f20377e98a0688652d270887fdacb86daa086ffda17937e6d20e4a82667f80ac7749a889b0d748e906d653f569b86de2b42b5819ade9c92970d4caeeed8cd5759d56fd38205215bd8401b2a5a000990afe6c9bea8d091171e85ed83f45bb5b9a8d74cae897cc36f1eaf0122693990b1fb57d0025ad6d92c90885accb649368fe237c4cf017787609fb93c9ea5b413847a9fcf2d2ccb6283345a278619abf8dc351682928187bf92551a820939ec73928eb9930c48f7088ed0a367882f4a8b20d754c5f06bc82990da02227923eb8d1cb73c23793ea0d19bed4a9986f0d48d7835733d1ed3396ec3cf15e1854473b05535261251f4f0af8a0743b3298888bec2f7656493d05eb2d9b848e6802845fb9f7835b50d6a0f0e6cfdaf9b1afc6caa6573b3350256e6f23cc4681316705e33eb0a5f664b79be556cb1bbdd0208430cdc95a35f61facbe7ca2a9bd329e4a1fa42aab9bb02f6519a5672346a4cfac1b96a969317480dd995e339af888fc0e43692332d583fec6215d
+
+COUNT = 3
+K = 1726706350c11e6883955f24ea11ab247ce3b2ab54d05e67ad9770b5564483dd
+C = b006f26a67d0e1e2cbeb5c23b6b300adc1526d1f17bbe964fe8237ae244878158e6b04cb488786b5258ac973c3a2eafd7fcf3a7ca6c825155659fbc53d112bc78b3a770cf059fdd5e68f2b4bfa36de3721231102e5041c947fba3d906bff39592ec3901a398da23035f1190e99b58659330cc2e856ee87ad4197dcc7d16e1f062275bced1ed5cd82163ae3e58da7368dc2aadac855385bd4fa0b8baadef608d0a5c27172d12b88c70b136eeccf37f36364361a990dc50815743cab1636e661bff04ca8345520c30b935a060b450526b1d6ac09170e5b0a327b88f42327b85c9a621d2ca745963c2815a2bfcf509d50b6058ed6e67f369b5608d2aa885238b67d1b8e0d83f9464aa473bf109350fcc02e360c2619236cbfbf895b607895530d8d3d2e41450750dad05b1c37ef15db7fb4707597ac252e8e58d4c1ab2713b427643d198164c908b5d8ff36e9700157284009c7b283633d8b27b378bb65eff8aa59b5fe5e6437a1d53a99c106c2c4d033d3d23950e313a10eb31d68524ae9f8e4f56437acf66db3e8f77407a15bbff4b393e5559908993146d93c673d2aeb7d4cb8fc8d0169de7ed6e2bbe6ce9958a0f5d201419e7acb17e47da827ba380d6b3ad3b5a8c2101c5fb501110c727169065f23297947f538ab3ec165d61edc1f6a9e1735e9b7fc06d4d3406cf8f9c6a68b196cf262324a986705fbc802cdd2e6b4ebcf68e6bb9e793ae644
+FAIL
+
+COUNT = 4
+K = 32e57ccfe7563dc0a20c14ee450837a33606c086ce1467fd7ec58467154338ab
+C = 977d9c5f6861a69e13cd854299434e348cd0690b4d04e08e0598b47eea621bcd8a22838dc9c35a72c35fb1a6434718d02fd24cb4b3dd90b0430334a938a218467eeb4c373d446a539810bc3ce1e923b7c20d9f58ea931d4f964c79613bce67b268efc44bdb9bb00a68d60037949aec7a399493defb2a466e33d4831efd63ad1cb89e00b530626d2f0165975ddfc4cc5e0f968d3875de0f674b3a517df26480b02b6236ebb377118268cebb30ff1ddf0e280fe1bff61902a017e8decf60753c642f35faf0565303bfe651ec8f0193cf34d4af010c9925b8871f0f8c934a149d874a3b659f78ad148428aacaeab80b1b25dec8b0f7ce54406287bc802ac2c0cca3db4adcaa8400a8636ea339b62f5e94d5e32fd3d1183b374507a2af620ca1346dccc9f83a4fe855b1c0e91db9e7c532828d0944d9a81b553ebdf35e24119ed8164bd0260627ea011e93bc103f208c76498ddb8bca15fd05324da5473157feedd54592aacaee68852968eb54c69eb1ddf607917c57493ea380de0cc6ae304dc49cab80a31b8b456986dc367c70f144e52dd604c8d5edbce5de5efb30d9470bc883445b34fa4414f44bb94a64362a12b546665721fa6db82f0c947f015978412b2ce136c471c98b1f908315a16c83e9318e64508c7e179a4429195a9b1ccc211a1c1d4e4df15c5ebc7ab90926fcf7da03657159e440e93adea31ee35f72f2399f5fe2f8c560c8826e23
+P = 80de48ff805a88f3b359451bb6df61def9cb3551e64fdd3a3a70a3b6d238a69311a85bc5924e395ce92ef394b1e5dc301233e9a212f7fb86272c42ddf5f4857c38d0dd259dc1d663c0d729e033d9b0f7f01ab1f8f1b7192d40921ee0d4696a3e35663c5ffcff5ed167660bf6b4c00e619512a2e827be33c90eecc539e18acc8c76eb332b28b1cc502af571242342f63d155271da3211352128aa0af70c9ce78dfdf084a13049b7bb6f2bd10dd385b412d60bf1ccc9fae1208f39dc53db471a04d0dcc3703b4f7b95e72ea815b64a1499865a7ccc5b740999e76338e1b251c740d75274150a96def8760a08c5a8a6f58273b079c06ee09f79a976eaacc8a04c365bc61a786b496811121c386d274c413a2fbfae9464db6ea775233193395740fc9a5eca1a3820d33f6f7b38a83ccbacaac16479225e108acdf46ca35e573151963721b73b3e1c9a12effff0c3a622eb9f07bef7ae712c96ee3ba245597fb8d511698d6e819a967e0d1868c0c6055333b7c13a98cf63d6a5d87779a95345ca8b7e9e597ec588e96f8fc2a7f0a0b8f1543d9e362a911dfb1f03132a4e6af71b503c41814d6b684a26b8df00cdc657ae129a1f2a18cf4b78a3981de68296b1268609fe3ecb9928b90df4553be37319fc508096fa54b35e4822328569da60a6c660f30c61f02f4c5ab2527cf36cb7da8d7dade4c714ea3fc2da8f65b4199090e114dd
+
+COUNT = 5
+K = e3982db2032f2b4ce658fc44b76f5964c45cd31bf803708982ae599186fc3765
+C = 975e49a4b9a770957a1bb2be920a4f39b9cfd69ba46983d2473d631c08132b9bf61c44510b8aa8bd48c70a86276aa1149d8fdefad511d15d2e2037d9e920e640cb71a97663d19eb90d0b74d9764d03e17cda87ebec6e35ac2003cb75bf9192920d910188d78e2e664255fdf6c9190319d34adb858162ff0830f37fe1dd44003d3d5a1f9451949e368f46ad1977ce622daadf8483a1f60359992b9b366e8a81ffbe96cee45d3aef2fd0ad8c17cc34927af77a0d6d0c5deef3b4a25c82ec388667a493bb0599ac492b351246cbad6d283bf1820883afa48bd909eb7304b9fc5b7d960344309133aab7a85c49f7de396926f50bc83c95900cd049eac1b387aae7fcba5345496425f9216e1fd15c20da75fbb26da176149b40a701e15a7bacfe899e3ecc534ab8bc5b7bd081fb825b5f40fa57e363d7bce40020e73f638acfa097b89c50cb9edb0bd6d71d429b8003aa5dcb7d61792eb3bcac795954c625a104209b373c28cf02038c3318916edd2b818e6719ec154cfa56afb2f337d333069f915d0d35edd6c278fae23c4440c40be462a1dcab23758e4a7fbe8436493f58e890092ea71cb8bcf1336e9ee16b852ccc488f21682dc9f02bdf6c56fe8ad04d84a3c69d8d06dee3d126c0a75f142d0c90c256139acd4b719573e588b80b4540024a05a35044cf58d89673923a534c3816492e62379797cd6e6a7464da5eaaf11ee7b9c27b9b03d7b53c03
+P = 43d38ca132545b154995cff07082611cc47a6467a980654d2d1f1ccfb3bcd387e9d7ffa281b0e0b00dc8669207e0d8033e9e36613c98978f8644bb7e505fbf491dcefbe19589254c8abf859dd65cb94dfc99e7b9d3d1f0a31f21285963e1f7b45c7490a522ff887786f7940fb6192f5081ce7181944bdca5c5bfcf2589f9173a682b78fcdf971dd9f4e8529033e15cde560984dcf796914206973dcc13f8c9a24b25dd00c11166ec6ecf33c6ad9b487847abd7bd29b4f3b9c8dc93a6a5a31723dc03245884bfadc12b2fddcc82409d7b14660af808d4e8216157bb6ba03a319193ad4dacbd37ac884550962a4de26ae923f8d74f2f694fcd0aa74f2e809da4689aad9f2820684b3b423ec4a7da0ce4a1b599fc21bd2779653283b0ee81d7b0d9fd3f6d1e75bd71af9620630aa87b73f7b12e68ddbdfa02ae86ae06b0b1aee4a997d34f61b466348b92e36f83652763084a215c47dcf689df17e36b64bae3ca1a2cc22c837b5907236833c2c1e5f3ddb74165fb6f0633990122cbe4af8b5920b1bb6961cdb144ea8d7b245d0128ab76f4fc0189ba97385717e89e0f99c962ee8c2b6e55546a18be0ba3dbebf7e4140eed6aa3558c43115b65b6f6e8e8fb4b9cfbe0b6eac006603667b28cefb4dec037f33568a3c94d9e36539e91b3199d728521a9a6b82b96ff1c29dd1d10366d0510f1b9a9494cd104db2390530be3fb6abdb7
diff --git a/src/test/crypto/KWP_AE_128.txt b/src/test/crypto/KWP_AE_128.txt
new file mode 100644
index 0000000000..a682d4bb32
--- /dev/null
+++ b/src/test/crypto/KWP_AE_128.txt
@@ -0,0 +1,35 @@
+# CAVS 21.4
+# 'NIST SP 800-38F KWP-AE with AES-128 cipher function' information for test-files
+# Seed = 7c0f161159cc9e338308ddc59a655450a030832b3d4576c568d63725d57b020073aee9c77d9fec34eab358b83bf5aae4fb52803343bc03d472283edbebbcf75c
+# Generated on Fri Apr 6 14:46:12 2018
+[PLAINTEXT LENGTH = 4096]
+
+COUNT = 0
+K = 0e54956a24c7d4a343f90269fb18a17f
+P = 817ddabdc5d215eee233adff97e92193c6beec52a71340477f70243a794ce954af51e356c9940e4ab198f0e68c543355f65ad179cb2d60dd369eaeb9ed141fb18c9e4054ac7fdc83506896990a4d20833d2d6e9a34938796ee67c9d7d23058544a4a35f2954103ce443a95a7e785602075ca0a73da37899e4568106bb2dbf1f901377d4d3380c70fa5175ebc550481ac6f15986a4407fde5c23ff317e37544c0a25f87117506597db5bb79850c86247b73a5d0090417d63e4c257ea0220c2c04db07a34f0ab7954e1dfa2007a1466795c4d0c2aa09ca3986c028185b43a466526594afc9c891c263a7c608304bc1957c9873f544dc71e6f847c48d32026ed03b2333825452ee7e12a50e1cd7d678319264c65f78001996d37fae7f9861fbd21cb506c2f8a3b0ee53c7debe17111b6e3f78a5c5677857b082c2c4943dfd1edf6337fea98a44fc25928361156ef38d865948b979cf6f4b46bd2119f12f0891cef7fc9d0638fd105fc05f9968d16948d1cb820751e82e44cb68e99d4f072ffd1577da6c0631b5827bec7e1b9ec72d18b74cf5f233e85013c1668ceb5d7a1f5e0f016b0ff726a0a9d41e2cea8e14a2f56492b14606d3fafd8ac141335f39f90d56863735628e8f17be90e100ef0785f3cd57db8b9d89a6b2189dc2ea00c285d2657983f8bd7883c215477e67a55556401f1d8b27d4e0d541c7fb7ace370c2e428884
+C = 876f3e53ba9cf4f6a521ac198bc813d0ede0f862ab6082e3e0a06ad82b4f279582f7c43bb63574608446bc2a05f401a68f74086cf2776b4b3df6b3679c2edfb91c024db54c6831e0752ae6f86c7596462de905ee0be908c1b9d043ecafe2ad1cbddb904e18ebc9b7a107031be3a87059516a3d1257812d9c801b0b9f21539e70c47150c128d87c5e58fa6e4371aedde69c7b5cd16b73ac422676328131f3ac48c602bb6e0741805aad9d23b33b3523b86cf0588cdf9dc6c4d5f9fa43d88ca17976eaf48fb37a41a598266da04144373df5631cc5126341c200a0c8499b29ae96e6e6e6c2bdf8d8903da62bf8ddae970569b695240e77f8ac5b191da5034008b6ef21936858e69bac372bbafd8794f6b03711503c1875528a9348681844edb199a0664d740f0f0b1f866c4248c80fe8b5700a3c4134cdddb17676e0cd37d6d81831a0f4adfba071bb0935502480eccd48b28be5954ea6c7d873b51b8bd2b709c5b6132ed31296510915073c18f7012f0eff6a9aad5340a19fd5e372d35260b718d9e4807b1954c24e6a4fd48e4dbb8f395474e99ab577367d2ab5ccaa18c947331047dc3986e213a878b41089aa221019dad4191a4feefd095f8606c2700a46d71cbb13efb6957df925ec26071c04d04d5a94e138e5fc5d1f059236aad76208077dcc607b1dd2086f9c04e33f955822b457eecd68bd5f24836ecedbac675e6ed93d8a787cb57ad68e
+
+COUNT = 1
+K = 6b8ba9cc9b31068ba175abfcc60c1338
+P = 8af887c58dfbc38ee0423eefcc0e032dcc79dd116638ca65ad75dca2a2459f13934dbe61a62cb26d8bbddbabf9bf52bbe137ef1d3e30eacf0fe456ec808d6798dc29fe54fa1f784aa3c11cf39405009581d3f1d596843813a6685e503fac8535e0c06ecca8561b6a1f22c578eefb691912be2e1667946101ae8c3501e6c66eb17e14f2608c9ce6fbab4a1597ed49ccb3930b1060f98c97d8dc4ce81e35279c4d30d1bf86c9b919a3ce4f0109e77929e58c4c3aeb5de1ec5e0afa38ae896df9121c72c255141f2f5c9a51be5072547cf8a3b067404e62f9615a02479cf8c202e7feb2e258314e0ebe62878a5c4ecd4e9df7dab2e1fa9a7b532c2169acedb7998d5cd8a7118848ce7ee9fb2f68e28c2b279ddc064db70ad73c6dbe10c5e1c56a709c1407f93a727cce1075103a4009ae2f7731b7d71756eee119b828ef4ed61eff164935532a94fa8fe62dc2e22cf20f168ae65f4b6785286c253f365f29453a479dc2824b8bdabd962da3b76ae9c8a720155e158fe389c8cc7fa6ad522c951b5c236bf964b5b1bfb098a39835759b95404b72b17f7dbcda936177ae059269f41ecdac81a49f5bbfd2e801392a043ef06873550a67fcbc039f0b5d30ce490baa979dbbaf9e53d45d7e2dff26b2f7e6628ded694217a39f454b288e7906b79faf4a407a7d207646f93096a157f0d1dca05a7f92e318fc1ff62ce2de7f129b187053
+C = aea19443d7f8ad7d4501c1ecadc6b5e3f1c23c29eca608905f9cabdd46e34a55e1f7ac8308e75c903675982bda99173a2ba57d2ccf2e01a02589f89dfd4b3c7fd229ec91c9d0c46ea5dee3c048cd4611bfeadc9bf26daa1e02cb72e222cf3dab120dd1e8c2dd9bd58bbefa5d14526abd1e8d2170a6ba8283c243ec2fd5ef07030b1ef5f69f9620e4b17a3639341005887b9ffc793533594703e5dcae67bd0ce7a3c98ca65815a4d067f27e6e66d6636cebb789732566a52ac3970e14c37310dc2fcee0e739a16291029fd2b4d534e30445474b26711a8b3e1ee3cc88b09e8b1745b6cc0f067624ecb232db750b01fe5457fdea77b251b10fe95d3eeedb083bdf109c41dba26cc9654f787bf95735ff07070b175cea8b62302e6087b91a0415474605691099f1a9e2b626c4b3bb7aeb8ead9922bc3617cb427c669b88be5f98aea7edb8b0063bec80af4c081f89778d7c7242ddae88e8d3aff1f80e575e1aab4a5d115bc27636fd14d19bc59433f697635ecd870d17e7f5b004dee4001cddc34ab6e377eeb3fb08e9476970765105d93e4558fe3d4fc6fe053aab9c6cf032f1116e70c2d65f7c8cdeb6ad63ac4291f93d467ebbb29ead265c05ac684d20a6bef09b71830f717e08bcb4f9d3773bec928f66eeb64dc451e958e357ebbfef5a342df28707ac4b8e3e8c854e8d691cb92e87c0d57558e44cd754424865c229c9e1abb28e003b6819400b
+
+COUNT = 2
+K = 5f0f2428ad83ea04642d8f90e31f00ad
+P = 05f65580a602fc839c5372778bc495219ab159935acf3caed10958ca0e135cbc44a896aa84b1f6b05d65ac45f5930029a223977ca46c7717809e80d8eb91e7ccbc525e69ca0a85d3e45b2f9bcf0c86980b7f9f9eb20e9b558479aa8c814e6113635a9b8b0e264055da68198303ae91683eb8ed3351dea3b8280a504d163e1cf284834e3641a26868e23fcf460d279d3577115d8c4cf0f54374df26ddfbf0c1df5bf71fe509231177ac531514ed878d3ef44a03685dee906ab394bfb69cb56ff973f19fb466ce550ae7b03008feca7046bae160ad47f28c9004da68e594bcd7f9ca8674a645c50768b5ee0ec2088f66602a429f95492e09dc567c11083eb18da97d85968a86f6ad05054f4a9c4bcd57ece2511bdb38afd78084b0a088112b81b993a4d37674d97e8081795ee055d9c37d2907d859be201e73580ebd439620918e7c4b42243f43a59b475095205c40a1d37c0f87a5df19450ccd5f274f186b0f45f7bc153d30ce3c873d7bba5dea5a8bd348b2f0b4f25461927642a2991c3c4bc3ad34d09c104322bf27e8eaa0ca99c0dd47a3c4c618af02599192e362e75de53db336a68b223567bcddcd7feecde0aba91a1eac0ce6327d3c9c5be36698494b2b67ec0efd36b744d794f521bee251b31dd29852a94e6b95f143a53f93f85ec8c2e86e1026ec949aaf97fc9b8d728c13e1deab628e9624ca966992efae61dcd6e5
+C = efa9d549d0e975789a6c84b99991c98cf20db4de6789be6f9f7ae109e322a6f04a1a1c1fe582e4242a5e2f2b390827d7e52f1bd97a4e745920dda69a25fe93fa5de82c09dce8a2593106667820dfa1f55c8a1152ee02ed0c607ab986e5209859cbb36baf90ca63285c3128fdceb18e5145b9d3efe6bc7b8bac7837637ce5553d634b13fa3888201112248bc31e0785f2dbae1a8eaa7f4ebc0b00d21242f03acd4e4b0f4ad5e9bb1e25301a331a4b5535c1253b9ccc7800cb2e80602ec9ef858865a39e0ba28006da93f9c26c1fb9ee8258dfcd7e79f02ab0d9c2df9ade309fa6c604fcbeb38641b2968ff96349bcc34927cc479c30153125f7ad07068e5a4ffaba903d27488c94bfb694d8277251061a7dcc5b5fbf47ea88065ea1df007cbd4c1bdbc2e3609d53bf99bf2c7de8ef457fd7b85882210fb52c5093415dd2b37084b75f83d2a05a669867143fd95705f241fa794526bc65c5f6422410da4783340497bc54852125e3da579bfabc9ba98ec398487444b3da454f40d8b59d384cdd3454ad77f38d4ce657d10b4708dd4ecb0fcc3238aca57258e082e8cb86d802b9722388284f232811ec35b49ebba151d5b0771feeb6120d8bee8d55fa9c4de45d8f950752b3b9c6526fc4fa8fdd02c645fb053cef8293437111c85b07e98b3b38ccd3de93cd6fd08fdfb8a3a94b619f9acd226be3bd3ff906f14b1dda93cccae783c04ed545db685bf0
+
+COUNT = 3
+K = 9b5c2e4a9e53d3e5f5670af2440439db
+P = c4c726199cc27b81c90ee59f1f44420abad2a4fe0f24647952116ec46f364a3dad1b48ba5bc6c48c056e34764ed4b4a2da04e7e45ff477df8364ee9f116c82dcff7e9ae2db6a7722f2261b6cee1ef7cc66820184988e002886a6b636264e7704383543817e119ba3582d42aa00aa78dcf84d0ac92ff69f09046f89e7b608143699f9fda37b07d3e2949e935bba230e0a42c2d41d285042feaa480bcc1ef8236d615f449107fad97b5be514ae39d586b98ed45cc39784c30a3a256d62336002b1f7f6cedc3e5418d6f444bacd74d7bc89b3ec65c52191b4a2f3b3c7b682f254e6e909c5dcbc11d5def56792f5d2183e5e7119f268b0395e7f4f25b8574bc921da8419af30113bd35b11c21debb979fb698752264595777be7665419052cf3265117bce96b61f27da4072f828bd9315ef25a9ab9eb4f5319907cdbce6104054c8563a997cf477523478e6d23527f91997dd43ebaf629003530de4b8406bdbfae49c0d97920b745c78a665c438ff064d24803382f79fcc2e24265adaf707394b832685def125b69b1787719aaf92e518645ac22dd94d0dcb03f052c87954854d177e2caa6ab5d9cf3dc4ed4ceca90d2fa0b235efd1b7d17ba9758987d127b11204047f22e75158720feec549f29b67a8158e577325e178d6b757770d56a30b85d12ab5fb164a07a5f613b53d075c44cd3c1e3a311515aeed2427f70b3bc018dd796
+C = 09540dbbe97e983a3420ec1ef7d1195f9cd32acc93e9f31d07e68dd9d963f32bb23565a37d97e857e5e40eae27946ef83a4da10ec29ea001fbe892133a4814b04aa3ebabfa295a19429c7e6b263293d07a48f5b2beea2930aabe6268aec00a8fbcf9f4865d4658905413a0f17bb08bbc6466f8d3e1744d9ffe6132f94ce79bec4ffea05785525eb5e071ecf30081281df58f1b5295842967a486ee262a7c366978140bcb8562abf1a417c7b1ffe07b1748978408cfae4beec8290b312f15d3223fa7447f5a89e52ddfdc4f79cf1d9fe4ecad3e674e98136c457e36748da001970cceed0c61d2ea5f9ee5f9660ad561b9fb808be3d4455fe76ae9778a49526c8461f99dc2c271ff9aa389d3e6a546a82d15967a82e8ec9b64cdf25fb73afccd7cc2fde8d07ef4a1345976970997facd5be16566c7761d23f4a31909a5a31bce35e33596a861e426fb56d9ea13e5f6219aceda2450434e04c454ec8569ef9076856cb48f36618e690a710dfe8717526001c3be7eccb671dbb4430289924ae975a30b6fc7079f8c9cbc8a268f5defc03d94d5bae0474987b7ebd4c9e29eec9080e8d85776c570089fcd235372518dc38e481f8a5edc2105bb4191cd81a19e51c60252a18b3058ac1ff9368af59a2caf82a06357764fb3721287bd28204d77dc2eadb0913cabff7cd63c73488e92c818341563476e69a50fbb781f151e02f2df4a9fe05c8419a3393d67
+
+COUNT = 4
+K = f2713273ad5b7add10d2b8d8b61d2f68
+P = 1c189d6d34b57c699dd8c79e30457c6735f3697aa6825e14257906e18413580727d676b6e667cd58e0a47b738abfd91144952940de60b161b0dd261d68af12573820f4f172b32d7c032ed7db3715fc6ceb92ec2d614a2005ea1aed14cc5015de143aaf6dc2db406e0fd31976562250e80eb79140bfc8d14fc79bea4159d66e1013704b474b0b1377220741dc55cecde0eed4e0931e58025e7fb672117a8288587cc91b1a41fb4d1e9e4a7ff59fdb814dbd9521dc148abfe16ab62fa45a3dc361cffb422482f4fd93466320a11d9a5438959947ba51d7761b2a3bb9e0b843ea6a71cba54a96bd14de8360ac8f16ece773d7bfb25df63c1aa67a7bcb587f618f2425bf9aa018eba5aebc2911b9fb6276e48d72dc74836a290fad0f5c545934f1231ed57c6944a464905fb08685e815ecddc8dbe4eeec1355f414fd2ce975c0dbe29bf4369e05c1582cfd9a7ddb7659247a0f6db38ba165f1f46a472760bcd2609ffc8bff49c6f54698063fa81305696f7ce25c95b237c6a4fecb3e48167060f556359609b73e738c5c3021d0e5bc62ed31a2a27f43dc417c0041f8a2402fa7bf76c72e2b290c060efd93ef65d0a893314b03c098a6674d0137f12eba3ebb63b3e65fd0aac89847bf9573d9eff0b1283843bd4edd6d459de6d559e48633d964dccd8b9fb5142596430b2373f9448606f9286157e810dd0f6a72843b0cb28ab452d7
+C = dbeedfe9aede047a07402d23b1b349a3e75490d3c404240c872470aa4f6b5a1723b6a0e455a8961fd25f88a9329b4fdc7d8e6bb2b3d7403bd65418f94b933467d882c1aaca702a47e7bf44916ce5eed0647250d572686c8a8c9f9437f2ceb13b361aef929d428c35cd4cf1d8970d98e75c46b4825c871003eceac2412adb6681f0b8a1d6d6bd7eb41c6e58ebe6909ab061d5e832f7580dce89eccb4ea2546204e895ccb54429fe388b0d9173922e08625d43f0a10743ad76230fa458b0971028b18e0f04f3c1049012a8bbc2f93934e0b51b943efd2fa4a14fdb795a1f21302529653583edcfa3c613ada2bf69a93e5a29a20a6f9d9df7891e3b438be4f9f714806cd57afcabc0d5f5753af804d885ca1ae23bb5fd2593fa361e92e7d12eea0805453056022256410369dab0e2e8d2d4c9ffda1d7df8ba5ee19c5bf62133d1ca9a713fa45157d327bc4e1d8e7a105012036939a5d5ae15a5fe8ef989c8323129b8b4e43402bb4f60f35483685893e9ea0b0dc4a8238148d0220cd9bdd7dc42590cc303f706cd81cc5f6ce761ffdc570c62cf734cfffe0ec7be17c7ae3ab34c65f1a53267b03de4da3a5f921cf01b068c4098c7594a3051bb03940cb6c5bac2aa04f95bd30bd15ccc9edb8e29091047de011ee9591cc4c83244750dceb8c2eedda797965c1243521c19228eb2d799775dd83ceab5aae082cfa743d58544b61b7104f3d10f632994a4
+
+COUNT = 5
+K = 8967c8058742169ab39d2d528130df80
+P = cbe7ac6c232b327f7d9154acf334891398fa0a1ea64a4f15243f488ecc784749fec41fab7a447e2a2cc75a44fb548a46a46e51f4c245f64901f3acbf18f468a28367c51060c34762c8a88780044761579846557f285c97bcb9c966aa08d4cf352c26d8a2f308803393155437b91776ee74941da9db39ef64d9ac9261f2c7fb6349e52111a7c146d5a4ef6d5d00717ac93338fea5ff0c90cdefe0d66b039045330877d8596de4c35987ce70f660aa8777276eed1b128002dffd90ce03224d505950eb2c73f5e958df92ee1cd8dee667bbb8cc71f87e4ce880585f004160c798ff2cc195b09921c73951332bd437840bbadcd10730272d46b3f23963a7c0396275407bb8041a6e5ceed4d1dc2a65a70d4843660cc039ce94a4ce96573811d2b8be00e91dec7486484ccb050dfd6e319001ba8b8fef48b03d17902cb91792dc420c79e823616cc79af4e738dc077519015c734c1be0150921657f3139e59ba4ad8e75e0f7b51806a96b7249709faac3cdcd53bfed8f0326d19000ebc1ae05a43e96c1fa02f2cfadefaf871250aff6eade8ce8d4b8d5a686d553e8a66d96412f2f189fb724e67c8acdb229501414e45c430a9533d211a217eae0030ff10db12a399567e94c900ce839157ebee1f4f876675794cb3642ace55fbdb261f1369807f284f4e6f8a2c8330eed88e18b8ff564fb754b96f8c85d28dc7a0b18805ecc3ddffc
+C = d29eb0bc54c43081030a3e1169f243eac73f1d0fb2b2f983fb25755684df4331a1932b02e237502c1b73a8d744619f521457b72fcd7401973ec48ee915054e5e12577ef1212d0e642c9dc7341cdaffd344dc480eaf9f29019355b07de2f05897cb91b00b4c53714609dea69f18c7c93429b73373311e0c4a45e2f2b3a6bd393b69f90a4125eb635a794563a68e092cc66730d236909cce6585d34f47a10c2968f161ed79c16c18a6c093c1fc455eb502770ee436dd48c091a4cefeaf449c5b27f2fd2f9150a601bccf8a37cda65b25917728ce3f1f12f55a51d9110c16132bde100b6479eb8a1ae670a9ca396f18a2ff19fb5543dbd3b189e51ba5f12516129071c997044df747f4ed70616e784bfb948a63211b224dfdfc5b9116b869fc5313c761d55c759a6e275e9ac53e83a5386e2cfa50f6c1b541cf752da800ff991ff896b79b9e691d67e96fffd166a06f0c300b7d9bb552f5ec7df6eea384751632ed9357df5a5a9fda931568930fedb0536f5491d4067a3951dc72903d1ca2f5f1b713e24cce5347b056e2f8829d34143032f53edb81a765d2ab3e6cbe12536cd85ac3230fea546ee50d991e13c562bea3d694d3990b1783a373a59fe12cfdb74fa0275e367d76bdef698dd33c30afadbbf0fe0dd7ddf288aec7a397e263a5d537df9f606f5c126565c06ed5e0bd6753d3eaf5002b472a1d7f4d7df7d9a8fe8aa2c595ecef8cfb3da3e7
diff --git a/src/test/crypto/KWP_AE_256.txt b/src/test/crypto/KWP_AE_256.txt
new file mode 100644
index 0000000000..66d8523987
--- /dev/null
+++ b/src/test/crypto/KWP_AE_256.txt
@@ -0,0 +1,35 @@
+# CAVS 21.4
+# 'NIST SP 800-38F KWP-AE with AES-256 cipher function' information for test-files
+# Seed = 6eb04695fd5dc2b0038aff221b225c302fc9cde3d6dbf4195d9d4d2756c8303906335e07679fee9a80e5d0d1580f7cdab8f0fdd78c0fa58f34afccdcba4820a5
+# Generated on Fri Apr 6 14:46:23 2018
+[PLAINTEXT LENGTH = 4096]
+
+COUNT = 0
+K = 20f31cded60b8ed8d9d3fd1e1fa6244e76c7cb7628bfd28a5d63ce8aa2c9494d
+P = f07225202842c8dede42215301e44b9bb7e625d3812f74f9b6ddbcd024ebd1f33e2cbf280b9004941f3cbf86c880a2357f88f92a6dcf8dad9da7dddcd00f3635efdff0af4382024e93c2af66b991e565eacca6b886f07178c9b4adad6f0d6ada5ff6aa7cd0712519a947a8089cea5e1e3e40ffe1806010b0149f9ffc7c4dd3c31b3d08d5ae1997c52369393d58611dff9bec501c1ab35e6ed3e7f9445a34e211010a8236686f154e0a5ae3433d6a844eb3884961aa6592216d93952b46bb58a4195aa80966ad0ccd4a7e23823912556a90d5ee9c3bb952ecbb9d895dabd3b11ab4f2e3a6c2582de50403289230ef4dc46e7c0d870a3f0cba9d643a0349503c1b162ddb6350e699589eb47bd563999f55a1adb6b78b52f006901b0427ea7d3394bb0adae4637b4f1ad5d5425e2c8ff3083506d7ad7ba4c7405a778b0a3a11760c96900a5256956cc9710091d073a19f46a985d004651fe2b6448ed761bf9bc81619cf273a6783d868d090753bf01318be21afd88d9f3a961a69f93e9d9fb822c80acc7b48cf14a08b5b7ef15c66975721b7cde9761a145b679155472a44dea8fedc0f86ae7ebf6283ecfde5f2444b51569e6723a7a19e28cdf8dec6791ccc14af95abad018f741575b343cb1a20a2a9adf4248f99728069a1e2e78ad8966c41c9918fb7019ef56c153a183a6247d22d9956564bb03075cbfd1b43d96818b28484
+C = a5b63618fc0c4512960f00a1f226d9837a90480baea75265453b9553b12a58c72153080842d7f8710f317f88fbbbf97caf879ab4bf416ba767ee9aeb34357f4a2d0e8b9571054d98e28804a70bc4d74807f2bfd95ee955bfdbb6f4d6969a0c3c3b541a514647d5cd8c9740ac3496095c3f145c50c97ec98b935158fbdf89705d5330015e48ece89188b8c1bcb2ad6825d865b375a9b9056b743dac720feeac033c9f757f6fe73dd7c4a747661b64cf490a0dd43b547cd791a5d78dac97efcd355f7ebac248fa2a33e4fad640dc34e0d40b0d36588aa32f0864c9446739a6b44ff84666d723bd7d646c5172cda932fec34ddaaba342b02a9604087ef042a2be4774194b5d32cb3fb112438fbf2801050b5424635fa2d3d3fb10332965c73e6669e65195310a3a30602640e9809179cdfc50de585aa1c0072423c626815d281a06eac3b6ffa137716318e288e3f9970e415ef0451bdc557968febf9eb6772c1f77cb8e95701246d9c567048142bb25e340351b87d7391822d9ee7fe51378bc0d08135f9f39cf44b348b87937939dc61f430dfe308cada632722e23aed5a0699e039cf0563ab8025163744b136a13ce3c62c748c89f5e17540f105e7c6ec9ba13515b504342f9e6dc7d65b9a633d8c0b5c9fa858dbb9b3a594406d478a81bb9abfa289730408c1e303c663a61d5caca00f615065312580042862397b9aa8c80ca812887664c439c8c68
+
+COUNT = 1
+K = 85c8aa6d9c83cbcb11550c95cb9fa991d49dc89fbe2531e10694269f38e9a309
+P = 58ffd9e73b5f64cb951b7df799a505d842ef0cca6d4b21027680793565a67d47db0bc872a7640691216c044fbfc0be1cd33ffe9fc73c5ad8cf38bcfe6d4b7b76639ef4200ced0eff2d361e9c948075f09398fbe5ef24d370a5a16b515c102d415813cbce56f6a8f7b5227db7aab33c4d8162ac3c178e2344f3c7e123f0e60142112b960137c0176c450248167e9714db8566e2d4af731a7ef3f88730a4be4cb99bf46eba39527ff279be67aee1f4af25b3396aef5883fe7086fed7285af5216c7945d9bd61d907a84095d6fad383a7a90ad4061f7241a84358cf9fbc7048c010c7652bf47f9ff58f1c0cfd62a5fef92b02074ecbf4762af71eeda9aeb8497d7b33beeb4fdbd80b72f2c6dc9a12409cf7ee3716fc07515970411ed7e113b7ad9865d822a502976bba1d2ef254dda4acdf6f55a04c4ac5f6a5e011375af4cfd405d1220a10465a4cd01d160d179fc7ce614ef4023d55dad363e3639f1eb8fec716f21d267ca21b0587186d162b4a3cd23f80e33ca330a8c3bd756e70c2d457306cd54be9e0433bd9416efd861ed42fcde10ccc98389652b5c14cff6c73341320c93ee457f9e3f320c5c5564fe50eeecf2d7a62a85639ddac4abdad7495e8926111b18f7866e6b3f95c788d9124ab9788210b1e25524427e86049726200052404d9b3585477bfc11f12c4fa761ed86b7449a5471ce18f964acee913946c0997980f
+C = d4112af26f37a7e2e9cdc0fd460cf4cf4f3985fc820348e63a095751ce30b2fa5d6e89a8d37b8ddd3dc2af8fa64e0db729f526bab89562ad7f2bbb9ba02668033ebf6eb742904f37fbf1d87b4d23b1dfc815529a6e3f24bb31ba1d4354820a47e0d7b0acb4e495a50912824715d24cdeab0be3597dc535326ee073eb39b925c7699115dfdffd4162b84afc61048c7802a76ddafec243b91791a26138f202bf365a0b0863555d48c953c50cce5e09fede4a97571d3ebf0582c57c39680714750fd2227a632274adece8a4bd13413825b51969b3e965ad417c96415f4041d8f0d0a3df04cbb24fcfbeb95b401be7ea840da43606cd5bd56bfde26034fe94aadc78bedfaa4eed77e64cdc3d059e3765ff6346e9344f0e0a183085d78711b13d4ddc356a669693d2ad090bb67e8013403c51c81951f8a6241269d3423a739b44215bfa403f8927fc1d100587d602a7efc505f28edfe51ae8df407361fa06c09d06c7e960cb7b7041f3603faa6b4767a8a3c3d1c8ab05f75143a3061396c7289503ec29237eea7a364eedb77f66da308fe8e194c9b1784364072dbcc8aa1145db1a03f3862c38cc1a08b569f8576623915a644700ab43f7eff7e59684eea5595c31b13778824704b4aaa9c1b5d2acc4a6cd2ae21c5b0b32bc1433d536e8bd7a4e6c5e6d1ceb70dcd2ac215bd4619aa1ea8c184033b3279cc9fd094336c5e6f5112bd9f36fbebf304ab04f
+
+COUNT = 2
+K = 5309c5dff8fb03b007c28bda0a219d329080120b430f4b5af9df60ebbe20890c
+P = 22b75fa402aa5b665aa71db0e24e76a6796329fea9128dffbbe03764e23d32443cadfbb6d557e833f841ecf5cc7c9200de81b2c58fc72a1a9451ce51c75abde6504b7868d1864a5824d6ef5dbd57c80e488a011102f7a321639fa57c40a5eb8479cbbc7428bb5497dd6295e270b53b00eaff17c2aac199a637e2baabce9c3f5af8d8c4698eea5390d7bdaa92ddf8190361a2452df2f44220b92519449605069661e0360cfef6841b734c8bdac87feeac243693a4e114995e1985d5c93e512d0f2fa27f12db8803e0aafd974d968cfb82cd9f0d4b09f34a440cff85e09a63462fe9820d5f1ba6bc6a489f30ed570e44c06668b6a4d3b977a6b7fdff3e781f0b2c7a1361108eb95f569c3ac5dfb1352cfe0965d9bea70f923d9f65cccfa83a9272947b3563f81c92f7f5caff203137b034af82847de0653bfee01aea476c9112267f97507b6949dd1d0d7510fa280432159a0b3bd67a8e628ef11599be4459238161914520b556fd5d91e92349169bd2ec175a9e728ca78b56eb2bacd56fd2af2696d1b116f5776e9df459c1023180a18f5d6e3718ac402bb0fdb806cd7b1cda6fc107e635f6a756005df79b720f3f274deffb31552cf238ec93182370d0ebe9f0b819e2e7225bc2b77e702d5df2825fe5d42005f1e39d67414ea539810db552049641ca22bd6539f1efefcb90dd479bf93080cb9df86a9aa99048f950297b8343
+C = c6a64b49464f5db516dd39f3d1dbc444528b5da57016b5f2a5649965a4781962264dee68d4c78e987f9446cdc815aabeff2590d3e2679a2583dbfd3aa6b8bd3bcd86d66d598daa6e766a6ab8fc6fe98a1bc89b5a137a342a59dfb52c7e7b2feb2a33d5f936503121476a9eee8d5c60b6c4c9aa7f8c56ddee5b4a9d7908890d3aa98f45cce20a074896aacefb6d7520e84b6675d02b819f37929746298d2c81407c20328a1ff4ee6da8298fd639fa9bba4ba27da5e19520434e27ae4bef59b73c3660aba0353701573d602d8135442a14cea4dbca3732a3fc8e5ac831929ad48ca30d4ee0f068f54faafb50bc7c6a784e81cf86b29225200f985af914ea6ed166e08c04e6324a372550603c98bb6c010ea7219699e76e4cdd9c4b8e6c9200d10c51e93a57f25b0019ae436dde220868596e96ebea3086089ece51b93a812c8120b8ba60435720ed690a2d1ff7c024cd514cf73bc7cb121a83222e4fba2dca65aaafd8cb51ac096511f83327458c53119f2462caf6cde5ae679fafd4f96756aa496ae6657ce417b2963c09463ab810ef51b4942dd1ecd120d69ae76f8928f3bcf20465f68e526beb7cef276533270f08136c4088182dcd626ffb2c3c4a357d09affb6b935e671bc827779ec6f260acf2c01f1faad57b162fb50579ed45909be1cb2a4a4dd07effcae6a2342022e0bad504261bb8d5d3b73eea932b0135fbb76fd6b8f8d3b6435db6d3
+
+COUNT = 3
+K = 8027a28b92435ac111a67ba0f4af5cdd50e6bbf965cb9077189143440db0327d
+P = 0d8f70b229ad687ab5f55f0b0ade81a4bcff76dba57aef40abb8074b9745ad74efcc8d8f2202b92a9788f8049c864e5973403279dd02e165b772ed7444795ebfdafd8e13a9a22ac1103d0c15d8d279c617dafff98dbdb9bcbea923a00967287ea9accdfcc90b2b4047021115d5454d057cf3a27a44988a8fa71578ab7751a5d3bbabd1a335248564f5e334c3925119a29c219aff6e71b3963f603b0f6e8213219fa56a36e33066f49409c643dfa1ab7478630d623c7b77efe07b0ad4811a522ed81a1c905831689747efb4edab65062cea3724c3f9decad27290c4ca0ee9c68f1ad9ce3981a0b712acbe2de17eefb7f9bfb3efd66aae691e63e5d64233715eae8957bce7924fe2f259309ddfe050cfca3cf76c52f96f8b7608cc833bb3876f35e37f6a83ac0a21f0c0fe859133c17c5f7422c72741137b8c3b5c3ed3e7737d5d8a29c063f1490558982496b7c852ec95bc452113c48bba93394dc1df0eee1f8dadd38715e93a1d7065c528f08525cd84583e250327320405bcf18d8011e3d864978f948b2839bb6d1d7bcef4b5b6871243a5c93a2afc1f7fc7d06c8307a025ebb522cf0fa1e1e9f8163f20854514504f4f873d757faa7938bb1ef91000df4ea64bb985d903136fcb9c4e9467561cd564f61146ac519c08dfa41f22b773cc821cd1252a3ba237c1e8c1c128ba40e0b75e67adf12cc448e47fb4037850b014f9c3
+C = c1e82d374a1d7c07b1f061656fd58b5df0a04c099f3390e98d2cd460b2e2713eb7274b9f646fc453a5d26c3fc0b9203521de72dc78292cf4d61095febeb0748658dc6764fdee4fa498b93a6bc053b3d81e10564cc9681aff2296ae0864fe3db495298ac4174f6ce06ae6b5bf86985c4603700757ef44c82c0e03beb7f2841df2dc4dd8022d5dc7ea2f711daacc1c8d1805798f646f1041146891e88edc4db9904f122c74656a5ea1694f8e6073920481e6cd30b20fac4cad7e03f58183f6b0beff8dde3ab4f781a1dc36ce0905e41e77be65c83dc9b6ab69d5d4c15a1de3cae6b081ea9568578207d678df77e20196483c2e00d2dba604784495949285a623f33e07e20687aadb86d57adb9935865e6467801a10add2ac7dfb6f8ba98e7635d6766eed38ca8fa8101f93018261b872a2a899065918b672423288cdaa417560701600424da6db15e0ebb7a5d1d0a7e1cc8aca714ac6bde51f779033563b49dcaeb8e0330112902b5ff0ebbdd00aa371c902092566244bd9a1e5e978af39adbe13a7067d056d04b718133575e420058529a42ec032cc5fa62035e872cec7acb3c32eb8b02e0cdd81c59afdba65b673c828b980756e8ff6b0160a1e796e221ed40c342863848b76e4390ef730abd3edc73ba975c037e6ba5e58162d7a60cd55b61643e9a8d4f6e5efa2485da7c45991ec4e8d4c6d9432e630e703ad1fc2f644d7577248ef7c7c974ce7
+
+COUNT = 4
+K = 1a5b5f43929215da78d1d8c8fd4ba4508a66273be95c85a892da606eed04d6da
+P = bd762d13e2385d333e435ecd792b8b3f13936d3b8372466fa7f65aa537bbd0bca184ec570acdb99ce59633bace041fb1374961be99276ff1d4b82cd49d78b89585aaa6cb1bf18498f4d01f2fc5e41e2b4fa56cedf5fe69e39eb1bed303d0795a715bb7eece6b389b3ae81b23b4f053137d54c03c2bdc68c79184fcca1758652c61a5e3c19d49b3af1a8946403b254da0f9ef227d2be6d0c528e1a7b78a55a2e79d79d6e420e9a3e8b40bb3ee74935be7c8e935414209be01532c991a871b90de0b35e963989475fbcfc37d81cbac674b1b29da1d0f9e78a6c3ef55575a8d3d7637b93895bae528d85130d059415d7c456a5c8d94897e567d6d6748e7c2689c98331ee5a7e1e25909901ace55828737e7464cb670399af3f806b996f416550d256f354dcdd1c332b4ca9eaef70114a7bc171efa08b88f1afd0429b2cf15366105cb812596a05e22b06debe1c69b06d904fe6cce284531ebefbae44b9c9eb156dedd01a4a73a1e631fcabc349e07636741580a916b35560d02e1cbbbea0a335b395487df8f972cf8600b48237e7b63bad5f08c85b85043eac0ca93d34df56cc486857d6eb3de0c332ee1ded06927e8acdaa3fce78636eb06c0f805788189c56023c8a1f5a2427c6f0f8b993f517a9b4a3bc3fa9c3e6318256dc49941c1e92973e3ae4814b87135c9ee767348045ef7877027c8ec441c50b1cab98bf556352333aa
+C = bed34ebd1c70d52665103651110f847d83e19a551f5156c9720e2d1b038ed5d5ac0be8cd97ad3d725a8fa3c127939cbe1a6946558015bda4c7fffff7e00662c4b0ae3d5477bf61e175f8e4ca14cf53ccc558b8fb5f371748329b6cfbfb277cd0828a2394ec99f943775b4a2c1c285127a2d4677ba3b918154908badde2a74b58f73ac8f92b440214a3cc990f1517372b2e53272db9bb191cc7711ea31ffba2de2b646e8e312f4eedbe9e51e182603cf075ff69bc4d845410bf66d20215c160df05ab007b49a5396bf69d702d193aa0e2a2f9d62b7f9fb46cdf11864314101c7b7072f5ba8262c32b12b14225edba48b440501163d027e7fcf2292c9b06ce19dd0eabc6b6639769544d71bf068b8e03c4834c4a785a6b2b927c077f839f29da61c6c52c39ef4354839eff993068b67438266e10e9a421a90a103835829320fb1ad9e7e0cf73bdd22aa268436f4df0d973c401f8bb20f55585954079b6de5a2707ab5ce3b9f42af09b7c1996a45a15af12d041f5fe0c055b969a9055f7656cc2806745e84836903804799ffcc15580f8a50362f4dab9694e8ebf6f32cacbc3ba3501959ffbf33db0d231722715705e685df9202f0466c3ac0f1361a71905e23b0aa9fcac76b01d6402598861084477338b5d0463a641e2576f76aab2de95016cb31d37d8f4617e3908a179b43e3d50c516bff1e388bfc0ca519430f85a0de5b0e274cd0bd021f20c34
+
+COUNT = 5
+K = 2a919617d24baf991779de6c06e68bbc4894a4e9a702214ba3eabfbd0015a7c0
+P = 8d01e11206e3b9285cfc73c2ad6b238536fefbe824f19588685601bf94e1ee2515c4ac281aa04feef877694840d15f2a6722661159a7216548d768b4754d65f1f1c522daf2baebbf88f638fb324b182e61d315f5f12b04c486022749f6a222d87c7d635c52605c7efc1853cbba466d138d3cda22979e7742e4fc8168f5139a72818109d2124c20ddcecc74fb388f34151e96fd33e8b269a7d7e51b6fc6ebb427219163c3d9f7de4a6831a7f8b596b4b490789f1104fa160115b6e8d191d1db885e42812c01724170e765634fdb8e05555cc7a85c3d8cc7a8e299b4f3708a4a5189871037c0d04216dca8e6127b449ec6004e7f076430d1f870f0d2c0c6812c810b4665f7fc07fbb04b56bd956b81267d2408c31be8dabade28a3c40c609eee3458b7340715272ee26df8d875d1139511649b629df067a8994e1fbcf7ec60f565e8da257db1b7ef3b2b765c77124c8120bf26db8c47f493d2afbbf8b89efe2e33c21c2a2a1141c0aa5b9dc1eeae50714304ff35a7cd34da963f5249bdce8e2c2a3026672bcd2ebf311f446cbfb7171fd35b446f0270320b30b554f96573eac72ea0594306d43c5ed48a6c2d01a363e88646a8759117075596fe17f18c0b36d00b7976098fddeb7abc4e34bf6d6b35238f81b59d169db4578a96c2bcfe78bd583fb31fc2e7fc97c6f3460147a5730bdf31ca197482acc341b7a719c4e1716cde56
+C = eb62ee28d9929093e3c6dbc0592c35dd0ffbe8faa37173195da391b84b1ffb342b9bd9f0df0947c33c7c74a071e7d71312f91c8e1f11d680865563f2d9604dea06892f2adaf2028246b74ecc82686bd7c369426a09f23012b69a89e7e66e7cf008b41dbf04f72cf950fd8f6ed63b3392de53b6feb34e45df3153e4aa6fdd685fcd7e3239f0509f474d8f6a58abfbd36bf14f93fecf913b2f9a0f0763c39cb6828651434ef0520cee88f46daef7932e5ce1549774ec106c2aba54cff64067e0202167f5a3ba1cd6396bda08023d2cfa9100043b44b8d90e2862d7d2a7f6e3f5eb6976dac2d5dbd2adeb69858ca391fdd4a582d3cb79a2d9fb57ad5b2cbd157b1e36ed49fb848e9960b0dd1c715b701f1d379027999fe8e35769e27a9ef60a45aeee56c7a3ebb39f6c50796d1236e721cd0e5d931e6a85f28fd87d652c3e5b706e2e97c12c4b33fc7df4585d60d0326267d8d252ab8970a2528be086d7ad18ee4b1531cc13b9c88cb3a9188a47b8a72cd276eb4d7d726a290398a98be0083e9917f349d2c8137a6fcd2232baedb6edf075d8e938319a12bfc31726b13d0e1a6c18ae33e11ad1f451fb9693f1774aec30fa7b473c98cbaf4dad1b3743d118ce69620503a753b87e4ae25f36564c2538d6650291087ae04d181e4fc44b6804b5bb0c6c8a0aa04a49895ea64d13f7d253e62216736a716258bc98c63349b5b19f5ae9aec3a176f98cb298
diff --git a/src/test/crypto/Makefile b/src/test/crypto/Makefile
new file mode 100644
index 0000000000..9b5fce5b30
--- /dev/null
+++ b/src/test/crypto/Makefile
@@ -0,0 +1,39 @@
+#------------------------------------------------------------------------
+#
+# Makefile for test crypto programs
+#
+# Copyright (c) 1998-2021, PostgreSQL Global Development Group
+#
+# src/test/crypto/Makefile
+#
+#------------------------------------------------------------------------
+
+PGFILEDESC = "testcrypto - test PG crypto routines"
+PGAPPICON=win32
+
+subdir = src/test/crypto
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+export with_ssl
+
+OBJS = \
+ $(WIN32RES) \
+ testcrypto.o
+
+PROGS = testcrypto
+
+all: $(PROGS)
+
+testcrypto: $(OBJS) | submake-libpgport
+ $(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+check: temp-install $(PROGS)
+ $(prove_check)
+
+installcheck: $(PROGS)
+ $(prove_installcheck)
+
+clean distclean maintainer-clean:
+ rm -f $(PROGS) $(OBJS)
+ rm -rf tmp_check log
diff --git a/src/test/crypto/README b/src/test/crypto/README
new file mode 100644
index 0000000000..3750346a83
--- /dev/null
+++ b/src/test/crypto/README
@@ -0,0 +1,33 @@
+src/test/crypto/README
+
+Regression tests for cluster file encryption
+============================================
+
+This directory contains scripts for testing cluster file encryption.
+The first two tests test the encryption/decryption using AES128 and
+AES256 in GCM mode and the Key Wrap with Padding (KWP) method.
+
+The third test tests that the data encryption keys can be encrypted and
+decrypted, and tests cluster key rotation via pg_alterckey. The fourth
+test checks that the database files are encrypted.
+
+Running the tests
+=================
+
+Run
+ make check
+or
+ make installcheck
+You can use "make installcheck" if you previously did "make install".
+In that case, the code in the installation tree is tested. With
+"make check", a temporary installation tree is built from the current
+sources and then tested.
+
+Either way, this test initializes, starts, and stops a test Postgres
+cluster.
+
+Requirements
+============
+
+OpenSSL must be compiled into the server for these test to run; if not
+the tests are skipped.
diff --git a/src/test/crypto/gcmDecrypt128.rsp b/src/test/crypto/gcmDecrypt128.rsp
new file mode 100644
index 0000000000..53f694d37a
--- /dev/null
+++ b/src/test/crypto/gcmDecrypt128.rsp
@@ -0,0 +1,129 @@
+# CAVS 14.0
+# GCM Decrypt with keysize 128 test information
+# Generated on Fri Aug 31 11:28:04 2012
+
+[Keylen = 128]
+[IVlen = 1024]
+[PTlen = 408]
+[AADlen = 0]
+[Taglen = 32]
+
+Count = 0
+Key = 137f40bc82b01b34047a407f40d5c434
+IV = 1f3de3a272227db8a88ba0cf7b23f7e57f84ce370b289e2f774a0c6184f44f9df9b3e645ae506e3d18cdc819fdc3b31f77d2dae7ee295ba2c776e6f1fefecec740021c4aa1118a6ed3813720411f27ace5c11437a78b3d626b5f421e2f3ffcd0677930e7878110a0cad535c057261c63f0531b538d8bf04f491c16e871f071ac
+CT = f6336f2001a5edc519aa64071fabec2fbe4654ff334871e9cb8b12c24ffb7ec9f0d316e43bbdb568b65608f937f24f64eab19b
+AAD =
+Tag = ea3d8cad
+PT = 60881890126afdf881dba4c1a7bd0f9d8c240073517b21c622bfa70c1f05d59544b9dccdd908bd924ca60d23697fe64aab927c
+
+Count = 1
+Key = 5e49fac6608ee4157a9f0ffde503893c
+IV = 53d854bba28eb9d9db71e68b5916f9e73964a0f95c6524411fb18f64977d012d611a15fe8d46ec49bcabf060cfc7989c4a8272567c660c0881f944066b88311992b5d52cd644bf07e9f01d2249e5f9b00ae1af9f2739da627b3a8838e28116b02e7d171d1938edc6dd5d7da3dac63ed148b367961b809bf423510f0a3f44a8d5
+CT = 23bff066410784e0999a3d0fe742ee28ce8095fe822fae0f50b0b17d70c8cdcd705c83e108fcef2d64c5b4db110e9a72d3c4e2
+AAD =
+Tag = 01aaf4e6
+FAIL
+
+Count = 2
+Key = 174b47de82c64196af18214d9d6dce2f
+IV = db9653e9ca4ff27f8180f45eaf4861541d30b3b147d2b9b64709983dd08312b5fa709a7bc6afc591fbdd08fa0f21f06a25758ee199a6c6415cffdd9777280e620d7bfaa6d551875c069f5672a730aa68137cf33f86d46cbd548248e68f589c2e699c554ae4680417e9440e5514c1cee03205324f789d953d63518b023f65abe6
+CT = f8b0c1317c5f3550363762dc7fcc847376a7917e1505cddafeb5451ecf4464ef52afd797697f1b0b1e8d3a2376600d6acc1c88
+AAD =
+Tag = f6fa70c6
+PT = 523986d433fbcca153af7765d177e2bad72ba52743440250db451e2da643913cb25b33fa364011a8f8d5cb563b461d46952923
+
+Count = 3
+Key = 5194e88020a501ddf21a7ba5525bff70
+IV = 444b0037fba6cb9d4f4e33df32289d44f262d8f535564a1921e6a4315611b98a4e1074b37114758b421ca44efda91a48ba8674c695c259649c0bd3c11bcdc8363d3d113a534bd790988a44e72559fa70a5e2deeb9da187e2fa89c7b636a000a38d7bfec2edf2831ddfc9326f0c54956c149e5ffba3f556c8f2a749dc6ed5407a
+CT = 31398e58d6e27c50e85c87a4dbfd73134f456ff042c9f869b47a41e3c10db7b24cffc103a2185a696fdc1f2d389cbf41379f04
+AAD =
+Tag = 2d28343a
+FAIL
+
+Count = 4
+Key = de250ebe0ea1212480f7193934d1ae84
+IV = 255f658334f1c5258d613ef92da7242079795e9c04c5c4e800aa1c03b6abe77f4dc28cf614278900b52f1de23550fc9c0ace8e8dd91a25d0240281dcd684dcb786839f376496692f3e35a5946bbf7b39cabf7d782f4e77eec53a875d2a5e3106238dae0d048e8b9ec7b05961d9d68489bcbe2805ab0f9cd42b5f8f62b4b28b00
+CT = 6eace8c7c10c713870c676e557c6b49134af26011af157de1e2c6ce0769fab666b9a7844756d6b948ccdf1df77fa98119c83a0
+AAD =
+Tag = 87b81d39
+FAIL
+
+Count = 5
+Key = d0d9c4aa70e5bcad8dd9d72d14c21c54
+IV = da20388ed0709e5af5a52a5293bc4060f02ed6d05d96f9ac3a05c14118a3613ff790b721ff2c888d63c04b9b28a7138ca00d0ebbfcb11b70868f5b90c0dce79c7220fb90d5f3b6e22c6ae3640e7dfaca72ec12813820debdb69f90c8393645d7e96cc36bdf62ecc4c0ec181c64593c65c8f707d6db61b16c6a5c2f05f8851692
+CT = 2f08e5a8027a88f0e223429e6749dbdf91ff9742c7c6130962ae837a434a7247600df8e2004fcf3e81f8ef64e6a23bda002f3e
+AAD =
+Tag = ef6c0d39
+FAIL
+
+Count = 6
+Key = 464609f485abe0bf4930102f9637fdee
+IV = d0a788fa1494d47bd1cddd55f88515229de5c1a2ca7dca054e08bf35d3c0aba677671d5290c3375fabe40de14e15465138d298edea5efe1ac62452dfee86920fec8e46577bae1c0b43d77cb6b8d1de6ba4557d7410796bede796b8b87b5771e30e05a8bfa55ed92194d506892c33f8ad6038ee8896b98772d1a88a7a40ceb650
+CT = 5e81bd883fbedb3e074ba273751081752c3aeb7a62b08091b768f8f8e6c0913cf633358fc26885150adf65bd1d708d9e45eff2
+AAD =
+Tag = 7e715546
+FAIL
+
+Count = 7
+Key = 018319ad18e5d877dbdd871316fa6610
+IV = 941f8715bbba7b1fdfa98c8c05546a50f86cd02b654862af6c1a70cf4d79618b37abd1f06b7a29bbed74e728a8c78a0ec8ef8312f2552fc4571f48024ec5c6d44c21d69311cb1ba390cda615f246cb175a9e8a522a20b671e6df973b8924fd003c5dab7e59af63f20edf56e9dbc9f98fcac25cc692cd87a3d8fbf8d4b5f150cf
+CT = c5a681b2b2e5f68e0cb698150aab9d13e93e900e0e6fb20170a4104839a7e0a5775bb3f20ae814edd109530552ee2b41cf5986
+AAD =
+Tag = 75a966e1
+FAIL
+
+Count = 8
+Key = 3556566a1cdec721469cc86412ade697
+IV = 731a29cc083422da193724a57c8126b587c3817fbc6b76bee106abb3ac48722bfd0cc15deb573ca73f2bc10ad7b60024f297bc52c31a7021bda91c5246c781a95d5d18978c685f096e9310d6f5ed151122262f7d3f5a2b265321683cd72e0693ff608e378199264eea876df44659817997b323e43457e70809c74737d015cf26
+CT = 753174df0aff575fcea79e54e5b8347e69175070a00d58720529b173e9dc64677fdf459ad72385b03244d90810670304129f2e
+AAD =
+Tag = 41147153
+PT = 7588b757f08b75fc799fc825034ac60585e24c56db94920b6982bbae86030e7a021ddc89a5131e5371e9295e414d3503ec11e3
+
+Count = 9
+Key = 2ec96059d1313923fe522bed1e942491
+IV = aafb096a63195cfc959bf93785a48cb7a78b98c60f861c9646129222643d249e50d46f706d670662f60c12a7f0a6ad65b908ede9a56b27c46f07d427754c11549154632e9b5ffa25c9306add4c0b13c7191abd59430a65ccc42c5420ec98093ad6ec6ad1c4d3bb895ca3546236c2c78cd075b4b1f44a6a03d115b91df02b96b9
+CT = 61c6b8bd44c9c6eb075fa9bba7ac0156bc64a6326d32afedb51f35daf232dfd068dbfd0505933519a753bce6de4560b79b31f1
+AAD =
+Tag = 26f53f5c
+PT = e4b86f8b7684d6a300cdca848ee8185c41fcfe061e61af6f3a563ceccab58751f1884b322019983d99150db217a1e92a8f15bb
+
+Count = 10
+Key = 48cbb46539dca9faa97b69b32015886f
+IV = 1d3152ccf3ff05a81837ac7c55f791c78aba8c95149c80de8724b1126dda847b983a1bfe54ac5206d5915dd0796944e4daf432b14a72c5fcf81822ce1e3e7ade345528641b94a2fdc90afac098b9e4b0b5ad159ba275dcbcc11fec04bba0f12db9b3d1473ea0ed7825d293fd0b5d2d07795b3de922bba5660778adbdb2cf8539
+CT = b2658e897b85d9acf437f9e895189954134c543aaa91fa82c586cb5bb1d96e04b189df64df5bfcb63a567e444f4653aee3fdbf
+AAD =
+Tag = 48d21620
+PT = ef58f7ad664e36dce523408e497acb871e9a579fe4688be6425adf41c4c41db85e905e9279b93a037648802dfc273b387e7216
+
+Count = 11
+Key = c5c2e83c4e5b336e30257c95c8f7f75d
+IV = 269c29118eb9cb4ae8fd8ff5b3a5d61f4a63a719416bf2cc5225544cabdae24741239ba7910d1a32364c341169a215264ae62964322b6ac64002d09455f11d9658a9454cc6bafa82dba5b5365b2243e854470dbe74a3043b5f5f82ebefadaca4d468317c83076691c188af6e800dc296cdd941fc1aaed8840874f8cbbf96827e
+CT = 891da6e799429c772edfe93d9fc3ad20c25fd5a172d272d0ee6b7063b59b611e4a2805923f182abf99373fbe4a1759339bb13d
+AAD =
+Tag = 6759d819
+PT = 474a2e7aa4a5422bf17f257b584059fc7abe2d25e7243b434a6f936f5a0eaed692616a43a2aef8b427d83ddf1bece29d96c581
+
+Count = 12
+Key = f4d7a0e84b7f024988f0507098415616
+IV = 9d58c03fe45a97d4c556bafa4292766c55bd84d0e467c6c26b0e42162015f3f506c9fa06caac63a3ef6a3f0ea62ca4e122bf453a767b2b00d6277a990329b76073c64ef260421d887eaf89b6deb488a0528608ee8c38faa5f5976b2b71d7e29a08ccbeb2342ec15c46232f3c8867ada7cb8cc8aed0728dc59d706913ba62124a
+CT = 11358bd3ea694574c0bde99aafc2621732fcd978626d3a3ef124e98a2c3dcb8261580306b51ffa9557d379952ee31bffd9ade0
+AAD =
+Tag = cc46943b
+PT = 4a289261f4ced6fbe544fa27d430f04c72ccfdbbe0f93e881fcc1b29a19e706a579a98304cb2b03cfb3aacbe187dc69f923ee5
+
+Count = 13
+Key = 31237ffe81db56b9541133156fd0e7d8
+IV = feea389ed83bddb360f42351cfd56321bdd0a8fbbdcc8787725a236365eafe5dbc9c6e9ec8e9ea0f74e623c33a574ea4223a43b1550add0c75d8348315add72ee9561e1750b69defd46214bcc507d4db4c66c6da23612aabc2ebc9b0e6d5aeee270c013a553ddbc7d7d7ac62ecb33a5a6e3db859471ca3d3686203c0c96eed2e
+CT = b965d7bb29a8811872c894d535e001d534a7d6e99eea5bd14ebf39c5e79c623a67d549bafbd0dcee1494245c04854f34de677e
+AAD =
+Tag = e6fac1d4
+FAIL
+
+Count = 14
+Key = b0e576fd7cc85f7c92d44e4b8c149b3c
+IV = 3855af97c5b740baebda1532c196d187b8af0da761489ee9e267fcbeb720bf6f73cd743b69d942f3f44893d68a5c70174a1863dfb831ced0ffbee668a03e1066e8b3dc03dc1471c3a848b3787c0645b20add2fc0cc37e5e0aa57ef08fc69c53897030fcb5579a831ee53c76df2f75d2de5bea93e9ddd8e4e1383bfa1a7c7aa77
+CT = 624aad7fd047a61c09638b8dd5065d7c00960035440303a03ff1b19caac02baa3b6835581f2b66c2381006798c9d5d63d65684
+AAD =
+Tag = e6142617
+PT = 0fb24ea93a14e3d76c4d7a991cb55e6bde4083504ac9f3a00d4b0ef53d3929953b53f5ce820c41b9aa75a985e09986d3f4baee
diff --git a/src/test/crypto/gcmDecrypt256.rsp b/src/test/crypto/gcmDecrypt256.rsp
new file mode 100644
index 0000000000..af72933a60
--- /dev/null
+++ b/src/test/crypto/gcmDecrypt256.rsp
@@ -0,0 +1,129 @@
+# CAVS 14.0
+# GCM Decrypt with keysize 256 test information
+# Generated on Fri Aug 31 11:31:25 2012
+
+[Keylen = 256]
+[IVlen = 1024]
+[PTlen = 408]
+[AADlen = 0]
+[Taglen = 32]
+
+Count = 0
+Key = 60c9f83fb0ddbdc727e70bf9eb1acc13b1b63e3056e64db7c2ac55c4f2068273
+IV = c33d34a3673b93bb78dd1e00f877c4e6e4cf628438b9effa61cfe81e159155cc9ca7c1418917527ed3f0a51daf2bedbdaca20fad687a7dd086ae086c8ff5094e9b31fd71bd6f8f1f1adbf96bb2690663386c37d7bce891137897aeef70be10a453cef7e31c1b8c0a24ac1baeaf08a46aac445ad5a8103804825fde86dd4720b4
+CT = df6586921250aacd9d25f432977e92b09ddf89a9403c83a80890ff15ce9c4559145ecd85d86f1573bbc1b48992859d22fc13b6
+AAD =
+Tag = b0dc70f0
+FAIL
+
+Count = 1
+Key = 8d7db520bf88c96d46778991a4f0b6de9aa7fd5d35cb6188a6f355499072af5c
+IV = 488a50706bfd8ec7fb4c508511bf4c897c8566ef289b5e58a4c59bcbf16b5ae85fbccaee4a1cc0d1ec74156ae911d36d497f5ee71f1fa51649819c9cb88cf65d62d2abb65d621c202bcb33d8d68018a858d04e79deb62b3486658730735a1c87829acb49e73301902c116c9b6ce110f23a6b1a4dd657e47a328e017c19f0ee52
+CT = 497e1a39ba1b38d263bcbf19cc2900ca4070ad37ec12bfdd30139a7068a889825eaac5012cb5c2dcc710a220cc658dcf069f60
+AAD =
+Tag = f20b9885
+FAIL
+
+Count = 2
+Key = 1ae1e42ee656973f5628e3cd11f0494dc8a563cbf5fbc5880cc2dc6d787bc9b9
+IV = 23e3c948cac6eba2ed11d667783557917f066ff6b93ab9409df9c7c84b27d26817dbdebb9fa9d0a64bbc572bfb2c7ef7f0c836528c9bd692505c8c5e522f57aecf6b479723449398e5b1f45cadd81264c5aca8059562d69deba26395034b4b01325d072dce92e540c159dba92d3e41e2d0947d873ad48f9f0b00f4807d420aac
+CT = 3839fae7008b88250b602cbdf295e932e3c4e3710d397a2b9a37289104efde75f73302b2820f14664c064e8dec45ae49a74036
+AAD =
+Tag = bce334cd
+FAIL
+
+Count = 3
+Key = 8af302bc8684cb91b4d7a6088cf8c94f9f6e027ba046c2b508956ba3c88f2d65
+IV = 3218736e931392e6510b91210a6a6a27680740ba8924062ea176048d6b42f44ed04a46ce31843b735ef4f63dd1d85643f28fb335d21fc2e3c673e97e6b845e363362d32844c9054a165f40658267bb177b74797a8828b1eea723d51b571d93748c758ea5c328103612b109e008f743f9505034ed3c42ab3dc310c20938f8627c
+CT = 1f1b133a1a7b58625fc77021f8ad1751bfa2b8addc0a9837dd5c44632cffe5ecc2e9e54b90cafb6cf8b652a8d2da116ecda3f5
+AAD =
+Tag = a673129d
+FAIL
+
+Count = 4
+Key = 13a7942b5a5cecb2bdc0e8b0348d4db5a98572544ee31918ea625b0691c10779
+IV = d92b4d05a549b296e18c90a8da55ec5bff3547a679697c489a1d49dc02bfe2dd85c8f050b32c389c4f857eb4b663f53354bcfe9c3a7e30019f2e3994421bcf3a3d1cc093768eed71bad5139f3f3078514d80a4a41d1284b5dc43ce07efac9c475d6ba2acb66dee50cdc62c463a05ca396e72d189f50d44ffb70d2c6112c6ef0e
+CT = afe058cf694d64706302b405243db77c7ae2fe4f33c6427416f8992ba92754c69d4e7c1a89e9b6987f2bc0a7b568dca9c9d273
+AAD =
+Tag = e0bfbdb0
+PT = d544d114e3d9ab8aa2b9ec588a112f780a6df74d637be3cd34fdefe14506f26281cacd2b98c26fc4adff837a7bd72173b962a2
+
+Count = 5
+Key = 7272e6ca6d6d76c483df9a55c6d07bd54fd8fad50b529ed52154959acf01b64a
+IV = 5c55bb8f4fe797ce34c0e281c3b04ba0bce8689493451ea569ba8cbacc74ea36ccf319776f77cb4d7f901fd0ff23cd28ff0ca77ad9d4adb0329fb68a60ff004a1c5b12111d2dd705ab1f7734178f14dbb356cfc0c5c208b91c277235f35afe8c2d46ebf43bd5e0a653e67e0c086ebcfca32a56d56dd5f810f562f769cce2794c
+CT = 7f378bc30cf2774f21078f42b5d6b66aa355c8c073d3a70f06775f3c7e5948539ec08a2cc50cae6f2ad9680ba47bac190c3068
+AAD =
+Tag = e612f4b2
+FAIL
+
+Count = 6
+Key = 82b1d2ffb53fc79f5ef88742a28eabcf404074836fe28b5b202ce7d5c68f6ebe
+IV = 33f834fe23b9639d30de763faf7c1a71568c5dea9d5d253f28723ccb3306a3cac3cac3beca638067a3485ff743b5133577633ec88dae0aec4fec08e894ab5d61c411f0939772df2fa66d5775f74b3ff36ee61695d7cd2726b9be4df80750011477705948b1276db0cafede5d7ac73ccdf01b73a5492a02c43b89632a501f6694
+CT = d9d92b33a10f4252fff828b57ca5f5f118885df0825be80ea5725a874b7e8721af40bd221e7f5c2c8b005d77af6266cd36ddd2
+AAD =
+Tag = 86e48fa3
+FAIL
+
+Count = 7
+Key = 5264831eaebdde1eadf741dbfd585cb0ef6437d1365bd5848d9cb3a22f57d420
+IV = ede74a8f53eac5dac276bc72518255831b616c9fb50a617eacdcdfa50e197d2941004f785f00f8c600e239cda77c8c06088793a674efb8759c98604dc0143e06665dc7e21d5031fd4751a7cd1b947304645e0987ec7e765db80a743122fbcaef9ec83849e8eee8d011dab67fb54317caddcfc472f585e93df91b1edce9695908
+CT = 9ad126b39dc2066542dd30c8fe81cd750b72123d74aa162113c6b0cf10a9cdb217d921e8f03b400f1ff719fc704f44e26ad463
+AAD =
+Tag = 5d7bed4e
+FAIL
+
+Count = 8
+Key = 6c2a43ce5610eab9dc40f43f035f7eed6651789dfdd166d4f106c95cef2a67ec
+IV = 60e3a8ddb899108c11550a461720bdbf9adef26c300f098c73c3767621b06eac4f5619b9855d96e4d972ddd38f4538f8e25b7524b46c6341e8780e22c3b42ccf43f41fddfc5680432b64fb4025b378204045bb2d7ea56f4340a4018a4c99eb8b91012b28024d1b2bdb603fa10a28130e84bce38384fbb7c43548c0072c5c657d
+CT = e073e948ddfc414948b12b4540d43dfeb9cbfa525b3cacccd21da89ecfb254c840722b9179057cb3ee69358f05e4ad0e41a543
+AAD =
+Tag = ac0497a5
+FAIL
+
+Count = 9
+Key = 7c36a9bb3033bc6f7395155eadf0e07c8e5b3441d0ad66b21625d4950760386c
+IV = c8393fb1d80ce92801a4fd906a568f7f404a82b02096e859e70e46d1ca5e231a073c5acbaa4cb4c33581e6887c402753bd55f95c76e68bfcbb1cb21bd37ab7a226e03d03e9dca6589c3020f5f916c50676e8c387f9b1710579a728ba7e7b60955ee5e383bb75d2b9d0f2abc72c02edd925bb32dc5a994f032e9a856931eb1ca3
+CT = 559b2ff3f5fd147b9889146f9fdea6758e5e0c716395cf1caf577dc2707764833099bda0910626c62bbb1ca010b66c54114982
+AAD =
+Tag = cd2ed4fc
+PT = 178745b297a23a897ec5cfe3a9e373befbdb840d9eb657885ad0423628c4a18f934e6fb57974a52436c517f4463cc5f9370c54
+
+Count = 10
+Key = 981afbf7e7b74f08d186616d1f71b682bccc3cff6c5560696d267ad455d111f6
+IV = a2d07ac3ef29978c44ebbd83e1ad330a8fcfad8213fe2e924390015bc966a944a0a76831189a011094ec4ef98535efeb56b871e7e1aa36748e639dd5f9d1bf3286a1b2965bfc029faa0f855622c30cad67331bd11dbcea51e397185cbc4f0f0341fd8e744d2f09b2e3c2bd03af15850dbe2a701855ed4247f97acf9754f5e4f9
+CT = cbb1f9a5bd84c4b1b8df2714f87db878f7d2658cc7c37f75d784e2157687398a391ecdfd1119e087bd12f6af79db50ae7711bc
+AAD =
+Tag = ad794c97
+PT = 06ea96ad8e6044978ea676056df8c647b7bdfce3923750983cdca875089841612737e6fe078496d77906b9606532b309851cdd
+
+Count = 11
+Key = 03183678896e28b84e16ac41ebb14f4f436efe386ee6df4e8ad2a7aaf11f17d6
+IV = 718ec99fa1b9b1d29a06ca3973d9c0323b14a2cc34cfa2816481aa2da97b435b0a075a2ed6412bb482bea23df9deddd16944492b1756c65138c3d189b8d2d695150667f46edce88755e868a2d90bf13f170d9b6bb29d9210f3c9f507663756866ede0b362aa5c859e15cd96da4f8c7f7852b3924bdf35ff3a515ba5150e1b017
+CT = 6ecd41a492ae5d6295e9c18290c9a36999c79c87f8b69ff20cb42ccb7c6678baaf159c75ecfb15cb87db99a3236734001545d2
+AAD =
+Tag = cad99689
+FAIL
+
+Count = 12
+Key = 2ca8b01d1cbb8d392bae40bd8a51205a9020be27a23533da51dfe1ad0c4c1d41
+IV = 5b5edcc2f17942afb9577c3d2ed7d5ecaf009ac3ebac985fcf1e0fac0dfdfe747fdfe3d05795337baf41cea3b26e4f35caee1c13fc52d1192da145f376b4ea810ce7dc94845a9ca9184203c3b8e803e7a9bfcbf4a310c85b28b04a007e8e9bd14ff0ae28a1966918a6e22ae8415334e7df0d530b0507a24f755f70f117581820
+CT = c420656fb66e89d5b10fbe3ec0929286683ddc4a34cbaca638493f5c09673609814127709b6b1bb765902f6857761a8d57d98f
+AAD =
+Tag = 23f1dc74
+FAIL
+
+Count = 13
+Key = 25cb38a4b7ff73bb632ecae5f75d46e45a108ffaf3ec2d6ad39d3af4b3c64ca5
+IV = 372510e18f877e0f74c1cc54b19d265b27a452cbe91339bb720aa1bdfaf9bbe5365c571ce8f01d2e96aef8bf089c3f4402f186213be72b46b200337c9ebf943bf3d2db1f68c8e655534d9198825737e623745c26f6b0a82585660a7cc3985a271dea9b20f93653701a8d383bbf3155864809decc03ffefc9ce018379d12d8bfc
+CT = ea01ae67abee8f8552ad260ca9d08ea5b35b53667a3455718545e007e5ac0c62c1ff0c5b06f8c031079fce5f2367889a6a068e
+AAD =
+Tag = c6a365f1
+PT = 3bc70116886ed9b4ef795e45c6ec8ea65f6285b3449174f89ceb1294ea73dae9b2f037107f57355be7242abb7da818c98d2755
+
+Count = 14
+Key = 19bb98022f5d140cdbb5b1c02aae8eeec1e96dc6eb489d70967588b6f414330a
+IV = 6d7b41c7f949f8ff3e9e18ff7af3d67eff5ddaa62eefdbc0b0a49dfb6fb07582998250d1c8e609d57510c859333a268f7e89bca06adf1646cdeb2e592bc86769aba402410cbd71f572dbe065beb37d8766ac61c12e7ac322d213407e073d4bb3c28848c42959cab21f9e39d7f4ff8debd50f40bfff96cbf81af07fbffb6bb2b0
+CT = 60a77e3d27fcea5e505221382d82e9ee39c2bfaa01d7d6ce0d293e7fc7bd0d7f900afa9a7f080c33c04cedd76573a914409e39
+AAD =
+Tag = b561ee30
+FAIL
diff --git a/src/test/crypto/gcmEncryptExtIV128.rsp b/src/test/crypto/gcmEncryptExtIV128.rsp
new file mode 100644
index 0000000000..69c74ca6c5
--- /dev/null
+++ b/src/test/crypto/gcmEncryptExtIV128.rsp
@@ -0,0 +1,129 @@
+# CAVS 14.0
+# GCM Encrypt with keysize 128 test information
+# Generated on Fri Aug 31 11:23:06 2012
+
+[Keylen = 128]
+[IVlen = 1024]
+[PTlen = 408]
+[AADlen = 0]
+[Taglen = 32]
+
+Count = 0
+Key = f8fe56171fa546a34b1b28e0b1d31cfb
+IV = 960d57f1336271e069c12f11044dd5a5bea996fc0290d37b5b2f47c8df3ae3ee37214a6871d963b830aec266026364984cfe31eb88c2a6229f5594ca9d3b6d26c7fadb91a0282cdd0a321714b745dd5e161e7cd192420cf2eacd552c4df5cee8fb5f0e06b7c353017b4b9523ce56899db770c344da720327817ba823a8f71382
+PT = fd229158f18f5b8c2a96c86fa3d8084014660eb2314bbab4ca09fa72c3a98b6faa2ebb83a1809de9ccbc8973d23af34014fb27
+AAD =
+CT = 4dc0fd07c86ec84c264f0544456bfee14f688af2109455d73aa1e58a3354727e05387c94568edf352f8342a6156c64d87c44d7
+Tag = ac350afd
+
+Count = 1
+Key = 803007b69146363999afad4433c0f3b5
+IV = 2fc7688faf0b3783094294526ac07c38a73ff961ab41f20deb66edcabfe68e094aa0f1fc52318706a0e2f9b3901a768346494e846cf17b662d05a3788d77c1468408a49ffe5c0cf68b3b8b26193dfd84c63c4631eebf0c7974283e05e39494d9aaad038018a6e999912b1f92681375214e634f5937cb32ccc4face42d013980c
+PT = bb311134866deba57fe506445c5a312ea1ba1ba16469731c1647c6a482bc84fa8349d82bbd01d3edb6cfe8f25b37ff8ec9d621
+AAD =
+CT = aa1d6ae4151aaec1030a6f0297c48e67b84d1c397e9eaf0c5c8c3d252bf8638bf8591342c5c20c7f88f41140b0334f55cb7b04
+Tag = 30076c5b
+
+Count = 2
+Key = bfb96f58535443205ce281e03bdc5c38
+IV = 3961f47e9e800f1c72526d73fe372cd6e69a7eb1d2692c58bf4297bb05503edf95e074a7cf2644981f72421f229d93866a6e1fcf5c13953b39bf36b56fcbd4c09b505e550b5ba0e9bc26efbe0b9621a47b81842caa8c945fa5bb606f0dab824a4bb5a2625668a916e47f0a0d8a995bcae6940e120724f6d53629545da5456008
+PT = 2c07d76ceac2a77809906bebd3452e2f898ea5467d47be1f17573b3f7fc7d11c9b868d1d1a24010b63dabe9c6c6b4e123df559
+AAD =
+CT = ca443acf08d121e1ae4221013ec40dca2237d035e89ae67040602132972417e07a7c770d75d96fbb5b8a38e048abb15bb978d2
+Tag = f61f75ee
+
+Count = 3
+Key = f3a44f15d104f81b4bb263eecf806737
+IV = bbf1b0646991c2b9735066dad5860fcc08a6b92944a5e90dfb120acf2a75403d3175f5e61a1d84b89a0c1bdd3b3414450faf6ffb8820ab1ea01a2b3cc05f1cd1de9bd48ee1308ddc7d87e6db33d3a171e7f63fce6b8e0417359afa833f6b5f293195bedf444ac56103ee0c8706a69c08fe59a95c8474f28a4a12661905f8781c
+PT = 62d05ea0b0ddff1c0418b01a21267230ebbf23b63a6c14caa769c9148150c2454c055cbe4a72a08e7cd8dcb456e1ec17bc3a63
+AAD =
+CT = df08c4d223a168d4dea9445f5b88d40ef0a796de3e77a6a116bfed840ccbe7b988345d070640f62f5757878420b5c50dd9f567
+Tag = 1518c2d1
+
+Count = 4
+Key = ccd58a6017ac344ce5f8ebebdbe03593
+IV = b2c53f013021f494f6637876f62ce5b5dec6d548ffe58d0952aa8fd8fd5c8d2b835165e1b0ca01e72c19f962e38cd4458229a3415d7b4f9afddf5bb63215999b750c07a080677ba4e40f6c5e42038882503c9923a6eb2cf0d3b82f9f94e624f9938830fc22430f16f6c93c362cb3c11cb05d63becdb4c572f03431e7108369c2
+PT = 0b89a455a2470b3b2b7e04afac15c45a0742061494b78f88c57f2505e1f5804f35c0a829f8b6443e427fb6ecd374642217b6f7
+AAD =
+CT = a8b6203b914391c61dbc123efc6902b892107ea9724341a22ecdd0deb16c48d885d606f68724cb43a956c07ef9ef654c042906
+Tag = 02aeaa63
+
+Count = 5
+Key = 50bcf114df40a431c0e1e88033154a25
+IV = 0ffcd9b14c6bf5f630c86b41cb3cf96004f3fa4fd48ee87b7235d34be0be1fdddeaa79abb3c0c9198a1eea0ccf04e8cfc8d24e4badbf438c59a70b435fbf07d44f55b75e5e48fa0f3d7714aaf9e34b430640614646d0008014ad432a464a252c66584c922b29a90cb2c2e4237c8545f913cd2ed3910a4f075062a55b71228411
+PT = 9f5206b2afe824ac4303d58a97255dea6f026b8531651105db9695f09acb16afd9488928060219307fd41edc16e49f01b7d646
+AAD =
+CT = 0143c0a035ce29d1418acb2561dfe88c74f24a9808a8672427797cc0ba01693c2d66c0e365961cfd58fd039bf08fb4c2b1be29
+Tag = 552e27c7
+
+Count = 6
+Key = 8dbf11f923374b8e8be93788de939806
+IV = 2fcb130962ba3e3eb8e31a7a26e2082a643f39d67cce11d8b2ba8a782f63d4df375b21b1fedaf67bd8d73b2208937ae941afc99420ebbe328214fe6a456bf00979d5ebfb22b79fd3cdea81056747bf4e4ded33f2f26f2d228965128a3d0a32696db44e4aff6ca5467d3c749830a5d2e9a41b4ffa3a422e5bb870cee84f4b64af
+PT = 1c57186740c4901e022e63b2e7b085f9cb60c83763e357a591c1968277920ade1987334e88ed9c3c96665e37fe492a975153d0
+AAD =
+CT = ce4c7e5fb87ed02424a13cd2cfcfaacf67f1072198dedc594db26d6453991821861ba6cc843c6d2750e846e162415123b7e01e
+Tag = ab10b040
+
+Count = 7
+Key = 6e88bed99ebe380c1f8297f46019d8a0
+IV = 80fb82c0016d054491f396a7722217b0a07bfd0fc954a7141bc1e2e7958cb24541a21492ec85d3c744489f93ae3abb9af101a78f2366226080389d29eef564d5205f377ab0902043bbf7ba64c30c9d2c945cd6f29654738106dd282194fa02344ec177b5547531061b31cebfac4a2b0f46b68e44c8c89f6942f9c9c13e50c58e
+PT = 6c90ea3dad3e172ae6784cf1b7c08fc04d0f9b4372f1103d11393a8f02e2f495b53c57cf46f1df6f55b2fdb184fd2b7d590402
+AAD =
+CT = f55d67b524b8e633019e9b1736f3db1a254e53cd71fecb48f2dceffa62256a0bfc3775f6506db52db6c6eb91971f5bb5688325
+Tag = b4585815
+
+Count = 8
+Key = 99de57ce03d1db62a751c3b2c7d38f3c
+IV = 2e74c283e216ab5ac9a1b214dd9280431d7ff942a17715fadc3a22978fa1ccc0743f969358baea3d79abe93388a7db82d82ad6a917bf67795fb4360543d7f22f7ee49f41029ebb87573ac03ddb7e279f1846f4a88b85bea63e2c9b9ecf6f91777434b3def0d4a42d3e025eb43a666a28d5f6c834c7e7991897bf051915e646c6
+PT = 72dc7e5533e862efe0d23c62095506b11c9b256d8d18d11511aa1ed4eae67b0017ec74e322f3a7a18e7d199e7093cadf26680a
+AAD =
+CT = af31d880d1820a35c9248ee0b1aa0da31339f90182e60451493bec8d4dec3baf922268741c2717831b8365bebf072aa7931ab4
+Tag = b1700396
+
+Count = 9
+Key = acdc98a8baa0f003c130ce196135334d
+IV = b7c9885d302842c41a2880b9382584806e8cb55b49183b80bc50403c82cfaf0d28a00fef813ba5b7e35b80dc1f0b7c3b0669a3bc739f499d77ed9cc47a467ca62fc34c5bf4c374ff396c01a472c3dfdab394e6926545a1c20363960c72dfad3eebf9a970e6579e3eff7a38f6bcf0373a8494d450d12445f9ff62c233dc1d2379
+PT = 30f1c7fb5fc152dad6911623ca4af1eb495e108ed94b6e6cc19eabeaa7b85262ea3cc4dc5297aa6f7cf504ac6e07db5db550e6
+AAD =
+CT = b2cbae4df8898ca223213824a5c08e16eda81f063916e813bf2d0d8c7e8a75b2d0a9f6de91e08d5422970534331cf1dd53fa8e
+Tag = 8ef08260
+
+Count = 10
+Key = 75d651a377ccdc0e743d73b8205fb38c
+IV = c7296796ef031d372284f7b1481a13862aec243792ca73f40e11bfe39da28984f11d591d3294c833babf05a1f19b603f4f4a9ca1102f201c1405b6cb45facc8ef408541963abcafa907b2eb8e5c1c404b0e4884a48bbb2d43add4dc1c44c526295bcfbd8f2b7041ed49189e835cdf4ed00dcbb450eef4070482f5e8b52360966
+PT = 13ab743d9db511857f0f79a84742c225b3692a9aee8a63697d42fe50d74fe028ccf95e18a2dd2d9b778392ee7a5d2f23b399eb
+AAD =
+CT = 09c3f153ef712fcb3b5fca2b9bf7f25740fb748bec64bc35576ed01682030ae2728d4282140264819e8c4dedd48e29199a6236
+Tag = ae374dd4
+
+Count = 11
+Key = 15414fda594aa87a3e7af69df769adbf
+IV = 7f021a78293e7cb4dd0af221efda3149b0cab87241b597865267cbab5aad530ea4aa4b10815ba9318a45fcb22fd0e6d692d7beecc2042fa2791f6cca5f9916b0bbae79e9d91133aa54d15a1397f8b063695a3d36b8e573a866fc94964f39016e9490c37189cbb0638db09548a91688d73e2c0d4542f5bd08e03ac0d75e36f519
+PT = bc8263ccf50d0224d088546bc16e2577925567ca52d98ec45cb43b190159bbd5f0326d4498a8a88c0ea0b0a79420b906cd5115
+AAD =
+CT = 34b07942793b066b74b7fb8a4ce71a04f9b29e40dca351f5b6e0939bc2819cd95e69bd58163a4df9729e3d6220a6df60a4ba9e
+Tag = bcf4aed9
+
+Count = 12
+Key = 4d73cae96ded98e1688104a63f462c76
+IV = 331b51fd88cd4731e0eca051717b642f86ea6d6941f9a7331ff361e0edc3f9ea4c013d585f3eff70004a696c4b51d7c589ca97e5fd30f4b2f99c0f3ac83769c2397e10669b7b83aef714a2388638b8941efde8e631098dd78742772f484edd568fefe26b9d981b437e4e4f3ad25445e1aa8c8608c655a5090d2cac158ee67e98
+PT = bf55d48ae5015c39bb167782cab391510d7b7d698e9d3faacd7d409fe4d86fea0a6f61128b6aa305ac0fe4cee4d582dd30e717
+AAD =
+CT = 3cf29f4d50feac5cda4c7cef91d563b2573096b7f32c723355d8d59b8e1a0e229dc2f6114f1db6bdc26212043d153709597c1d
+Tag = 76fcb529
+
+Count = 13
+Key = 7045a9d7caae2c0a39f58998720974ec
+IV = a9028008f708ae19dd28e75a02c6e84a05096779781ce0a908047152208468cc4b2a57d25608767f936cf70fbe82dd8a61497a180e1fc967caf4e6310ed850082e6919c922e021ec070dc64b040ed9edbbf5883676630d69e48953068b2bd006bd6d5417038604ee5aed04980c9ed2316c531ce6a3a73dfad90c04c58596d5e9
+PT = 635b946047c38533bb2cb4c9799b44f6eae0e63626901b0741f6dcf3c2bb02270343c7708a72dfe303b20f7805cb732386b341
+AAD =
+CT = aebb8c1e914ea9bc1ada9b01f84cc8dbbc611f2cd386d5fc89497d37e5a469b28fe2fdf0ab0f1c882dcce50620b1b18a2d8343
+Tag = 27bcaf06
+
+Count = 14
+Key = 79d2a35efbf03f57b66e875c232e10d5
+IV = 7cc96b48afff401adf4bab2ebedd021377b18a819f3c3af39fda42e24c5d62e67ec30f8bcab00263dc5a9bb06cbfe1750c98555901d34d775fdcc86841bd08fbeb44ba68ee794dc351a29a1a9de576d83c17a730d50db79cab88d538a441bb9ff6aa073a2a976de820ab5cc61f834753220d4e472a275dcd13db3e51a23a84a5
+PT = d91e1b2811b3e3894b46c563e6ea0b4a33990ba4fce8a354c941e1effc5691671de5d97c4c1a35e3730b43584944695f00544a
+AAD =
+CT = 6a45beb05c0dba6c38c997f8c37ef07c7cf78eacff6ff4dd6fa000e745e8053d2d270a746994f29c8628f41fc7fecdac158655
+Tag = 36caee75
diff --git a/src/test/crypto/gcmEncryptExtIV256.rsp b/src/test/crypto/gcmEncryptExtIV256.rsp
new file mode 100644
index 0000000000..3faa8956bb
--- /dev/null
+++ b/src/test/crypto/gcmEncryptExtIV256.rsp
@@ -0,0 +1,129 @@
+# CAVS 14.0
+# GCM Encrypt with keysize 256 test information
+# Generated on Fri Aug 31 11:26:26 2012
+
+[Keylen = 256]
+[IVlen = 1024]
+[PTlen = 408]
+[AADlen = 0]
+[Taglen = 32]
+
+Count = 0
+Key = 0df25c2bc9444b4a01e26d357a3ac0635fb6ff2e65ce1e759aae491a17772243
+IV = 54652573ba189cffed3bcfa60efbfb417eb4b0e8de80c7e53765d018cbddfae74617269eb35f29faf628d28c40737f9d9e1eb0b8757c984d94340ecc5ddd108f0a5a0e96335ea805950d378fea7569b98693e88bc1cfd9f6eb8d25de177122fc774a5b1957bcb80e92230c12fe401a8e00d0c04897e234644bda36ae761ea619
+PT = b29af460c6a5dbe56f1d67751346d7182c93413a6c328c6d85176cd8dea8ecce1cad3063c8708c0be9ae73d42bbb10421e73f1
+AAD =
+CT = 3c786a3d0c8945bf320a21ca63f3b8bf5c6bf56a8412f836d7894e42c9e0695a8e41bf59b23fc52d17f8b341183f1cdce02e22
+Tag = 0b31bffe
+
+Count = 1
+Key = 1b6b7b8d00e543f0a17bd0bf595319a4a1f8a55ff41ce2380381d4e83c83243b
+IV = 791d9546f180b838b50bb7b66b68f7f1aa09a2cef411a1dbf2e64c5ed026613200ac8f0e5b961925853621a1d4339322ea4b7bbc4adcbc008efeaaee0dc948a916b22ff8693a7d441620d0ff67680b23567f9582e22eec529408c6d0a00de1bd1ee5a11ed7fa29b7567f990e412ce90ee12d5b1d8ba8c2b528d0d9963fdc49a5
+PT = 0a76090de676d71d9f5ee8511864f0c9440a0fa12b5155a5bffc36127bc957b293d4fb624af3b956385783124f28f6f3c0f0e8
+AAD =
+CT = fdaa9ef2c65d7666b61367f843863a3b273249192551b5633cfaf84a5ab9ecba42916395177f9a16c1ad385e77393cd93d71b3
+Tag = fd854d63
+
+Count = 2
+Key = 261a0382c739093634502303d60dd0d2a568b5155147d661e7789f7bd70de82c
+IV = e22330a2981b9b60354a740c49dcd17c9016cff50423977f7fc8500fc36a81610c979eb37a1f9c4d54d11b790906492205263178dca6d269a230a595ee95edeb60e45d92a2d6169877dfe5514b23db143dbcf3ca44e13cf5b7402e4f95a9e6760451be2d57d1d5fefb015ffe4b69456d87338865ee8775d1cce238f75c345dad
+PT = 42c8c79988c52bcf38865a6f341a5165294af0947bc4fa597c6f648a1172070851c4ee154604d9a21c8d53f7eba77fbb72cf2b
+AAD =
+CT = 0fb368563c6ad0c097207239a791bc685274d434c4a54cb0ea97a98b4cee9eb100c4cd626ba82c338a3f51cb29391b1da3a44f
+Tag = a9607e42
+
+Count = 3
+Key = 5f3d6b5c76cb7ad0be3d771f1f7d1bafd88c8c7c7c84922f89f45f3b7453eb44
+IV = 7072d302e4acb642090c5d48e4549a8823725be1389316df0152e68c41f77937eadc4ee1d164101717cdcdd3bc9be3c2669c0b1394953090ac787fb117500b6275122f608e70a3f5063b21fae42ca4b00724b21c50e27c37a77d4befb118018ada7999888442e1271410f6b804c36a27a41e95a438792de4f06fd7122b177cc3
+PT = 00d82cc00c07d3cf0fd9b31ddc091d351aab9b58af07d2c59f3e28c7202ef6e3ec35726cbdd14f8f1985a770d092470f115be7
+AAD =
+CT = 0aab0dbb31c113b33b116659294e0af5497c987af870ae8ce55288d48e8a4f1b3e8377cb34d06460daf3cc99fbefb90724a39f
+Tag = 26e56b13
+
+Count = 4
+Key = 1d4b6b6ba43269d46ca5c0fab38b61f1d2bf3d24dae46181fac73420a8e6194f
+IV = 01c283a90c588dafc585f437dc111e94dd8ea98a622a5d2e554a87086ec10e8dac9b205fae70be3c024b3f9fbdc26208a6e44d082ad92a51a7fa1f861d0e93e7e74d4e41426e70b2e7ef7fbcbc5a302aed4bc8a42963ca53334259f1924e74225c3bf9aa2ffdf97a2be6474c72f6dedba7c454e36fbc3537596a63b61a4cf3ce
+PT = 4eb036d945c14de26c4e0c83d446dacb57e91b5bdec2ef612bbed8135e57ce62b28843fbac555580ef23c74959eb869a017872
+AAD =
+CT = 24d67f2e94294255b9fd90c16b67c7a8b3a47a42782954aa15e80ffe732d64342b1ec65aa66fc4d6b8fe9dfcdbf4d4c1a979e7
+Tag = 37358e86
+
+Count = 5
+Key = cbfbb15a4fae4af3de55ffcaa4b8cdbd515ecc72fd50060b22acc8fad57a5f4b
+IV = 86e47ba98d8ed360262f94afc364df373d1e44c788b4ee5ed7542822858aaeca2b3c07b5464d6f7edefca9759107bbb64c086d526cc4c9a6b9a7cd6dbb50acec2297597612fd436d8c8ddbc83664305b214cf2c3b6fab3e545c499c7f1cbba1897041475f94952d8a4fc10110986b5ccc7a10c1d9e6427dabba8d942ada9290a
+PT = 5470a60f2818ffef374be3cc591084096a4a8caa0dac0024c12d304382c301264526b02efb674dfbcc3be5c818a1d88a7c19f1
+AAD =
+CT = a6a8ebadfe5c5ac754e4a2a9cdc25fdf3588c69a287b0f9aba2347bfcdc110a1134426a45c62d577e52a8d61fae39ea38bad51
+Tag = 97406b4b
+
+Count = 6
+Key = 3ced5c63078a36f6b02d3bec0debacd19021c8dbc501a9d86f556c4ad2cdef8a
+IV = b9629801cba22f7493b4f6394620d49a76ec686e524f5ebcb3a76b5a189473484060cbb01bcbf10048427ef21527626085c8a75aa5264b6338abbc26171c2c3a44f6b3b5c3fd05a892c8290a8f99be962deea48d7ae4e626c45a45ffda5efaad6e54e98ba876b039a5dde3d6061f217e57da4774acfaf1f5da9495083aad4dad
+PT = c29788107799077ac6dbccad29a346727f263676a8510fbcadb9b5bf53df978b3382fdfc1b5c3312eaa0f7621b6efdaeebe1c0
+AAD =
+CT = 5811d4a96169b39cd8f8ff3f931efdda68550f17558b48de8bb5adf455af5179c8c5ff4a73f363f8819daf846427132daf6a17
+Tag = 06e8e7fa
+
+Count = 7
+Key = 39ae4fdfe56b74a71325a1a685a6593b44936892890cd05d2719a420c97a7c64
+IV = 8f110d0fd20ece7f35c35b2eed32316fd742ae9346fc6907bd749361a4436427f80185b376b16a36fb95429cfaf2e22e46c210442fe5efc14985a9d9d847c3ceb5db02e0d999acbc3ba0afdfecbfaf65024258cc7f6fc8e3f568cbedec1c7eabe3ff3ab3c7331722b6400429d46b54820bd1f88ac03cfba5cbd0812d91342c4b
+PT = 8b086d11be7ff55312addad86b49585ea38ea1ee7c4200964cd269a4bb5cfe0f518e6f9b733efb4ba3ce35ce2e803b0ad47d24
+AAD =
+CT = be46d6e61dc11b2fc8ac4c9c5f49ceca0fb6fe1fc7221c7cc5d8ba254a92282500b1b31528314035cd125578de960b3bafb69d
+Tag = 7c00aa14
+
+Count = 8
+Key = c244e9afd90ec810acb7e586cf7a06386e6892e01c7d111d5c1455fc95250d1b
+IV = 6ff9103f9751ff4743d856c5cd54491e1a537384260fbfe076f772ad0d66ff6ddd0aaff57023885b0a4d60d2b25c80b1cdc54802607770a61a2503c23cf26f1fe529573c8d4745b19dedaff5769a6a796c01540776d4fa99be9057ad87cdc973e7938640f9497753e88c4cdc358c1cfa06f1ddd826c33f44c55e2d183927baaa
+PT = cd1296133cefe6f5cf6f8ec68b37172bfb793a8582d92a539f24f3582569bfcfebb706fe9f276716b185558fbfd6fe8ea99a1b
+AAD =
+CT = dd1694b178f15612c454885b3100c576c7b68206c57898161d4fdc51e75a428840c5cfee104c3d85fcbd92edbd1d80f22d8e64
+Tag = 741953f5
+
+Count = 9
+Key = 785ea9cd2403ecaeec3e4940dce7c41ec012203a2610c780bca5d15af64748c5
+IV = fbc177ec9d47f69e2fcaba9fef7de30735e46a5d20dc66bf66c8a76a382051d780f58dccae8e2054bbe437a5bc0814381bac2b0efa99202aa1f1bf7f51b842907dc9b60f83987c31eae086e26e2018243bdd47a291b523ac6905b40dfb442ba239c876cdfaa581b2ef0683456ac944829a97b663ecd48c116d06f1f054ddfcdb
+PT = a3465ad9a4009e26c39acb1d424b7ba6556a74dcc2b78a5ab65b5d07c2a97f382aadb7415395fa6fc90bd137f6894a75f70907
+AAD =
+CT = 2b3eb5ab251282da1795a25f9b43ccaa5a27643d042fc315ad662947b0c5cddeb5848e6c69869ae5e81f5c76729bbaceda9889
+Tag = 64173526
+
+Count = 10
+Key = 0becf9c7a069db5b9a25f2871fb0594e452126262ec1c48bca3d3024d85d0c51
+IV = e05643a72ab0b1fc42a6f17302d6446fc507bbb4f0ad59178f5084530f02534df2e673e92e67802629f93221bba545a13fb9143eb2a2ee4bce047be621a9d96a450a19951d93c527eb698ac7e132e4f00985dff91e079ae791549d37da3105c77ef5d8cb7649f1aa761b5a5fbc9e0d7dfb5aade98f3df4a641cca02f33eed55c
+PT = 70ba9822801760e5ec647a41e27cf6069978be6a28be0b1da0f3661e124847823dec7ec6af737d6dea597ce5bd5baa5d6f2651
+AAD =
+CT = 0d45b5ad77226524374a3bca3680528787cefd9978b3f1407688886ce28fea86e7d9005d7d6acde2a7c2fb158479918c06bba4
+Tag = 43dfd859
+
+Count = 11
+Key = d1f374702b4481b83e7c7783853c1850e887b0c80cd28c0686b4cab6adc744c2
+IV = 5e15c44cef36c1c89e02b2979f4e7d275fe5973a80580753f6fd51ea477ba84346a030b90b22a9adfff02096bc0b1691c37ee21cdc1b5f4862696d195859cd2bf0b423f5d19da5e1475bcc99f96b8c7c51fe85930aea0e97304d4c025b52bd386666d0537036360e939a6fbdfd6ce3b012e551d333fa74f3cbb9d33a59477364
+PT = 996428a232b1b9ac81ff5260eb77938f0d531a4a0ed3bd774c72c18128af72e964f8d05fb7ec4e0ac8e37056a48c85713a3a01
+AAD =
+CT = aa9c103ae51bf556f7c57f9948d28c859f458a74e22f039fea183e9e262b023bdac9bb5a2f167b34cde4a694c555eb0c905f34
+Tag = d1ed9cf0
+
+Count = 12
+Key = a6a69524d66f015b4afb746ad0942410baf06d1aba18ef1a8b40e35633f04ce1
+IV = 5e1df758c63728b1269a1d3f610b6e3724bc7ff797dcb4a7aca9dfe1c8e1717ce4281d1c5b4b33d0aafcee342a4f4eb30089eeab2b0470d3f9b709622bde4654d41b3bbc6bf59c11edea28f26b099d83d4fefdc63ab7218221238fe1c230b1290235465b445f60eaa9822eaa8da2c08f6f2fdaaef87dacf74c7e5d6cea191c74
+PT = 0ffc6c3090817fc1a0e6f3802269e263e40d17772fabf5cbee905962878d77c8bc4223e5671bd9f310d8db56ceaebae41fc79e
+AAD =
+CT = f24537db3fe27eb07f1d2b9b7059dfae97df86d4e609930491c4ec3154462df308ba1c85204dd754521d8de9619603f90f8c44
+Tag = 360adc36
+
+Count = 13
+Key = ff0007317f9ecc43c1601708614eb8d443a407318cbf3c085c0a9a7c67faafa8
+IV = 8186f92bd394ecfbf4bcbb9e7e442bd6faa1e59f8785d9aec82551fee38aa10212a8477be3055927379a906902c153598d50c63e316c4e4f3956ef04ce40e5b6f1901fb5d08a23b913e5ad53512b02bb75ce96f8c50a2cebdfa058ded996fe1dc5370ee50c6d90e948129d544dc89c2f28dda8429bb338d6aed0d9557f96889e
+PT = e7f4063319fd31e07192a5fc9e486fc0a2a3470671a46356ab0d32989803259c0dae103a4033c533fbde585866fc2af5eba151
+AAD =
+CT = 8f86644614e4f6483f28eec02fab33e7cbbd56381a8f878522c2015b91d48652472e356f608c62361939dc983e1cd364a28665
+Tag = 83335b3a
+
+Count = 14
+Key = 43646d9ddebec23447febe71596e6f9b2387965db1faf3f06feec8bc5c808342
+IV = 0d3774607261b07427dc77f0dcf57b026226dbd2c1df1b9d74598582da44c677af36a6bd80b6bc1f000632c84c5dfa701f8b51b4da228d340d8b4ccc4d2f5d7b5fad00809133eae9250ecd18d7a8741bb57c394396feb81c20bec23de8520f883b8e22b362dde6e6bd9dad73e8919695384a04c09a28bdb6de2fa356b0d259b1
+PT = 2aab9a484b817b759bd9d876967c90160a18208cfa753e7bccd4f73a715aaa6acc6ce666e97bc22fbcf11f263dfed332418707
+AAD =
+CT = dd80974e31ad15ba26ec8a8dacfa5f57f0bc69f7113cc6c39fb038ddce37ffeee3f789a02cd86b0418fe16ca104b5fbdb26432
+Tag = 75c35f4c
diff --git a/src/test/crypto/t/001_testcrypto.pl b/src/test/crypto/t/001_testcrypto.pl
new file mode 100644
index 0000000000..586a37c5bd
--- /dev/null
+++ b/src/test/crypto/t/001_testcrypto.pl
@@ -0,0 +1,137 @@
+# Reads and parses .rsp files and runs them through testcrypto
+#
+# Test vectors downloaded from:
+#
+# https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/cavp-testing-block-cipher-modes
+#
+# Specifically GCM Test Vectors (SP 800-38D):
+#
+# https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
+#
+# Note that the AADlen > 0 cases were removed from our set, since we don't support that and
+# the test code will just skip them currently, and we also don't bother testing 192-bit,
+# but one could download the full set and run all of the files through if they wished and
+# all should pass (they did when this test suite was written originally).
+
+use strict;
+use warnings;
+use TestLib;
+use Test::More;
+
+if ($ENV{with_ssl} eq 'openssl')
+{
+ plan tests => 56;
+}
+else
+{
+ plan skip_all => 'SSL not supported by this build';
+}
+
+
+# XXX add CTR tests here
+
+my $algo = "AES-GCM";
+my @rspfiles = (
+ "gcmDecrypt128.rsp", "gcmDecrypt256.rsp",
+ "gcmEncryptExtIV128.rsp", "gcmEncryptExtIV256.rsp");
+
+note "running tests";
+
+foreach my $rspfile (@rspfiles)
+{
+ open(my $in_rspfile, '<', $rspfile) || die;
+ my %testrun;
+ my %lengths;
+
+ while (my $line = <$in_rspfile>)
+ {
+
+ chomp($line);
+
+ # Remove CR, if it's there.
+ $line =~ s/\r$//;
+
+ # Skip comments
+ if ($line =~ /^[[:space:]]*#/) { next; }
+
+ # If we hit a blank, time to run a test
+ if ($line =~ /^[[:space:]]*$/)
+ {
+ if (%testrun)
+ {
+ my @testargs;
+
+ # Set up the command to run
+ push(@testargs, ("$ENV{TESTDIR}/testcrypto", '-a', $algo));
+
+ if ($testrun{'Key'})
+ {
+ push(@testargs, ('-k', $testrun{'Key'}));
+ }
+
+ if ($testrun{'IV'})
+ {
+ push(@testargs, ('-i', $testrun{'IV'}));
+ }
+
+ if ($testrun{'CT'})
+ {
+ push(@testargs, ('-c', $testrun{'CT'}));
+ }
+
+ if ($testrun{'AAD'})
+ {
+ # Don't currently support AAD
+ undef(%testrun);
+ next;
+ }
+
+ if ($testrun{'Tag'})
+ {
+ push(@testargs, ('-t', $testrun{'Tag'}));
+ }
+
+ if ($testrun{'PT'})
+ {
+ push(@testargs, ('-p', $testrun{'PT'}));
+ }
+
+ if ($testrun{fail})
+ {
+ command_exit_is(\@testargs, 1,
+ "Run $testrun{Count} of Keylen: $lengths{Keylen}, IVlen: $lengths{IVlen}, PTlen: $lengths{PTlen}, AADlen: $lengths{AADlen}, Taglen: $lengths{Taglen}"
+ );
+ }
+ else
+ {
+ command_ok(\@testargs,
+ "Run $testrun{Count} of Keylen: $lengths{Keylen}, IVlen: $lengths{IVlen}, PTlen: $lengths{PTlen}, AADlen: $lengths{AADlen}, Taglen: $lengths{Taglen}"
+ );
+ }
+ undef(%testrun);
+ undef(%lengths);
+ }
+ else
+ {
+ next;
+ }
+ }
+
+ # Grab length information, just to have.
+ if ($line =~ /^\[([A-Za-z]*) = ([0-9]*)]$/)
+ {
+ $lengths{$1} = $2;
+ next;
+ }
+
+ if ($line =~ /^([A-Za-z]*) = ([a-f0-9]*)$/)
+ {
+ $testrun{$1} = $2;
+ }
+
+ if ($line =~ /^FAIL$/)
+ {
+ $testrun{fail} = 1;
+ }
+ }
+}
diff --git a/src/test/crypto/t/002_testkwp.pl b/src/test/crypto/t/002_testkwp.pl
new file mode 100644
index 0000000000..25911cfd9f
--- /dev/null
+++ b/src/test/crypto/t/002_testkwp.pl
@@ -0,0 +1,126 @@
+# Reads and parses .rsp files and runs them through testcrypto
+#
+# (Partial) Test vectors downloaded from:
+#
+# https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/cavp-testing-block-cipher-modes
+#
+# Specifically Key Wrap Test Vectors (SP 800-38F):
+#
+# https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/kwtestvectors.zip
+#
+# We don't include the 192-bit tests, though they all worked when this test suite was written.
+# We also don't include the _inv tests as those aren't supported in OpenSSL yet.
+
+use strict;
+use warnings;
+use TestLib;
+use Test::More;
+
+if ($ENV{with_ssl} eq 'openssl')
+{
+ plan tests => 20;
+}
+else
+{
+ plan skip_all => 'SSL not supported by this build';
+}
+
+
+my $algo;
+# my @txtfiles = ("KWP_AD_128.txt", "KWP_AD_192.txt", "KWP_AD_256.txt", "KWP_AE_128.txt", "KWP_AE_192.txt", "KWP_AE_256.txt");
+my @txtfiles =
+ ("KWP_AD_128.txt", "KWP_AD_256.txt", "KWP_AE_128.txt", "KWP_AE_256.txt");
+
+note "running tests";
+
+foreach my $txtfile (@txtfiles)
+{
+ open(my $in_txtfile, '<', $txtfile) || die;
+ my %testrun;
+ my %lengths;
+
+ if ($txtfile =~ /^KWP_/)
+ {
+ $algo = 'AES-KWP';
+ }
+
+ while (my $line = <$in_txtfile>)
+ {
+
+ chomp($line);
+
+ # Remove CR, if it's there.
+ $line =~ s/\r$//;
+
+ # Skip comments
+ if ($line =~ /^[[:space:]]*#/) { next; }
+
+ # If we hit a blank, time to run a test
+ if ($line =~ /^[[:space:]]*$/)
+ {
+ if (%testrun)
+ {
+ my @testargs;
+
+ # Set up the command to run
+ push(@testargs, ("$ENV{TESTDIR}/testcrypto", '-a', $algo));
+
+ if ($testrun{'K'})
+ {
+ push(@testargs, ('-k', $testrun{'K'}));
+ }
+
+ if ($testrun{'C'})
+ {
+ push(@testargs, ('-c', $testrun{'C'}));
+ }
+
+ if ($testrun{'P'})
+ {
+ push(@testargs, ('-p', $testrun{'P'}));
+ }
+
+ if ($testrun{fail})
+ {
+ command_exit_is(\@testargs, 1,
+ "Run $testrun{COUNT} of Plaintext Length: $lengths{'PLAINTEXT LENGTH'}"
+ );
+ }
+ else
+ {
+ command_ok(\@testargs,
+ "Run $testrun{COUNT} of Plaintext Length: $lengths{'PLAINTEXT LENGTH'}"
+ );
+ }
+ undef(%testrun);
+ undef(%lengths);
+ }
+ else
+ {
+ next;
+ }
+ }
+
+ # Grab length information, just to have.
+ if ($line =~ /^\[([A-Za-z ]*) = ([0-9]*)]$/)
+ {
+ $lengths{$1} = $2;
+ next;
+ }
+
+ if ($line =~ /^([A-Z]) = ([a-f0-9]*)$/)
+ {
+ $testrun{$1} = $2;
+ }
+
+ if ($line =~ /^COUNT = ([0-9]*)$/)
+ {
+ $testrun{COUNT} = $1;
+ }
+
+ if ($line =~ /^FAIL$/)
+ {
+ $testrun{fail} = 1;
+ }
+ }
+}
diff --git a/src/test/crypto/t/003_clusterkey.pl b/src/test/crypto/t/003_clusterkey.pl
new file mode 100644
index 0000000000..5b8a255ace
--- /dev/null
+++ b/src/test/crypto/t/003_clusterkey.pl
@@ -0,0 +1,93 @@
+# Test cluster file encryption key managment
+#
+
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+if ($ENV{with_ssl} eq 'openssl')
+{
+ plan tests => 6;
+}
+else
+{
+ plan skip_all => "tests cannot run without OpenSSL";
+}
+
+# generate two cluster file encryption keys of random hex digits
+my ($rand_hex, $rand_hex2);
+$rand_hex .= sprintf("%x", rand 16) for 1 .. 64;
+$rand_hex2 .= sprintf("%x", rand 16) for 1 .. 64;
+
+# initialize cluster using the first cluster key
+my $node = get_new_node('node');
+$node->init(
+ extra => [
+ '--file-encryption-method', 'AES256',
+ '--cluster-key-command', "echo $rand_hex"
+ ]);
+
+# Set wal_level to 'replica'; encryption can't use 'minimal'
+$node->append_conf('postgresql.conf', 'wal_level=replica');
+
+$node->start;
+
+# check encryption method
+my $file_encryption_method =
+ $node->safe_psql('postgres', 'SHOW file_encryption_method;');
+ok($file_encryption_method eq 'AES256', 'file_encryption_method is valid');
+
+# record pg_proc count
+my $old_pg_proc_count =
+ $node->safe_psql('postgres', 'SELECT COUNT(*) FROM pg_proc;');
+ok($old_pg_proc_count > 0, 'pg_proc count is valid');
+
+# create permanent table
+$node->safe_psql('postgres',
+ 'CREATE TABLE perm_table (x) AS SELECT * FROM generate_series(1, 100);');
+
+# create unlogged table
+# Non-permanent tables like unlogged tables use a special nonce bit, so test those here.
+$node->safe_psql('postgres',
+ 'CREATE UNLOGGED TABLE unlog_table (x) AS SELECT * FROM generate_series(1, 200);'
+);
+
+# We can run pg_alterckey and change the cluster_key_command here
+# without affecting the running server.
+system_or_bail(
+ 'pg_alterckey',
+ "echo $rand_hex",
+ "echo $rand_hex2",
+ $node->data_dir);
+
+$node->safe_psql('postgres',
+ "ALTER SYSTEM SET cluster_key_command TO 'echo $rand_hex2'");
+
+$node->stop;
+
+# start/stop with new cluster key
+$node->start;
+
+# check encryption method
+$file_encryption_method =
+ $node->safe_psql('postgres', 'SHOW file_encryption_method;');
+ok($file_encryption_method eq 'AES256', 'file_encryption_method is valid');
+
+# check pg_proc count
+my $new_pg_proc_count =
+ $node->safe_psql('postgres', 'SELECT COUNT(*) FROM pg_proc;');
+ok($new_pg_proc_count == $old_pg_proc_count, 'old/new pg_proc counts match');
+
+# check permanent table count
+my $perm_table_count =
+ $node->safe_psql('postgres', 'SELECT COUNT(*) FROM perm_table;');
+ok($perm_table_count == 100, 'perm_table_count count matches');
+
+# check unlogged table count
+my $unlog_table_count =
+ $node->safe_psql('postgres', 'SELECT COUNT(*) FROM unlog_table;');
+ok($unlog_table_count == 200, 'unlog_table_count count matches');
+
+$node->stop;
diff --git a/src/test/crypto/t/004_buffers.pl b/src/test/crypto/t/004_buffers.pl
new file mode 100644
index 0000000000..a1379b4772
--- /dev/null
+++ b/src/test/crypto/t/004_buffers.pl
@@ -0,0 +1,157 @@
+# Test cluster file encryption buffer management
+
+# This tests that an encrypted server actually encrypts the database
+# files. It does this by checking for strings in the database files in
+# both non-encrypted and encrypted clusters. We test a system table, a
+# permanent relation, and a unlogged/non-permanent table.
+# (Non-permanent relations use a special nonce bit, which is why we test
+# it here.)
+
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+if ($ENV{with_ssl} eq 'openssl')
+{
+ plan tests => 17;
+}
+else
+{
+ plan skip_all => "tests cannot run without OpenSSL";
+}
+
+my %file_match_count;
+
+sub get_cluster_file_contents
+{
+ my $node = shift();
+ my %relnode;
+
+ # get postgres database oid
+ my $postgres_db_oid = $node->safe_psql('postgres',
+ "SELECT oid FROM pg_database WHERE datname = 'postgres';");
+ ok($postgres_db_oid != 0, 'retrieving postgres database oid');
+
+ # get pg_proc relfilenode
+ $relnode{pg_proc} =
+ $node->safe_psql('postgres', "SELECT pg_relation_filenode('pg_proc');");
+ ok($relnode{pg_proc} != 0, 'retrieving pg_proc relfilenode');
+
+ # create permanent table
+ $node->safe_psql('postgres',
+ "CREATE TABLE perm_table (x) AS SELECT 'aaaaaaaa' FROM generate_series(1, 100);"
+ );
+
+ # get permanent table relfilenode
+ $relnode{perm} = $node->safe_psql('postgres',
+ "SELECT pg_relation_filenode('perm_table');");
+ ok($relnode{perm} != 0, 'retrieving permanent table relfilenode');
+
+ # create unlogged table
+ $node->safe_psql('postgres',
+ "CREATE UNLOGGED TABLE unlog_table (x) AS SELECT 'bbbbbbbb' FROM generate_series(1, 200);"
+ );
+
+ # get unlogged table relfilenode
+ $relnode{unlog} = $node->safe_psql('postgres',
+ "SELECT pg_relation_filenode('unlog_table');");
+ ok($relnode{unlog} != 0, 'retrieving unlogged table relfilenode');
+
+ my $file_contents =
+ slurp_file($node->basedir .
+ '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{pg_proc});
+ # () converts to list context
+ $file_match_count{pg_proc} = () = $file_contents =~ m/pg_[a-z]{3,}/g;
+
+ $file_contents =
+ slurp_file($node->basedir .
+ '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{perm});
+ $file_match_count{perm} = () = $file_contents =~ m/a{8,}/g;
+
+ $file_contents =
+ slurp_file($node->basedir .
+ '/pgdata/base/' . $postgres_db_oid . '/' . $relnode{unlog});
+ $file_match_count{unlog} = () = $file_contents =~ m/b{8,}/g;
+}
+
+#
+# Test with disabled encryption
+#
+
+# initialize cluster
+my $non_encrypted_node = get_new_node('non_encrypted_node');
+$non_encrypted_node->init();
+
+$non_encrypted_node->start;
+
+# check encryption is disabled
+my $file_encryption_method =
+ $non_encrypted_node->safe_psql('postgres', 'SHOW file_encryption_method;');
+ok($file_encryption_method eq '', 'file_encryption_method is valid');
+
+get_cluster_file_contents($non_encrypted_node);
+
+# record pg_proc count
+my $query_pg_proc_count = $non_encrypted_node->safe_psql('postgres',
+ "SELECT COUNT(*) FROM pg_proc WHERE proname ~ '^pg_[a-z]{3,}';");
+ok($query_pg_proc_count > 0, 'pg_proc count is valid');
+
+# check pg_proc count
+ok($file_match_count{pg_proc} != $query_pg_proc_count,
+ 'SQL/file pg_proc counts match');
+
+# check permanent table count
+ok($file_match_count{perm} != 100, 'perm_table count matches');
+
+# check unlogged table count
+ok($file_match_count{unlog} != 200, 'unlog_table count matches');
+
+$non_encrypted_node->stop;
+
+
+#---------------------------------------------------------------------------
+
+#
+# Test with enabled encryption
+#
+
+my $rand_hex;
+$rand_hex .= sprintf("%x", rand 16) for 1 .. 64;
+
+# initialize cluster using a cluster key
+my $encrypted_node = get_new_node('encrypted_node');
+$encrypted_node->init(
+ extra => [
+ # We tested AES256 in 003, so use AES128
+ '--file-encryption-method', 'AES128',
+ '--cluster-key-command', "echo $rand_hex"
+ ]);
+
+# Set wal_level to 'replica'; encryption can't use 'minimal'
+$encrypted_node->append_conf('postgresql.conf', 'wal_level=replica');
+
+$encrypted_node->start;
+
+# check encryption method
+$file_encryption_method =
+ $encrypted_node->safe_psql('postgres', 'SHOW file_encryption_method;');
+ok($file_encryption_method eq 'AES128', 'file_encryption_method is valid');
+
+get_cluster_file_contents($encrypted_node);
+
+# Because the files are encrypted, we should get zero matches for all
+# comparisons below. However, technically the encrypted data might
+# match the desired string, so we allow one such match.
+
+# check pg_proc
+ok($file_match_count{pg_proc} <= 1, 'pg_proc is encrypted');
+
+# check permanent table
+ok($file_match_count{perm} <= 1, 'perm_table is encrypted');
+
+# check unlogged table
+ok($file_match_count{unlog} <= 1, 'unlog_table is encrypted');
+
+$encrypted_node->stop;
diff --git a/src/test/crypto/testcrypto.c b/src/test/crypto/testcrypto.c
new file mode 100644
index 0000000000..85bf921c09
--- /dev/null
+++ b/src/test/crypto/testcrypto.c
@@ -0,0 +1,458 @@
+/*-------------------------------------------------------------------------
+ *
+ * testcrypto.c
+ * A utility to test our encryption / decryption routines.
+ *
+ * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/test/crypto/testcrypto.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#define FRONTEND 1
+
+#define EXITSUCCESS 0
+#define EXITDECRYPTFAIL 1
+#define EXITFAILURE 2
+
+#include "postgres_fe.h"
+
+#include <signal.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/stat.h>
+
+#include "common/hex.h"
+#include "common/cipher.h"
+#include "common/logging.h"
+#include "getopt_long.h"
+#include "pg_getopt.h"
+
+static const char *progname;
+
+static void
+usage(const char *progname)
+{
+ printf(_("%s tests encryption/decryption routines in PG common library.\n\n"), progname);
+ printf(_("Usage:\n"));
+ printf(_(" %s [OPTION]\n"), progname);
+ printf("\n");
+ printf(_(" Performs one encryption and one decryption run.\n"));
+ printf("\n");
+ printf(_(" Encrypts the provided plaintext (or the empty string if none given) and generates a tag, if using AES-GCM, using the key and IV given.\n"));
+ printf(_(" After encryption, compares provided ciphertext to resulting ciphertext.\n"));
+ printf(_(" Compares provided tag, if any, to resulting tag.\n"));
+ printf(_(" If no tag is provided, then the tag created during encryption is used during decryption.\n"));
+ printf("\n");
+ printf(_(" Decrypts the provided ciphertext (or the empty string if none given) using the key, and IV + tag given if using AES-GCM.\n"));
+ printf(_(" After successful decryption (requires tag to match for AES-GCM), compares provided plaintext to resulting plaintext.\n"));
+ printf(_(" Exits with '1' if decryption fails.\n"));
+ printf("\n");
+ printf(_(" Exits with '2' for any other failure.\n"));
+ printf("\n");
+ printf(_(" Key is always required, IV is required for AES-GCM mode.\n"));
+ printf("\n");
+ printf(_(" Algorithms supported are AES-GCM and AES-KWP.\n"));
+ printf("\n");
+ printf(_("\nOptions:\n"));
+ printf(_(" -a, --algorithm=ALG Crypto algorithm to use\n"));
+ printf(_(" -i, --init-vector=IV Initialization vector to use\n"));
+ printf(_(" -k, --key=KEY Key to use, in hex\n"));
+ printf(_(" -p, --plain-text=PT Plain text to encrypt\n"));
+ printf(_(" -c, --cipher-text=CT Cipher text to decrypt\n"));
+ printf(_(" -t, --tag=TAG Tag to use for decryption\n"));
+ printf(_(" -T, --tag-length=LEN Length of tag to use for encryption\n"));
+ printf(_(" -v, --verbose verbose output\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
+ printf("\n");
+ printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
+}
+
+
+int
+main(int argc, char *argv[])
+{
+ char *algorithm = NULL,
+ *iv_hex = NULL,
+ *key_hex = NULL,
+ *plaintext_hex = NULL,
+ *ciphertext_hex = NULL,
+ *tag_hex = NULL;
+
+ unsigned char *plaintext = NULL,
+ *ciphertext = NULL,
+ *key = NULL,
+ *iv = NULL,
+ *tag = NULL,
+ *tag_result = NULL,
+ *result = NULL;
+
+ int verbose = 0,
+ plaintext_len = 0,
+ ciphertext_len = 0,
+ key_len = 0,
+ iv_len = 0,
+ tag_len = 16,
+ result_len = 0,
+ blocksize = 0,
+ cipher = PG_CIPHER_AES_GCM;
+
+ PgCipherCtx *ctx = NULL;
+
+ static struct option long_options[] = {
+ {"algorithm", required_argument, NULL, 'a'},
+ {"init-vector", required_argument, NULL, 'i'},
+ {"key", required_argument, NULL, 'k'},
+ {"plain-text", required_argument, NULL, 'p'},
+ {"cipher-text", required_argument, NULL, 'c'},
+ {"tag", required_argument, NULL, 't'},
+ {"tag-length", required_argument, NULL, 'T'},
+ {"verbose", required_argument, NULL, 'v'},
+ {NULL, 0, NULL, 0}
+ };
+
+ int c;
+
+ pg_logging_init(argv[0]);
+ set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("testcrypto"));
+ progname = get_progname(argv[0]);
+
+ if (argc > 1)
+ {
+ if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
+ {
+ usage(progname);
+ exit(EXITSUCCESS);
+ }
+ if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
+ {
+ puts("testcrypto (PostgreSQL) " PG_VERSION);
+ exit(EXITSUCCESS);
+ }
+ }
+
+ /* Process command-line argument */
+
+ while ((c = getopt_long(argc, argv, "a:i:k:p:c:t:T:v", long_options, NULL)) != -1)
+ {
+ switch (c)
+ {
+ case 'a':
+ algorithm = pg_strdup(optarg);
+ break;
+
+ case 'i':
+ iv_hex = pg_strdup(optarg);
+ break;
+
+ case 'k':
+ key_hex = pg_strdup(optarg);
+ break;
+
+ case 'p':
+ plaintext_hex = pg_strdup(optarg);
+ break;
+
+ case 'c':
+ ciphertext_hex = pg_strdup(optarg);
+ break;
+
+ case 't':
+ tag_hex = pg_strdup(optarg);
+ break;
+
+ case 'T':
+ tag_len = atoi(optarg);
+ break;
+
+ case 'v':
+ verbose = 1;
+ break;
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ exit(EXITFAILURE);
+ }
+ }
+
+ /* Complain if any arguments remain */
+ if (optind < argc)
+ {
+ pg_log_error("too many command-line arguments (first is \"%s\")",
+ argv[optind]);
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(EXITFAILURE);
+ }
+
+ /* Check options passed in */
+ if (algorithm)
+ {
+ if (strcmp(algorithm, "AES-GCM") == 0)
+ cipher = PG_CIPHER_AES_GCM;
+ else if (strcmp(algorithm, "AES-KWP") == 0)
+ cipher = PG_CIPHER_AES_KWP;
+ else
+ {
+ pg_log_error("Unsupported algorithm selected.");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(EXITFAILURE);
+ }
+ }
+
+ if (key_hex == NULL)
+ {
+ pg_log_error("Key must be provided");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(EXITFAILURE);
+ }
+ else
+ {
+ size_t key_hex_len = strlen(key_hex);
+
+ key_len = pg_hex_dec_len(key_hex_len);
+
+ key = pg_malloc0(key_len);
+ pg_hex_decode(key_hex, key_hex_len, (char *) key, key_len);
+ }
+
+ if (cipher == PG_CIPHER_AES_GCM && iv_hex == NULL)
+ {
+ pg_log_error("Initialization vector must be provided");
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+ progname);
+ exit(EXITFAILURE);
+ }
+ else if (cipher == PG_CIPHER_AES_GCM)
+ {
+ size_t iv_hex_len = strlen(iv_hex);
+
+ iv_len = pg_hex_dec_len(iv_hex_len);
+
+ iv = pg_malloc0(iv_len);
+ pg_hex_decode(iv_hex, iv_hex_len, (char *) iv, iv_len);
+ }
+
+ if (plaintext_hex)
+ {
+ size_t plaintext_hex_len = strlen(plaintext_hex);
+
+ plaintext_len = pg_hex_dec_len(plaintext_hex_len);
+
+ plaintext = pg_malloc0(plaintext_len);
+ pg_hex_decode(plaintext_hex, plaintext_hex_len, (char *) plaintext,
+ plaintext_len);
+ }
+
+ /*
+ * OpenSSL 1.1.1d and earlier crashes on some zero-length plaintext and
+ * ciphertext strings. It crashes on an encryption call to
+ * EVP_EncryptFinal_ex(() in GCM mode of zero-length strings if plaintext
+ * is NULL, even though plaintext_len is zero. Setting plaintext to
+ * non-NULL allows it to work. In KWP mode, zero-length strings fail if
+ * plaintext_len = 0 and plaintext is non-NULL (the opposite). OpenSSL
+ * 1.1.1e+ is fine with all options.
+ */
+ else if (cipher == PG_CIPHER_AES_GCM)
+ {
+ plaintext_len = 0;
+ plaintext = pg_malloc0(1);
+ }
+
+ if (ciphertext_hex)
+ {
+ size_t ciphertext_hex_len = strlen(ciphertext_hex);
+
+ ciphertext_len = pg_hex_dec_len(ciphertext_hex_len);
+
+ ciphertext = pg_malloc0(ciphertext_len);
+ pg_hex_decode(ciphertext_hex, ciphertext_hex_len,
+ (char *) ciphertext, ciphertext_len);
+ }
+ /* see OpenSSL 1.1.1d item above, though crash only happens in GCM mode */
+ else if (cipher == PG_CIPHER_AES_GCM)
+ {
+ ciphertext_len = 0;
+ ciphertext = pg_malloc0(1);
+ }
+
+ if (cipher == PG_CIPHER_AES_GCM)
+ tag_result = pg_malloc0(tag_len);
+
+ if (tag_hex)
+ {
+ size_t tag_hex_len = strlen(tag_hex);
+
+ tag_len = pg_hex_dec_len(tag_hex_len);
+
+ tag = pg_malloc0(tag_len);
+ pg_hex_decode(tag_hex, tag_hex_len, (char *) tag, tag_len);
+ }
+ else
+ tag = tag_result;
+
+ if (verbose)
+ {
+ printf("Alrogithm: %d\n", cipher);
+ printf("Key length: %d (%d bits)\n", key_len, key_len * 8);
+ printf("IV length: %d (%d bits)\n", iv_len, iv_len * 8);
+ printf("Tag length: %d (%d bits)\n", tag_len, tag_len * 8);
+ printf("Plaintext length: %d\n", plaintext_len);
+ printf("Ciphertext length: %d\n", ciphertext_len);
+ }
+
+ /*
+ * Encryption
+ *
+ * We run through the encryption even if there wasn't a plaintext
+ * provided- in that case we just encrypt the empty string.
+ */
+ ctx = pg_cipher_ctx_create(cipher, key, key_len, true);
+ if (!ctx)
+ {
+ pg_log_error("Error creating encryption context, be sure key is of supported length");
+ exit(EXITFAILURE);
+ }
+
+ blocksize = pg_cipher_blocksize(ctx);
+
+ /* If we were provided with a plaintext input */
+ if (plaintext_len != 0)
+ {
+ /* Encryption might result in as much as input length + blocksize */
+ result_len = plaintext_len + blocksize;
+ result = palloc0(result_len);
+
+ if (ciphertext_hex == NULL)
+ {
+ ciphertext = result;
+ ciphertext_len = plaintext_len;
+ }
+ }
+
+ if (cipher == PG_CIPHER_AES_GCM)
+ {
+ if (!pg_cipher_encrypt(ctx, cipher,
+ plaintext, plaintext_len,
+ result, &result_len,
+ iv, iv_len,
+ tag_result, tag_len))
+ {
+ pg_log_error("Error during encryption.");
+ exit(EXITFAILURE);
+ }
+ }
+ else if (cipher == PG_CIPHER_AES_KWP)
+ {
+ if (!pg_cipher_keywrap(ctx,
+ plaintext, plaintext_len,
+ result, &result_len))
+ {
+ pg_log_error("Error during encryption.");
+ exit(EXITFAILURE);
+ }
+ }
+
+ if (verbose || ciphertext == NULL)
+ {
+ uint64 result_hex_len = pg_hex_enc_len(result_len);
+ char *result_hex = palloc0(result_hex_len + 1);
+
+ pg_hex_encode((char *) result, result_len, result_hex, result_hex_len);
+ result_hex[result_hex_len] = '\0';
+
+ printf("ciphertext: %s\n", result_hex);
+
+ if (cipher == PG_CIPHER_AES_GCM)
+ {
+ result_hex_len = pg_hex_enc_len(tag_len);
+ result_hex = palloc0(result_hex_len + 1);
+
+ pg_hex_encode((char *) tag_result, tag_len, result_hex, result_hex_len);
+ result_hex[result_hex_len] = '\0';
+
+ printf("tag: %s\n", result_hex);
+ }
+ }
+
+ /*
+ * Report on non-matching results, but still go through the decryption
+ * routine to make sure that we get the correct result, and then error
+ * out.
+ */
+ if (plaintext_len != 0 && ciphertext != NULL && memcmp(ciphertext, result, plaintext_len) != 0)
+ pg_log_error("Provided ciphertext does not match");
+
+ if (cipher == PG_CIPHER_AES_GCM && tag != tag_result && memcmp(tag, tag_result, tag_len) != 0)
+ pg_log_error("Provided tag does not match");
+
+ /*
+ * If a ciphertext was provided then use that as the max size of our
+ * plaintext result. We shouldn't ever get a result larger.
+ */
+ if (ciphertext_len != 0)
+ {
+ result_len = ciphertext_len;
+ result = palloc0(result_len);
+ }
+
+ /*
+ * Decryption
+ *
+ * We run through the decryption even if there wasn't a ciphertext
+ * provided- in that case we just decrypt the empty string.
+ */
+ ctx = pg_cipher_ctx_create(cipher, key, key_len, false);
+ if (!ctx)
+ {
+ pg_log_error("Error creating decryption context, be sure key is of supported length");
+ exit(EXITFAILURE);
+ }
+
+ if (cipher == PG_CIPHER_AES_GCM)
+ {
+ if (!pg_cipher_decrypt(ctx, cipher,
+ ciphertext, ciphertext_len,
+ result, &result_len,
+ iv, iv_len,
+ tag, tag_len))
+ {
+ pg_log_error("Error during decryption.");
+ exit(EXITDECRYPTFAIL);
+ }
+ }
+ else if (cipher == PG_CIPHER_AES_KWP)
+ {
+ if (!pg_cipher_keyunwrap(ctx,
+ ciphertext, ciphertext_len,
+ result, &result_len))
+ {
+ pg_log_error("Error during decryption.");
+ exit(EXITDECRYPTFAIL);
+ }
+ }
+
+ if (verbose || plaintext == NULL)
+ {
+ uint64 result_hex_len = pg_hex_enc_len(result_len);
+ char *result_hex = palloc0(result_hex_len + 1);
+
+ pg_hex_encode((char *) result, result_len, result_hex, result_hex_len);
+ result_hex[result_hex_len] = '\0';
+
+ printf("plaintext: %s\n", result_hex);
+ }
+
+ if (ciphertext_len != 0 && plaintext != NULL && memcmp(plaintext, result, plaintext_len) != 0)
+ {
+ pg_log_error("Provided plaintext does not match");
+ exit(EXITFAILURE);
+ }
+
+ exit(EXITSUCCESS);
+}
--
2.20.1
cfe-10-hint_over_cfe-09-test.diff (text/x-diff, 14.3K) download
From 85d251c614bb8ea7692bb1f403a982a2c6c007da Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:46 -0400
Subject: [PATCH] cfe-10-hint_over_cfe-09-test squash commit
---
doc/src/sgml/config.sgml | 7 +-
src/backend/access/rmgrdesc/xlogdesc.c | 6 +-
src/backend/access/transam/xlog.c | 4 +
src/backend/access/transam/xloginsert.c | 24 +++++
src/backend/replication/logical/decode.c | 1 +
src/backend/storage/buffer/bufmgr.c | 128 +++++++++++++++++------
src/include/access/xlog.h | 14 +--
src/include/access/xloginsert.h | 2 +
src/include/catalog/pg_control.h | 1 +
9 files changed, 143 insertions(+), 44 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d7b6e368d0..6bcd9c64df 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3093,9 +3093,10 @@ include_dir 'conf.d'
</para>
<para>
- If data checksums are enabled, hint bit updates are always WAL-logged
- and this setting is ignored. You can use this setting to test how much
- extra WAL-logging would occur if your database had data checksums
+ If data checksums or cluster file encryption is enabled,
+ hint bit updates are always WAL-logged and this setting is
+ ignored. You can use this setting to test how much extra
+ WAL-logging would occur if your database had data checksums
enabled.
</para>
diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c
index e6090a9dad..d5474990d6 100644
--- a/src/backend/access/rmgrdesc/xlogdesc.c
+++ b/src/backend/access/rmgrdesc/xlogdesc.c
@@ -80,7 +80,8 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
appendStringInfoString(buf, xlrec->rp_name);
}
- else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT)
+ else if (info == XLOG_FPI || info == XLOG_FPI_FOR_HINT ||
+ info == XLOG_ENCRYPTION_LSN)
{
/* no further information to print */
}
@@ -184,6 +185,9 @@ xlog_identify(uint8 info)
case XLOG_FPI_FOR_HINT:
id = "FPI_FOR_HINT";
break;
+ case XLOG_ENCRYPTION_LSN:
+ id = "ENCRYPTION_LSN";
+ break;
}
return id;
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 51165f6593..dba690bc5e 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10353,6 +10353,10 @@ xlog_redo(XLogReaderState *record)
UnlockReleaseBuffer(buffer);
}
}
+ else if (info == XLOG_ENCRYPTION_LSN)
+ {
+ /* nothing to do here */
+ }
else if (info == XLOG_BACKUP_END)
{
XLogRecPtr startpoint;
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 7052dc245e..c76ed1f881 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -980,6 +980,30 @@ XLogSaveBufferForHint(Buffer buffer, bool buffer_std)
return recptr;
}
+/*
+ * This function returns either a WAL or fake LSN, for use by encryption.
+ */
+XLogRecPtr
+LSNForEncryption(bool use_wal_lsn)
+{
+ if (use_wal_lsn)
+ {
+ int dummy = 0;
+
+ Assert(FileEncryptionEnabled);
+ /*
+ * Records other than SWITCH_WAL must have content. We use an integer 0 to
+ * follow the restriction.
+ */
+ XLogBeginInsert();
+ XLogSetRecordFlags(XLOG_MARK_UNIMPORTANT);
+ XLogRegisterData((char *) &dummy, sizeof(dummy));
+ return XLogInsert(RM_XLOG_ID, XLOG_ENCRYPTION_LSN);
+ }
+ else
+ return GetFakeLSNForUnloggedRel();
+}
+
/*
* Write a WAL record containing a full image of a page. Caller is responsible
* for writing the page to disk after calling this routine.
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 97be4b0f23..48400fde2a 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -224,6 +224,7 @@ DecodeXLogOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
case XLOG_FPW_CHANGE:
case XLOG_FPI_FOR_HINT:
case XLOG_FPI:
+ case XLOG_ENCRYPTION_LSN:
break;
default:
elog(ERROR, "unexpected RM_XLOG_ID record type: %u", info);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 852138f9c9..367fed5e85 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -3766,11 +3766,12 @@ IncrBufferRefCount(Buffer buffer)
* This is essentially the same as MarkBufferDirty, except:
*
* 1. The caller does not write WAL; so if checksums are enabled, we may need
- * to write an XLOG_FPI_FOR_HINT WAL record to protect against torn pages.
+ * to write an XLOG_FPI_FOR_HINT record to protect against torn pages, or
+ * XLOG_ENCRYPTION_LSN to generate a new LSN for the page.
* 2. The caller might have only share-lock instead of exclusive-lock on the
- * buffer's content lock.
+ * buffer's content lock.
* 3. This function does not guarantee that the buffer is always marked dirty
- * (due to a race condition), so it cannot be used for important changes.
+ * (due to a race condition), so it cannot be used for important changes.
*/
void
MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
@@ -3816,51 +3817,110 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std)
* If we need to protect hint bit updates from torn writes, WAL-log a
* full page image of the page. This full page image is only necessary
* if the hint bit update is the first change to the page since the
- * last checkpoint.
+ * last checkpoint. If cluster file encryption is enabled, we also
+ * need to generate new page LSNs for all other cases of page writes.
*
* We don't check full_page_writes here because that logic is included
* when we call XLogInsert() since the value changes dynamically.
*/
- if (XLogHintBitIsNeeded() &&
- (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT))
+ if (XLogHintBitIsNeeded())
{
/*
- * If we must not write WAL, due to a relfilenode-specific
- * condition or being in recovery, don't dirty the page. We can
- * set the hint, just not dirty the page as a result so the hint
- * is lost when we evict the page or shutdown.
+ * If we must not write WAL during recovery so don't dirty the page.
+ * We can set the hint, just not dirty the page as a result so the
+ * hint is lost when we evict the page or shutdown.
*
* See src/backend/storage/page/README for longer discussion.
+ * XXX Can this be improved?
*/
- if (RecoveryInProgress() ||
- RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ if (RecoveryInProgress() ||
+ (RelFileNodeSkippingWAL(bufHdr->tag.rnode) &&
+ !FileEncryptionEnabled))
return;
/*
- * If the block is already dirty because we either made a change
- * or set a hint already, then we don't need to write a full page
- * image. Note that aggressive cleaning of blocks dirtied by hint
- * bit setting would increase the call rate. Bulk setting of hint
- * bits would reduce the call rate...
- *
- * We must issue the WAL record before we mark the buffer dirty.
- * Otherwise we might write the page before we write the WAL. That
- * causes a race condition, since a checkpoint might occur between
- * writing the WAL record and marking the buffer dirty. We solve
- * that with a kluge, but one that is already in use during
- * transaction commit to prevent race conditions. Basically, we
- * simply prevent the checkpoint WAL record from being written
- * until we have marked the buffer dirty. We don't start the
- * checkpoint flush until we have marked dirty, so our checkpoint
- * must flush the change to disk successfully or the checkpoint
- * never gets written, so crash recovery will fix.
+ * Non-BM_PERMANENT objects don't need full page images because
+ * they are not restored. WAL-skipped relfilenodes should never
+ * have full page images generated.
+ */
+ if (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT &&
+ !RelFileNodeSkippingWAL(bufHdr->tag.rnode))
+ {
+ /*
+ * If the block is already dirty because we either made a change
+ * or set a hint already, then we don't need to write a full
+ * page image. Note that aggressive cleaning of blocks dirtied
+ * by hint bit setting would increase the call rate. Bulk
+ * setting of hint bits would reduce the call rate...
+ *
+ * We must issue the WAL record before we mark the buffer
+ * dirty. Otherwise we might write the page before we write
+ * the WAL. That causes a race condition, since a checkpoint
+ * might occur between writing the WAL record and marking the
+ * buffer dirty. We solve that with a kluge, but one that is
+ * already in use during transaction commit to prevent race
+ * conditions. Basically, we simply prevent the checkpoint WAL
+ * record from being written until we have marked the buffer
+ * dirty. We don't start the checkpoint flush until we have
+ * marked dirty, so our checkpoint must flush the change to disk
+ * successfully or the checkpoint never gets written, so crash
+ * recovery will fix.
+ *
+ * It's possible we may enter here without an xid, so it
+ * is essential that CreateCheckpoint waits for virtual
+ * transactions rather than full transactionids.
+ */
+ MyProc->delayChkpt = delayChkpt = true;
+ lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ }
+
+ /*
+ * Above, for hint bit changes, we might have generated a new page
+ * LSN and a full-page WAL record for a page's first-clean-to-dirty
+ * during a checkpoint for permanent, non-WAL-skipped relfilenodes.
+ * If we didn't (the lsn variable is invalid), and we are
+ * doing cluster file encryption, we must generate a new
+ * page LSN here for either non-permanent relations or page
+ * non-first-clean-to-dirty during a checkpoint. (Cluster file
+ * encryption does not support WAL-skip relfilenodes.) We must
+ * update the page LSN even if the page with the hint bit change is
+ * later overwritten in the file system with an earlier version of
+ * the page during crash recovery.
*
- * It's possible we may enter here without an xid, so it is
- * essential that CreateCheckpoint waits for virtual transactions
- * rather than full transactionids.
+ * XXX Can we rely on the full page write above with no lock being
+ * held to avoid torn pages? Above, the LSN and page image are
+ * tied together, but here is just the page LSN update.
*/
- MyProc->delayChkpt = delayChkpt = true;
- lsn = XLogSaveBufferForHint(buffer, buffer_std);
+ if (XLogRecPtrIsInvalid(lsn) && FileEncryptionEnabled)
+ {
+ /*
+ * For cluster file encryption we need a new page LSN because
+ * the LSN is used, with the page number and permanent flag, as
+ * part of the nonce, and the nonce must be unique for every
+ * page write. If we reencrypt a page with hint bit changes
+ * using the same nonce as previous writes, it would expose the
+ * hint bit change locations. To avoid this, we write a simple
+ * WAL record to advance the lsn, which can then be assigned to
+ * the page below.
+ *
+ * Above we are relying on the full page writes to revert
+ * any partial pages writes caused by this LSN change for
+ * permanent, non-WAL-skip relfilenodes. For non-permanent
+ * relations, they crash recover as empty. For WAL-skip
+ * relfilenodes, they recover with their original contents, so
+ * that works too.
+ */
+ /* XXX Do we need the checkpoint delay here? */
+ MyProc->delayChkpt = delayChkpt = true;
+ /*
+ * XXX We probably don't need to replay this WAL on the primary
+ * since the full page image is restored, but do we have
+ * to replay this on the repicas (for relations that are
+ * replicated)?
+ */
+ lsn = LSNForEncryption(
+ pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT);
+ }
}
buf_state = LockBufHdr(bufHdr);
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index ab66a660a5..52f5a42f50 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -203,13 +203,15 @@ extern PGDLLIMPORT int wal_level;
/*
* Is a full-page image needed for hint bit updates?
*
- * Normally, we don't WAL-log hint bit updates, but if checksums are enabled,
- * we have to protect them against torn page writes. When you only set
- * individual bits on a page, it's still consistent no matter what combination
- * of the bits make it to disk, but the checksum wouldn't match. Also WAL-log
- * them if forced by wal_log_hints=on.
+ * Normally, we don't WAL-log hint bit updates, but if checksums or encryption
+ * is enabled, we have to protect them against torn page writes. When you
+ * only set individual bits on a page, it's still consistent no matter what
+ * combination of the bits make it to disk, but the checksum wouldn't match.
+ * Cluster file encryption requires a new LSN for hint bit changes, and can't
+ * tolerate torn pages. Also WAL-log them if forced by wal_log_hints=on.
*/
-#define XLogHintBitIsNeeded() (DataChecksumsEnabled() || wal_log_hints)
+#define XLogHintBitIsNeeded() \
+ (DataChecksumsEnabled() || FileEncryptionEnabled || wal_log_hints)
/* Do we need to WAL-log information required only for Hot Standby and logical replication? */
#define XLogStandbyInfoActive() (wal_level >= WAL_LEVEL_REPLICA)
diff --git a/src/include/access/xloginsert.h b/src/include/access/xloginsert.h
index f1d8c39edf..e7ee674dd4 100644
--- a/src/include/access/xloginsert.h
+++ b/src/include/access/xloginsert.h
@@ -61,6 +61,8 @@ extern void log_newpage_range(Relation rel, ForkNumber forkNum,
BlockNumber startblk, BlockNumber endblk, bool page_std);
extern XLogRecPtr XLogSaveBufferForHint(Buffer buffer, bool buffer_std);
+extern XLogRecPtr LSNForEncryption(bool use_wal_lsn);
+
extern void InitXLogInsert(void);
#endif /* XLOGINSERT_H */
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 6216d1faf5..e23693da4f 100644
--- a/src/include/catalog/pg_control.h
+++ b/src/include/catalog/pg_control.h
@@ -76,6 +76,7 @@ typedef struct CheckPoint
#define XLOG_END_OF_RECOVERY 0x90
#define XLOG_FPI_FOR_HINT 0xA0
#define XLOG_FPI 0xB0
+#define XLOG_ENCRYPTION_LSN 0xC0
/*
--
2.20.1
cfe-11-gist_over_cfe-10-hint.diff (text/x-diff, 6.6K) download
From 4b180ea5e7c5430cb5868bc464d0e3dc2f5e8be0 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:48 -0400
Subject: [PATCH] cfe-11-gist_over_cfe-10-hint squash commit
---
src/backend/access/gist/README | 10 ++++++++++
src/backend/access/gist/gist.c | 21 +++++++++++++++------
src/backend/access/gist/gistbuild.c | 9 ++++++---
src/backend/access/gist/gistvacuum.c | 18 ++++++++++++------
4 files changed, 43 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/gist/README b/src/backend/access/gist/README
index 25cab0047b..6b1f84634f 100644
--- a/src/backend/access/gist/README
+++ b/src/backend/access/gist/README
@@ -461,6 +461,16 @@ value. The page is not recycled, until that XID is no longer visible to
anyone. That's much more conservative than necessary, but let's keep it
simple.
+GiST and Encryption
+-------------------
+
+GiST uses LSNs and NSNs for concurrency. This means that unlogged and
+temporary relations also need LSNs. GiST has a mechanism to assign LSNs
+to such relations, but sometimes uses fixed LSNs and or calls
+gistGetFakeLSN(), which can cause duplicate LSNs to be used. Therefore,
+when encryption is enabled and fake LSNs are needed, the GiST code calls
+LSNForEncryption(). See src/backend/crypto/README for more details.
+
Authors:
Teodor Sigaev <teodor@sigaev.ru>
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 0683f42c25..5ca5f17db3 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -501,13 +501,16 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
* we don't need to be able to detect concurrent splits yet.)
*/
if (is_build)
- recptr = GistBuildLSN;
+ recptr = !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(rel));
else
{
if (RelationNeedsWAL(rel))
recptr = gistXLogSplit(is_leaf,
dist, oldrlink, oldnsn, leftchildbuf,
markfollowright);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
recptr = gistGetFakeLSN(rel);
}
@@ -568,7 +571,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
MarkBufferDirty(leftchildbuf);
if (is_build)
- recptr = GistBuildLSN;
+ recptr = !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(rel));
else
{
if (RelationNeedsWAL(rel))
@@ -586,6 +590,8 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
deloffs, ndeloffs, itup, ntup,
leftchildbuf);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
recptr = gistGetFakeLSN(rel);
}
@@ -1665,6 +1671,8 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel)
if (ndeletable > 0)
{
+ XLogRecPtr recptr;
+
TransactionId latestRemovedXid = InvalidTransactionId;
if (XLogStandbyInfoActive() && RelationNeedsWAL(rel))
@@ -1690,16 +1698,17 @@ gistprunepage(Relation rel, Page page, Buffer buffer, Relation heapRel)
/* XLOG stuff */
if (RelationNeedsWAL(rel))
{
- XLogRecPtr recptr;
-
recptr = gistXLogDelete(buffer,
deletable, ndeletable,
latestRemovedXid);
- PageSetLSN(page, recptr);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- PageSetLSN(page, gistGetFakeLSN(rel));
+ recptr = gistGetFakeLSN(rel);
+
+ PageSetLSN(page, recptr);
END_CRIT_SECTION();
}
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index 1054f6f1f2..a8b82483e8 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -299,7 +299,8 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
GISTInitBuffer(buffer, F_LEAF);
MarkBufferDirty(buffer);
- PageSetLSN(page, GistBuildLSN);
+ PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(index)));
UnlockReleaseBuffer(buffer);
@@ -451,7 +452,8 @@ gist_indexsortbuild(GISTBuildState *state)
/* Write out the root */
RelationOpenSmgr(state->indexrel);
- PageSetLSN(pagestate->page, GistBuildLSN);
+ PageSetLSN(pagestate->page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(state->indexrel)));
PageSetChecksumInplace(pagestate->page, GIST_ROOT_BLKNO);
smgrwrite(state->indexrel->rd_smgr, MAIN_FORKNUM, GIST_ROOT_BLKNO,
pagestate->page, true);
@@ -573,7 +575,8 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state)
if (blkno != state->pages_written)
elog(ERROR, "unexpected block number to flush GiST sorting build");
- PageSetLSN(page, GistBuildLSN);
+ PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
+ LSNForEncryption(RelationIsPermanent(state->indexrel)));
PageSetChecksumInplace(page, blkno);
smgrextend(state->indexrel->rd_smgr, MAIN_FORKNUM, blkno, page, true);
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index 0663193531..f783edd627 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -174,6 +174,8 @@ gistvacuumscan(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
vstate.callback_state = callback_state;
if (RelationNeedsWAL(rel))
vstate.startNSN = GetInsertRecPtr();
+ else if (FileEncryptionEnabled)
+ vstate.startNSN = LSNForEncryption(RelationIsPermanent(rel));
else
vstate.startNSN = gistGetFakeLSN(rel);
@@ -353,6 +355,8 @@ restart:
*/
if (ntodelete > 0)
{
+ XLogRecPtr recptr;
+
START_CRIT_SECTION();
MarkBufferDirty(buffer);
@@ -361,16 +365,15 @@ restart:
GistMarkTuplesDeleted(page);
if (RelationNeedsWAL(rel))
- {
- XLogRecPtr recptr;
-
recptr = gistXLogUpdate(buffer,
todelete, ntodelete,
NULL, 0, InvalidBuffer);
- PageSetLSN(page, recptr);
- }
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- PageSetLSN(page, gistGetFakeLSN(rel));
+ recptr = gistGetFakeLSN(rel);
+
+ PageSetLSN(page, recptr);
END_CRIT_SECTION();
@@ -657,8 +660,11 @@ gistdeletepage(IndexVacuumInfo *info, IndexBulkDeleteResult *stats,
if (RelationNeedsWAL(info->index))
recptr = gistXLogPageDelete(leafBuffer, txid, parentBuffer, downlink);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(info->index));
else
recptr = gistGetFakeLSN(info->index);
+
PageSetLSN(parentPage, recptr);
PageSetLSN(leafPage, recptr);
--
2.20.1
cfe-12-rel_over_cfe-11-gist.diff (text/x-diff, 31.8K) download
From 9180179166525bd5b6ac66ddda4c74501cbf1161 Mon Sep 17 00:00:00 2001
From: Bruce Momjian <bruce@momjian.us>
Date: Tue, 6 Apr 2021 14:23:49 -0400
Subject: [PATCH] cfe-12-rel_over_cfe-11-gist squash commit
---
contrib/bloom/blinsert.c | 3 +
src/backend/access/gist/gist.c | 51 ++++----
src/backend/access/gist/gistbuild.c | 12 ++
src/backend/access/hash/hashpage.c | 1 +
src/backend/access/heap/rewriteheap.c | 7 ++
src/backend/access/nbtree/nbtree.c | 3 +
src/backend/access/nbtree/nbtsort.c | 4 +-
src/backend/access/spgist/spginsert.c | 6 +
src/backend/access/transam/xlog.c | 2 +
src/backend/bootstrap/bootstrap.c | 1 +
src/backend/catalog/storage.c | 7 +-
src/backend/crypto/Makefile | 1 +
src/backend/crypto/bufenc.c (new) | 174 ++++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 2 +
src/backend/storage/buffer/bufmgr.c | 22 +++-
src/backend/storage/buffer/localbuf.c | 8 +-
src/backend/storage/file/copydir.c | 27 +++-
src/backend/storage/file/reinit.c | 2 +-
src/backend/storage/page/bufpage.c | 51 +++++++-
src/backend/tcop/postgres.c | 2 +
src/include/access/gist.h | 5 +-
src/include/crypto/bufenc.h (new) | 28 +++++
src/include/storage/bufpage.h | 18 ++-
src/include/storage/copydir.h | 2 +-
24 files changed, 392 insertions(+), 47 deletions(-)
diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c
index d37ceef753..cb04aac963 100644
--- a/contrib/bloom/blinsert.c
+++ b/contrib/bloom/blinsert.c
@@ -177,7 +177,10 @@ blbuildempty(Relation index)
* XLOG_DBASE_CREATE or XLOG_TBLSPC_CREATE record. Therefore, we need
* this even when wal_level=minimal.
*/
+ PageEncryptInplace(metapage, INIT_FORKNUM, RelationIsPermanent(index),
+ BLOOM_METAPAGE_BLKNO);
PageSetChecksumInplace(metapage, BLOOM_METAPAGE_BLKNO);
+
smgrwrite(index->rd_smgr, INIT_FORKNUM, BLOOM_METAPAGE_BLKNO,
(char *) metapage, true);
log_newpage(&index->rd_smgr->smgr_rnode.node, INIT_FORKNUM,
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 5ca5f17db3..5d5f109055 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -503,17 +503,14 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
if (is_build)
recptr = !FileEncryptionEnabled ? GistBuildLSN :
LSNForEncryption(RelationIsPermanent(rel));
+ else if (RelationNeedsWAL(rel))
+ recptr = gistXLogSplit(is_leaf,
+ dist, oldrlink, oldnsn, leftchildbuf,
+ markfollowright);
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
else
- {
- if (RelationNeedsWAL(rel))
- recptr = gistXLogSplit(is_leaf,
- dist, oldrlink, oldnsn, leftchildbuf,
- markfollowright);
- else if (FileEncryptionEnabled)
- recptr = LSNForEncryption(RelationIsPermanent(rel));
- else
- recptr = gistGetFakeLSN(rel);
- }
+ recptr = gistGetFakeLSN(rel);
for (ptr = dist; ptr; ptr = ptr->next)
PageSetLSN(ptr->page, recptr);
@@ -573,28 +570,26 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
if (is_build)
recptr = !FileEncryptionEnabled ? GistBuildLSN :
LSNForEncryption(RelationIsPermanent(rel));
- else
+ else if (RelationNeedsWAL(rel))
{
- if (RelationNeedsWAL(rel))
- {
- OffsetNumber ndeloffs = 0,
- deloffs[1];
+ OffsetNumber ndeloffs = 0,
+ deloffs[1];
- if (OffsetNumberIsValid(oldoffnum))
- {
- deloffs[0] = oldoffnum;
- ndeloffs = 1;
- }
-
- recptr = gistXLogUpdate(buffer,
- deloffs, ndeloffs, itup, ntup,
- leftchildbuf);
+ if (OffsetNumberIsValid(oldoffnum))
+ {
+ deloffs[0] = oldoffnum;
+ ndeloffs = 1;
}
- else if (FileEncryptionEnabled)
- recptr = LSNForEncryption(RelationIsPermanent(rel));
- else
- recptr = gistGetFakeLSN(rel);
+
+ recptr = gistXLogUpdate(buffer,
+ deloffs, ndeloffs, itup, ntup,
+ leftchildbuf);
}
+ else if (FileEncryptionEnabled)
+ recptr = LSNForEncryption(RelationIsPermanent(rel));
+ else
+ recptr = gistGetFakeLSN(rel);
+
PageSetLSN(page, recptr);
if (newblkno)
diff --git a/src/backend/access/gist/gistbuild.c b/src/backend/access/gist/gistbuild.c
index a8b82483e8..bd3dc96eb9 100644
--- a/src/backend/access/gist/gistbuild.c
+++ b/src/backend/access/gist/gistbuild.c
@@ -454,6 +454,12 @@ gist_indexsortbuild(GISTBuildState *state)
RelationOpenSmgr(state->indexrel);
PageSetLSN(pagestate->page, !FileEncryptionEnabled ? GistBuildLSN :
LSNForEncryption(RelationIsPermanent(state->indexrel)));
+ /* Make sure LSNs are vaild, and if encryption, are not constant. */
+ Assert(!XLogRecPtrIsInvalid(PageGetLSN(pagestate->page)) &&
+ (!FileEncryptionEnabled ||
+ PageGetLSN(pagestate->page) != GistBuildLSN));
+ PageEncryptInplace(pagestate->page, MAIN_FORKNUM, RelationIsPermanent(state->indexrel),
+ GIST_ROOT_BLKNO);
PageSetChecksumInplace(pagestate->page, GIST_ROOT_BLKNO);
smgrwrite(state->indexrel->rd_smgr, MAIN_FORKNUM, GIST_ROOT_BLKNO,
pagestate->page, true);
@@ -577,6 +583,12 @@ gist_indexsortbuild_flush_ready_pages(GISTBuildState *state)
PageSetLSN(page, !FileEncryptionEnabled ? GistBuildLSN :
LSNForEncryption(RelationIsPermanent(state->indexrel)));
+ /* Make sure LSNs are vaild, and if encryption, are not constant. */
+ Assert(!XLogRecPtrIsInvalid(PageGetLSN(page)) &&
+ (!FileEncryptionEnabled ||
+ PageGetLSN(page) != GistBuildLSN));
+ PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(state->indexrel),
+ blkno);
PageSetChecksumInplace(page, blkno);
smgrextend(state->indexrel->rd_smgr, MAIN_FORKNUM, blkno, page, true);
diff --git a/src/backend/access/hash/hashpage.c b/src/backend/access/hash/hashpage.c
index 49a9867787..eae9a9f04a 100644
--- a/src/backend/access/hash/hashpage.c
+++ b/src/backend/access/hash/hashpage.c
@@ -1025,6 +1025,7 @@ _hash_alloc_buckets(Relation rel, BlockNumber firstblock, uint32 nblocks)
true);
RelationOpenSmgr(rel);
+ PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(rel), lastblock);
PageSetChecksumInplace(page, lastblock);
smgrextend(rel->rd_smgr, MAIN_FORKNUM, lastblock, zerobuf.data, false);
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 1aff62cd42..981b73c101 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -324,6 +324,9 @@ end_heap_rewrite(RewriteState state)
state->rs_buffer,
true);
+ PageEncryptInplace(state->rs_buffer, MAIN_FORKNUM,
+ RelationIsPermanent(state->rs_new_rel),
+ state->rs_blockno);
PageSetChecksumInplace(state->rs_buffer, state->rs_blockno);
RelationOpenSmgr(state->rs_new_rel);
@@ -697,6 +700,10 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
*/
RelationOpenSmgr(state->rs_new_rel);
+ /* XXX Do we need to encrypt a copy of the page here? */
+ PageEncryptInplace(page, MAIN_FORKNUM,
+ RelationIsPermanent(state->rs_old_rel),
+ state->rs_blockno);
PageSetChecksumInplace(page, state->rs_blockno);
smgrextend(state->rs_new_rel->rd_smgr, MAIN_FORKNUM,
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.c
index 9282c9ea22..2dbb2d3ebc 100644
--- a/src/backend/access/nbtree/nbtree.c
+++ b/src/backend/access/nbtree/nbtree.c
@@ -162,7 +162,10 @@ btbuildempty(Relation index)
* XLOG_DBASE_CREATE or XLOG_TBLSPC_CREATE record. Therefore, we need
* this even when wal_level=minimal.
*/
+ PageEncryptInplace(metapage, INIT_FORKNUM, RelationIsPermanent(index),
+ BTREE_METAPAGE);
PageSetChecksumInplace(metapage, BTREE_METAPAGE);
+
smgrwrite(index->rd_smgr, INIT_FORKNUM, BTREE_METAPAGE,
(char *) metapage, true);
log_newpage(&index->rd_smgr->smgr_rnode.node, INIT_FORKNUM,
diff --git a/src/backend/access/nbtree/nbtsort.c b/src/backend/access/nbtree/nbtsort.c
index 2c4d7f6e25..6dbc1b6f25 100644
--- a/src/backend/access/nbtree/nbtsort.c
+++ b/src/backend/access/nbtree/nbtsort.c
@@ -657,13 +657,15 @@ _bt_blwritepage(BTWriteState *wstate, Page page, BlockNumber blkno)
{
if (!wstate->btws_zeropage)
wstate->btws_zeropage = (Page) palloc0(BLCKSZ);
- /* don't set checksum for all-zero page */
+ /* don't set checksum or encryption for all-zero page */
smgrextend(wstate->index->rd_smgr, MAIN_FORKNUM,
wstate->btws_pages_written++,
(char *) wstate->btws_zeropage,
true);
}
+ PageEncryptInplace(page, MAIN_FORKNUM, RelationIsPermanent(wstate->index),
+ blkno);
PageSetChecksumInplace(page, blkno);
/*
diff --git a/src/backend/access/spgist/spginsert.c b/src/backend/access/spgist/spginsert.c
index 1af0af7da2..4fd5645c6f 100644
--- a/src/backend/access/spgist/spginsert.c
+++ b/src/backend/access/spgist/spginsert.c
@@ -168,6 +168,8 @@ spgbuildempty(Relation index)
* of their existing content when the corresponding create records are
* replayed.
*/
+ PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index),
+ SPGIST_METAPAGE_BLKNO);
PageSetChecksumInplace(page, SPGIST_METAPAGE_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_METAPAGE_BLKNO,
(char *) page, true);
@@ -177,6 +179,8 @@ spgbuildempty(Relation index)
/* Likewise for the root page. */
SpGistInitPage(page, SPGIST_LEAF);
+ PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index),
+ SPGIST_ROOT_BLKNO);
PageSetChecksumInplace(page, SPGIST_ROOT_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_ROOT_BLKNO,
(char *) page, true);
@@ -186,6 +190,8 @@ spgbuildempty(Relation index)
/* Likewise for the null-tuples root page. */
SpGistInitPage(page, SPGIST_LEAF | SPGIST_NULLS);
+ PageEncryptInplace(page, INIT_FORKNUM, RelationIsPermanent(index),
+ SPGIST_NULL_BLKNO);
PageSetChecksumInplace(page, SPGIST_NULL_BLKNO);
smgrwrite(index->rd_smgr, INIT_FORKNUM, SPGIST_NULL_BLKNO,
(char *) page, true);
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dba690bc5e..7ac0ca8b4c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -45,6 +45,7 @@
#include "common/controldata_utils.h"
#include "crypto/kmgr.h"
#include "executor/instrument.h"
+#include "crypto/bufenc.h"
#include "miscadmin.h"
#include "pg_trace.h"
#include "pgstat.h"
@@ -5413,6 +5414,7 @@ BootStrapXLOG(void)
WriteControlFile();
BootStrapKmgr();
+ InitializeBufferEncryption();
if (terminal_fd != -1)
{
diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c
index ce6419ddd6..e8545a72b8 100644
--- a/src/backend/bootstrap/bootstrap.c
+++ b/src/backend/bootstrap/bootstrap.c
@@ -29,6 +29,7 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_type.h"
#include "common/link-canary.h"
+#include "crypto/bufenc.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c
index cba7a9ada0..7f18020b9f 100644
--- a/src/backend/catalog/storage.c
+++ b/src/backend/catalog/storage.c
@@ -443,8 +443,9 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
smgrread(src, forkNum, blkno, buf.data);
- if (!PageIsVerifiedExtended(page, blkno,
- PIV_LOG_WARNING | PIV_REPORT_STAT))
+ if (!PageIsVerifiedExtended(page, forkNum,
+ relpersistence == RELPERSISTENCE_PERMANENT,
+ blkno, PIV_LOG_WARNING | PIV_REPORT_STAT))
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg("invalid page in block %u of relation %s",
@@ -461,6 +462,8 @@ RelationCopyStorage(SMgrRelation src, SMgrRelation dst,
if (use_wal)
log_newpage(&dst->smgr_rnode.node, forkNum, blkno, page, false);
+ PageEncryptInplace(page, forkNum,
+ relpersistence == RELPERSISTENCE_PERMANENT, blkno);
PageSetChecksumInplace(page, blkno);
/*
diff --git a/src/backend/crypto/Makefile b/src/backend/crypto/Makefile
index c27362029d..8985a66875 100644
--- a/src/backend/crypto/Makefile
+++ b/src/backend/crypto/Makefile
@@ -13,6 +13,7 @@ top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
OBJS = \
+ bufenc.o \
kmgr.o
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/crypto/bufenc.c b/src/backend/crypto/bufenc.c
new file mode 100644
index 0000000000..0c70c131e1
--- /dev/null
+++ b/src/backend/crypto/bufenc.c
@@ -0,0 +1,174 @@
+/*-------------------------------------------------------------------------
+ *
+ * bufenc.c
+ *
+ * Copyright (c) 2020, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/crypto/bufenc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "miscadmin.h"
+#include "lib/stringinfo.h"
+
+#include "access/gist.h"
+#include "access/xlog.h"
+#include "crypto/bufenc.h"
+#include "storage/bufpage.h"
+#include "storage/fd.h"
+
+/*
+ * We use the page LSN, page number, and permanent-bit to indicate if a fake
+ * LSN was used to create a nonce for each page.
+ */
+#define BUFENC_IV_SIZE 16
+
+static unsigned char buf_encryption_iv[BUFENC_IV_SIZE];
+
+PgCipherCtx *BufEncCtx = NULL;
+PgCipherCtx *BufDecCtx = NULL;
+
+static void set_buffer_encryption_iv(Page page, BlockNumber blkno,
+ bool relation_is_permanent);
+
+void
+InitializeBufferEncryption(void)
+{
+ const CryptoKey *key;
+
+ if (!FileEncryptionEnabled)
+ return;
+
+ key = KmgrGetKey(KMGR_KEY_ID_REL);
+
+ BufEncCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR,
+ (unsigned char *) key->key,
+ (key->klen), true);
+ if (!BufEncCtx)
+ elog(ERROR, "cannot intialize encryption context");
+
+ BufDecCtx = pg_cipher_ctx_create(PG_CIPHER_AES_CTR,
+ (unsigned char *) key->key,
+ (key->klen), false);
+ if (!BufDecCtx)
+ elog(ERROR, "cannot intialize decryption context");
+}
+
+/* Encrypt the given page with the relation key */
+void
+EncryptPage(Page page, bool relation_is_permanent, BlockNumber blkno)
+{
+ unsigned char *ptr = (unsigned char *) page + PageEncryptOffset;
+ bool is_gist_page_or_similar;
+
+ int enclen;
+
+ Assert(BufEncCtx != NULL);
+
+ /*
+ * Permanent pages have valid LSNs, and non-permanent pages usually have
+ * invalid (not set) LSNs. (One exception are GiST fake LSNs, see below.)
+ * However, we need valid ones on all pages for encryption. There are too
+ * many places that set the page LSN for permanent pages to do the same
+ * for non-permanent pages, so we just set it here.
+ *
+ * Also, while permanent relations get new LSNs every time the page is
+ * modified, for non-permanent relations do not, so we just update the LSN
+ * here before it is encrypted.
+ *
+ * GiST indexes uses LSNs, which are also stored in NSN fields, to detect
+ * page splits. Therefore, we allow the GiST code to assign LSNs and we
+ * don't change them here.
+ */
+
+ /* Permanent relations should already have valid LSNs. */
+ Assert(!XLogRecPtrIsInvalid(PageGetLSN(page)) || !relation_is_permanent);
+
+ /*
+ * Check if the page has a special size == GISTPageOpaqueData, a valid
+ * GIST_PAGE_ID, no invalid GiST flag bits are set, and a valid LSN. This
+ * is true for all GiST pages, and perhaps a few pages that are not. The
+ * only downside of guessing wrong is that we might not update the LSN for
+ * some non-permanent relation page changes, and therefore reuse the IV,
+ * which seems acceptable.
+ */
+ is_gist_page_or_similar =
+ (PageGetSpecialSize(page) == MAXALIGN(sizeof(GISTPageOpaqueData)) &&
+ GistPageGetOpaque(page)->gist_page_id == GIST_PAGE_ID &&
+ (GistPageGetOpaque(page)->flags & ~GIST_FLAG_BITMASK) == 0 &&
+ !XLogRecPtrIsInvalid(PageGetLSN(page)));
+
+ if (!relation_is_permanent && !is_gist_page_or_similar)
+ PageSetLSN(page, LSNForEncryption(relation_is_permanent));
+
+ set_buffer_encryption_iv(page, blkno, relation_is_permanent);
+ if (unlikely(!pg_cipher_encrypt(BufEncCtx, PG_CIPHER_AES_CTR,
+ (const unsigned char *) ptr, /* input */
+ SizeOfPageEncryption,
+ ptr, /* length */
+ &enclen, /* resulting length */
+ buf_encryption_iv, /* iv */
+ BUFENC_IV_SIZE,
+ NULL, 0)))
+ elog(ERROR, "cannot encrypt page %u", blkno);
+
+ Assert(enclen == SizeOfPageEncryption);
+}
+
+/* Decrypt the given page with the relation key */
+void
+DecryptPage(Page page, bool relation_is_permanent, BlockNumber blkno)
+{
+ unsigned char *ptr = (unsigned char *) page + PageEncryptOffset;
+ int enclen;
+
+ Assert(BufDecCtx != NULL);
+
+ set_buffer_encryption_iv(page, blkno, relation_is_permanent);
+ if (unlikely(!pg_cipher_decrypt(BufDecCtx, PG_CIPHER_AES_CTR,
+ (const unsigned char *) ptr, /* input */
+ SizeOfPageEncryption,
+ ptr, /* output */
+ &enclen, /* resulting length */
+ buf_encryption_iv, /* iv */
+ BUFENC_IV_SIZE,
+ NULL, 0)))
+ elog(ERROR, "cannot decrypt page %u", blkno);
+
+ Assert(enclen == SizeOfPageEncryption);
+}
+
+/* Construct iv for the given page */
+static void
+set_buffer_encryption_iv(Page page, BlockNumber blkno,
+ bool relation_is_permanent)
+{
+ unsigned char *p = buf_encryption_iv;
+
+ MemSet(buf_encryption_iv, 0, BUFENC_IV_SIZE);
+
+ /* page lsn (8 byte) */
+ memcpy(p, &((PageHeader) page)->pd_lsn, sizeof(PageXLogRecPtr));
+ p += sizeof(PageXLogRecPtr);
+
+ /* block number (4 byte) */
+ memcpy(p, &blkno, sizeof(BlockNumber));
+ p += sizeof(BlockNumber);
+
+ /*
+ * Mark use of fake LSNs in IV so if the real and fake LSN counters
+ * overlap, the IV will remain unique. XXX Is there a better value?
+ */
+ if (!relation_is_permanent)
+ *p++ = 0x80;
+
+ /*
+ * The maximum required counter for AES-CTR is 2048, which fits in the
+ * last three bytes.
+ */
+}
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 11dd3eadff..df238c6260 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -100,6 +100,7 @@
#include "common/file_perm.h"
#include "common/ip.h"
#include "common/string.h"
+#include "crypto/bufenc.h"
#include "lib/ilist.h"
#include "libpq/auth.h"
#include "libpq/libpq.h"
@@ -1339,6 +1340,7 @@ PostmasterMain(int argc, char *argv[])
RemovePgTempFiles();
InitializeKmgr();
+ InitializeBufferEncryption();
if (terminal_fd != -1)
close(terminal_fd);
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 367fed5e85..7d913e038a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -37,6 +37,7 @@
#include "access/xlog.h"
#include "catalog/catalog.h"
#include "catalog/storage.h"
+#include "crypto/bufenc.h"
#include "executor/instrument.h"
#include "lib/binaryheap.h"
#include "miscadmin.h"
@@ -930,7 +931,9 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
}
/* check for garbage data */
- if (!PageIsVerifiedExtended((Page) bufBlock, blockNum,
+ if (!PageIsVerifiedExtended((Page) bufBlock, forkNum,
+ relpersistence == RELPERSISTENCE_PERMANENT,
+ blockNum,
PIV_LOG_WARNING | PIV_REPORT_STAT))
{
if (mode == RBM_ZERO_ON_ERROR || zero_damaged_pages)
@@ -2810,12 +2813,24 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln)
*/
bufBlock = BufHdrGetBlock(buf);
+ if (FileEncryptionEnabled)
+ {
+ /*
+ * Technically BM_PERMANENT could indicate an init fork, but that's
+ * okay since forkNum would also tell us not to encrypt init forks.
+ */
+ bufToWrite = PageEncryptCopy((Page) bufBlock, buf->tag.forkNum,
+ buf_state & BM_PERMANENT, buf->tag.blockNum);
+ bufToWrite = PageSetChecksumCopy((Page) bufToWrite, buf->tag.blockNum);
+ }
+ else
+ bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum);
+
/*
* Update page checksum if desired. Since we have only shared lock on the
* buffer, other processes might be updating hint bits in it, so we must
* copy the page to private storage if we do checksumming.
*/
- bufToWrite = PageSetChecksumCopy((Page) bufBlock, buf->tag.blockNum);
if (track_io_timing)
INSTR_TIME_SET_CURRENT(io_start);
@@ -3474,6 +3489,9 @@ FlushRelationBuffers(Relation rel)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* XXX should we be writing a copy of the page here? */
+ PageEncryptInplace(localpage, bufHdr->tag.forkNum,
+ RelationIsPermanent(rel), bufHdr->tag.blockNum);
PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
smgrwrite(rel->rd_smgr,
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea3..1033467885 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -159,7 +159,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
}
return bufHdr;
}
-
+
#ifdef LBDEBUG
fprintf(stderr, "LB ALLOC (%u,%d,%d) %d\n",
smgr->smgr_rnode.node.relNode, forkNum, blockNum,
@@ -217,6 +217,12 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
/* Find smgr relation for buffer */
oreln = smgropen(bufHdr->tag.rnode, MyBackendId);
+ /*
+ * Technically BM_PERMANENT could indicate an init fork, but that's
+ * okay since forkNum would also tell us not to encrypt init forks.
+ */
+ PageEncryptInplace(localpage, bufHdr->tag.forkNum,
+ buf_state & BM_PERMANENT, bufHdr->tag.blockNum);
PageSetChecksumInplace(localpage, bufHdr->tag.blockNum);
/* And write... */
diff --git a/src/backend/storage/file/copydir.c b/src/backend/storage/file/copydir.c
index da8b7cbeca..f41387abd3 100644
--- a/src/backend/storage/file/copydir.c
+++ b/src/backend/storage/file/copydir.c
@@ -22,6 +22,8 @@
#include <unistd.h>
#include <sys/stat.h>
+#include "access/xloginsert.h"
+#include "crypto/bufenc.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/copydir.h"
@@ -74,7 +76,7 @@ copydir(char *fromdir, char *todir, bool recurse)
copydir(fromfile, tofile, true);
}
else if (S_ISREG(fst.st_mode))
- copy_file(fromfile, tofile);
+ copy_file(fromfile, tofile, false);
}
FreeDir(xldir);
@@ -124,7 +126,7 @@ copydir(char *fromdir, char *todir, bool recurse)
* copy one file
*/
void
-copy_file(char *fromfile, char *tofile)
+copy_file(char *fromfile, char *tofile, bool encrypt_init_file)
{
char *buffer;
int srcfd;
@@ -132,9 +134,8 @@ copy_file(char *fromfile, char *tofile)
int nbytes;
off_t offset;
off_t flush_offset;
-
/* Size of copy buffer (read and write requests) */
-#define COPY_BUF_SIZE (8 * BLCKSZ)
+ int copy_buf_size = (encrypt_init_file) ? BLCKSZ : 8 * BLCKSZ;
/*
* Size of data flush requests. It seems beneficial on most platforms to
@@ -149,7 +150,7 @@ copy_file(char *fromfile, char *tofile)
#endif
/* Use palloc to ensure we get a maxaligned buffer */
- buffer = palloc(COPY_BUF_SIZE);
+ buffer = palloc(copy_buf_size);
/*
* Open the files
@@ -187,7 +188,7 @@ copy_file(char *fromfile, char *tofile)
}
pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_READ);
- nbytes = read(srcfd, buffer, COPY_BUF_SIZE);
+ nbytes = read(srcfd, buffer, copy_buf_size);
pgstat_report_wait_end();
if (nbytes < 0)
ereport(ERROR,
@@ -195,6 +196,20 @@ copy_file(char *fromfile, char *tofile)
errmsg("could not read file \"%s\": %m", fromfile)));
if (nbytes == 0)
break;
+ /*
+ * When we copy an init fork page to be part of an empty unlogged
+ * relation, its real LSN must be replaced with a fake one, and the
+ * page encrypted.
+ */
+ if (encrypt_init_file)
+ {
+ Page page = (Page) buffer;
+
+ Assert(nbytes == BLCKSZ);
+ PageSetLSN(page, LSNForEncryption(false));
+ PageEncryptInplace(page, MAIN_FORKNUM, false, offset / BLCKSZ);
+ }
+
errno = 0;
pgstat_report_wait_start(WAIT_EVENT_COPY_FILE_WRITE);
if ((int) write(dstfd, buffer, nbytes) != nbytes)
diff --git a/src/backend/storage/file/reinit.c b/src/backend/storage/file/reinit.c
index 40c758d789..9953bb2e1f 100644
--- a/src/backend/storage/file/reinit.c
+++ b/src/backend/storage/file/reinit.c
@@ -300,7 +300,7 @@ ResetUnloggedRelationsInDbspaceDir(const char *dbspacedirname, int op)
/* OK, we're ready to perform the actual copy. */
elog(DEBUG2, "copying %s to %s", srcpath, dstpath);
- copy_file(srcpath, dstpath);
+ copy_file(srcpath, dstpath, true);
}
FreeDir(dbspace_dir);
diff --git a/src/backend/storage/page/bufpage.c b/src/backend/storage/page/bufpage.c
index 5d5989c2f5..28cc9d3b36 100644
--- a/src/backend/storage/page/bufpage.c
+++ b/src/backend/storage/page/bufpage.c
@@ -17,6 +17,7 @@
#include "access/htup_details.h"
#include "access/itup.h"
#include "access/xlog.h"
+#include "crypto/bufenc.h"
#include "pgstat.h"
#include "storage/checksum.h"
#include "utils/memdebug.h"
@@ -85,7 +86,8 @@ PageInit(Page page, Size pageSize, Size specialSize)
* to pgstat.
*/
bool
-PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags)
+PageIsVerifiedExtended(Page page, ForkNumber forknum, bool relation_is_permanent,
+ BlockNumber blkno, int flags)
{
PageHeader p = (PageHeader) page;
size_t *pagebytes;
@@ -108,6 +110,8 @@ PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags)
checksum_failure = true;
}
+ PageDecryptInplace(page, forknum, relation_is_permanent, blkno);
+
/*
* The following checks don't prove the header is correct, only that
* it looks sane enough to allow into the buffer pool. Later usage of
@@ -1433,3 +1437,48 @@ PageSetChecksumInplace(Page page, BlockNumber blkno)
((PageHeader) page)->pd_checksum = pg_checksum_page((char *) page, blkno);
}
+
+char *
+PageEncryptCopy(Page page, ForkNumber forknum, bool relation_is_permanent,
+ BlockNumber blkno)
+{
+ static char *pageCopy = NULL;
+
+ /* If we don't need a checksum, just return the passed-in data */
+ if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum))
+ return (char *) page;
+
+ /*
+ * We allocate the copy space once and use it over on each subsequent
+ * call. The point of palloc'ing here, rather than having a static char
+ * array, is first to ensure adequate alignment for the checksumming code
+ * and second to avoid wasting space in processes that never call this.
+ */
+ if (pageCopy == NULL)
+ pageCopy = MemoryContextAlloc(TopMemoryContext, BLCKSZ);
+
+ memcpy(pageCopy, (char *) page, BLCKSZ);
+ EncryptPage(pageCopy, relation_is_permanent, blkno);
+ return pageCopy;
+}
+
+void
+PageEncryptInplace(Page page, ForkNumber forknum, bool relation_is_permanent,
+ BlockNumber blkno)
+{
+ if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum))
+ return;
+
+ EncryptPage(page, relation_is_permanent, blkno);
+}
+
+
+void
+PageDecryptInplace(Page page, ForkNumber forknum, bool relation_is_permanent,
+ BlockNumber blkno)
+{
+ if (PageIsNew(page) || !PageNeedsToBeEncrypted(forknum))
+ return;
+
+ DecryptPage(page, relation_is_permanent, blkno);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d36ac60da3..d667e6fbdb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -42,6 +42,7 @@
#include "catalog/pg_type.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "crypto/bufenc.h"
#include "executor/spi.h"
#include "jit/jit.h"
#include "libpq/libpq.h"
@@ -4039,6 +4040,7 @@ PostgresMain(int argc, char *argv[],
if (!IsUnderPostmaster)
{
InitializeKmgr();
+ InitializeBufferEncryption();
if (terminal_fd != -1)
close(terminal_fd);
diff --git a/src/include/access/gist.h b/src/include/access/gist.h
index 4b06575d98..04f11fb369 100644
--- a/src/include/access/gist.h
+++ b/src/include/access/gist.h
@@ -41,7 +41,7 @@
#define GISTNProcs 11
/*
- * Page opaque data in a GiST index page.
+ * Page opaque data flags in a GiST index page.
*/
#define F_LEAF (1 << 0) /* leaf page */
#define F_DELETED (1 << 1) /* the page has been deleted */
@@ -51,6 +51,9 @@
#define F_HAS_GARBAGE (1 << 4) /* some tuples on the page are dead,
* but not deleted yet */
+/* Specifies the bits that can be set in the GiST flags field */
+#define GIST_FLAG_BITMASK 0x1F
+
/*
* NSN (node sequence number) is a special-purpose LSN which is stored on each
* index page in GISTPageOpaqueData and updated only during page splits. By
diff --git a/src/include/crypto/bufenc.h b/src/include/crypto/bufenc.h
new file mode 100644
index 0000000000..cc86ceb821
--- /dev/null
+++ b/src/include/crypto/bufenc.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * bufenc.h
+ *
+ * Portions Copyright (c) 2021, PostgreSQL Global Development Group
+ *
+ * src/include/crypto/bufenc.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef BUFENC_H
+#define BUFENC_H
+
+#include "storage/bufmgr.h"
+#include "crypto/kmgr.h"
+
+/* Cluster encryption encrypts only main forks */
+#define PageNeedsToBeEncrypted(forknum) \
+ (FileEncryptionEnabled && (forknum) == MAIN_FORKNUM)
+
+
+extern void InitializeBufferEncryption(void);
+extern void EncryptPage(Page page, bool relation_is_permanent,
+ BlockNumber blkno);
+extern void DecryptPage(Page page, bool relation_is_permanent,
+ BlockNumber blkno);
+
+#endif /* BUFENC_H */
diff --git a/src/include/storage/bufpage.h b/src/include/storage/bufpage.h
index 359b749f7f..f9d97e7c04 100644
--- a/src/include/storage/bufpage.h
+++ b/src/include/storage/bufpage.h
@@ -15,6 +15,7 @@
#define BUFPAGE_H
#include "access/xlogdefs.h"
+#include "common/relpath.h"
#include "storage/block.h"
#include "storage/item.h"
#include "storage/off.h"
@@ -164,6 +165,8 @@ typedef struct PageHeaderData
} PageHeaderData;
typedef PageHeaderData *PageHeader;
+#define PageEncryptOffset offsetof(PageHeaderData, pd_special)
+#define SizeOfPageEncryption (BLCKSZ - PageEncryptOffset)
/*
* pd_flags contains the following flag bits. Undefined bits are initialized
@@ -418,8 +421,8 @@ do { \
((overwrite) ? PAI_OVERWRITE : 0) | \
((is_heap) ? PAI_IS_HEAP : 0))
-#define PageIsVerified(page, blkno) \
- PageIsVerifiedExtended(page, blkno, \
+#define PageIsVerified(page, relation_is_permanent, blkno) \
+ PageIsVerifiedExtended(page, MAIN_FORKNUM, relation_is_permanent, blkno, \
PIV_LOG_WARNING | PIV_REPORT_STAT)
/*
@@ -433,7 +436,10 @@ StaticAssertDecl(BLCKSZ == ((BLCKSZ / sizeof(size_t)) * sizeof(size_t)),
"BLCKSZ has to be a multiple of sizeof(size_t)");
extern void PageInit(Page page, Size pageSize, Size specialSize);
-extern bool PageIsVerifiedExtended(Page page, BlockNumber blkno, int flags);
+extern bool PageIsVerifiedExtended(Page page, ForkNumber forknum,
+ bool relation_is_permanent,
+ BlockNumber blkno,
+ int flags);
extern OffsetNumber PageAddItemExtended(Page page, Item item, Size size,
OffsetNumber offsetNumber, int flags);
extern Page PageGetTempPage(Page page);
@@ -452,5 +458,11 @@ extern bool PageIndexTupleOverwrite(Page page, OffsetNumber offnum,
Item newtup, Size newsize);
extern char *PageSetChecksumCopy(Page page, BlockNumber blkno);
extern void PageSetChecksumInplace(Page page, BlockNumber blkno);
+extern char *PageEncryptCopy(Page page, ForkNumber forknum,
+ bool relation_is_permanent, BlockNumber blkno);
+extern void PageEncryptInplace(Page page, ForkNumber forknum,
+ bool relation_is_permanent, BlockNumber blkno);
+extern void PageDecryptInplace(Page page, ForkNumber forknum,
+ bool relation_is_permanent, BlockNumber blkno);
#endif /* BUFPAGE_H */
diff --git a/src/include/storage/copydir.h b/src/include/storage/copydir.h
index 2c3936b0da..669c04d581 100644
--- a/src/include/storage/copydir.h
+++ b/src/include/storage/copydir.h
@@ -14,6 +14,6 @@
#define COPYDIR_H
extern void copydir(char *fromdir, char *todir, bool recurse);
-extern void copy_file(char *fromfile, char *tofile);
+extern void copy_file(char *fromfile, char *tofile, bool encrypt_init_file);
#endif /* COPYDIR_H */
--
2.20.1
end of thread, other threads:[~2021-04-06 18:42 UTC | newest]
Thread overview: 114+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-01-01 04:50 Key management with tests Bruce Momjian <bruce@momjian.us>
2021-01-01 06:07 ` Bruce Momjian <bruce@momjian.us>
2021-01-07 15:02 ` Bruce Momjian <bruce@momjian.us>
2021-01-07 19:08 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-08 00:39 ` Bruce Momjian <bruce@momjian.us>
2021-01-08 20:33 ` Stephen Frost <sfrost@snowman.net>
2021-01-08 21:17 ` Bruce Momjian <bruce@momjian.us>
2021-01-08 21:24 ` Stephen Frost <sfrost@snowman.net>
2021-01-08 00:46 ` Bruce Momjian <bruce@momjian.us>
2021-01-08 20:34 ` Stephen Frost <sfrost@snowman.net>
2021-01-08 21:18 ` Bruce Momjian <bruce@momjian.us>
2021-01-09 18:17 ` Bruce Momjian <bruce@momjian.us>
2021-01-10 01:08 ` Bruce Momjian <bruce@momjian.us>
2021-01-10 02:45 ` Bruce Momjian <bruce@momjian.us>
2021-01-10 05:04 ` Thomas Munro <thomas.munro@gmail.com>
2021-01-10 14:51 ` Bruce Momjian <bruce@momjian.us>
2021-01-11 11:12 ` Masahiko Sawada <sawada.mshk@gmail.com>
2021-01-11 17:22 ` Bruce Momjian <bruce@momjian.us>
2021-01-11 17:54 ` Stephen Frost <sfrost@snowman.net>
2021-01-11 18:04 ` Bruce Momjian <bruce@momjian.us>
2021-01-11 18:23 ` Stephen Frost <sfrost@snowman.net>
2021-01-11 19:06 ` Bruce Momjian <bruce@momjian.us>
2021-01-11 19:19 ` Stephen Frost <sfrost@snowman.net>
2021-01-11 20:22 ` Bruce Momjian <bruce@momjian.us>
2021-01-11 20:52 ` Stephen Frost <sfrost@snowman.net>
2021-01-12 03:40 ` Neil Chen <carpenter.nail.cz@gmail.com>
2021-01-12 16:08 ` Stephen Frost <sfrost@snowman.net>
2021-01-13 02:26 ` Neil Chen <carpenter.nail.cz@gmail.com>
2021-01-12 00:32 ` Masahiko Sawada <sawada.mshk@gmail.com>
2021-01-12 02:09 ` Bruce Momjian <bruce@momjian.us>
2021-01-12 12:40 ` Masahiko Sawada <sawada.mshk@gmail.com>
2021-01-12 18:03 ` Bruce Momjian <bruce@momjian.us>
2021-01-12 18:11 ` Andres Freund <andres@anarazel.de>
2021-01-12 18:11 ` Stephen Frost <sfrost@snowman.net>
2021-01-12 18:15 ` Bruce Momjian <bruce@momjian.us>
2021-01-12 18:44 ` Stephen Frost <sfrost@snowman.net>
2021-01-12 18:51 ` Bruce Momjian <bruce@momjian.us>
2021-01-12 18:57 ` Stephen Frost <sfrost@snowman.net>
2021-01-12 19:32 ` Bruce Momjian <bruce@momjian.us>
2021-01-12 18:46 ` Bruce Momjian <bruce@momjian.us>
2021-01-14 00:49 ` Bruce Momjian <bruce@momjian.us>
2021-01-12 19:34 ` Andres Freund <andres@anarazel.de>
2021-01-12 17:04 ` Bruce Momjian <bruce@momjian.us>
2021-01-15 20:49 ` Bruce Momjian <bruce@momjian.us>
2021-01-15 21:23 ` Robert Haas <robertmhaas@gmail.com>
2021-01-15 21:47 ` Bruce Momjian <bruce@momjian.us>
2021-01-15 21:59 ` Robert Haas <robertmhaas@gmail.com>
2021-01-15 22:07 ` David G. Johnston <david.g.johnston@gmail.com>
2021-01-15 22:07 ` Bruce Momjian <bruce@momjian.us>
2021-01-15 22:37 ` Andres Freund <andres@anarazel.de>
2021-01-16 00:21 ` Bruce Momjian <bruce@momjian.us>
2021-01-16 00:56 ` Andres Freund <andres@anarazel.de>
2021-01-16 01:49 ` Bruce Momjian <bruce@momjian.us>
2021-01-16 04:20 ` Andres Freund <andres@anarazel.de>
2021-01-16 04:39 ` Michael Paquier <michael@paquier.xyz>
2021-01-17 00:08 ` Tom Kincaid <tomjohnkincaid@gmail.com>
2021-01-17 06:24 ` Amit Kapila <amit.kapila16@gmail.com>
2021-01-17 06:58 ` Andres Freund <andres@anarazel.de>
2021-01-18 15:51 ` Bruce Momjian <bruce@momjian.us>
2021-01-18 15:50 ` Bruce Momjian <bruce@momjian.us>
2021-01-18 17:06 ` Bruce Momjian <bruce@momjian.us>
2021-01-18 17:42 ` Andres Freund <andres@anarazel.de>
2021-01-18 18:58 ` Bruce Momjian <bruce@momjian.us>
2021-01-18 19:00 ` Andres Freund <andres@anarazel.de>
2021-01-18 22:47 ` Tom Kincaid <tomjohnkincaid@gmail.com>
2021-01-25 19:18 ` Bruce Momjian <bruce@momjian.us>
2021-01-25 23:12 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-01-26 00:09 ` Bruce Momjian <bruce@momjian.us>
2021-01-26 03:27 ` Bruce Momjian <bruce@momjian.us>
2021-01-26 16:15 ` Bruce Momjian <bruce@momjian.us>
2021-01-26 20:24 ` Robert Haas <robertmhaas@gmail.com>
2021-01-26 22:53 ` Bruce Momjian <bruce@momjian.us>
2021-01-27 14:11 ` Bruce Momjian <bruce@momjian.us>
2021-01-28 19:41 ` Tom Kincaid <tomjohnkincaid@gmail.com>
2021-01-28 20:22 ` Bruce Momjian <bruce@momjian.us>
2021-01-29 08:05 ` Masahiko Sawada <sawada.mshk@gmail.com>
2021-01-29 22:40 ` Stephen Frost <sfrost@snowman.net>
2021-01-30 13:23 ` Tom Kincaid <tomjohnkincaid@gmail.com>
2021-02-01 02:05 ` Moon, Insung <tsukiwamoon.pgsql@gmail.com>
2021-02-01 22:44 ` Bruce Momjian <bruce@momjian.us>
2021-02-01 23:34 ` Stephen Frost <sfrost@snowman.net>
2021-02-01 23:52 ` Bruce Momjian <bruce@momjian.us>
2021-02-01 22:43 ` Bruce Momjian <bruce@momjian.us>
2021-02-01 23:31 ` Stephen Frost <sfrost@snowman.net>
2021-02-02 00:47 ` Bruce Momjian <bruce@momjian.us>
2021-02-02 15:58 ` Bruce Momjian <bruce@momjian.us>
2021-02-03 15:33 ` Stephen Frost <sfrost@snowman.net>
2021-02-03 18:16 ` Bruce Momjian <bruce@momjian.us>
2021-02-05 16:40 ` Bruce Momjian <bruce@momjian.us>
2021-02-05 18:14 ` Stephen Frost <sfrost@snowman.net>
2021-02-05 20:37 ` Bruce Momjian <bruce@momjian.us>
2021-02-05 22:21 ` Stephen Frost <sfrost@snowman.net>
2021-02-06 00:53 ` Bruce Momjian <bruce@momjian.us>
2021-02-07 19:00 ` Bruce Momjian <bruce@momjian.us>
2021-03-12 03:31 ` Bruce Momjian <bruce@momjian.us>
2021-03-15 16:50 ` Bruce Momjian <bruce@momjian.us>
2021-03-18 15:31 ` Stephen Frost <sfrost@snowman.net>
2021-03-18 16:55 ` Bruce Momjian <bruce@momjian.us>
2021-03-23 00:38 ` Bruce Momjian <bruce@momjian.us>
2021-03-23 02:19 ` Bruce Momjian <bruce@momjian.us>
2021-03-18 17:13 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-18 17:21 ` Stephen Frost <sfrost@snowman.net>
2021-03-18 17:37 ` Alvaro Herrera <alvherre@alvh.no-ip.org>
2021-03-18 17:46 ` Stephen Frost <sfrost@snowman.net>
2021-03-18 18:59 ` Bruce Momjian <bruce@momjian.us>
2021-03-18 18:49 ` Bruce Momjian <bruce@momjian.us>
2021-04-06 08:56 ` Neil Chen <carpenter.nail.cz@gmail.com>
2021-04-06 18:42 ` Bruce Momjian <bruce@momjian.us>
2021-02-01 22:16 ` Bruce Momjian <bruce@momjian.us>
2021-01-18 19:00 ` Tom Kincaid <tomjohnkincaid@gmail.com>
2021-01-18 21:38 ` Robert Haas <robertmhaas@gmail.com>
2021-01-18 23:21 ` Bruce Momjian <bruce@momjian.us>
2021-01-18 00:50 ` Robert Haas <robertmhaas@gmail.com>
2021-01-18 15:43 ` Bruce Momjian <bruce@momjian.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox